mv command fails silently for 4.2 GB file at 450 MB

dawnofman · 2022-06-13 02:05:49

Does mv have some limitations/issues with large files ? (I really doubt it but I need to ask anyway to begin with)

someone:workstation:~ $ cd /tmp-workspace-server/common/ [A]
someone:workstation:common $ ll -h
-r-xr-xr-x 1 someone stuff 4.2G 2020-05-11 03:41:12 install.wim

someone:workstation:common $ mv install.wim /tmp-workspace/common/ [B][C]
someone:workstation:common $ ll -h /tmp-workspace/common/
-rwx------ 1 someone stuff 590M 2022-06-12 22:00:54 install.wim

someone:workstation:common $ cp --preserve install.wim /tmp-workspace/common/ [D]
someone:workstation:common $ ll -h /tmp-workspace/common/
-r-xr-xr-x 1 someone stuff 4.2G 2020-05-11 03:41:12 install.wim

[A]      server tmpfs @ 8 GB (8 GB free) (shared with NFSv4.2 as RW) having some 4 GB file
[B] workstation tmpfs @ 8 GB (8 GB free)
[C] moving the file from the server to the workstation (both tmpfs directories) fails silently at 590 MB for target leaving the source intact
[D] using cp --preserve (instead of mv) afterwards completes successfully

The above example happened today: I have a server and a workstation both having plenty of RAM so I placed 8 GB ram drives on both to move things back-and-forth at ease. I mainly use them for editing large files remuxing media containers and things like that in order to preserve the SSDs as much as possible -I even have the home user's cache directories on the workstation there and both ram drives work flawlessly.

When I hit the above failed mv it instantly reminded me of a previous issue I encountered some years ago: moving large file from Windows to arch with cifs-utils stops at 45 MB (ivanborodin was my previous user ID for which for whatever reasons lost my credentials probably while migrating mail accounts).

This past issue happened all the time while attempting to move large files between a Windows server and my first arch-linux laptop (and my first arch-linux by the way). Those were on a wired connection at 100-Mbps. Now I am on a wired connection at 1-Gbps. They previously (silently) failed (same symptoms) at 45 MB and now at 450 MB ... exactly 10X higher so these values seem to be related, I don't think they just match by sheer randomness. In the past I was transferring via Ethernet on normal (MTU) frames and now I am doing on Jumbo frames.

Needless to say the network connection between the server and the workstation works flawlessly since I installed both machines three months ago more-or-less. I use them all the time: NFS file server, nginx intranet web server, BIND server, etc. I copy large files all the time between the two; eg: large image files for linux/BSD distros and I always check the checksums for those thereafter -never got a bad checksum; all those files get downloaded on my workstation and go to the server via those ram drives; I have lots of such images; 70 GB or so. I also do server backups with rsync now to a local drive on the workstation. No problems at all.

I seldom use the mv command with large files; I almost use cp and now I am starting to use rsync more often. Today was the exception, and ... bump !

I wonder what could be the issue here ... can anyone point me in the right direction please ?

PS: linux is 5.17.5.

dawnofman · 2022-06-13 03:09:20

dawnofman wrote:

I wonder what could be the issue here ...

a follow up: this is starting to get funny:

- move from workstation to server with xfe: OK, MD5 checksum OK, source removed
- move from workstation to server with mv: OK, MD5 checksum OK but source is not removed

- move from server to workstation with mv: FAILS: stops at 548 MB on workstation; source left on server
- move from server to workstation with xfe: OK, MD5 checksum OK, source removed

- move from workstation to server with mv (once again): OK, MD5 checksum OK but source is not removed

- move from server to workstation with mv: FAILS: stops at 546 MB on workstation; source left on server
- move from server to workstation with caja: OK, MD5 checksum OK, source removed

As you can see both xfe and caja don't have issues with the move while mv at console keeps failing silently.

PS#1: xfe is xfe file explorer (custom-built package) running on MATE on the workstation
PS#2: caja is caja file explorer running on MATE on the workstation
PS#3: the server has no GUI installed
PS#4: for all cases: network traffic is 100~105 MB/s (30 seconds or-so elapsed for whole transfer): point-to-point wired connection with jumbo frames; no other traffic goes here, just a direct inter-server link for file-sharing; all other traffic like outbound connections for the internet go through a different interface.

Last edited by dawnofman (2022-06-13 03:10:21)

seth · 2022-06-13 15:09:25

Do you get a "failed to close" "Input/output error"?

dawnofman · 2022-06-13 16:15:35

seth wrote:

Do you get a "failed to close" "Input/output error"?

Nothing at all --not even in the journal. That's why I used the term "silently".

Arch Linux

#1 2022-06-13 02:05:49

mv command fails silently for 4.2 GB file at 450 MB

#2 2022-06-13 03:09:20

Re: mv command fails silently for 4.2 GB file at 450 MB

#3 2022-06-13 15:09:25

Re: mv command fails silently for 4.2 GB file at 450 MB

#4 2022-06-13 16:15:35

Re: mv command fails silently for 4.2 GB file at 450 MB

Board footer