Arch Linux

b4283

Member

Registered: 2008-11-26

Posts: 123

[HALF-SOLVED] rsync crash when processing large amount of files

I heard that XFS or JFS over-performs the traditional EXT3, so I try to use them on a new partition.

However, problems just keep coming and coming, I wonder if archlinux's stock kernel has buggy module for them?

At first, I use mkfs.xfs to format, boot into Xorg, so far seems good.
But when I started to use rsync to recover datas (>150GB) from remote servers, the program will crash and the whole system goes into halt after a certain period of time (it doesn't crash from the beginning).

I try to fix this by switch to vty2-6 to kill rsync, but after i press ctrl-alt-f2, it gets worse. Even the cursor stop responding.

I don't think I have more power to try out Reiser4FS, especially when I'm force to use another kernel. Now I have switched back to EXT3, the rsync is still running in the background.

Does anyone have similar experiences ?

Last edited by b4283 (2008-12-19 16:54:13)

b4283

Member

Registered: 2008-11-26

Posts: 123

Re: [HALF-SOLVED] rsync crash when processing large amount of files

UPDATE:
ext3 still crashes, but it happens slower than xfs or jfs or reiserfs
apparently it has something to do with rsync
or the fact that i'm using network interface to restore backup files

btw, is linux vulnerable to ethernet attack if one of the lan computer (winXP) is affected with virus ?
usually there's no stopping packet in my lan

UPDATE:
cp crashes too
so ruling out rsync is problematic
so either the kernel is buggy or it's a network issue
another possibility is local samba protocol may be broken against large amount of files

so i'm trying out all these options and report my results later
(hope not to crash)

Last edited by b4283 (2008-12-17 16:10:01)

wuischke

Member

From: Suisse Romande

Registered: 2007-01-06

Posts: 630

Re: [HALF-SOLVED] rsync crash when processing large amount of files

Can you check the SMART status of your hdd?

b4283

Member

Registered: 2008-11-26

Posts: 123

Re: [HALF-SOLVED] rsync crash when processing large amount of files

smartctl -H reported a passed, so i'm checking if theres are any bad blocks now.

fukawi2

Ex-Administratorino

From: .vic.au

Registered: 2007-09-28

Posts: 6,224

Website

Re: [HALF-SOLVED] rsync crash when processing large amount of files

I seriously doubt it's anything to do with JFS or XFS... I use both with rsync regularly without issue.

If you're syncing from a Windows machine, I recall having very similar issues with a large amount of data once upon a time - it was a long time ago and I can't remember anything useful, but it was defiantely a problem with Samba / SMB / CIFS or the actual Windows machine.

---

Are you familiar with our Forum Rules, and How To Ask Questions The Smart Way?

BlueHackers // fscanary // resticctl

b4283

Member

Registered: 2008-11-26

Posts: 123

Re: [HALF-SOLVED] rsync crash when processing large amount of files

it was a long time ago and I can't remember anything useful, but it was defiantely a problem with Samba / SMB / CIFS or the actual Windows machine.

yes, thank you. i'm feeling the same.
because the harddrive passed SMART and badblock tests, so i don't think it could be anything about that.

last night i tried to use a different protocol (ftp) to transfer, it still crashes my system.

so, anyelse suggestions ?

i'm thinking maybe a kernel update can resolve this but i really doubt it since it's a stock kernel.

UPDATE: nope, update kernel does nothing.

UPDATE:
after running under pure vty (avoid xorg), i discovered that this action of copying files not just break down THAT partition only (in my case /dev/sdc2), it acutally makes the entire /dev/sdc stop responding.
the reason is that i also saw some ext3-3g errors in the output, which /dev/sdc1 is using.

ps: does it has anything to do with the fact that i'm using this to connect my hard drives?

Last edited by b4283 (2008-12-18 05:08:36)

b4283

Member

Registered: 2008-11-26

Posts: 123

Re: [HALF-SOLVED] rsync crash when processing large amount of files

The problem was indeed caused by the Windows XP system. I couldn't get large amount of remote files from there.
So, in alternative, I used a USB harddrive to copy those files and it turned out to be alright.
Apparently there's some incompatibilities exist, either the Windows NTFS filesystem is sending the wrong message, or, Windows XP network interface issues. I doubt that it's most likely former, cause remember that USB drives on Linux is using ntfs-3g driver instead of Windows XP's own driver, and this makes me suspicious.

Anyway, this problem can't really be called "solved".
If you are experiencing the same problem, try to start a Linux server on that terminal instead of using original Windows.