Arch Linux

jletzel

Member

Registered: 2012-11-04

Posts: 7

Problems with external USB drive using LUKS and rsync

Hi,

I am using encrypted external USB drives via USB3 for backup. To backup I am using "rsync" with comparing file content instead timestamp.

Yesterday I did a backup as usual and done a full system upgrade. Now when I start the backup, it runs (maybe a little slow) and then at a specific point it just stuck. One of my four CPU cores shows 100%. The command "iotop"  show no significant I/O.

Is there maybe a already known problem with that ?

Thanks in advance.

Regards.

J. Letzel

frostschutz

Member

Registered: 2013-11-15

Posts: 1,418

Re: Problems with external USB drive using LUKS and rsync

Which type of drive is it? I had issues with a "Seagate Backup Plus Hub".

Anything in dmesg?

jletzel

Member

Registered: 2012-11-04

Posts: 7

Re: Problems with external USB drive using LUKS and rsync

Hi !

I am using three WD Elements disks.

Your tip with dmesg was great: I got an I/O error in a specific sector with flags 0700  referring to my PCs hard disk.
So the problems seems to be the source disk.

I hope I can fix this.

Thanks a lot.

Regards.

jletzel

Member

Registered: 2012-11-04

Posts: 7

Re: Problems with external USB drive using LUKS and rsync

Hi, again.

Sorry I was wrong. The target disk (USB disk) was the disk which causes the I/O error. I will check a second backup disk which makes problems and will look at what dmesg says.

Stay tuned

jletzel

Member

Registered: 2012-11-04

Posts: 7

Re: Problems with external USB drive using LUKS and rsync

Hi !

I checked the second backup disk to. Also here the backup process stops suddenly with 100% on one core and zero I/O.

dmesg shows also some I/O error but not at the end. Somehow this is a little strange.

[  309.983110] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 393216 bytes), total 32768 (slots), used 1 (slots)
[  309.983146] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 393216 bytes), total 32768 (slots), used 1 (slots)
[  309.983182] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 393216 bytes), total 32768 (slots), used 1 (slots)

[  310.899591] usb 6-1: reset SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[  310.922934] sd 6:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
[  310.922939] sd 6:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 65 66 bb 00 00 03 00 00
[  310.922942] print_req_error: I/O error, dev sda, sector 1701231360 flags 80700
[  314.986203] swiotlb_tbl_map_single: 45158 callbacks suppressed [  314.986207] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 376832 bytes), total 32768 (slots), used 329 (slots)

[  314.986293] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 376832 bytes), total 32768 (slots), used 329 (slots)
[  314.986376] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 376832 bytes), total 32768 (slots), used 329 (slots)
[  314.986461] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 376832 bytes), total 32768 (slots), used 329 (slots)
[  314.986544] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 376832 bytes), total 32768 (slots), used 329 (slots)
...
[  319.989554] swiotlb_tbl_map_single: 59895 callbacks suppressed
[  319.989557] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 376832 bytes), total 32768 (slots), used 329 (slots)
[  319.989640] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 376832 bytes), total 32768 (slots), used 329 (slots)
...
[  329.996996] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 37683 2bytes), total 32768 (slots), used 329 (slots)
[  329.997079] xhci_hcd 0000:05:00.0: swiotlb buffer is full (sz: 376832 bytes), total 32768 (slots), used 329 (slots)

I think about to try it with a linux-lts environment.

Any other ideas ?

Thanks in advance.

Regards.

jletzel

Member

Registered: 2012-11-04

Posts: 7

Re: Problems with external USB drive using LUKS and rsync

Hi !

Here are results of my testing so far:

I booted the LTS kernel (without graphical UI because of missing nvidia-lts) and mounted the external USB drive by foot.
Then I started my backup-script and all went fine.

So the conclusion is: There is something with the latest kernel. I don't know what but its fishy somehow.

Regards

ua4000

Member

Registered: 2015-10-14

Posts: 420

Re: Problems with external USB drive using LUKS and rsync

Interesting, I ran into the same issue with my external backup, USB3.0 + LUKS + rsync.
Also I saw " I/O error, dev sda, sector "

I had planned to run a long smart....

But your idea is better, I will revert to another kernel and retest.
I think I have to google for the real issue here: xhci_hcd xxxx swiotlb buffer is full
https://www.spinics.net/lists/iommu/msg36333.html

Edit:
Does anyone knows if ther exists another workaround with kernel’s command-line parameters ?
Like something with MMU or swiotlb ?

Last edited by ua4000 (2019-07-23 16:12:33)

Thme

Member

From: Raleigh NC

Registered: 2012-01-22

Posts: 105

Re: Problems with external USB drive using LUKS and rsync

Hi,

I am using encrypted external USB drives via USB3 for backup. To backup I am using "rsync" with comparing file content instead timestamp.

Yesterday I did a backup as usual and done a full system upgrade. Now when I start the backup, it runs (maybe a little slow) and then at a specific point it just stuck. One of my four CPU cores shows 100%. The command "iotop"  show no significant I/O.

Is there maybe a already known problem with that ?

Thanks in advance.

Regards.

J. Letzel

How are you rsync'ing?
command arguments used? are you doing incremental or differential snapshots or just one snapshot? rsync transfers data differently, and comparing file content is pretty much it's default in archive mode "-a" unless you tell it otherwise, it only changes timestamps if checksums are the same. if you're using -v try -i instead. That will actually show whats being changed  --itemize-changes(long option for -i ) in the man page explains what the letters in the output mean. rsync is a lot more intelligently designed than many tend to think... many third party solutions are built around it (and why I don't use them).

---

"Hidden are the ways for those who pass by, for light is perished and darkness comes into being." Nephthys:
Ancient Egyptian Coffin Texts

ua4000

Member

Registered: 2015-10-14

Posts: 420

Re: Problems with external USB drive using LUKS and rsync

It's a bug,
https://patchwork.kernel.org/patch/10992541/

This patch fixes an issue that the following error happens on
swiotlb environment:
    xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 524288 bytes), total 32768 (slots), used 1338 (slots)
...

Thme

Member

From: Raleigh NC

Registered: 2012-01-22

Posts: 105

Re: Problems with external USB drive using LUKS and rsync

Well, That's good to know I'm setting up a USB3.0 stick with LUKS encryption for portable offline snapshots and a bootable installation for recovery/rescue purposes. etc. Opened a different thread Under GNU?Linux discossion looking for some feedback on the methods I'm preparing to use, Looks like I'll Fallback to an LTS kernel while setting this up until patch reaches the stable upstream release.

Last edited by Thme (2019-07-24 21:39:16)

---

"Hidden are the ways for those who pass by, for light is perished and darkness comes into being." Nephthys:
Ancient Egyptian Coffin Texts

ua4000

Member

Registered: 2015-10-14

Posts: 420

Re: Problems with external USB drive using LUKS and rsync

The issue is still in the latest kernel, tested today with linux-5.2.3.arch1-1-x86_64.pkg.tar.xz
Using the LTS kernel, linux-lts-4.19.61-1-x86_64.pkg.tar.xz solved it for now.

cat /sys/block/sdX/queue/max_segment_size

on the device "X" connected to USB3 gives 65536 on a working kernel, 4294967295 on the faulty one - which causes the buffer full.

ezacaria

Member

Registered: 2007-12-10

Posts: 113

Re: Problems with external USB drive using LUKS and rsync

It seems that 5.2.9-arch1-1-ARCH solves the issue, at least to some extent.

I tested one USB3 stick, first connected on USB2 port and then on a USB3 port. I do not see any "buffer is full" message in the dmesg output.

However, max_segment_size behaves oddly even if data transfers are possible on both USB2 and USB3.
Initially, I got max_segment_size 65536 on USB2 and 4294967295 on USB3.

Re-plugging the device on any USB2 port still gives  max_segment_size 4294967295. But as mentioned, data transfer is still possible.
On the other hand, the transfer speed oscillates a lot and in average the whole thing is pretty slow. This may be because for some reason the write cache comes up disabled for the stick.