Only 6MB/s on encfs on ntfs-3g

lynix · 2011-07-15 14:30:29

I switched from LUKS to an encfs-encrypted setup for one partiton to be able to access it from other (non-*nix) operating systems.
Layering is now: RAID1-Volume --> ntfs-3g --> encfs.

The problem is the write speed I'm currently having while copying the contents over: ~6.63MB/s. The RAID can do ~65MB/s, and I see no CPU load (Core i7-920).
I've read many articles about performance concerns with user-mode filesystems and the additional context-switches (or better mode-switches) they require, but come on - 6MB/s??

Any ideas or hints are welcome

lynix · 2011-07-15 15:05:11

Well, as I have access to the system now I think I could add some more details:

encfs:
- EncFS 1.7.4
- cipher: AES-256
- blocksize: 1024
- unique IV
- chained name IVs

ntfs-3g:
- ntfs-3g 2011.4.12-1
- blocksize: 4096
- mount options: uid=lynix,gid=users,fmask=113,dmask=002

system:
- kernel26-ck-core2 2.6.39.3-1
- bfq I/O scheduler registered for the RAID volumes

Please ask if there's something missing

lilsirecho · 2011-07-15 15:13:30

Perhaps your drive is not master........................

lynix · 2011-07-15 15:17:27

What do you mean with "master"?

They're all SATA drives...

lilsirecho · 2011-07-15 15:32:00

My system has four SataII ports. They are arranged in pairs, master and slave,,ID ch2M ..ch2S..ch3M..ch3S.

In raid arrays, I must use master(s).

This is what I mean by master.

My system is x86_64 with GA-945GCM-S2C mobo.

If I don't use master devices, my raid speed is cut in half(if only one of three devices is a slave).

Your system may not have master/slave arrangement so disregard.................

lynix · 2011-07-15 15:37:03

Thanks for the explanation, I didn't know there can be designated ports.
I have write speeds of ~65 MB/s on the other volumes of the RAID array, so I assume this is not the bottleneck.

lilsirecho · 2011-07-15 15:46:02

My system has three raid devices in a bootable archlinux 64bit system using SataII to CF cards with adapters. Raid0.....

Each drive has ~60-65MB/s in hdparm and the raid device reports 186MB/s in hdparm.

I needed pcie adapter for Sata to obtain the third master connection.

I plan on a fourth install today with a Kingston 600x device (16GB for eachdevice). I have one pcie port open, an independent port.

Last edited by lilsirecho (2011-07-15 15:46:52)

JGC · 2011-07-18 14:47:28

You're layering a FUSE filesystem on top of another FUSE filesystem and you're still expecting to have performance?

As for master/slave: that's only an issue if you still use IDE emulation, and probably only with certain chipsets. Modern SATA controllers in AHCI or RAID mode don't suffer if you populate SATA0+1 instead of SATA0+2.

lynix · 2011-07-18 16:31:40

I'm not expecting a level of performance that is comparable with LUKS or no encryption at all
But at least I would expect a visible cpu load while having so bad transfer speed. I understand that layering FUSE filesystems is not very clever, but if there are so many context switches why can't I se any load?

lynix · 2011-07-18 17:19:39

By the way, isn't truecrypt FUSE-based too? I just killed the volume and setup a new one with truecrypt.
51.77 MB/s write speed now...

JGC · 2011-07-19 05:10:51

You'll probably get high load because things go in D-state. You won't notice much CPU load simply because the kernel will be much slower processing your data:

Plain filesystem:
user app -> kernel fs/block

FUSE filesystem:
user app -> kernel -> user through FUSE -> kernel block

FUSE filesystem on top of FUSE filesystem:
user app -> kernel -> user through FUSE -> kernel -> back to FUSE -> kernel block

Doing that wastes a lot of clockcycles, and wasted clockcycles don't count up for CPU usage. Try creating a program that sleeps a lot: same slowness, but no CPU load.

lynix · 2011-07-19 15:45:41

Okay I understand that all these memcpy() from user to kernel-space don't count als real load.

But as I already asked: what is so different with truecrypt? There should be the same layering and buffer copying...

Awebb · 2011-07-19 19:54:56

In the beginnung you had: RAID1-Volume --> ntfs-3g --> encfs
Now you have: RAID1-Volume --> truecrypt-filesystem.

That's one damn slow punk less in the middle. I also had issues with NTFS partitions. They gave me a maximum read of 20 mb/s and a maximum write of 10 mb/s. They also blocked my cpu completely, I couldn't even move windows in the process. While I know that this is a problem that might need fixing (ntfs-3g is not bad at all, usually), I switched to EXT4 for all my old data partitions and now I have up to 80 mb/s throughput (non-raid) and no extra process eating my cpu power away.

lynix · 2011-07-19 20:32:21

I still have ntfs-3g as truecrypt doesn't have its own filesystem:

RAID1 --> truecrypt (/dev/mapper/truecrypt1) --> ntfs-3g

Same amount of indirections, but truecrypt instead of encfs and ntfs-3g last instead of second.

As shown by mount:

truecrypt on /tmp/.truecrypt_aux_mnt1 type fuse.truecrypt (rw,nosuid,nodev,allow_other)
/dev/mapper/truecrypt1 on /mnt/storage type fuseblk (rw,nosuid,nodev,noatime,allow_other,blksize=4096,default_permissions)

So I suspect it might be somehow the order of the layering? I also do not have the speed issues with NTFS-partitions you describe: 75 MB/s read with ntfs-3g under ~14% cpu load.

Last edited by lynix (2011-07-19 20:33:48)

Arch Linux

#1 2011-07-15 14:30:29

Only 6MB/s on encfs on ntfs-3g

#2 2011-07-15 15:05:11

Re: Only 6MB/s on encfs on ntfs-3g

#3 2011-07-15 15:13:30

Re: Only 6MB/s on encfs on ntfs-3g

#4 2011-07-15 15:17:27

Re: Only 6MB/s on encfs on ntfs-3g

#5 2011-07-15 15:32:00

Re: Only 6MB/s on encfs on ntfs-3g

#6 2011-07-15 15:37:03

Re: Only 6MB/s on encfs on ntfs-3g

#7 2011-07-15 15:46:02

Re: Only 6MB/s on encfs on ntfs-3g

#8 2011-07-18 14:47:28

Re: Only 6MB/s on encfs on ntfs-3g

#9 2011-07-18 16:31:40

Re: Only 6MB/s on encfs on ntfs-3g

#10 2011-07-18 17:19:39

Re: Only 6MB/s on encfs on ntfs-3g

#11 2011-07-19 05:10:51

Re: Only 6MB/s on encfs on ntfs-3g

#12 2011-07-19 15:45:41

Re: Only 6MB/s on encfs on ntfs-3g

#13 2011-07-19 19:54:56

Re: Only 6MB/s on encfs on ntfs-3g

#14 2011-07-19 20:32:21

Re: Only 6MB/s on encfs on ntfs-3g

Board footer