You are not logged in.

#1 2019-02-03 16:35:12

Maniaxx
Member
Registered: 2014-05-14
Posts: 732

[SOLVED]Compressed folder in tmpfs?

Hallo,
i want to have a compressed folder in tmpfs (for logs, txt files). I've looked at zram but it seems it can only handle swap devices that doesn't allow a specific folder to be compressed.

Any ideas how to achieve that?

Last edited by Maniaxx (2019-02-05 16:38:49)


sys2064

Offline

#2 2019-02-03 17:18:34

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: [SOLVED]Compressed folder in tmpfs?

Not sure if tmpfs supports compression... limited memory on your system?  In any case, did you google 'tmpfs compression'?


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#3 2019-02-03 19:51:47

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: [SOLVED]Compressed folder in tmpfs?

There's https://code.google.com/archive/p/fusecompress/ but event the "newer" version is unmaintained and (apparently different from the legacy implementation) prone to compromise data integrity…
If you're ok w/ a ro FS, squashfs is the natural choice.

On a sidenote: a compressing FS is not necessarily a good choice for a continuously written logfile.

Offline

#4 2019-02-03 20:44:46

Maniaxx
Member
Registered: 2014-05-14
Posts: 732

Re: [SOLVED]Compressed folder in tmpfs?

Yes, there's low memory on that system. If there's no other choice i probably have to consider log rotation.


sys2064

Offline

#5 2019-02-03 20:48:44

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: [SOLVED]Compressed folder in tmpfs?

tmpfs does not support compression but it supports swapping. so file contents stored in tmpfs can be transparently swapped out.

swap then can be compressed through zram/zswap. so indirectly, tmpfs might work as a compressing swap file system.

that said, I've never actually tried it.

a completely different approach would be to rotate the logs, that's how you usually compress them on disk (in a filesystem that does not support compression either)

----

otherwise using zram for a filesystem:

# modprobe zram
# echo lz4 > /sys/block/zram0/comp_algorithm
# echo 4G > /sys/block/zram0/disksize
# mkfs.ext2 /dev/zram0
mke2fs 1.44.5 (15-Dec-2018)
Discarding device blocks: done                            
Creating filesystem with 1048576 4k blocks and 262144 inodes
Filesystem UUID: 30584cfd-eaae-4706-a8b4-ff6ee2f40b30
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done 

# mount /dev/zram0 /mnt/tmp/
# df -h /mnt/tmp/
Filesystem      Size  Used Avail Use% Mounted on
/dev/zram0      4.0G  8.0M  3.8G   1% /mnt/tmp
# yes > /mnt/tmp/yes.txt
yes: standard output: No space left on device
# df -h /mnt/tmp/
Filesystem      Size  Used Avail Use% Mounted on
/dev/zram0      4.0G  4.0G     0 100% /mnt/tmp
# free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi       1.3Gi       9.0Gi       247Mi       5.2Gi        13Gi
Swap:            0B          0B          0B

it's not tmpfs anymore but compressed ext2 in ram is similar to it, yes?

only you have to be careful with such things, if you put non-compressible data in there it might actually use that much ram - and OOM killer will not get rid of it for you, so it's possible to crash a system if you use too much. same if tmpfs if you have too many instances and each of them 50% of RAM size.

Last edited by frostschutz (2019-02-03 21:03:24)

Offline

#6 2019-02-03 22:08:18

jamespharvey20
Member
Registered: 2015-06-09
Posts: 129

Re: [SOLVED]Compressed folder in tmpfs?

Not sure if this is better, worse, or just different than frostschutz' solution.

What I thought of first was to create a tmpfs, allocate a file at maximum size, mount the file as a loop device, and make a btrfs filesystem using compression on top.

Offline

#7 2019-02-03 22:32:12

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: [SOLVED]Compressed folder in tmpfs?

Oh yes, one thing I forgot:

for both zram filesystem, and loop mounted filesystem as the previous suggestion, you actually have to enable 'discard' flag to make sure free space is really free.

Otherwise, overwritten and deleted files will continue to consume ram space.

Last edited by frostschutz (2019-02-03 22:33:02)

Offline

#8 2019-02-04 16:07:09

Maniaxx
Member
Registered: 2014-05-14
Posts: 732

Re: [SOLVED]Compressed folder in tmpfs?

Thanks. That looks exactly what i'm looking for. I probably will use a subfolder in tmpfs just for well compressible data. If it proves well i will keep it.


sys2064

Offline

Board footer

Powered by FluxBB