You are not logged in.
- Topics: Active | Unanswered
#1 2009-11-01 12:45:32
- gen1s
- Member
- From: Russian Federation
- Registered: 2009-08-04
- Posts: 17
[How-To] SquashFS with LZMA compression
SquashFS (.sfs) is a compressed read-only file system for Linux. SquashFS compresses files, inodes and directories, and supports block sizes up to 1 MB for greater compression. SquashFS is also free software (licensed under the GPL) for accessing SquashFS filesystems.
SquashFS is intended for general read-only file system use and in constrained block device/memory systems (e.g. embedded systems) where low overhead is needed. The standard version of SquashFS uses gzip compression.
Here is my How-To on how to get LZMA compressed SquashFS in Archlinux.
Requirements
1) squashfs-tools with LZMA patches. This package provides tools (mksquashfs, unsquashfs) to create Squashfs filesystem. With standard version (from /extra repository) you can create only gzip-compressed Squashfs filesystem. To get LZMA support you must install package:
squashfs-tools-lzma
This package doesn't conflict with package from /extra. It provides mksquashfs.lzma, unsquashfs.lzma. So if you want create LZMA compressed filesystem you must use:
mksquashfs.lzma source destOtherwise for GZIP compressed filesystem:
mksquashfs source dest2) squashfs module with LZMA patches. Without it you cant't mount your LZMA compressed filesystems. Here is two ways on how to get it.
2.1 Patch kernel with SquashFS LZMA patches.
Before proceed i recommend to read Custom Kernel Compilation with ABS
You must apply on sources three patches:
051-squashfs_pcomp.patch
052-pcomp_lzma_support.patch
053-squashfs_lzma.patch
And set in kernel config:
CONFIG_SQUASHFS=m|y
CONFIG_SQUASHFS_SUPPORT_ZLIB=y
CONFIG_SQUASHFS_SUPPORT_LZMA=y
CONFIG_CRYPTO_UNLZMA=m|ym|y means that you can build it as a module (m) or as a part of kernel (y).
Build kernel as usual and after that you can mount LZMA compressed filesystems.
2.2 Build standalone module without patching and rebuilding the whole kernel.
Warning. I am not Linux guru. And i think that this method is not optimal (but it works). Any suggestions are welcome from experienced users.
For ARCH kernel here is PKGBUILD on AUR:
kernel26-squashfs-lzma-modules
Create directory (called for example squashfs-module) somewhere on your hard disk. Cd to that directory;
Download here Linux kernel sources (i strongly recommend to use same sources as in PKGBUILD of your kernel);
Download LZMA patches listed in 2.1;
Now you need kernel config file. Get it from your kernel package;
Unpack linux kernel sources. Cd to that directory;
Copy kernel config file here and rename it to .config;
Now we can patch our kernel:
patch -Np1 -i /pathto_051-squashfs_pcomp.patch
patch -Np1 -i /pathto_052-pcomp_lzma_support.patch
patch -Np1 -i /pathto_053-squashfs_lzma.patch
Before building modules we need some preparations:
make prepare
make modules_prepare
The kernel is patched and all preparations are ready. We can build modules:
make modules SUBDIRS=crypto
make modules SUBDIRS=fs/squashfs
Modules are ready. Now we need to copy it to our kernel modules directory:
cp crypto/unlzma.ko /lib/modules/KERNELVERSION/misc
cp fs/squashfs/squashfs.ko /lib/modules/KERNELVERSION/misc/squashfs_lzma.ko
Now execute:
depmod KERNELVERSION
Modules are installed and ready for use.Now you can load modules:
modprobe unlzma
modprobe squashfs_lzmaRemember to load both of them (unlzma and squashfs_lzma) and in that order.
I don't like this way. I think it is to possible to simplify this process. And this modules doesn't contain information about dependency. So you must manually load unlzma module.
How to use it
1) First we need to create SquashFS filesystem.
For GZIP:
mksquashfs source destFor LZMA:
mksquashfs.lzma source destFor example:
sudo mksquashfs.lzma /usr /usr.sqfs2) Now we can mount it as usual.
sudo mount -t squashfs -o loop /usr.sqfs /mountpointBenchmark
du -hs /usr/
3.3G /usr/GZIP (standard):
sudo mksquashfs /usr/ /usr.sqfs
Parallel mksquashfs: Using 2 processors
Creating 4.0 filesystem on /usr.sqfs, block size 131072.
[=============================\] 137140/137140 100%
Exportable Squashfs 4.0 filesystem, data block size 131072
compressed data, compressed metadata, compressed fragments
duplicates are removed
Filesystem size 1449060.58 Kbytes (1415.10 Mbytes)
46.41% of uncompressed filesystem size (3122543.79 Kbytes)
Inode table size 1365724 bytes (1333.71 Kbytes)
29.28% of uncompressed inode table size (4664165 bytes)
Directory table size 1437349 bytes (1403.66 Kbytes)
42.30% of uncompressed directory table size (3397611 bytes)
Number of duplicate files found 21775
Number of inodes 141741
Number of files 127438
Number of fragments 7820
Number of symbolic links 3853
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 10450
Number of ids (unique uids + gids) 6
Number of uids 2
root (0)
policykit (102)
Number of gids 6
root (0)
locate (21)
mail (12)
tty (5)
policykit (102)
dbus (81)
real 7m28.167s
user 11m18.825s
sys 0m4.214sdu -h /usr.sqfs
1.4G /usr.sqfsLZMA (patched):
sudo mksquashfs.lzma /usr/ /usr.lzma.sqfs
Parallel mksquashfs: Using 2 processors
Creating 4.0 filesystem on /usr.lzma.sqfs, block size 131072.
[=============================|] 137140/137140 100%
Exportable Squashfs 4.0 filesystem, data block size 131072
compressed data, compressed metadata, compressed fragments
duplicates are removed
Filesystem size 1326846.64 Kbytes (1295.75 Mbytes)
42.49% of uncompressed filesystem size (3122543.79 Kbytes)
Inode table size 1067747 bytes (1042.72 Kbytes)
22.89% of uncompressed inode table size (4664165 bytes)
Directory table size 1360891 bytes (1329.00 Kbytes)
40.05% of uncompressed directory table size (3397611 bytes)
Number of duplicate files found 21775
Number of inodes 141741
Number of files 127438
Number of fragments 7820
Number of symbolic links 3853
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 10450
Number of ids (unique uids + gids) 6
Number of uids 2
root (0)
policykit (102)
Number of gids 6
root (0)
locate (21)
mail (12)
tty (5)
policykit (102)
dbus (81)
real 22m51.122s
user 28m39.617s
sys 9m46.870sdu -h /usr.lzma.sqfs
1.3G /usr.lzma.sqfsResults
Size of /usr: 3.1 Gib (3.327.028.002) 131.201 files, 10.642 sub-folders
Size of usr.sqfs: 1.4 Gib (1.483.841.536)
Size of usr.lzma.sqfs: 1.3 GiB (1.358.692.352)
P.s. Sorry for my poor english. Any corrections are welcome.
P.p.s. Any suggestions, improvements are welcome.
Offline
#2 2009-11-01 14:47:52
- Dheart
- Member
- From: Sofia, Bulgaria
- Registered: 2006-10-26
- Posts: 956
Re: [How-To] SquashFS with LZMA compression
Such things should be posted in the wiki not the forums, this is not ubuntu forum.
My victim you are meant to be
No, you cannot hide nor flee
You know what I'm looking for
Pleasure your torture, I will endure...
Offline
#3 2009-11-01 15:56:07
- Svenstaro
- Administrator
- From: Germany
- Registered: 2008-11-19
- Posts: 388
Re: [How-To] SquashFS with LZMA compression
I appreciate this howto, although performance seems bad. I also suggest you put it on the wiki instead of the forums because more people will be able to find it that way. Any idea why performance seems so bad? 1.4G vs 1.3G doesn't seem like a good trade-off, especially considering the time it takes to do the lzma based compression.
Offline