fast compression of primitive ASCII?

dp · 2004-11-25 11:58:55

the problem:
--------------

big files (~200-1600MB) with primitive contents (genetic sequences: containing only "1" "2" "3" "4" and "-" and "|" chars)

for a project i'm doing myself (coded in java) i need this data to be plain (because i didn't add any compressor/decompressor lib)

what i'm now searching is a way to use less harddisk space but still be able to access (scroll, jump ...) the sequences (the big files) at a fast speed (say: goto(538285) would jump imediately to char 538285 and read +1000 and -1000 chars from this position)

the ways i'm thinking about:
--------------------------------

- adding some lib to the project to let java compress/decompress the data (PRO easy implementable CONTRA slow )

- using a cloop partition (PRO faster CONTRA no idea how to do it with java (out of the app))

- ???

to anyone with more experience in coding and compression: what would you suggest? any literature you would suggest me to read about compressing?

thanx in advance

z4ziggy · 2004-11-25 15:52:33

i think any use of compressed filesystem like cloop should be be better, since all the compression will be transparent to your java program, and u dont need to alter your sources (lazy-ass coders like me...), and u can always revet easily to non-compressed data... downside is, your program rely on the user providing the compression... but i see this also as advantage.
[EDIT]
also, user compression lets u change the compression library used on the fly.

Michel · 2004-11-25 19:25:30

There is a file-attribute to enable compression for that file or not, but it isn't supported wel yet I believe ...

dp · 2004-11-25 19:51:16

z4ziggy wrote:

i think any use of compressed filesystem like cloop should be be better, since all the compression will be transparent to your java program, and u dont need to alter your sources (lazy-ass coders like me...), and u can always revet easily to non-compressed data... downside is, your program rely on the user providing the compression... but i see this also as advantage.
[EDIT]
also, user compression lets u change the compression library used on the fly.

seen like this, you are absolutely right! as addition, the data would be still human-readable, as it is transperently compressed

i'll go for the cloop solution - thanx for reply

Arch Linux

#1 2004-11-25 11:58:55

fast compression of primitive ASCII?

#2 2004-11-25 15:52:33

Re: fast compression of primitive ASCII?

#3 2004-11-25 19:25:30

Re: fast compression of primitive ASCII?

#4 2004-11-25 19:51:16

Re: fast compression of primitive ASCII?

Board footer