Arch Linux

Bellum

Member

Registered: 2011-08-24

Posts: 230

[SOLVED] Book suggestions?

Alright, I'd like to store a bunch of binary data in a potentially large file and quickly (read: need random access) get the portions of it I need. Seems like a pretty simple every-day task, but it's a gap in my education. What sort of cool magic tricks are good for this task, and what do I need to read to cover them?

Last edited by Bellum (2012-02-15 01:09:00)

Goran

Member

Registered: 2012-01-24

Posts: 92

Re: [SOLVED] Book suggestions?

Alright, I'd like to store a bunch of binary data in a potentially large file and quickly (read: need random access) get the portions of it I need. Seems like a pretty simple every-day task, but it's a gap in my education. What sort of cool magic tricks are good for this task, and what do I need to read to cover them?

Well, as far as I know, you can just seek to the beginning of the desired portion, and then read a block of bytes from that point.

If your portions are well defined  (they're all a set size in bytes), getting the the 50000th portion in the file is just a matter of doing something like this:

fseek(fp, 49999 * PORTION_SIZE, SEEK_SET);

portion_buffer = (char*) malloc (sizeof(char) * PORTION_SIZE);

fread(portion_buffer, 1, PORTION_SIZE, fp);

I think something along those lines would probably accomplish what you want, or did I misunderstand the problem?

Blµb

Member

Registered: 2008-02-10

Posts: 224

Re: [SOLVED] Book suggestions?

You want to take a look at mmap(2) (or 3).
This basically maps the file into memory and gives you a pointer to the data - you can specify offset into the file, and length.
This is usually a tiny bit faster than fseek/fread since it doesn't actually forcefully *read* the whole data you want to map, but works on an on-access basis AFAIK.
It also supports writing of course - you can sync your changes back to the file with msync() (which also happens on munmap())

---

You know you're paranoid when you start thinking random letters while typing a password.
A good post about vim
Python has no multithreading.

Bellum

Member

Registered: 2011-08-24

Posts: 230

Re: [SOLVED] Book suggestions?

mmap... That looks like a good tool for this job, thanks!