Arch Linux

Gullible Jones

Member

Registered: 2004-12-29

Posts: 4,863

Python, mmap, and making stuff stay (usable) in memory

I was looking at the source code of go-preload and saw that it used mmap() to load stuff into RAM/swap. Python also lets you mmap stuff, so I thought, "Why not write something like this in Python?"

Unfortunately I don't really know what I'm doing. I wrote this little test script:

import os 
import glob 
import mmap 
import time 
 
path = '/usr/lib/firefox-12.0/' 
for infile in glob.glob(os.path.join(path, '*.so*')): 
    f = open(infile, "r") 
    mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ) 
    f.close()
while True: 
    time.sleep(1)

If I drop all caches and run it, it grinds the hard drive briefly. As far as I can tell, it might as well be doing nothing; it doesn't make any difference in Firefox start times after dropping caches (even though mmapped files are supposed to be sharable by default in Python). Clearly, I am doing something very wrong.

So: how can I load a file into memory, keep it in memory, and make it possible for other things to access it in memory?

P.S. Also, what's the best way in Python to get a list of the shared libraries a program uses?

Edit: tried something that I figured would be better:

import os
import mmap
import time

path = '/usr/lib/libreoffice/'
for root, dirs, files in os.walk(path):
    for afile in files:
        filepath = os.path.join(root, afile)
        f = open(filepath, "r")
        if (os.path.getsize(filepath) != 0):
            mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
        f.close()
while True:
    time.sleep(1)

This quite clearly is not reading anything into memory at all.

What's going on? Does each mmapped file have to be assigned to a separate variable?

Edit again: putting each mmapped file in a variable and appending that to a global list didn't do anything either. The files are still not being loaded into memory.

Last edited by Gullible Jones (2012-05-18 19:19:52)

Tes--

Member

Registered: 2009-11-13

Posts: 44

Re: Python, mmap, and making stuff stay (usable) in memory

Well, it works as intended then. mmap mapped areas will be flushed from memory when needed (as they are always backed by a file).

If you want to load the files into the cache, just read them. (open(filepath).read() or in a loop for big files) You can also read from the mmap, but that adds nothing (for this purpose at least).

Gullible Jones

Member

Registered: 2004-12-29

Posts: 4,863

Re: Python, mmap, and making stuff stay (usable) in memory

Oh... Thanks.

FWIW I did try using readlines(), and that definitely read the files, but they were not cached - the same files were read again later when I started LibreOffice. Does read() work differently?

Edit: read() works, thank you!

Last edited by Gullible Jones (2012-05-19 16:32:04)