How Linux accesses files

LiteHacker · 2008-12-10 15:48:38

Hello everyone,

I'm thinking of writing up a lot of programs that will all need to reference some type of configuration. (This configuration should be changeable by the user and by some helper configuration program). I'm thinking that probably the more simple way of doing this is by simply accessing some type of configuration file.

Now I'm coming from the Windows world so please excuse me if part of my question sound somewhat noobish.
I see that the majority of configuration files in Arch Linux are written in a way that is user friendly and standard. (ie, in /etc/rc.conf you could have comments and standard
someproperty = "value"

The only other type of configuration file I have seen is something of more like a shell batch file, like for wmii, there is wmiirc.
However it seems to me that this type is less efficient than the one noted above as a shell batch file requires having to repeatedly send various shell prompts to bash (meaning potentially running various programs. Running each program takes time as it requires placing each program into memory.)

Based on the standard form of these configuration files, I gather that there is some type of efficient program that quickly reads through any configuration file and parses it in whatever fashion required.

Am I correct? If not then I would have to manually read and parse this configuration file each time I have one of my programs turning on.
I thus ask how Linux reads files. If these programs keep accessing the file, would it mean Linux would take less time to access it? Note that the programs will only be reading this file once. I also assume that these programs probably won't be repeatedly run too many times.

Is there some alternative way to do this otherwise? Perhaps have some program in the background holding the configuration that would some how send the data without having to use the harddrive? (This background program would probably read the file when turning on.)
The only way I can think of doing that is by using local loopback socket. However I don't know what is the efficiency of doing that.

Well, anybody have any suggestions?
LiteHacker

Daenyth · 2008-12-10 16:10:59

The best way to do it depends on your specific configuration complexity and what programming language you are using.

Dusty · 2008-12-10 16:11:55

There are dozens of configuration syntaxes and, thankfully, also dozens of configuration parsers out there. That means you don't *really* need to worry about all these questions unless you have a specific reason to write your own parser (eg: to understand what's going on better).

As a python coder, I typically use config files written in python. Typically they just contain name = value pairs, but if the user wants to write python code that imports from elsewhere or loops over some configuration items, they're welcome to do so. The same applies to config files that use bash syntax; rc.conf is literally sourced by the arch initscripts and interpretted is bash code. Sometimes that gives them too much power over your program though.

In those cases, I would normally use the ConfigParser (http://docs.python.org/library/configparser.html), which basically allows you to use a windows ini syntax. Once you've let configParser parse your config file, it will typically store them in memory and make the values available to your program as variables or method calls.

There are also parsers for json, xml, and pretty much everything else, and there are bindings and libraries for pretty much every language. Its usually safe to assume these will use the more efficient methods of file access. I'd search google for "<insert preferred programming language> configuration file parser"

Dusty

LiteHacker · 2008-12-10 17:57:56

I am writing all of these programs in C, if that helps. The configuration will mainly consist of instructions of how to display something. For example to display an extremely basic button, you need a black line on the top, bottom, left, and right. ...the configuration would show how to draw the button. (The line instructions). These instructions would be used in varied ways. For example, when referencing the button information, you must note the width and height of the button.

What type of parser would you suggest for such use?

LiteHacker

phrakture · 2008-12-10 19:52:10

If you're doing it in C and want it to be as simple as possible, I would read the file line-by-line, use strtok to find the first '=' and set that to NULL, using the original string and the token++ as the key/value pairs.

Here's how pacman does it:
http://projects.archlinux.org/?p=pacman … =HEAD#l573

ornitorrincos · 2008-12-10 21:40:00

or something like

[section1]
option1
option2
option3

[section2]
...

you look for '[section1]\n'
and read until you find the blank line ('\n')

then look for '[section2]\n'

it's the easiest way I've find to parse a config file with several options per section(I've only done it with python, but should be also applicable to C, there might be easier and more efficient ones though)

LiteHacker · 2008-12-11 13:34:57

Indeed, I can always create some type of configuration standard, or use some type of configuration parser. There is no question of how to do it.
What I am asking is the efficiency of using a file that would have to be accessed from the hard drive each time. (maybe cached by Linux? If so, for how long?)
Would it be more efficient to have some program running in the background that sends the configuration via loopback socket?
Is there some type of background process already does something like this in Arch Linux for all programs that require it?
Perhaps an alternative method?

The whole goal of this is to have minimal start-up time. By having to read through a config file at start up, specially if the file is not in the cache, then the start-up time would take a bit longer.

Does anybody have insight on the efficiency of this stuff?

Thank you,
LiteHacker

peets · 2008-12-11 14:25:30

to find out how much time it takes to read a file, you can use something like

for i in {1..10000}; do echo "blablablablabla" >> thefile; done
time cat thefile

to get an approximation.

Hint: it's most probably negligible. Reading from a harddrive is "slow", but still very fast by "macrotemporal" standards.

phrakture · 2008-12-11 17:20:21

LiteHacker wrote:

What I am asking is the efficiency of using a file that would have to be accessed from the hard drive each time. (maybe cached by Linux? If so, for how long?)
Would it be more efficient to have some program running in the background that sends the configuration via loopback socket?
Is there some type of background process already does something like this in Arch Linux for all programs that require it?
Perhaps an alternative method?

This is way way overboard. Have you even coded it yet? It seems you're assuming you have a startup bottleneck before you actually have a startup bottleneck.

Code first, then profile, then fix the bottlenecks.

Knuth wrote:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Arkane · 2008-12-12 07:47:03

Also, and if this can help you rest easier, files read by linux are cached in RAM and stay there until the space is needed for something else. That's not really predictable but it works pretty well under reasonable loads.

Arch Linux

#1 2008-12-10 15:48:38

How Linux accesses files

#2 2008-12-10 16:10:59

Re: How Linux accesses files

#3 2008-12-10 16:11:55

Re: How Linux accesses files

#4 2008-12-10 17:57:56

Re: How Linux accesses files

#5 2008-12-10 19:52:10

Re: How Linux accesses files

#6 2008-12-10 21:40:00

Re: How Linux accesses files

#7 2008-12-11 13:34:57

Re: How Linux accesses files

#8 2008-12-11 14:25:30

Re: How Linux accesses files

#9 2008-12-11 17:20:21

Re: How Linux accesses files

#10 2008-12-12 07:47:03

Re: How Linux accesses files

Board footer