fipolate: interpolate text files through FIFOs

Xyne · 2013-01-19 07:45:22

Project page: http://xyne.archlinux.ca/projects/fipolate/

I needed a simple way to pass files with passwords to wpa_supplicant without writing them to disk. I ended up with fipolate. It can interpolate arbitrary variables in a text file based on a user-defied regular expression. It can then pipe the interpolated text through a fifo (default) or write it to a file (e.g. on a ramdisk).

There may be betters ways to do this (feel free to share) but I found this useful.

If anyone knows how to check if something is listening on the other end of the fifo, please explain.

As always, question, comments, suggestions, bugs and other feedback are welcome.

Lord Bo · 2013-01-29 21:29:49

Xyne wrote:

If anyone knows how to check if something is listening on the other end of the fifo, please explain.

The answer is purly theoretical, since I never did that: Define a signal handle for the writing process to catch the SIGPIPE signal.
From Wikipedia:
SIGPIPE
The SIGPIPE signal is sent to a process when it attempts to write to a pipe without a process connected to the other end.
( http://en.wikipedia.org/wiki/Signal_(computing) )

Benjamin

Last edited by Lord Bo (2013-01-29 21:30:13)

Xyne · 2013-01-30 00:52:00

Thanks for the suggestion but it doesn't work in this case. The problem that I am trying to solve is postponing writing until the other end is connected. You can write to an unconnected pipe and it will just store the data in the buffer until something else connects. The SIGPIPE error is only thrown when writing to the pipe after something connects and then closes it.

The problem appears when there is no delay in the loop that write to the pipe and the other end is opened, read and closed in rapid succession. There is some race condition that enables the write loop to proceed multiple times while the other end is trying to read the file, so the content is multiplied on the other end for a single read. The interval prevents the race condition.

After reading your post I found another workaround: recreate the pipe after a write in the absence of an interval. This seems to do the trick and it is now an option, but recreating the FIFO multiple times is not ideal.

ackalker · 2013-01-30 03:00:19

Xyne wrote:

If anyone knows how to check if something is listening on the other end of the fifo, please explain.

Might seem difficult, as `fuser -v <fifo>` shows nothing. Why not just swap things around: let the reader create the fifo, so the writer can test for its existence (idea taken from http://www.chemie.fu-berlin.de/chemnet/ … bc_10.html).

If you don't want to spin in a sleep / test -p <fifo> loop waiting for the reader to come online, create another fifo in the writer and have it read a single line from that. The reader then can write a single line to this <signaling fifo> to signal the writer that it is ready.
No, this is not just a complicated way of simply writing to <fifo> in the first place, setting up <signaling fifo> enables you to postpone starting the process which is to write to <fifo> until there's someone to read from it, instead of that process blocking on write, tying up resources.

I hope this is clear enough, without using a diagram :-)

Last edited by ackalker (2013-01-30 03:34:27)

Xyne · 2013-01-30 10:42:43

Thanks for the rpely, but the goal of fipolate is to make sensitive data available to applications that expect a text file without writing that data to disk. The reader is completely independent.

For example. one of the uses of fipolate is to provide wpa_supplicant configuration files that contain passwords without writing them to disk. wpa_supplicant is not aware of fipolate and will simply try to read the specificed file. To achieve the solution that you describe, users would have to patch wpa_supplicant and any other application that they wish to use with fipolate. They may as well just patch that software to directly prompt for the sensitive data.

ackalker · 2013-01-30 20:27:31

No need to patch, simply use a wrapper. My guess is that systemd and even the venerable tcp wrappers use the same trick, but with sockets instead of fifo's.

Let's say I want a 'service' which tells users the exact date and time when they read from a fifo, and I want to be notified when that happens.
Here's what it looks like:

Setup the fifo:

$ mkfifo date-fifo

date-wrapper.sh:

#!/bin/sh
(
  echo -n "" # Prime the fifo
  echo Client connected... >&2
  exec date # Replace us with `date`, keeping stdout connected to the fifo
) > date-fifo

date-reader.sh:

#!/bin/sh
read d < date-fifo
echo "It is now: $d"

In one terminal, start the wrapper:

$ ./date-wrapper.sh

and in another, the client:

$ ./date-reader.sh

(This outputs the connection message to stderr, but you can just as easily write it to another fifo.)
The trick is in 'priming' the fifo by writing a zero-length string to it. This way both sides of the fifo will be open as soon as a client connects. but the date will still be 'exact', not something written to the fifo's buffer some time ago.
Although you will have to protect date-wrapper from modification, it doesn't (have to) know anything about the communication between `date` and the client.
You will also have to have something running which respawns 'date-wrapper', but there's no hurry, if a client connects before 'date-wrapper' is up again, it will simply block on read. When the message finally comes, it is correct. No race, no hassle.

Last edited by ackalker (2013-01-30 21:05:30)

Xyne · 2013-01-30 21:46:21

ackalker wrote:

My guess is that systemd and even the venerable tcp wrappers use the same trick, but with sockets instead of fifo's.

Sockets behave differently because both ends must be simultaneously connected. This is not so with fifos.

As for the rest of your post, I suspect that you are still misunderstanding the problem. The problem is not that the data is stale when it is written. The data is static.

The problem is that there is a race condition which can lead to the data being printed multiple times after the other end has connected but before it has read the fifo buffer.

Fipolate needs to run in a loop and keep making the data available, i.e. it has to open the fifo, write the data, close it, wait for the other end to read and close, then repeat. Opening is not a problem. Writing is not a problem either as it waits for the other end. Closing is fine as well. The problem is the loop. Unless there is a delay or the fifo is recreated, the loop can repeat fast enough for the fifo to be opened for writing again while the other ends is still reading.

A wrapper will not work. First, it requires control of the other process, which the user may not have (e.g. nested file reads within a given application such as netcfg/wpa_supplicant that reads the configuration file multiple times). Second, "you will also have to have something running which respawns..." is not a trivial afterthought. That is basically the problem that I am trying to solve (or rather, have solved, by recreating the pipe to ensure all data is flushed before additional writes).

Here's an example.

fipolite.py

#!/usr/bin/env python3

while True:
  try:
    with open('tmp.fifo', 'w') as f:
      f.write('foo')
      f.close()
      print("data written")
  except BrokenPipeError:
    continue

reader.sh

#!/bin/bash

while true
do
  cat tmp.fifo
  echo
done

Save both, then do this in one terminal:

$mkfifo tmp.fifo
$python3 fipolite.py

Now in another terminal:

$ cat tmp.fifo
foo
$ cat tmp.fifo
foo

That seem to work as expected, but try running "cat tmp.fifo" in rapid succession, which is what reader.sh does:

$ sh reader.sh
foo
foofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoo
foo
foo
foo
foo
foo
foo
foofoo
foo
foo
foo
foo
foo
foo
foofoo
foo
foo
foo
foofoo
foofoo
foo
foofoo
foo
foo
foo
foo
foofoofoo
foo
foo
foofoo
foo
foo
foofoofoofoofoofoo
foofoofoofoofoo
foo
foo
foofoofoofoo

Every line on which "foo" appears more than once is an instance of multiple write loops occurring during one reader loop.

ackalker · 2013-01-30 22:14:05

If you have no control over the other application, and no means for it to communicate its future intentions, how can you ever escape from these problems? If the reader has less delay between the reads than your writer has between (write / recreate fifo) then the reader might fail in all kinds of unsuspected ways (no such file, reading from the wrong fifo because its filesystem metadata hasn't been updated yet, etc.).

Have you considered making the 'file with sensitive data' available via FUSE?

Last edited by ackalker (2013-01-30 22:16:53)

Xyne · 2013-01-30 23:57:22

ackalker wrote:

If you have no control over the other application, and no means for it to communicate its future intentions, how can you ever escape from these problems? If the reader has less delay between the reads than your writer has between (write / recreate fifo) then the reader might fail in all kinds of unsuspected ways (no such file, reading from the wrong fifo because its filesystem metadata hasn't been updated yet, etc.).

The problem seems solved in the current version. If you find a bug, post steps to reproduce it.

Have you considered making the 'file with sensitive data' available via FUSE?

I would like a "cleaner" solution than recreating the fifo, but FUSE is definitely not it.

Last edited by Xyne (2013-01-30 23:57:39)

ackalker · 2013-01-31 04:34:11

Xyne wrote:

I would like a "cleaner" solution than recreating the fifo, but FUSE is definitely not it.

If you would like to elaborate a bit on why you don't like a solution like FUSE, I might be better able to help. Just stating that it is "definitely not it" isn't very informative.

Xyne · 2013-01-31 13:51:09

ackalker wrote:

If you would like to elaborate a bit on why you don't like a solution like FUSE, I might be better able to help. Just stating that it is "definitely not it" isn't very informative.

Using a user-space filesystem to provide a single file is overkill. It adds an extra dependency and requires a dedicated directory for a single file in this case.* It also requires the application to correctly manage permissions and attributes. Overall it will increase the complexity of the current code base while offering very little extra.

* Sure, you could use one instance and put all interpolated files in that directory, but that still lacks versatility and introduces further complexity for prompting the user for data and removing individual files.

avx · 2013-02-10 19:58:13

I never tried if FIFOs work as normal files in relation to inotify, but if they do, wouldn't it be enough using `inotifywait -e {access,modify} fifo`?

side1: initialize with "", wait for access, write real data
side2: check if "", thus producing access, check for modify and if yes, read again
(looping as needed if >1 read is needed?)

Xyne · 2013-02-12 19:04:30

@avx
I only have control of side 1, but the idea was workable. I now open the fifo, write the data, close it, and then watch for the IN_CLOSE_NOWRITE event to know when I can open the file for writing again.

It's not as dead-simple as I would like, but it avoids the FIFO recreation and other kludges that I had.

Thanks for the ideas.

Arch Linux

#1 2013-01-19 07:45:22

fipolate: interpolate text files through FIFOs

#2 2013-01-29 21:29:49

Re: fipolate: interpolate text files through FIFOs

#3 2013-01-30 00:52:00

Re: fipolate: interpolate text files through FIFOs

#4 2013-01-30 03:00:19

Re: fipolate: interpolate text files through FIFOs

#5 2013-01-30 10:42:43

Re: fipolate: interpolate text files through FIFOs

#6 2013-01-30 20:27:31

Re: fipolate: interpolate text files through FIFOs

#7 2013-01-30 21:46:21

Re: fipolate: interpolate text files through FIFOs

#8 2013-01-30 22:14:05

Re: fipolate: interpolate text files through FIFOs

#9 2013-01-30 23:57:22

Re: fipolate: interpolate text files through FIFOs

#10 2013-01-31 04:34:11

Re: fipolate: interpolate text files through FIFOs

#11 2013-01-31 13:51:09

Re: fipolate: interpolate text files through FIFOs

#12 2013-02-10 19:58:13

Re: fipolate: interpolate text files through FIFOs

#13 2013-02-12 19:04:30

Re: fipolate: interpolate text files through FIFOs

Board footer