GNU parallel treats input as a single token [SOLVED]

graysky · 2012-11-22 15:48:15

I have a bash script that expects three tokens that I'd like to push through parallel. The problem I am experiencing is that when I cat a text file and pipe it into parallel, the spaces in the tokens are not treated as such; they are treated as a single token.

Here is a simplified script: ~/bin/action

#!/bin/bash
echo "first = $1"
echo "second = $2"
echo "third = $3"

If I simply run it like so, I get the expected output:

% ~/bin/action a b c
first = a
second = b
third = c

So to use parallel, I place the "a b c" into a text file that I cat and pipe into parallel but all of the tokens become $1 as you can see:

% echo "a b c" > worklist
% cat worklist | parallel ~/bin/action
first = a b c
second = 
third =

Please help me understand the error of my way

Last edited by graysky (2012-11-24 14:38:40)

falconindy · 2012-11-22 16:02:48

Did you not find the manpage?

--delimiter delim
-d delim
Input items are terminated by the specified character. Quotes and backslash are not special; every character in the input is taken literally. Disables the end-of-file string, which is treated like any other argument. This can be used when the input consists of simply newline-separated items, although it is almost always better to design your program to use --null where this is possible. The specified delimiter may be a single character, a C-style character escape such as \n, or an octal or hexadecimal escape code. Octal and hexadecimal escape codes are understood as for the printf command. Multibyte characters are not supported.

Last edited by falconindy (2012-11-22 16:03:07)

graysky · 2012-11-22 16:11:50

@falconindy - I saw that but am unclear how to make use of it to treat items in a line as tokens... for example: my goal is to have a whole list of items in the worklist and have them treated as $1 $2 and $3 on a per-line basis.

% cat worklist
a b c
d e f
g h i
j k l
m n o
p q r
s t u
v w x
y z 1

So the first command should be processed as:

~/bin/action a b c

The 2nd as:

~/bin/action d e f

The 3rd as:

~/bin/action g h i

...you get the idea.

Last edited by graysky (2012-11-22 16:12:52)

falconindy · 2012-11-23 00:59:04

I would think this would be easier just using bash, then...

#!/bin/bash

worker() {
  dostuff "$@"
}

maxjobs=4
while read -ra v; do
  # ratelimit max inflight jobs
  while (( $(jobs | wc -l) >= maxjobs )); do
    sleep 0.1
    jobs >/dev/null
  done

  # dispatch in the background
  worker "${v[@]}" &
done <"$inputfile"

# let children finish before the script exits
wait

graysky · 2012-11-24 14:38:29

@falconindy - Thank you for the suggestion. It seems that the correct way to invoke parallel in the fashion I want is to use the -a switch so you were close

parallel -a worklist --colsep ' ' ~/bin/action
first = a
second = b
third = c
first = d
second = e
third = f
first = g
second = h
third = i
first = j
second = k
third = l
first = m
second = n
third = o
first = p
second = q
third = r
first = s
second = t
third = u
first = v
second = w
third = x
first = y
second = z
third = 1

http://www.gnu.org/software/parallel/ma … e_as_input

Powerful stuff once you learn how to invoke it

Last edited by graysky (2012-11-24 15:11:40)

Arch Linux

#1 2012-11-22 15:48:15

GNU parallel treats input as a single token [SOLVED]

#2 2012-11-22 16:02:48

Re: GNU parallel treats input as a single token [SOLVED]

#3 2012-11-22 16:11:50

Re: GNU parallel treats input as a single token [SOLVED]

#4 2012-11-23 00:59:04

Re: GNU parallel treats input as a single token [SOLVED]

#5 2012-11-24 14:38:29

Re: GNU parallel treats input as a single token [SOLVED]

Board footer