Bash Overflow [SOLVED]

Jclifton44 · 2018-11-18 14:07:27

Hello,

I have been writing a script that will help me find all dependencies for any given non-binary executable.

The script parses the file and will find the package owner for each program that will run in an abitrary other script. As an intermediate script I've created a script that gets all unique programs within a file.

As an example, suppose I am writing a script to get the UUID for my fs.

#fsUUID.sh


fsUUID=$(grub-probe --target=fs_uuid /boot/EFI/Microsoft/Boot/bootMicrosoft.efi)
stringHints=$(grub-probe --target=hints_string /boot/EFI/Microsoft/Boot/bootMicrosoft.efi)
sed "s/#MicrosoftHints/search --fs-uuid --set=root $stringHints $fsUUID/" grub.cfg > file
echo "test"
echo $fsUUID

./getPrograms.sh fsUUID.sh

If I run my script for fsUUID.sh it will find sed and echo and create a unique list of programs in the script.

It works for larger preloaded programs.

./getPrograms.sh $(which mkinitcpio)

When I run this program I've created for the binary executable sed, it may overflow into the /usr/bash shell I'm using. My shell becomes symbols and signs that make no sense.

It may be because of the size of sed is considerably larger than the other files I've tested it with. I'm sure this is not intended functionality for the Bourne Again Shell. This may be a bug.

https://ibb.co/hypo70

Link to the script:

https://github.com/jclifton44/shell-scr … rograms.sh

Reproduce with:

./getPrograms.sh $(which sed)

Last edited by Jclifton44 (2025-01-28 12:53:14)

Trilby · 2018-11-18 14:20:08

So you have some script that you have no provided that implements some unspecified algorithm to attempt to get certain names out of a text-file script but then you run it on a binary file it gives binary junk.

Yeah. No surprise.

If you want help, show the script and give exact commands being used.

Please read the following:
https://wiki.archlinux.org/index.php/Co … ow_to_post
http://catb.org/~esr/faqs/smart-questions.html

Last edited by Trilby (2018-11-18 14:21:52)

WorMzy · 2018-11-18 14:35:23

You're doing something that garbles your terminal. You can run 'reset' to fix that, but without knowing what it is you're actually doing, we can't help.

Mod note: Moving to NC.

Jclifton44 · 2018-11-18 14:35:32

I've edited my original post to include the code and steps to reproduce.

Reset made my shell readable again.

Last edited by Jclifton44 (2018-11-18 14:38:27)

Trilby · 2018-11-18 14:46:37

Note the following are equivalent, but the second *much* more efficient:

# yours
while read line; do
        command=$(echo $line | sed -s 's/^\s*//g' | sed -s 's/ .*$//g')
        [ ! -z $command ] && map[$command]="true"
done < $1
# mine
sed 's/^\s*//;s/ .*$//' "$1" | while read command; do
   [ ! -z $command ] && map[$command]="true"
done
# or even better, just don't grab zero-length strings:
sed 's/^\s*\([^\s][^\s]*\).*$/\1/' "$1" | while read command; do
   map[$command]="true"
done
# of course at that point, just get rid of the loop:
map=( $(sed 's/^\s*\([^\s][^\s]*\).*$/\1/' "$1") )

After that the second half of your script just loops through the array (with a needless subshell and echo) then tests that each entry is not an empty string (which was already done) before printing it. Just print the array. This whole script could be a sed one-liner:

sed 's/^\s*\([^\s][^\s]*\).*$/\1/' scriptfile

But this is not getting the commands within a script, it's literally just getting the first (non-whitespace) token from every line. So if you are just grabbing random 'non-space' elements of the file provided, why is it surprising that when you run this on a binary file it gives binary output?

Last edited by Trilby (2018-11-18 14:50:42)

Jclifton44 · 2018-11-18 15:14:02

It's not very surprising. The output is binary because the file is binary. What is surprising is that all output thereafter on the shell is also binary.

Trilby · 2018-11-18 15:17:29

Is this solved then? If so, please edit your first post to prepend [SOLVED] to the title (and consider revising the title to actually represent the problem).

Jclifton44 · 2018-11-18 15:23:42

I don't think it's solved. The Bourne Again Shell is not functioning like a shell. The garbled text is cleared with reset.

Try to reproduce the bug. The shell isn't behaving as expected.

Trilby · 2018-11-18 16:59:34

It looks to me like it is behaving exactly as expected. What do you expect it to do? How does what you see differ from what you expect?

You wrote a script that extracts portions of a file and prints them to the screen, then you ran that script on a binary file; it extracted portions of the binary file and printed them to the screen. The shell is doing exactly what you told it to do.

On another note, parsing the text of a script to see what binaries it calls will never work. Use `set -x` or parse `strace -f` output, the former is a bit simpler, but will also list shell builtins:

PS4='' /path/to/script.sh 2>&1 1>/dev/null | cut -d' ' -f 1 | sort -u

# or, probably better:
strace  -f  /path/to/script.sh 2>&1 1>/dev/null | sed -n 's/.*execve("\([^"]*\).*/\1/p'

# and to get packages
strace  -f  /path/to/script.sh 2>&1 1>/dev/null | \
   sed -n 's/.*execve("\([^"]*\).*/\1/p' | \
   pacman -Qqo - | sort -u

Last edited by Trilby (2018-11-18 17:29:40)

Jclifton44 · 2018-11-18 17:38:11

Your scripting looks like it will be very helpful for what I was originally trying to do. I will test it as soon as I have access to a shell. Thank you.

I expect the script I created to output random characters when parsing a binary file with random nonsensical characters. When that script completes execution, it should return control back to the calling shell. Instead it returns control back to a shell that outputs the same nonsensical characters.

I will post more info when I have access to a shell.

Morganamilo · 2018-11-18 17:47:57

The terminal uses escape sequences[1] to do things such as setting colours, moving the cursor and stuff.
When you send random output to the terminal you're likely to end up printing some of those escape codes and causing strange behaviour.
This is not a bug, it's expected behaviour.

[1] https://en.wikipedia.org/wiki/ANSI_escape_code

Last edited by Morganamilo (2018-11-18 17:48:17)

Trilby · 2018-11-18 18:57:31

Jclifton44 wrote:

Instead it returns control back to a shell that outputs the same nonsensical characters.

It's the same shell. Just characters are displayed as apparently nonsensical. Commands will still work fine, you just will not be able to read them. But the fix to this has already been provided: `reset`. I'm not sure what more of a solution you could want.

Jclifton44 · 2018-11-18 22:12:36

I'm not seeking a solution. If I wanted to gain my sensical shell I could just reset the computer. I am saying that the shell should not be outputting those characters.

From the user's perspective the shell will seem unusable when the shell text is garbled. Sure you can type r e s e t and press enter. From the user's perspective there is no shell active. The screen just displays symbols.

I've simplified steps to reproduce a bit.

 Array=()
while read line
do
    Array+=($(echo $line | cut-c-2))
done < $(which sed)
echo ${Array[98]}

It's not a problem I'm unable to get around.

I'm thinking array entry 98 is not null terminated and is overflowing into the shell space.

Last edited by Jclifton44 (2018-11-18 22:13:29)

Trilby · 2018-11-18 22:18:21

Jclifton44 wrote:

I'm not seeking a solution.

Then you are posting on the wrong website.

Jclifton44 wrote:

I am saying that the shell should not be outputting those characters.

It's only doing so because you specifically told it to. But feel free to file a bug report with the GNU to complain that core utilities should not do what they are told.

eschwartz · 2018-11-18 22:22:43

I'm thinking if you object to this happening, then you should complain to the bash developers, since this is not Arch Linux specific and will happen on all Linux distributions, macOS, the *BSDs, and Windows, in addition to all past and future operating systems without exception.

And furthermore consider the fact that the cause of this problem has been known for decades, and the canonical solution has been declared to be "don't do that, and if you accidentally did, use the reset command".

Also: Don't do that, you'll mess up your shell. Your script is buggy.

Last edited by eschwartz (2018-11-18 22:23:32)

Jclifton44 · 2018-11-18 22:22:51

Maybe. It's operating specifically how it's been given instructions to operate. This could be only a graphical error.

Maybe there are some escape characters that cause output to go wild.

I marked it as solved. I can save this for another time.

Last edited by Jclifton44 (2018-11-18 22:38:06)

Jclifton44 · 2018-11-19 00:00:36

Trilby,

The sed commands you listed don't quite accomplish what I am aiming for.
I can obtain the first token ending in whitespace with that I have.
I'm considering using 'file' to determine what kind of file the program is (E.g. file $(which sed)) and from there recursively call the script or find the package owner with pacman Qo.

Trilby · 2018-11-19 00:50:30

My main point was that dependencies of a script and the first token on every line would not have any correlation:

#!/bin/sh
mounts=$( findmnt )
echo ${mounts[1]} | grep -q 'sda1' && echo "yup"

This script requires `findmnt` and `grep` to run, but grabbing the first token on the line would not detect these. Instead the first tokens would be "mounts=$(" (which is gibberish and at best would cause 'file' to error out) and "echo". Use the strace-based solution if you need to find what the script actually calls.

Jclifton44 · 2018-11-19 01:51:20

I'm considering both implementations. I've already created a solution in the way I was describing. I call Which for each parsed token. If the process returns 0, I move onto pacman Qo.

Consider parsing the file for all $ symbols. Would that catch almost everything? I'm trying to avoid downloading large libraries, packages and programs. I want this to function on the standard Arch ISO with no additional installation.

eschwartz · 2018-11-19 01:55:22

What???

Look, unless you manage to write an actually effective AST parser for bash, you're never going to approach the same accuracy as simply executing the script and stracing it. It's simply madness -- you cannot really do static analysis of bash. bash is not that language. The very best tools are only capable of guessing at common patterns.

Jclifton44 · 2018-11-19 02:19:27

I wasn't thinking of making a syntax tree. I would start with C or C++ to do something like that; however, with indirect references it may be possible.

Plus, you'd have to get into the nitty gritty details. It sounds like it would be a big project.

Last edited by Jclifton44 (2018-11-19 02:21:35)

Arch Linux

#1 2018-11-18 14:07:27

Bash Overflow [SOLVED]

#2 2018-11-18 14:20:08

Re: Bash Overflow [SOLVED]

#3 2018-11-18 14:35:23

Re: Bash Overflow [SOLVED]

#4 2018-11-18 14:35:32

Re: Bash Overflow [SOLVED]

#5 2018-11-18 14:46:37

Re: Bash Overflow [SOLVED]

#6 2018-11-18 15:14:02

Re: Bash Overflow [SOLVED]

#7 2018-11-18 15:17:29

Re: Bash Overflow [SOLVED]

#8 2018-11-18 15:23:42

Re: Bash Overflow [SOLVED]

#9 2018-11-18 16:59:34

Re: Bash Overflow [SOLVED]

#10 2018-11-18 17:38:11

Re: Bash Overflow [SOLVED]

#11 2018-11-18 17:47:57

Re: Bash Overflow [SOLVED]

#12 2018-11-18 18:57:31

Re: Bash Overflow [SOLVED]

#13 2018-11-18 22:12:36

Re: Bash Overflow [SOLVED]

#14 2018-11-18 22:18:21

Re: Bash Overflow [SOLVED]

#15 2018-11-18 22:22:43

Re: Bash Overflow [SOLVED]

#16 2018-11-18 22:22:51

Re: Bash Overflow [SOLVED]

#17 2018-11-19 00:00:36

Re: Bash Overflow [SOLVED]

#18 2018-11-19 00:50:30

Re: Bash Overflow [SOLVED]

#19 2018-11-19 01:51:20

Re: Bash Overflow [SOLVED]

#20 2018-11-19 01:55:22

Re: Bash Overflow [SOLVED]

#21 2018-11-19 02:19:27

Re: Bash Overflow [SOLVED]

Board footer