[Solved] Trying to grep a hex pattern from a binary dump - not working

Baz_ · 2015-04-01 23:25:25

Hi all, I'm not sure if this is the right forum but as it is related to a scripting course I'm doing I couldn't see anywhere more suitable.

I have a raw disk dump that contains several jpeg images etc... I'm trying to make a carving utility to extract the images using bash and ultimately grep to do the heavy lifting.

What I'm trying to do is to grep hex escaped character values within the raw disk dump to find the location where a jpeg image starts. The jpeg image file header in hex is "ffd8ff". The most straight forward example I could find for how to achieve this was from here: http://www.commandlinefu.com/commands/v … l-patterns

According to that page I should be able to execute grep -P "\xff\xd8\xff" diskdump.raw in order to be able to get some sort of match, however in spite of my certainty that there are a number of jpeg images in the disk dump (I created the disk dump) and the fact that I can use hex dump utilities to confirm the presence of the matching header grep is refusing to play ball and acknowledge a match. I don't know what to do, other people on the course are able to use grep this way with different distributions.

Does anyone know if this is a known issue, has anyone come across it before and, better yet, does anyone know how I can get grep to bend to my will on Arch?

Thanks in advance,

B

Last edited by Baz_ (2015-04-03 08:55:11)

ewaller · 2015-04-02 00:04:35

A quick and nasty (and buggy) test would be:

od -xAx theFileName | grep "ff \?d8 \?ff"

The od means "octal" dump. The x means show the data in hex, the Ax means show the address in hex. Pipe through grep which looks for ff, zero or one spaces, d8, zero or one spaces, and ff. If it finds everything it prints the entire line.
It will fail when the first or last byte are not on the same line as other two. It might also do weird things if the address start to match. Having said that, dropping the Ax from the od command will keep the addresses from matching, but the addresses will be in octal -- which should have gone the way of the passenger pigeon when 18 bit machines ceased to exist.

Edit: Fixed my extinct pigeon reference

Last edited by ewaller (2015-04-02 00:07:47)

Trilby · 2015-04-02 00:07:33

I think you'll need the -a flag for grep (unless you use ewaller's od approach):

man grep wrote:

-a, --text
Process a binary file as if it were text; this is equivalent to the --binary-files=text option.

Baz_ · 2015-04-02 21:29:56

Trilby wrote:

I think you'll need the -a flag for grep (unless you use ewaller's od approach):
man grep wrote:
-a, --text
Process a binary file as if it were text; this is equivalent to the --binary-files=text option.

I have tried "grep -a ...", "grep -Pa ..." and the version that works for my class mate "grep -obPa ..." plus several other variations of the above and none of them work.

I have a working version of a carver similar to ewaller's idea using a hexdump utility (xxd) rather than od but because of the fact that the header and footer could roll over from one line to another it is extremely inefficient as it requires two searches through the file to ensure that you have found the earliest header (the second search pads a few characters to the front of the raw dump depending on the size of the header).

The ability to search for a string of characters by their hex values with grep would lead to much more efficient code.

I don't suppose grep is due an update soon that might cause it to do what I want? Or even better is someone knows how to get grep as it stands to do what I want?

Thanks for your posts so far guys, much appreciated.

B

Baz_ · 2015-04-02 23:59:55

Okay so I started off by downgrading my grep (believing the version to be at fault) and initially came up with the error text: "grep: invalid UTF-8 byte sequence in input" in version 2.20.

A search for this text led me to this page: https://bugs.debian.org/cgi-bin/bugrepo … bug=730472

This page mentions a bit of fiddling with the LC_ALL environment variable, and when I set "LC_ALL=C" followed immediately by the grep search I wished to carry out it worked, so I restored my version of grep to be fully updated tried again and voila! It still appears to work.

Although I'm not a complete noob, I'm still noobish and if anyone could test that I'm on the right track or could give a reasonably simple explanation as to why this works I would be much appreciative.

Either way, I'm much happier now...

B

Baz_ · 2015-04-03 00:03:50

Is there any point marking this as solved, or can I even do it?

ewaller · 2015-04-03 05:45:07

As we don't know when you consider it solved, we leave it to you. Edit your first post in the thread and you can change the title. Just prepend [SOLVED] to the title and you are all set.

Edit... But yo have a long title, so [SOLVED] might not fit. Do your best. Either shorten the title or just ignore marking it solved.

Last edited by ewaller (2015-04-03 05:46:35)

Arch Linux

#1 2015-04-01 23:25:25

[Solved] Trying to grep a hex pattern from a binary dump - not working

#2 2015-04-02 00:04:35

Re: [Solved] Trying to grep a hex pattern from a binary dump - not working

#3 2015-04-02 00:07:33

Re: [Solved] Trying to grep a hex pattern from a binary dump - not working

#4 2015-04-02 21:29:56

Re: [Solved] Trying to grep a hex pattern from a binary dump - not working

#5 2015-04-02 23:59:55

Re: [Solved] Trying to grep a hex pattern from a binary dump - not working

#6 2015-04-03 00:03:50

Re: [Solved] Trying to grep a hex pattern from a binary dump - not working

#7 2015-04-03 05:45:07

Re: [Solved] Trying to grep a hex pattern from a binary dump - not working

Board footer