You are not logged in.

#1 2008-02-22 05:41:54

print
Member
Registered: 2007-02-27
Posts: 174

split binary data on delimiter?

I have some long wav files that are delimited at uneven intervals by a single, uniform "beep" tone.

I have done a lot of coding with text.  So I think in text.  Here's what I know how to do, for example in Ruby:

my_new_array = "aaaaa,bbbbb,cc,ddd,eeeeeeeeeeeeeeee,f,,g,h,iiiiiiiiiii".split(/,/)

That would give me an array of the individual elements split on the comma regex.  But if the data is binary, and the regex is also a chunk of binary data . . . how do I do this?  Normally I would just write a test script, but I don't even know how to manipulate binary that I could use to test my theory!  Can I use binary as an argument to a regex?  Can I just extract a piece and grep for it within the file?

Any suggestions or pointers much appreciated.

Last edited by print (2008-04-07 04:56:20)


% whereis whatis whence which whoami whois who

Offline

#2 2008-02-22 16:21:58

raeven
Member
From: the Herts of Crunchy Nut Land
Registered: 2008-02-19
Posts: 31
Website

Re: split binary data on delimiter?

i assume this 'beep' is BEL, which is ascii:7, you could try splitting the say, chr(7), or '\x07' . this is what i'd do in python, i dunno anything about ruby, but i don't think it'd be much if at all different.

if not, and you can't simply copy the delimiter into the .split() then maybe you could find out what its codes are and just use the chr() form. i.e beep_char = ord('*BEEP*')

e.g: beep_split.py

handle = open('file.wav', 'rb')
wav = handle.read().split('\x07')
handle.close()

you now have a python list of the chunks

hope that helped.

Last edited by raeven (2008-02-22 16:23:26)

Offline

#3 2008-02-22 22:48:03

print
Member
Registered: 2007-02-27
Posts: 174

Re: split binary data on delimiter?

Thanks so much for reading, and thanks for your reply.  Next question:  I am pretty sure it is just a random, though programmatically generated tone.  So how do I read the beep to get its binary "signature"?  If I zoom in on it using a spectral analyzer and save it to a file is there some way to read the value of the delimiter?  Sorry for the way I am describing this but I just don't have the vocabulary to describe it more precisely . . .

print

Last edited by print (2008-02-22 22:49:58)


% whereis whatis whence which whoami whois who

Offline

#4 2008-02-23 03:52:49

raeven
Member
From: the Herts of Crunchy Nut Land
Registered: 2008-02-19
Posts: 31
Website

Re: split binary data on delimiter?

hmm, i think this might be possible using something like gstreamer, to play it back and track the byte position,
then simply extract a couple of the beeps. then play it back in say audacity(if it even plays anymore lol) and this way we can try to narrow
the extraction so that we can say well all these beeps occur within a 128-136 byte range, if that's possible we may
be able to split the files in such a way that allows us to search each chunk and compare the hamming distance against our extracted beep/s

and hopefully that should be a relatively safe way to do it,
I know next to nothing about multimedia processing, so i can't say if this is possible. but it seems that way to me. wink

I'll do some experiments this weekend if i have time, and see what i can find, hopefully there's some multimedia genius in these forums that can provide some insight.

Offline

#5 2008-02-24 09:32:14

print
Member
Registered: 2007-02-27
Posts: 174

Re: split binary data on delimiter?

raeven:  Thanks for sticking with me.  Your comments gave me some great ideas.  Here's what I came up with, sorry it's not Python!

First thing I did was create some test files.  You can get them here:
http://downbe.at/pub/splitwav.rb/

That way I could actually tell what was up.  Then I spend an hour trying to figure out why it wouldn't split, until bham!  Remembered reading that wav files have headers.  Now I know what's it up, it was pretty easy . . .

if ARGV.length != 2
  puts "usage:  #{$0} <main audio file> <raw wav delimiter>"
  exit
end

tracks  = File.open("#{ARGV[0]}",'r').readline.to_s
delim   = File.open("#{ARGV[1]}",'r').readline.to_s
tracks  = tracks.split(delim).compact
pfmt    = "#{ARGV[0].split('.')[0]}_%0#{tracks.length.to_s.length}d"

i=0
tracks.each do |track| 
  fn = "#{sprintf(pfmt,i)}.wav"
  IO.popen("sox -t raw -r 44100 -1 -u -c 2 - -t wav #{fn}","r+") do |pipe|
    pipe << track
    pipe.close_write
    puts pipe.read
  end
  puts "wrote #{fn}"
  i += 1
end

Two caveats:
1)  Wave, I have learned, is a very pissy format, meaning that sox has strict requirements that you must specify the rate, number of channels, encoding, and sample size to get it to play nice.  Therefore, if your wav file is in a different format from mine, the above sox line would need to be adjusted.

2)  The way the above works is that you just pass a delimited wav file (normal wav) and a whatever the delimiter is, also a wav file, but the delimiter _must be a raw wav file.  (Raw wav is documented in the sox manpage.)  If you were to pass a normal wav file, it would contain a header which was not present in the delimiter itself, and therefore the call to split wouldn't work.  Poorly explained, but you get my drift.

Thanks again for your help, much appreciated.


% whereis whatis whence which whoami whois who

Offline

#6 2008-02-24 11:13:56

raeven
Member
From: the Herts of Crunchy Nut Land
Registered: 2008-02-19
Posts: 31
Website

Re: split binary data on delimiter?

your welcome, this is very interesting indeed. i think i might tinker with this when i get bored
maybe write a python implementation just for the sake of it, i dunno about ruby
but python's wave module allow for the extraction of such info as the # of channels and sample width
i like questions like these, it makes me look forward to becoming a computer scientist(i.e i won't be stuck doing boring research),
if I'm lucky enough...

Offline

#7 2008-02-24 19:54:13

raeven
Member
From: the Herts of Crunchy Nut Land
Registered: 2008-02-19
Posts: 31
Website

Re: split binary data on delimiter?

swave.py
differences to splitwav.rb:
  only depends on python,
  delimiter can be a valid wave file,
  optional prefix,
  requires no custom configuration -- all the required info is extracted from the original wave file.
that's about it i think

#!/usr/bin/env python
'''
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
#~ swave.py is written by Raeven K. Mason <kumico@nrk.cc>                    ~#
#~ this script is Public Domain                                              ~#
#~ no warrantly is given nor implied, although great care has been taken to- ~#
#~ ensure a high quality piece of code, you use it at your own risc.         ~#
#~                                                                           ~#
#~ a python implimetation of splitwav.rb by Noah K. Tilton <noah@downbe.at>  ~#
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
'''

import sys
import wave

def print_help():
    ''' print misc and usage info '''
    print 'A little program to split audio-delimited audio.'
    print 'Python version of `splitwav.rb\' by Noah K. Tilton <noah@downbe.at>'
    print
    print 'usage: %s origin.wav delim.wav [prefix]' % sys.argv[0]
    print ' `origin\' must the a valid wav file with header and footer in-tact'
    print ' `delim\' is automatically stripped if necessary'
    print ' \prefix\' is optional, and is derived from origin in absence'

def main():
    ''' exec script '''
    print 'swave.py, splitwav.rb but in python\n'
    
    argc = len(sys.argv)
    if argc < 3:
        print_help()
        sys.exit(1)

    if argc > 3:
        prefix = sys.argv[3]
    else:
        prefix = '%s_' % sys.argv[1].split('/')[-1].split('.')[0]

    try:
        wav = wave.open(sys.argv[1], 'rb')
        rate = wav.getframerate()
        width = wav.getsampwidth()
        channels = wav.getnchannels()
        origin = wav.readframes(wav.getnframes())
        wav.close()
    except (wave.Error, IOError):
        sys.stderr.write('Cannot load Origin: %s\n,' % sys.argv[1])
        sys.stderr.write(' maybe it\'s not a valid filename or wavefile.\n')
        sys.exit(1)

    try:
        wav = wave.open(sys.argv[2], 'rb')
        delim = wav.readframes(wav.getnframes())
        wav.close()
    except (wave.Error, IOError):
        try:
            wav = open(sys.argv[2], 'rb')
            delim = wav.read()
            wav.close()
        except IOError:
            delim = None
        if not delim:
            sys.stderr.write('Cannot load Delim %s\n' % sys.argv[2])
            sys.exit(1)

    segments = origin.split(delim)
    seg_len = len(segments)
    for (i, segment) in enumerate(segments):
        i += 1
        seg_name = '%s%d.wav' % (prefix, i)
        sys.stdout.write('writing part(%d/%d): %s ..' % (i, seg_len, seg_name))
        try:
            seg = wave.open(seg_name, 'wb')
            seg.setframerate(rate)
            seg.setsampwidth(width)
            seg.setnchannels(channels)
            seg.writeframes(segment)
            seg.close()
            print '. Done'
        except IOError:
            print '. Fail'
    print 'All Done!'

if __name__ == '__main__':
    main()

Last edited by raeven (2008-02-24 19:56:18)

Offline

#8 2008-02-25 13:10:03

print
Member
Registered: 2007-02-27
Posts: 174

Re: split binary data on delimiter?

Cool wink


% whereis whatis whence which whoami whois who

Offline

Board footer

Powered by FluxBB