Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

graysky · 2015-06-14 16:34:22

I'd like to take the output of `pwsafe --exportdb > database.txt` and convert it to a KeePassX XML friendly format (feature request in pwsafe).

I found flat file converter but the syntax is beyond me with this example. Solutions are welcomed.

More details
Here is the pwsafe --> KeePassX XML translations. The pwsafe export is simply a txt file with 6 fields (the first field can be ignored):

uuid= doesn't translate
group= group>title
name= entry>title
login= entry>username
passwd= entry>password
notes= entry>comment

Example txt file for conversion (exported from pwsafe):

# passwordsafe version 2.0 database"
uuid	group	name	login	passwd	notes
"123d9-daf-df-3423423"	"retail"	"amazon"	"myamazonuser"	"sjfJ849"	"superfluous comment"
"4599d934-dsfs-324"	"retail"	"netflix"	"netflixuser"	"dj3W$#"	""
"4kdfkd-434-jj"	"email"	"gmail"	"mygmail"	"dfkpass"	""

Example xml in keepassx xml:

<!DOCTYPE KEEPASSX_DATABASE>
<database>
 <group>
  <title>Internet</title>
  <entry>
   <title>github</title>
   <username>githubusername</username>
   <password>githubpassword</password>
   <comment>optional comment</comment>
  </entry>
 </group>
 <group>
  <title>retail</title>
  <entry>
   <title>amazon</title>
   <username>username</username>
   <password>myamazonpw</password>
  </entry>
 </group>
 <group>
  <title>retail</title>
  <entry>
   <title>netflix</title>
   <username>username</username>
   <password>mynfxpw</password>
  </entry>
 </group>
</database>

Last edited by graysky (2015-06-14 18:27:17)

mauritiusdadd · 2015-06-14 22:44:06

It seems that ffe does not work quite well with tab/space separated list (at least I was not able to get it working). However you can use sed to replace tabs with commas:

sed -e "s/\t/,/g" input.txt | ffe -c pwsafe.rc  -s example.txt  -l 2>/dev/null

and this is pwsafe.rc

     structure pwsafe {
         type separated , *
         output group
         quoted  
         header first
         record line {
             field uuid * * noprint
             field title
             field title * * entry
             field username * * entry
             field passwordd * * entry
             field comment * * entry
         }
     }

     output noprint {
         data ""
         no-data-print no
     }

     output entry {
         data "     <%n>%t</%n>\n"
         no-data-print no
     }

     output group {
         file_header "<!DOCTYPE KEEPASSX_DATABASE>\n <database>\n"
         record_header "  <group>\n"
         data "   <%n>%t</%n>\n    <entry>\n"
         record_trailer "    <entry>\n  </group>\n"
         file_trailer "</database>\n"
         no-data-print no
     }

If you are not bound to ffe, maybe it is easier to use a python script which is simpler to modify and it is more flexible. Something like the following should work:

#!/usr/bin/python

import sys
from xml.dom import minidom


class Converter(object):

    def __init__(self, filename):
        self.url = filename

    def convert(self):
        inp_f = open(self.url, 'r')

        data = inp_f.readlines()
        inp_f.close()

        # xml document model
        doc = minidom.Document()

        root = doc.createElement('database')
        doc.appendChild(root)

        for line in data:

            if '"' not in line:
                continue

            fields = line.split('\t')

            if len(fields) < 6:
                continue

            # uuid = fields[0].strip('"') # unused
            group = fields[1].strip('" ')
            name = fields[2].strip('" ')
            login = fields[3].strip('" ')
            passwd = fields[4].strip('" ')
            notes = fields[5].strip('" \n')

            group_node = doc.createElement('group')
            root.appendChild(group_node)

            # <group>
            group_title_node = doc.createElement('title')
            group_node.appendChild(group_title_node)
            group_title_node.appendChild(doc.createTextNode(group))

            # one <entry> per <group>
            entry_node = doc.createElement('entry')
            group_node.appendChild(entry_node)

            # <entry> -> <title>
            entry_title_node = doc.createElement('title')
            entry_title_node.appendChild(doc.createTextNode(name))
            entry_node.appendChild(entry_title_node)

            # <entry> -> <username>
            entry_uname_node = doc.createElement('username')
            entry_uname_node.appendChild(doc.createTextNode(login))
            entry_node.appendChild(entry_uname_node)

            # <entry> -> <password>
            entry_passwd_node = doc.createElement('password')
            entry_passwd_node.appendChild(doc.createTextNode(passwd))
            entry_node.appendChild(entry_passwd_node)

            # <entry> -> <comments>
            entry_comment_node = doc.createElement('comment')
            entry_comment_node.appendChild(doc.createTextNode(notes))
            entry_node.appendChild(entry_comment_node)

        print('<!DOCTYPE KEEPASSX_DATABASE>')
        print(root.toprettyxml(' ', '\n'))

if __name__ == "__main__":

    try:
        ifname = sys.argv[1]
    except IndexError:
        print("Input file name required")
        sys.exit(1)

    cc = Converter(ifname)
    cc.convert()

NOTE: I'm assuming from the example you posted that every <group> node contains only one <entry> child node.

--edit[0]: corrected a typo in the python code (just realized that it is <comment> instead of <comments ) and made the parsing function more reliable

Last edited by mauritiusdadd (2015-06-15 09:29:01)

graysky · 2015-06-14 23:17:30

Nice. No it doesn't need to be ffe at all... that was the most promising I found googling. Your python code works very well for this task. Thanks for putting it together.

mauritiusdadd · 2015-06-15 05:47:06

No problem, I've also updated the code because I noted that it was not parsing well the notes, showing

<comments>&quot;</comments>

instead of

<comment></comment>

A personal reminder: I should never code at night when I cannot sleep

Last edited by mauritiusdadd (2015-06-15 05:47:49)

graysky · 2015-06-15 09:27:28

Do you mind if I post your script to a github repo... or better yet do you have one of your own?

mauritiusdadd · 2015-06-15 09:46:16

Done, I've pushed the script on my github: https://github.com/mauritiusdadd/pwsafe2keepassx

graysky · 2015-06-15 09:54:19

Nice, thanks! I did find an error (due to my miscommunicaton, not your code): if users have multiple entries under the same group in the pwsafe dump, they are not correctly grouped within the keepassx XML file your script generates and thus are not grouped in the keepassx database.

To illustrate, take the example I provided above: the two "retail" groups should appear together in the generated xml so some sorting is needed. Here is the correctly formatted XML with the two retail entries grouped. Is it simple for you to modify?

<!DOCTYPE KEEPASSX_DATABASE>
<database>
 <group>
  <title>email</title>
  <entry>
   <title>gmail</title>
   <username>mygmail</username>
   <password>dfkpass</password>
   <comment></comment>
  </entry>
 </group>
 <group>
  <title>retail</title>
  <entry>
   <title>amazon</title>
   <username>myamazonuser</username>
   <password>sjfJ849</password>
   <comment>superfluous comment</comment>
  </entry>
  <entry>
   <title>netflix</title>
   <username>netflixuser</username>
   <password>dj3W$#</password>
   <comment></comment>
  </entry>
 </group>
</database>

Last edited by graysky (2015-06-15 09:55:45)

mauritiusdadd · 2015-06-15 09:58:04

It does not seems hard to do, I will update the script as soon as I can.

mauritiusdadd · 2015-06-15 10:27:51

Done, see if it works now and let me know if it needs other fixes.

graysky · 2015-06-15 10:34:40

mauritiusdadd wrote:

Done, see if it works now and let me know if it needs other fixes.

Works beautifully! Thanks again. I have a sizable pwsafe export I will work on and report back.

Arch Linux

#1 2015-06-14 16:34:22

Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

#2 2015-06-14 22:44:06

Re: Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

#3 2015-06-14 23:17:30

Re: Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

#4 2015-06-15 05:47:06

Re: Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

#5 2015-06-15 09:27:28

Re: Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

#6 2015-06-15 09:46:16

Re: Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

#7 2015-06-15 09:54:19

Re: Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

#8 2015-06-15 09:58:04

Re: Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

#9 2015-06-15 10:27:51

Re: Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

#10 2015-06-15 10:34:40

Re: Convert tab separated text to non-trivial xml. (pwsafe --> KeePassx)

Board footer