Arch Linux

catnap

Member

Registered: 2016-10-03

Posts: 131

[SOLVED] Ignoring files in Git's FastImport

I'm currently attempting to convert all my Bazaar repositories to Git repositories so that I could combine all my version controlled repositories under one master repository for easy universal syncing over different computers and backup devices. Mostly I followed the instructions given here, https://development.robinwinslow.uk/201 … t-and-bzr/, but instead of installing the Bazaar's FastImport plugin manually, I used the command aurman -S bzr-fastimport, which automatically installed the relevant Python module. When I ran the command bzr fast-export --plain . | git fast-import  , I encoutered the following error:

fatal: Missing space after source: R "My Maps/Muu/Getting past "no".mm" My Maps/Neuvottelu/Getting past "no".mm
fast-import: dumping crash report to .git/fast_import_crash_16118
bzr: broken pipe

It seems like the error has to do with the formating of the file name My Maps/Muu/Getting past "no".mm which includes special characters. I know that the problem is in Git's FastImport rather than in Bazaar's exporting functionality because I had success trying only the first part, bzr fast-export --plain . > /dev/null , of the above command.

How could I make the importing program to ignore this part of the repository or to alter the name of the file in the history of the repository? The current name of the file has already been changed to steer away from special characters but without an effect that would solve the problem.

An important CORRECTION (added 2018-06-05 19:50):
The problem necessarily has to be in Bazaar's export rather than in Git's import because the error includes the message bzr: broken pipe. Why the first part of the command seems to succeed, I cannot tell.

Last edited by catnap (2018-06-06 15:37:22)

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 77,909

Re: [SOLVED] Ignoring files in Git's FastImport

You could pipe it through sed to (probably that's the problem?) escape the quotes?

sed 's/"/\\"/g'

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

catnap

Member

Registered: 2016-10-03

Posts: 131

Re: [SOLVED] Ignoring files in Git's FastImport

You could pipe it through sed to (probably that's the problem?) escape the quotes?

I have now tried this solution. Unfortunately, the command also seems to escape something that should not be escaped because I get another error message.

17:55:45 Calculating the revisions to include ...
17:55:45 Starting export of 214 revisions ...
fatal: Unsupported command: < ... some random text here, omitted for privacy reasons ...   >
fast-import: dumping crash report to .git/fast_import_crash_18153
Unpacking objects: 100% (3/3), done.
bzr: broken pipe

At least, the error message changed which is often the hall-mark of progress with issues like these.

eschwartz

Fellow

Registered: 2014-08-08

Posts: 4,097

Re: [SOLVED] Ignoring files in Git's FastImport

Maybe http://www.catb.org/esr/reposurgeon/ would let you fix this properly? It's designed to process the fast-export dump before importing it into a new VCS, so you should be able to edit history and use sane filenames.

---

Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

catnap

Member

Registered: 2016-10-03

Posts: 131

Re: [SOLVED] Ignoring files in Git's FastImport

Maybe http://www.catb.org/esr/reposurgeon/ would let you fix this properly?

This probably really is what I need. However, the program does not seem intuitive. Can you pinpoint any simple usecase of this program that would get me started with it.

catnap

Member

Registered: 2016-10-03

Posts: 131

Re: [SOLVED] Ignoring files in Git's FastImport

Maybe http://www.catb.org/esr/reposurgeon/ would let you fix this properly?

Unfortunately Reposurgeon crashes about the same way that Bazaar export did. It gives the same error message: bzr: broken pipe

eschwartz

Fellow

Registered: 2014-08-08

Posts: 4,097

Re: [SOLVED] Ignoring files in Git's FastImport

I've never used it, just heard about it.

Is the error message coming from git fast-import, or from reposurgeon? I believe it eventually tries to use git fast-import on the post-processed contents, so maybe you never modified the filename at all.

---

Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

catnap

Member

Registered: 2016-10-03

Posts: 131

Re: [SOLVED] Ignoring files in Git's FastImport

Is the error message coming from git fast-import, or from reposurgeon? I believe it eventually tries to use git fast-import on the post-processed contents, so maybe you never modified the filename at all.

I only used the read . command. I did not use edit or any of the other various commands. Maybe I just mistook how the program works. If I find a tutorial, I will share a link for future reference.

catnap

Member

Registered: 2016-10-03

Posts: 131

Re: [SOLVED] Ignoring files in Git's FastImport

You could pipe it through sed to (probably that's the problem?) escape the quotes?

I tried a small alteration

bzr fast-export --plain . | sed 's/"no"/\\"no\\"/g'   |  git fast-import

The error message here is a bit surprizing:

fatal: Unsupported command: .

progandy

Member

Registered: 2012-05-17

Posts: 5,321

Re: [SOLVED] Ignoring files in Git's FastImport

R "My Maps/Muu/Getting past "no".mm" My Maps/Neuvottelu/Getting past "no".mm

Here the first "no" must be escaped, but the second has to be kept without backslashes. the bzr exporter doesn't do that correctly.

You could export to a file, fix the file manually and then import it in a third step.

https://git-scm.com/docs/git-fast-import
...

A <path> string must use UNIX-style directory separators (forward slash /), may contain any byte other than LF, and must not start with double quote (").

A path can use C-style string quoting; this is accepted in all cases and mandatory if the filename starts with double quote or contains LF. In C-style quoting, the complete name should be surrounded with double quotes, and any LF, backslash, or double quote characters must be escaped by preceding them with a backslash (e.g., "path/with\n, \\ and \" in it").

The value of <path> must be in canonical form. That is it must not:

    contain an empty directory component (e.g. foo//bar is invalid),

    end with a directory separator (e.g. foo/ is invalid),

    start with a directory separator (e.g. /foo is invalid),

    contain the special component . or .. (e.g. foo/./bar and foo/../bar are invalid).

The root of the tree can be represented by an empty string as <path>.

It is recommended that <path> always be encoded using UTF-8.

...

filerename

Renames an existing file or subdirectory to a different location within the branch. The existing file or directory must exist. If the destination exists it will be replaced by the source directory.

    'R' SP <path> SP <path> LF

here the first <path> is the source location and the second <path> is the destination. See filemodify above for a detailed description of what <path> may look like. To use a source path that contains SP the path must be quoted.

...

Last edited by progandy (2018-06-05 19:22:05)

---

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' | alias ENGLISH='LANG=C.UTF-8 ' |

catnap

Member

Registered: 2016-10-03

Posts: 131

Re: [SOLVED] Ignoring files in Git's FastImport

This might not really count as a solution because it does not show where the problem is but there seems to be a workaround for the problem. Encouraged by the absence of error messages when streaming the output to /dev/null, I attempted to write the output to an intermediary text file.

bzr fast-export --plain . > bzr-fast-export.db

Amazingly, this really did avoid the problem with the broken pipe, which was one of the former error messages. After implementing this apparently superficial but still workable change, I filtered the contents of the file with sed and streamed it to Git's FastImport.

cat bzr-fast-export.db | sed 's/Getting past "no"/Getting past no/g'   |  git fast-import   

When I check Git's log, my Bazaar commit messages are there. But Git status shows that all the files have been deleted and it lists my files as untracked. Is this usual in the conversion and should I just add the same files again with git add *?

progandy

Member

Registered: 2012-05-17

Posts: 5,321

Re: [SOLVED] Ignoring files in Git's FastImport

If you changed a filename, then of course git has different files than bazaar and the files with quotes in the working directory are untracked.

Maybe this sed does what is required?

sed 's/\(R ".*\)"no"\(.*" \)/\1\\"no\\"\2/'

---

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' | alias ENGLISH='LANG=C.UTF-8 ' |

catnap

Member

Registered: 2016-10-03

Posts: 131

Re: [SOLVED] Ignoring files in Git's FastImport

If you changed a filename, then of course git has different files than bazaar and the files with quotes in the working directory are untracked.

I might have caused the mindmap Getting past no.mm to fall outside the tracking by changing its name but the problem is the same with all the other files too. Also, I created a test repository bzr2git-test/ where I only had a simple text file without special characters in its name. After the conversion, git status gives

On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        deleted:    mytestfile.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .bzr/
        export.db
        mytestfile.txt

This might only be an counterintuitive feature of FastExport. When I added the file and the committed changes, everything seemed to be working fine.

catnap

Member

Registered: 2016-10-03

Posts: 131

Re: [SOLVED] Ignoring files in Git's FastImport

Maybe this sed does what is required?

sed 's/\(R ".*\)"no"\(.*" \)/\1\\"no\\"\2/'

This works.

progandy

Member

Registered: 2012-05-17

Posts: 5,321

Re: [SOLVED] Ignoring files in Git's FastImport

If the tracked files are displayed wrong, then you can probably do a simple "git reset".

---

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' | alias ENGLISH='LANG=C.UTF-8 ' |