ebook metadata lookup similar to musicbrainz picard?

Xyne · 2020-04-25 23:19:32

MusicBrainz Picard is a gui app that can look up music file metadata online and use it to update metadata tags in files and rename them. If can use the following to retrieve metadata:

file name
existing metadata in the file (e.g. IDv* tags, CD info)
file fingerprint (via chromaprint)
interactive online search via the MusicBrainz website

Once the metadata is found, you can use a template string to rename the files and move them to a custom location (e.g. audio/<artist>/<album>/<track number>-<title>.<extension>).

I'm looking for something similar for ebooks (mainly pdf and djvu). I know of the following but they don't really fit my needs:

calibre: There doesn't seem to be an option to rename the files and move them to a user-configured directory. Calibre just copies them into an internal directory and forces me to export them via the interface. If there is an option that I missed, please let me know.
zotero: It's not really meant for importing local files and the metadata lookup from imported files is unreliable, while manual entry tedious.

The ideal would be a command-line tool that I can use in a script to scan and rename files, but a gui would be fine too. Any suggestions?

dwidmann · 2020-04-26 02:25:06

https://github.com/na--/ebook-tools

This seems to be a tool that can (among other things) rename/reorganize books based on things it gets from calibre. I was bored, so I gave it a try; native option didn't seem to work but the docker image did.

Xyne · 2020-04-26 04:48:25

That looks promising, thanks!

eschwartz · 2020-04-26 06:08:17

Xyne wrote:

calibre: There doesn't seem to be an option to rename the files and move them to a user-configured directory. Calibre just copies them into an internal directory and forces me to export them via the interface. If there is an option that I missed, please let me know.

calibre's GUI forces you to store all books within it, which is something most users do anyway as it has various useful features that its users tend to heavily use; if you buy into this, why not go all the way?

That being said, it's heavily scriptable if that's what floats your boat. You can use the fetch-ebook-metadata command-line utility for this, look up metadata based on title or author or isbn or whatever.

This is what the suggested ebook-tools repo does, with some scripting to read the results. I'm not entirely sure why it uses ebook-convert to convert the ebook to txt format before grepping it for an isbn, though, since calibre's ebook-meta command can just directly look them up if the existing metadata has even the slightest quality, and also set metadata by accepting an opf file produced by fetch-ebook-metadata.

Honestly, you'd want to review the fetched metadata anyway, and possibly choose between multiple search results, which would mean using the GUI as it has a very nice dialog for this. I've never once been unhappy letting it handle all my books via the internal directory.

Xyne · 2020-04-26 13:25:30

Thanks. I wasn't aware that calibre was scriptable. If I can just use the metadata lookup in my own scripts that would probably work.

Also, I'm not saying that using an internal hierarchy is a bad thing. It just doesn't seem to suit my particular needs. Then again, if it's scriptable, I suppose it's possible to automatically export to a custom directory structure based on metadata so I will take a look.

eschwartz · 2020-04-26 14:32:00

https://manual.calibre-ebook.com/genera … index.html for the CLI documentation, also available as packaged manpages for all documented commands.

You can use

calibredb export --template="{author_sort}/{title}/{title} - {authors}"

to export some or all books.

...

There is also a stable API for several parts of calibre, including the database interaction. See https://manual.calibre-ebook.com/develo … of-calibre
It's mainly intended for use in plugins, but you can also write python scripts with a #!/usr/bin/calibre-debug shebang -- I've done it before for extremely specialized scripting to set certain metadata fields in many files at once, as it's much more efficient than repeated calibredb commands in a shellscript loop.

Since the entire application is written in python, you can also, of course, access any internals via python (and various popular plugins do that), and things don't usually change around a lot, but "no promises".

Xyne · 2020-04-26 14:44:55

Is calibre-python3 mature enough for this, or should I stick to plain calibre for scripting for now?

eschwartz · 2020-04-26 15:11:51

While experimenting with fetch-ebook-metadata, inspired by this very thread, I discovered and reported the python3 bug fixed by this commit: https://github.com/kovidgoyal/calibre/c … bb367246f3

That being said, you can install both at the same time and switch between them (see `calibre-alternatives help` for details) and I'd be delighted if you used it and reported any python3 bugs you might find. It's usable enough for me to use python3 by default, and Fedora and debian only provide python3 versions. We find the occasional python3 bugs still, but they are quickly fixed once found.

Xyne · 2020-04-27 17:28:36

Just to follow up, I've played around a bit with calibre and the provided scripts. The success rate of metadata extraction with calibre just isn't high enough for me (and the ISBN extractor plugin doesn't seem to work at all), but the metadata fetcher script is good once you have an ISBN.

I'm writing my own tool to handle the ISBN extraction and fetch the metadata with calibre's script. At the moment I seem to have a somewhat reliable ISBN extraction method for pdf and djvu files based on pdftotext, djvutxt and ISBN checksum calculations. The plan is to provide a simple yet extensible command-line tool that accepts format strings for renaming files based on the metadata. Once I have something usable I'll probably publish and package it.

I haven't tested ebook-tools yet because I was initially put off by using docker and now I'm having fun trying my own solution, but I plan to come back to it if I get stuck.

I'm not marking the thread as solved yet because I am still interesting in other suggestions if anyone has them.

eschwartz · 2020-04-27 17:57:57

Xyne wrote:

Just to follow up, I've played around a bit with calibre and the provided scripts. The success rate of metadata extraction with calibre just isn't high enough for me (and the ISBN extractor plugin doesn't seem to work at all)

Do you mean this one? https://www.mobileread.com/forums/showt … p?t=126727

Note that metadata extraction in calibre is pretty simplistic in that it's about reading actual properly-tagged ebooks. e.g. for EPUB to find an ISBN it *needs* to be already present as

<dc:identifier opf:scheme="ISBN">1234567890<dc:identifier>

And many ebooks are badly tagged, *especially* less convenient formats like PDF.

Xyne wrote:

, but the metadata fetcher script is good once you have an ISBN.
I'm writing my own tool to handle the ISBN extraction and fetch the metadata with calibre's script. At the moment I seem to have a somewhat reliable ISBN extraction method for pdf and djvu files based on pdftotext, djvutxt and ISBN checksum calculations. The plan is to provide a simple yet extensible command-line tool that accepts format strings for renaming files based on the metadata. Once I have something usable I'll probably publish and package it.

One thing which would be neat is to extend the fairly popular Extract ISBN plugin to do whatever else you need it to do, and add a `calibre-debug --run-plugin "Extract ISBN"` option (if none exists yet) to enhance scripting. https://manual.calibre-ebook.com/plugin … n.cli_main

calibre guarantees pdftohtml exists in upstream binary builds since it actually uses this via subprocess, but not pdftotext (even though distro builds will have both utils). Not djvutxt though.

Xyne · 2020-04-27 18:52:35

Yeah, I think that's the plugin. I loaded it via Calibre's dialogue.

I plan to make the external tools (pdftotext, djvutxt) optional via a json or yaml file that maps MIME type to text extractor commands. That should provide enough flexibility for everyone. I haven't tested it yet but it should work with pdftohtml as long as it doesn't insert tags inside of the ISBN text.

All I'm doing is running a regex over the text looking for sequences of at least 9 digits optionally interspersed with dashes and spaces and maybe ending in "x" (ISBN-10 check digit representing 10). Sequences of 13 or 10 digits are checked for ISBN validity. Anything longer than 10 but not 13 digits long is scanned in windows of 13 or 10 digits for a valid checksum. It's fairly naive but so far the results have been promising. There's only a 1 in 10 chance that a random sequence of 10 or 13 digits validates as an ISBN sequence. The variability in formats between different books makes it difficult to use more restrictive criteria. There's also currently an assumption that the first valid result is the book's ISBN in the front matter, but that could eventually be replaced with an algorithm for ranking potential ISBNs by context and file position, or by looking up metadata for all candidates and prompting the user to select the correct book.

It's a work in progress and I'm still examining the results and exploring ways to make it more robust. Once the code is stable, I'll take a look at how it could be added to the Extract ISBN plugin as a fallback option when the metadata is absent.

The approach should work for extracting DOIs too.

Arch Linux

#1 2020-04-25 23:19:32

ebook metadata lookup similar to musicbrainz picard?

#2 2020-04-26 02:25:06

Re: ebook metadata lookup similar to musicbrainz picard?

#3 2020-04-26 04:48:25

Re: ebook metadata lookup similar to musicbrainz picard?

#4 2020-04-26 06:08:17

Re: ebook metadata lookup similar to musicbrainz picard?

#5 2020-04-26 13:25:30

Re: ebook metadata lookup similar to musicbrainz picard?

#6 2020-04-26 14:32:00

Re: ebook metadata lookup similar to musicbrainz picard?

#7 2020-04-26 14:44:55

Re: ebook metadata lookup similar to musicbrainz picard?

#8 2020-04-26 15:11:51

Re: ebook metadata lookup similar to musicbrainz picard?

#9 2020-04-27 17:28:36

Re: ebook metadata lookup similar to musicbrainz picard?

#10 2020-04-27 17:57:57

Re: ebook metadata lookup similar to musicbrainz picard?

#11 2020-04-27 18:52:35

Re: ebook metadata lookup similar to musicbrainz picard?

Board footer