smsync: synch & conversion of big music collections

mipi · 2018-07-22 11:11:21

Hi,
I have created a small program to sync and convert big music collections. smsync is available in AUR. The source code is on Github.

smsync is made for use cases where you have a folder structure for your high quality lossless or lossy but high bit rate music that acts as a "master". From this master you replicate your music to "slaves", such as a smartphone or an SD card / hard drive for your car etc. On a smartphone or in the car you either don't have or you don't want to spend that much storage capacity that you might have for you master music storage. smsync takes care of the replication/conversion while keeping the folder structure. Find more details here

First, I tried to use Make for my use case (there is an approach with Make mentioned in the Arch Linux wiki), but there are some problems with that approach. Therefore, I decided to write something new that solves my problem.

Would be great to get some feedback.

Thanks,
-mipi

Rasi · 2018-07-23 00:01:17

Very nice. Will this also detect changes in master (e.g. renamed/deleted directories) and reflect these in the sync?

Also: please support ogg and opus.

mipi · 2018-07-23 04:42:22

Yes, these changes will be detected.

Implementing support of ogg shouldn't be a problem. Regarding opus, I have to see.

BTW: You're welcome to contribute.

mipi · 2018-08-04 11:25:56

Support of ogg(vorbis) and opus is now implemented :-) Also wav is supported as source format. :-)

Last edited by mipi (2018-08-04 11:26:10)

dizwell · 2018-10-22 10:42:08

The documentation is a bit strange. Copying your config file contents as shown at Github, exactly, I get: "Error: Rule #1: 'copy' is not a valid conversion". There is then no help anywhere I could see that would tell you what valid conversion options actually were.

I removed the Rule 1 and Rule 2 shown in the Github doco and the conversion process started well. But it wasn't an encouraging start when the only available doco appears wrong!

Incidentally, the doco contains another error, too. It shows for rule 0 "conversion = abr:192|cl:3" and then goes on to describe "In the example, [rule0] tells smsync to convert FLAC files (i.e. files with the extension '.flac') to MP3, using the conversion vbr:5|cl:3" Which rule 0 clearly doesn't do as written (i.e., the rule itself says "abr", and the commentary says "vbr").

Minor points: it's a very useful tool. But a bit of doco tidying might be in order, I think.

Nevertheless, thanks for the great functionality.

Last edited by dizwell (2018-10-22 10:42:52)

dizwell · 2018-10-23 07:22:54

As mentioned, it's a very useful tool, so what follows are just a couple of suggestions.

1. It would be good if the tool was able to be run non-interactively (eg, a command-line option such as --no-confirm). That is, at the moment, when you type 'smsync', it asks, "Start synchronization (Y/n)?", which is fine -but means setting up crontab to periodically ensure a directory is synched to its source is trickier than it ought to be.

2. When I did a manual ffmpeg of one the files that smsync baulked at converting, I saw this in the output:

Stream mapping:
  Stream #0:1 -> #0:0 (mjpeg (native) -> png (native))
  Stream #0:0 -> #0:1 (flac (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
[swscaler @ 0x55a4fed78900] deprecated pixel format used, make sure you did set range correctly
[mp3 @ 0x55a4fecb9340] Frame rate very high for a muxer not efficiently supporting it.
Please consider specifying a lower framerate, a different muxer or -vsync 2

Which I think means that ffmpeg is not fond of the cover art I have embedded in that particular flac. No problem: the manual convert concluded correctly and output a working, listenable mp3 file.

Except smsync doesn't. It instead simply declares an 'error' in the log file, and refuses to convert the file. As a result, I'm missing about 2000 files in my MP3 directory, and I don't feel like altering the cover art in that many source FLACs in order to have another go to get source and destination properly in sync!

If there's a way to configure 'gentler' error handling, such that non-audio-related errors are ignored, that would be useful. Of course, smsync might have been reacting to the 'Frame rate very high' warning -but again, it's only a warning, and didn't stop ffmpeg finishing the job when run manually, so I don't quite think smsync should just quit at the job when receiving it.

3. Feedback: there isn't much of it apart from a simple file count/files processed type thing. If it were able to display the file name it was working on (or something!), that would be helpful when processing a very large source music collection in interactive mode. Otherwise, it's difficult to work out where smsync has reached in order to check on (say) the output quality of a freshly-created MP3.

Again, it's a very useful tool I'm pleased to have found. But #2 in particular is a bit of a show-stopper for me.

Regards
HJR

mipi · 2018-10-27 07:59:48

Hi dizwell,
thanks for reporting this. I will check the docu and correct it. Regarding your other points:
#1: Good idea. Adding that is not a big deal. Will do that asap.
#2: This requires some more investigation. I need to figure out how to distinguish (a) between errors and warnings and (b) between the types of errors that ffmpeg throws. But overall your idea makes sense to me. BTW: Your ffmpeg message looks like you want to convert the cover art from JPG to PNG?
#3: Could add another cli option to make it more chatty.

Great that you like the tool overall. I wrote it because I needed something like that and there was nothing else with this functionality.

mipi

Rasi · 2018-11-24 15:04:33

Hmm, don't know how to get Opus conversion to work.

[general]
source_dir = /mnt/dump2/Audio/Rips/flac
num_cpus   = 4
num_wrkrs  = 4

[rule0]
source = flac
target = opus
conversion = vbr:128

gives me:

Rule #0: 'vbr:128' is not a valid conversion

mipi · 2018-11-25 12:33:10

Will look into it, latest during the Xmas vacation.

mipi · 2018-12-17 20:05:03

Version 3.0.3 is out now. Contains fixes for the reported bugs and new features. See changelog.

Note, that the config is now done in a YAML file (see https://github.com/mipimipi/smsync/blob … .md#config). smsync takes care of converting the old INI config file (SMSYNC.CONF) into the new YAML config file (smsync.yaml). Of course, you could also create the YAML file manually.

Feedback is welcome, as always.

Thanks,
-mipi

mipi · 2018-12-24 08:28:13

dizwell wrote:

...
2. When I did a manual ffmpeg of one the files that smsync baulked at converting, I saw this in the output:
Stream mapping:
  Stream #0:1 -> #0:0 (mjpeg (native) -> png (native))
  Stream #0:0 -> #0:1 (flac (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
[swscaler @ 0x55a4fed78900] deprecated pixel format used, make sure you did set range correctly
[mp3 @ 0x55a4fecb9340] Frame rate very high for a muxer not efficiently supporting it.
Please consider specifying a lower framerate, a different muxer or -vsync 2
Which I think means that ffmpeg is not fond of the cover art I have embedded in that particular flac. No problem: the manual convert concluded correctly and output a working, listenable mp3 file.
Except smsync doesn't. It instead simply declares an 'error' in the log file, and refuses to convert the file. As a result, I'm missing about 2000 files in my MP3 directory, and I don't feel like altering the cover art in that many source FLACs in order to have another go to get source and destination properly in sync!
If there's a way to configure 'gentler' error handling, such that non-audio-related errors are ignored, that would be useful. Of course, smsync might have been reacting to the 'Frame rate very high' warning -but again, it's only a warning, and didn't stop ffmpeg finishing the job when run manually, so I don't quite think smsync should just quit at the job when receiving it.
...

Unfortunately, it is quite hard to find out the cause of an ffmpeg error automatically (OK, you could parse the log, but that's quite cumbersome and error-prone). In the latest release (3.2.0), smsync stores the ffmpeg log in a file if an error occurred during the conversion, see README

Merry Christmas.
mipi

Rasi · 2018-12-25 08:45:20

Still getting the same error.

source_dir: /mnt/dump2/Audio/Rips/flac               
num_cpus: 4
num_wrkrs: 4
rules:
- source: flac
  target: opus
  conversion: vbr:128

and the result:

Rule #1: 'vbr:128' is not a valid conversion

mipi · 2018-12-26 03:48:36

Rasi wrote:

Still getting the same error.

source_dir: /mnt/dump2/Audio/Rips/flac               
num_cpus: 4
num_wrkrs: 4
rules:
- source: flac
  target: opus
  conversion: vbr:128

and the result:

Rule #1: 'vbr:128' is not a valid conversion

For opus, in the config file "abr" is expected to trigger variable bitrate conversion (instead of "vbr"), which is confusing. I corrected that. Now, "vbr" is expected. Just update to 3.2.1 and your config should work.

Rasi · 2018-12-29 12:01:29

I have a feature request. Since many use cases for such a tool include throwing your Transcodes onto a phone, it would be handy if smsync could produce character-safe output, that works with fat32/mtp.
Of course the tool would have to check the target dir for changed filenames before encoding so one does not get duplicates.

Last edited by Rasi (2018-12-29 12:02:38)

mipi · 2018-12-29 16:18:29

Rasi wrote:

I have a feature request. Since many use cases for such a tool include throwing your Transcodes onto a phone, it would be handy if smsync could produce character-safe output, that works with fat32/mtp.
Of course the tool would have to check the target dir for changed filenames before encoding so one does not get duplicates.

What exactly is the problem the requested feature would solve? And what means "character-safe output"? I am using smsync since many months to sync my music (~35K+ songs) on a smartphone (connected via mtp) and on an sdcard (FAT32). I never had problems with file names.

Rasi · 2018-12-29 21:38:30

Many special characters are not allowed on fat32 most noticable "?" but also some others.

mipi · 2019-01-02 08:23:59

Rasi wrote:

Many special characters are not allowed on fat32 most noticable "?" but also some others.

Got it. I put that on the backlog. But again: I did not have problems with file names and mtp or fat32 so far.

Arch Linux

#1 2018-07-22 11:11:21

smsync: synch & conversion of big music collections

#2 2018-07-23 00:01:17

Re: smsync: synch & conversion of big music collections

#3 2018-07-23 04:42:22

Re: smsync: synch & conversion of big music collections

#4 2018-08-04 11:25:56

Re: smsync: synch & conversion of big music collections

#5 2018-10-22 10:42:08

Re: smsync: synch & conversion of big music collections

#6 2018-10-23 07:22:54

Re: smsync: synch & conversion of big music collections

#7 2018-10-27 07:59:48

Re: smsync: synch & conversion of big music collections

#8 2018-11-24 15:04:33

Re: smsync: synch & conversion of big music collections

#9 2018-11-25 12:33:10

Re: smsync: synch & conversion of big music collections

#10 2018-12-17 20:05:03

Re: smsync: synch & conversion of big music collections

#11 2018-12-24 08:28:13

Re: smsync: synch & conversion of big music collections

#12 2018-12-25 08:45:20

Re: smsync: synch & conversion of big music collections

#13 2018-12-26 03:48:36

Re: smsync: synch & conversion of big music collections

#14 2018-12-29 12:01:29

Re: smsync: synch & conversion of big music collections

#15 2018-12-29 16:18:29

Re: smsync: synch & conversion of big music collections

#16 2018-12-29 21:38:30

Re: smsync: synch & conversion of big music collections

#17 2019-01-02 08:23:59

Re: smsync: synch & conversion of big music collections

Board footer