You are not logged in.

#1 2018-07-22 11:11:21

mipi
Member
From: Germany
Registered: 2018-01-03
Posts: 36

smsync: synch & conversion of big music collections

Hi,
I have created a small program to sync and convert big music collections. smsync is available in AUR. The source code is on Github.

smsync is made for use cases where you have a folder structure for your high quality lossless or lossy but high bit rate music that acts as a "master". From this master you replicate your music to "slaves", such as a smartphone or an SD card / hard drive for your car etc. On a smartphone or in the car you either don't have or you don't want to spend that much storage capacity that you might have for you master music storage. smsync takes care of the replication/conversion while keeping the folder structure. Find more details here

First, I tried to use Make for my use case (there is an approach with Make mentioned in the Arch Linux wiki), but there are some problems with that approach. Therefore, I decided to write something new that solves my problem.

Would be great to get some feedback.

Thanks,
-mipi

Offline

#2 2018-07-23 00:01:17

Rasi
Member
From: Germany
Registered: 2007-08-14
Posts: 1,912
Website

Re: smsync: synch & conversion of big music collections

Very nice. Will this also detect changes in master (e.g. renamed/deleted directories) and reflect these in the sync?

Also: please support ogg and opus.


He hoped and prayed that there wasn't an afterlife. Then he realized there was a contradiction involved here and merely hoped that there wasn't an afterlife.

Douglas Adams

Offline

#3 2018-07-23 04:42:22

mipi
Member
From: Germany
Registered: 2018-01-03
Posts: 36

Re: smsync: synch & conversion of big music collections

Yes, these changes will be detected.

Implementing support of ogg shouldn't be a problem. Regarding opus, I have to see.

BTW: You're welcome to contribute.

Offline

#4 2018-08-04 11:25:56

mipi
Member
From: Germany
Registered: 2018-01-03
Posts: 36

Re: smsync: synch & conversion of big music collections

Support of ogg(vorbis) and opus is now implemented :-) Also wav is supported as source format. :-)

Last edited by mipi (2018-08-04 11:26:10)

Offline

#5 2018-10-22 10:42:08

dizwell
Member
Registered: 2018-10-22
Posts: 7

Re: smsync: synch & conversion of big music collections

The documentation is a bit strange. Copying your config file contents as shown at Github, exactly, I get: "Error: Rule #1: 'copy' is not a valid conversion". There is then no help anywhere I could see that would tell you what valid conversion options actually were.

I removed the Rule 1 and Rule 2 shown in the Github doco and the conversion process started well. But it wasn't an encouraging start when the only available doco appears wrong!

Incidentally, the doco contains another error, too. It shows for rule 0 "conversion = abr:192|cl:3" and then goes on to describe "In the example, [rule0] tells smsync to convert FLAC files (i.e. files with the extension '.flac') to MP3, using the conversion vbr:5|cl:3" Which rule 0 clearly doesn't do as written (i.e., the rule itself says "abr", and the commentary says "vbr").

Minor points: it's a very useful tool. But a bit of doco tidying might be in order, I think.

Nevertheless, thanks for the great functionality.

Last edited by dizwell (2018-10-22 10:42:52)

Offline

#6 2018-10-23 07:22:54

dizwell
Member
Registered: 2018-10-22
Posts: 7

Re: smsync: synch & conversion of big music collections

As mentioned, it's a very useful tool, so what follows are just a couple of suggestions.

1. It would be good if the tool was able to be run non-interactively (eg, a command-line option such as --no-confirm). That is, at the moment, when you type 'smsync', it asks, "Start synchronization (Y/n)?", which is fine -but means setting up crontab to periodically ensure a directory is synched to its source is trickier than it ought to be.

2. When I did a manual ffmpeg of one the files that smsync baulked at converting, I saw this in the output:

Stream mapping:
  Stream #0:1 -> #0:0 (mjpeg (native) -> png (native))
  Stream #0:0 -> #0:1 (flac (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
[swscaler @ 0x55a4fed78900] deprecated pixel format used, make sure you did set range correctly
[mp3 @ 0x55a4fecb9340] Frame rate very high for a muxer not efficiently supporting it.
Please consider specifying a lower framerate, a different muxer or -vsync 2

Which I think means that ffmpeg is not fond of the cover art I have embedded in that particular flac. No problem: the manual convert concluded correctly and output a working, listenable mp3 file.

Except smsync doesn't. It instead simply declares an 'error' in the log file, and refuses to convert the file. As a result, I'm missing about 2000 files in my MP3 directory, and I don't feel like altering the cover art in that many source FLACs in order to have another go to get source and destination properly in sync!

If there's a way to configure 'gentler' error handling, such that non-audio-related errors are ignored, that would be useful. Of course, smsync might have been reacting to the 'Frame rate very high' warning -but again, it's only a warning, and didn't stop ffmpeg finishing the job when run manually, so I don't quite think smsync should just quit at the job when receiving it.

3. Feedback: there isn't much of it apart from a simple file count/files processed type thing. If it were able to display the file name it was working on (or something!), that would be helpful when processing a very large source music collection in interactive mode. Otherwise, it's difficult to work out where smsync has reached in order to check on (say) the output quality of a freshly-created MP3.

Again, it's a very useful tool I'm pleased to have found. But #2 in particular is a bit of a show-stopper for me.

Regards
HJR

Offline

#7 2018-10-27 07:59:48

mipi
Member
From: Germany
Registered: 2018-01-03
Posts: 36

Re: smsync: synch & conversion of big music collections

Hi dizwell,
thanks for reporting this. I will check the docu and correct it. Regarding your other points:
#1: Good idea. Adding that is not a big deal. Will do that asap.
#2: This requires some more investigation. I need to figure out how to distinguish (a) between errors and warnings and (b) between the types of errors that ffmpeg throws. But overall your idea makes sense to me. BTW: Your ffmpeg message looks like you want to convert the cover art from JPG to PNG?
#3: Could add another cli option to make it more chatty.

Great that you like the tool overall. I wrote it because I needed something like that and there was nothing else with this functionality.

mipi

Offline

#8 2018-11-24 15:04:33

Rasi
Member
From: Germany
Registered: 2007-08-14
Posts: 1,912
Website

Re: smsync: synch & conversion of big music collections

Hmm, don't know how to get Opus conversion to work.

[general]
source_dir = /mnt/dump2/Audio/Rips/flac
num_cpus   = 4
num_wrkrs  = 4

[rule0]
source = flac
target = opus
conversion = vbr:128                   

gives me:

Rule #0: 'vbr:128' is not a valid conversion

He hoped and prayed that there wasn't an afterlife. Then he realized there was a contradiction involved here and merely hoped that there wasn't an afterlife.

Douglas Adams

Offline

#9 2018-11-25 12:33:10

mipi
Member
From: Germany
Registered: 2018-01-03
Posts: 36

Re: smsync: synch & conversion of big music collections

Will look into it, latest during the Xmas vacation.

Offline

#10 2018-12-17 20:05:03

mipi
Member
From: Germany
Registered: 2018-01-03
Posts: 36

Re: smsync: synch & conversion of big music collections

Version 3.0.3 is out now. Contains fixes for the reported bugs and new features. See changelog.

Note, that the config is now done in a YAML file (see https://github.com/mipimipi/smsync/blob … .md#config). smsync takes care of converting the old INI config file (SMSYNC.CONF) into the new YAML config file (smsync.yaml). Of course, you could also create the YAML file manually.

Feedback is welcome, as always.

Thanks,
-mipi

Offline

#11 2018-12-24 08:28:13

mipi
Member
From: Germany
Registered: 2018-01-03
Posts: 36

Re: smsync: synch & conversion of big music collections

dizwell wrote:

...
2. When I did a manual ffmpeg of one the files that smsync baulked at converting, I saw this in the output:

Stream mapping:
  Stream #0:1 -> #0:0 (mjpeg (native) -> png (native))
  Stream #0:0 -> #0:1 (flac (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
[swscaler @ 0x55a4fed78900] deprecated pixel format used, make sure you did set range correctly
[mp3 @ 0x55a4fecb9340] Frame rate very high for a muxer not efficiently supporting it.
Please consider specifying a lower framerate, a different muxer or -vsync 2

Which I think means that ffmpeg is not fond of the cover art I have embedded in that particular flac. No problem: the manual convert concluded correctly and output a working, listenable mp3 file.

Except smsync doesn't. It instead simply declares an 'error' in the log file, and refuses to convert the file. As a result, I'm missing about 2000 files in my MP3 directory, and I don't feel like altering the cover art in that many source FLACs in order to have another go to get source and destination properly in sync!

If there's a way to configure 'gentler' error handling, such that non-audio-related errors are ignored, that would be useful. Of course, smsync might have been reacting to the 'Frame rate very high' warning -but again, it's only a warning, and didn't stop ffmpeg finishing the job when run manually, so I don't quite think smsync should just quit at the job when receiving it.
...

Unfortunately, it is quite hard to find out the cause of an ffmpeg error automatically (OK, you could parse the log, but that's quite cumbersome and error-prone). In the latest release (3.2.0), smsync stores the ffmpeg log in a file if an error occurred during the conversion, see README

Merry Christmas.
mipi

Offline

#12 2018-12-25 08:45:20

Rasi
Member
From: Germany
Registered: 2007-08-14
Posts: 1,912
Website

Re: smsync: synch & conversion of big music collections

Still getting the same error.

source_dir: /mnt/dump2/Audio/Rips/flac               
num_cpus: 4
num_wrkrs: 4
rules:
- source: flac
  target: opus
  conversion: vbr:128

and the result:

Rule #1: 'vbr:128' is not a valid conversion

He hoped and prayed that there wasn't an afterlife. Then he realized there was a contradiction involved here and merely hoped that there wasn't an afterlife.

Douglas Adams

Offline

#13 2018-12-26 03:48:36

mipi
Member
From: Germany
Registered: 2018-01-03
Posts: 36

Re: smsync: synch & conversion of big music collections

Rasi wrote:

Still getting the same error.

source_dir: /mnt/dump2/Audio/Rips/flac               
num_cpus: 4
num_wrkrs: 4
rules:
- source: flac
  target: opus
  conversion: vbr:128

and the result:

Rule #1: 'vbr:128' is not a valid conversion

For opus, in the config file "abr" is expected to trigger variable bitrate conversion (instead of "vbr"), which is confusing. I corrected that. Now, "vbr" is expected. Just update to 3.2.1 and your config should work.

Offline

#14 2018-12-29 12:01:29

Rasi
Member
From: Germany
Registered: 2007-08-14
Posts: 1,912
Website

Re: smsync: synch & conversion of big music collections

I have a feature request. Since many use cases for such a tool include throwing your Transcodes onto a phone, it would be handy if smsync could produce character-safe output, that works with fat32/mtp.
Of course the tool would have to check the target dir for changed filenames before encoding so one does not get duplicates.

Last edited by Rasi (2018-12-29 12:02:38)


He hoped and prayed that there wasn't an afterlife. Then he realized there was a contradiction involved here and merely hoped that there wasn't an afterlife.

Douglas Adams

Offline

#15 2018-12-29 16:18:29

mipi
Member
From: Germany
Registered: 2018-01-03
Posts: 36

Re: smsync: synch & conversion of big music collections

Rasi wrote:

I have a feature request. Since many use cases for such a tool include throwing your Transcodes onto a phone, it would be handy if smsync could produce character-safe output, that works with fat32/mtp.
Of course the tool would have to check the target dir for changed filenames before encoding so one does not get duplicates.

What exactly is the problem the requested feature would solve? And what means "character-safe output"? I am using smsync since many months to sync my music (~35K+ songs) on a smartphone  (connected via mtp) and on an sdcard (FAT32). I never had problems with file names.

Offline

#16 2018-12-29 21:38:30

Rasi
Member
From: Germany
Registered: 2007-08-14
Posts: 1,912
Website

Re: smsync: synch & conversion of big music collections

Many special characters are not allowed on fat32 most noticable "?" but also some others.


He hoped and prayed that there wasn't an afterlife. Then he realized there was a contradiction involved here and merely hoped that there wasn't an afterlife.

Douglas Adams

Offline

#17 2019-01-02 08:23:59

mipi
Member
From: Germany
Registered: 2018-01-03
Posts: 36

Re: smsync: synch & conversion of big music collections

Rasi wrote:

Many special characters are not allowed on fat32 most noticable "?" but also some others.

Got it. I put that on the backlog. But again: I did not have problems with file names and mtp or fat32 so far.

Offline

Board footer

Powered by FluxBB