Arch Linux

dpx

Member

Registered: 2017-01-09

Posts: 48

saf - "one backup is saf, two are safe, three are safer"

saf is a simple, reliable, rsync-based, battle tested, well rounded backup system, written in Python.

It uses rsync to incrementally back up your data to a different directory, hard disk or remote server via SSH. All operations are incremental, atomic and optionally possible to resume.

saf is using very limited set of commands in the background, so it plays well with storage shells with limited set of commands, like Hetzner Storage Box.

Main backup code is used in production for many years and has proved its reliability, while recent saf updates only improve user facing commands. No guarantees but should be safe to use.

https://github.com/dusanx/saf

Contributors welcome.

dpx

Member

Registered: 2017-01-09

Posts: 48

Re: saf - "one backup is saf, two are safe, three are safer"

Soon with time machine like browsing trough revisions on any locally backed up folder, to easily restore or compare any file. Initially in Vifm as Lua plugin, fully integrated.

Contributors are welcome to:
- Improve code
- Improve documentation
- Package saf
- Plugins for other file managers are welcome. I use Vifm so can't maintain too many plugins.

Who's with me?

dpx

Member

Registered: 2017-01-09

Posts: 48

Re: saf - "one backup is saf, two are safe, three are safer"

It was brought to my attention, on some other forum, that people don't react to call to open source the way they used to. I published such call on very few places, including here, as here is my home forum. Only couple of years ago it was enough to publish notification/call the way I did and intrigued people would check code and decide if it is legit and if they are interested or not. Now it seems I have to entertain people with my history and accomplishments before anybody bothers to look, on open source forum(s) no less. From my perspective that's wrong and waste of programmers time, damn good code is damn good code, but let me fulfill new expectations, I'll entertain you and I may even dance later. Not.

I am very ancient guy with over 30 years of software carrier, with many public and private open source contributions. I have been mostly using Arch for over 10+ (15+?) years. Among many other things, I am occasional vim contributor so I brought 'guiligatures' and ligatures support to gvim: clickable link! pictures!

Saf is a very good and reliable backup, that could use the help of more eyes, less in code area and more in doc/package area. It is a legit non commercial and open source project, with MIT license. Contributors are welcome but not crucial. Use it or not, all the same to me.

If open source community has slipped into such consumer mindset that it can't be bothered unless spoon fed, I am not sure I have time for that. This is not purely Arch related but since Arch is my home distro it seems fit to bother to publish here. That being said, I'll tap dance myself off the stage. Monorail!

On the serious note, if somebody volunteers to improve readme or put saf trough Arch wiki I'd be shocked and grateful but mostly grateful.

schard

Forum Moderator

From: Hannover

Registered: 2016-05-06

Posts: 1,992

Website

Re: saf - "one backup is saf, two are safe, three are safer"

You asked, so...

I had a look at the source code and consider its quality quite low.
- There are no docstrings in the sources
- The comments are mostly useless
- Weird "helper" constructs such as "dotdict" are questionable at best
- No PEP-8 compliance
- No type hints
- A lot of long methods and a few "megamoths"
- All the business logic is forced into one class "saf" that is used as a singleton with just one exposed method that is used, namely "process_command_line". This suggests, that you did not need to write a class in the first place.
- Zero unit tests

Given the high mental load the code exhibits combined with zero unit tests, I ask myself how you could possibly come to the conclusion that this code is "reliable".
I certainly won't use it in its current state.

Last edited by schard (2023-08-23 11:25:48)

---

macro_rules! yolo { { $($tokens:tt)* } => { unsafe { $($tokens)* } }; }

dpx

Member

Registered: 2017-01-09

Posts: 48

Re: saf - "one backup is saf, two are safe, three are safer"

- Zero unit tests

Unit tests are sometimes useful but mostly gimmicky way to pretend that we make safer code. Or at least they are gimmicky in one or two programmers environment. If lots of patches start arriving then yes I could consider adding them. Contributors are welcome.

Given the high mental load the code exhibits combined with zero unit tests, I ask myself how you could possibly come to the conclusion that this code is "reliable".

By writing good and reliable code, in many languages, for longer than I care to remember? Or by using saf core code (without user friendly switches, I added that recently) on many backup systems (personal/companies), with many different backup endpoints (local/remote) for over five years? In my book that's pretty good indication of being "reliable".

I certainly won't use it in its current state.

Thank you for letting me know, I respect that.

schard

Forum Moderator

From: Hannover

Registered: 2016-05-06

Posts: 1,992

Website

Re: saf - "one backup is saf, two are safe, three are safer"

Given the high mental load the code exhibits combined with zero unit tests, I ask myself how you could possibly come to the conclusion that this code is "reliable".

By writing good and reliable code, in many languages, for longer than I care to remember?

That mindset can get one into trouble easily. I too am an experienced professional SW dev and I would never conclude that my code is reliable, because I have loads of experience.
In fact, I still can write bugs into code every day. Unit test, in certain circumstances I admit, can help me detect those.

Or by using saf core code (without user friendly switches, I added that recently) on many backup systems (personal/companies), with many different backup endpoints (local/remote) for over five years? In my book that's pretty good indication of being "reliable".

The GitHub repo says the code is 5 days old. How can you be running it for five years?
Why are there no commits over the last five years?
This makes it hard to reproduce the steps taken during development, which also does not bode well on auditability.

Last edited by schard (2023-08-23 11:42:34)

---

macro_rules! yolo { { $($tokens:tt)* } => { unsafe { $($tokens)* } }; }

dpx

Member

Registered: 2017-01-09

Posts: 48

Re: saf - "one backup is saf, two are safe, three are safer"

That mindset can get one into trouble easily. I too am an experienced professional SW dev and I would never conclude that my code is reliable, because I have loads of experience.
In fact, I still can write bugs into code every day. Unit test, in certain circumstances I admit, can help me detect those.

I disagree. Unit tests are beautiful way of wasting time if you are on the timed pay and mostly useless if you try unit testing your own code (same brain trying to fix an break code while following the same patterns). I found them useful only once, many years ago, where I joined the team of young programmers in a who-the-hell-cares breakneck commit mood, where unit tests saved my code multiple times from being broken by new careless commits. Apart from that I respect if anybody needs unit tests but in general I mostly despise them.

Truly curious, I am aware it is off topic: how do you write unit test that catches your own bad code, why don't you spend that time on writing code that would not trigger unit test in the first place? Could be age gap that makes it hard for me to understand this.

The GitHub repo says the code is 5 days old. How can you be running it for five years?
Why are there no commits over the last five years?
This makes it hard to reproduce the steps taken during development, which also does not bode well on auditability.

From HN announcement I had recently: "I had this backup code working reliably for years, using local file system, vps/dedicated server, or remote storage for backup, then I finally get time to wrap README, iron few missing switches and publish. Should be production ready and reliable, so it could be useful to others. Contributors are welcome."

So in short I thought that community could benefit from sharing this code, with no intention to win beauty content but with respect for your data reliability. It was never version controlled before, in fact main backup logic wasn't barely touched for 3+ years (I think) so there is no history to share.

dpx

Member

Registered: 2017-01-09

Posts: 48

Re: saf - "one backup is saf, two are safe, three are safer"

As announced before, saf 0.16 is now available at https://github.com/dusanx/saf

This version adds "saf revisions" command that will list all the backup directories, scattered across all backups in local backup target, where specified path content differs. In other words it will provide short list of backup folders where content changed, providing easy way to restore files or folders.

Using "saf revisions", there is new Lua plugin for Vifm, that provides time-machine like functionality to quickly browse trough backup revisions, then to compare or restore file(s) to the original backup source. Only thing missing is star-field animation (planned), rest of the time-machine functionality is there and working well.

At first I wanted to record asciinema screencast of using saf plugin in Vifm but really nah -- I keep believing that people using Arch can be bothered to imagine or try and don't need to be spoon fed.

I would also like to use this opportunity to deny all the rumours going around that I was always right and never wrong. That's simply not true, sorry. I clearly remember the last time it happened, in fall of 1986, I was wrong for a brief period of time, because I thought I made a mistake. So only a human, rest assured.

With this update saf will most probably enter much calmer phase, because it has reached target functionality I wanted to achieve. As always contributors, merge requests, and questions are welcome.

3beb6e7c46a615a

Member

Registered: 2021-03-27

Posts: 165

Re: saf - "one backup is saf, two are safe, three are safer"

I think you may be overestimating the relevance of your thing. Don't confuse a lack of interest with a "consumer mindset". 

You're not the first to write yet another rsync frontend and won't be the last either.  The community has seen countless of these come and go.  If you'd like people take a look at it, you'll need to tell them why this particular rsync frontend, unlike all the others before, is worth it.

"I'm good, really, I swear, and this has never lost data in five years, promise" doesn't really do it. That's just what everyone said about their pet backup script, ever.

Meanwhile people are taking a look at borg or restic which for all their own faults have actually contributed some novel aspects to backups, beyond just invoking rsync in another funky way.

Sorry for the rant but you kinda started it...

Last edited by 3beb6e7c46a615a (2023-08-24 15:51:25)

dpx

Member

Registered: 2017-01-09

Posts: 48

Re: saf - "one backup is saf, two are safe, three are safer"

Don't confuse a lack of interest with a "consumer mindset".
... 
If you'd like people take a look at it, you'll need to tell them why this particular rsync frontend, unlike all the others before, is worth it.

Paradox, isn't it?

I think you may be overestimating the relevance of your thing.

Thanks for the rant Rest assured I am not overestimating anything -- I take my work seriously but not myself / don't think I am smartest cookie here / don't care if people like me for my work / don't want people to tiptoe around me and my work. If somebody uses saf and finds it useful I'd be happy. If nobody uses it but me all the same for me, right?

Way I advertise my work may rub some people in a wrong way but then again may provoke them to do some useful work on the project. I bet/hope @schard tried poking holes in the code, to find some real non-aesthetics issues. Than maybe he didn't, who knows? I despise unit tests but won't reject PR containing them, same as if somebody wants to chase current standards and wants to make code more pleasing to the eye.

I don't appreciate new normal of everybody speaking like there are cylindrical objects up our behinds, if you read my posts above, all exaggerations are jokes on my account, or at least should register like that. I am surprised nobody picked up the stupid Chuck Norris joke (about being wrong) in the post above, can't be that people seriously think... I don't even want to consider that.

To recap:
Q: How good is saf?
A: Damn good.
Q: Why is it that good? What are the differences?
A: saf has some nice commands but you'll have to try it to find out.
Q: I don't want to try it unless you tell me and be very nice to really convince me!
A: Then don't try it, you are missing the biggest thing since big bang.

As far as I am concerned please feel free to rant again

(typo edit)

Last edited by dpx (2023-08-24 18:00:38)

schard

Forum Moderator

From: Hannover

Registered: 2016-05-06

Posts: 1,992

Website

Re: saf - "one backup is saf, two are safe, three are safer"

I bet/hope @schard tried poking holes in the code, to find some real non-aesthetics issues. Than maybe he didn't, who knows? I despise unit tests but won't reject PR containing them, same as if somebody wants to chase current standards and wants to make code more pleasing to the eye.

If you consider my points made to only address "aesthetic" issues, that'd be revealing...

If you're asking whether I found actual bugs, yes:

0 ✓ neumann@workstation ~/test $ ~/saf init --name test "[fe80::aaa1:59ff:feb8:86ae]:8080/test"
Initialized .saf.conf in /home/neumann/test, with backup target location test.
You can edit manually or use "saf configedit" before first use.
0 ✓ neumann@workstation ~/test $ ~/saf list
Safety check failed - the destination does not appear to be a backup directory or drive (marker file not found)
If it is indeed a backup directory, you may add the marker file by running the following commands:

    ssh -q -p 22 [fe80 mkdir -p -- ""
    ssh -q -p 22 [fe80 touch "/backup.marker"
1 ✗ neumann@workstation ~/test $ 

split_server_location() fails horribly on IPv6 targets.
Unit testing possible inputs, such as IPv6 addresses that contain colons, could have prevented that.
I bet that there's even more issues with this code, but I will not try to find bugs before the code has been properly formatted refactored to be even remotely auditable.

Last edited by schard (2023-08-25 09:00:50)

---

macro_rules! yolo { { $($tokens:tt)* } => { unsafe { $($tokens)* } }; }

dpx

Member

Registered: 2017-01-09

Posts: 48

Re: saf - "one backup is saf, two are safe, three are safer"

If you consider my points made to only address "aesthetic" issues, that'd be revealing...

Thank for the input, I was kind of hoping on very high motivation to break it.

"[fe80::aaa1:59ff:feb8:86ae]:8080/test"

Good start but you could do better, manual marker creation is there to check if address works or not, regardless of reason, including saf not being able to process some edge case target location. It failed as intended, before even starting any backup, so I miss the a-ha moment. You certainly won't claim to ssh into that location the same way, without specifying that server in ~/.ssh/config?

I am curious why :8080 on the command line? There is "ssh-options = -p 23" to specify ssh parameters and port in .saf.conf even if you didn't do that already in ~/.ssh/config.

Also, backup content being holy thing for saf, considering saf using ssh+bash commands to reach remote shells, sometimes very limited, I would fix but not consider the bug if "saf list" or "saf realsizes" fail -- servers can send all kinds of garbage and helper commands are not crucial.

However if you find "saf prune" or "saf backup" bug, I would be very grateful and speak highly of your skills among my peers.

dpx

Member

Registered: 2017-01-09

Posts: 48

Re: saf - "one backup is saf, two are safe, three are safer"

While we wait, kind of funny story yesterday, somewhat unit tests related: I am using Vifm as a file manager and Vifm has very nice Lua plugin interface so I decided to publish plugin that, as some say, provides complete Apple time-machine experience in Vifm/terminal.

https://github.com/vifm/vifm/pull/931

So I love Vifm dearly and I _do_ understand why big project with many contributors needs unit tests, no problem with that. However Lua plugins are completely outside of main Vifm code, don't affect it at all, although they are within Vifm repo dir structure. So even PR with Lua plugin triggers all kinds of tests, no matter that it does not touch anything. It looks beautiful and I _did_ feel important but they are also useless, right? Unit tests are usually 10-20 minutes of 2==2 testing and in many cases, like with my plugin, don't even test anything remotely related to PR. I must not be the only person seeing humor in this?

Also all the wasted resources? Heat generated by (mostly) useless tests? Do you not care about the planet at all?