alternatives to borg for backups

bulletmark · 2021-06-23 22:36:42

Perhaps people can recall this thread and that as I stated above I created a borg-bin AUR package at the time which I have maintained faithfully ever since. I have always updated it the same day of a new borg release. It is based on the standalone binary so is not brittle to python version changes like the official python source package. Today, a user requested that the AUR package be deleted as a duplicate and a TU immediately did so. I suggest/request that the AUR system add a mechanism to appeal against arbitary deletions like this.

Slithery · 2021-06-23 22:41:43

bulletmark wrote:

Today, a user requested that the AUR package be deleted as a duplicate and a TU immediately did so. I suggest/request that the AUR system add a mechanism to appeal against arbitary deletions like this.

There already is such a mechanism, it's the aur-requests mailing list...
https://lists.archlinux.org/listinfo/aur-requests

bulletmark · 2021-06-23 23:12:33

Well that is a mailing list (and it looks to be noise machine generated, not for users?). It is not really part of the AUR. Do I really have to resort to such a 30 year old mechanism? That is TFH.

Last edited by bulletmark (2021-06-23 23:16:04)

Alad · 2021-06-23 23:15:36

Yes, you do. If you don't like it, come up with a different mechanism.

Slithery · 2021-06-23 23:21:22

It's part of the official AUR infrastructure.

Some of it is machine generated by requests on the web front-end but all you need to do is send a single email, surely that's well within the capabilities of someone that has already submitted a package.

bulletmark · 2021-06-27 23:59:17

@Slithery and @Alad, I sent an email to aur-requests a few days ago but got no response/bounce and haven't seen it appear on the archive. Do I actually have to subscribe to that list to be able to post? The info page doesn't seem to require that.

Xyne · 2021-07-06 00:36:50

@bulletmark
I've restored the package (I had to bump the version but no other changes were made) and left a comment on the page to explain why it should not be deleted. Please re-adopt it

I didn't track down the report to see who deleted it but it was likely an honest mistake. The name itself is slightly misleading because the "-bin" suffix usually means that a package is built from precompiled binaries instead of compiled from source. The "-bin" packages are therefore allowed as alternatives to source packages on the AUR to spare user's the compilation time, but not for official packages which are compiled by the packager and provided compiled to the user.

Perhaps it should be renamed to "borg-standalone" to avoid confusion?

Edit: For future reference, issues with AUR moderation can be discussed on aur-general

Last edited by Xyne (2021-07-06 00:42:48)

bulletmark · 2021-07-06 02:11:41

@Xyne, thanks very much for that. I have re-adopted the package. Yes, I was also going to suggest renaming the package to "borg-standalone" if my appeal dialog had progressed. Please change it if you can, or should I just make a new package and request deletion?

Also (and unfortunately this brings another hairy issue), I had another related package called "borg-arm-bin" which was also deleted. That is the equivalent standalone package for ARM machines like the Raspberry Pi. It was package separately because those upstream binaries are built independently of the borg project and so can take a few weeks to be updated so I did not want to hold up release of a new borg-bin due to that. Now I know that Arch does not technically support ARM, but would you please consider re-instating that as well, and renaming it to borg-standalone-arm.

Xyne · 2021-07-06 02:43:25

@bulletmark
Change the names and reupload both packages to the AUR, then file a request to merge borg-bin into borg-standalone so that my comment is transferred. If someone flags the package or tries to delete it again, cite this thread and the comment on the AUR. If it ends up as a discussion on aur-general, I'll weigh in there.

And thanks for maintaining the packages!

bulletmark · 2021-07-06 04:46:22

@Xyne, one more thing. I think the newly renamed packages should be called borg-standalone-bin and borg-standalone-arm-bin because they are built from pre-compiled binaries, not source. The -bin seems to be appropriate. Do you agree?

Xyne · 2021-07-06 11:45:45

bulletmark wrote:

@Xyne, one more thing. I think the newly renamed packages should be called borg-standalone-bin and borg-standalone-arm-bin because they are built from pre-compiled binaries, not source. The -bin seems to be appropriate. Do you agree?

I agree.

eschwartz · 2021-07-07 00:26:29

The name borg-bin is correct, since this repackages the upstream binaries (which we have historically allowed just fine) and has nothing to do with "standalone".

I agree that it should not have been deleted... it was flagged for deletion by a user as "Version is available in community" and that deletion request IMO should have been rejected.

borg-arm-bin is a different matter entirely. Its deletion request was submitted on the grounds that the package is ARM-specific and the correct way to package software is to *NOT* be specific to ARM, e.g. by providing additional arch=() variants for a generic -bin package... I'm not sure why "these are completely unofficial non-upstream binaries which are always late to be uploaded by the third party in charge" is a compelling rationale to e.g. go against the guidance at https://wiki.archlinux.org/title/PKGBUILD#arch

Possible workarounds might be:

temporarily drop ARM support whenever the latest version is unavailable
wait for ARM support after a release, before updating, assuming that the wait is typically a day or two
do some hacky clever bash to make pkgver be the old version, but only when CARCH = arm* or aarch64
not bother with ARM at all, since IIRC upstream "solved" the msgpack situation by vendoring an extremely old version into borg.somethingvendored.msgpack rather than backporting a trivial change from the unstable branch with an if/else to be compatible with both, and the current Arch/ALARM packages use that (no module dependencies listed by pacman), so this reduces down to personal preference rather than concern that community/borg is dangerously unstable, hence ALARM support may not be an overriding priority

bulletmark · 2021-07-07 01:32:52

Note that after he deleted borg-bin I sent a detailed email to the TU explaining the rationale for creation of the AUR package and pointing him to this thread etc, but he merely replied "The submitted PKGBUILDs must not build applications already in any of the official binary repositories under any circumstances". I felt he had not read my email at all. I then sent the same appeal email to aur-requests but it never appeared, and I never got any response.

I agree with Xyne that borg-standalone-bin is a better name. After all, I am packaging a separate product produced by the borg project which they call Standalone Binary. It is independent to their Distribution Package as packaged by Arch.

I was aware of potential objections about the ARM package, that is why I described it above as a "hairy issue". I never appealed that deletion. The upstream ARM packages can take a few days to a week or more. In fact, since taking up these packages, it has been me(!) every time who chases up the independent borg ARM maintainer to manually update those upstream packages. eschwartz proposes 4 "work-arounds" but none of them are appealing compared to my "work-around" which was to create a separate package. The simple fact is that Arch does allow AUR creators to specify foreign architectures in a PKGBUILD so therefore Arch does implicitly allow support of other architecture distros (Arch 32 and Arch ARM very notably). So is it really a problem if I make that support explicit by maintaining a separate package? As far as I can tell, the packaging guidelines do not explicitly prohibit it.

Xyne · 2021-07-07 01:35:31

I admit that I haven't examined the official borg package in a while. Previously it depended on external Python libraries which is what lead to breakage that started this thread. If the current version is functionally equivalent to the standalone binary provided by upstream then the correct name is indeed "borg-bin". But in that case the binary package should not exist in the AUR.

I agree with your analysis of the -bin package for ARM and suggested workarounds. I would go for option 2 or 3, or just push the updated version and let the ARM binaries catch up. The previous version can be left in the PKGBUILD along with a comment. Users who need to build the package in the interim can edit the pkgver.

edit: Actually, that won't work because the package requires the source checksums. So option 2 or 3.

edit 2:

I saw the last reply after posting. My current stance is that if the official package is functionally equivalent to the standalone binary then it is indeed against our policy to include it in the AUR. I was going on the assumption that it was not equivalent when I undeleted the package and I should have checked.

In that case, the ARM binary package is problematic as well. Given the previous point, borg-bin should not exist, even if just for the ARM version. And putting "arm" in the name goes against the policy of not making it architecture-specific. But at the same time, the AUR is used for other architectures and I lean towards pragmatism and utility so I'm inclined to keep the package in the AUR in some way.

I honestly don't know what the best solution is right now so I'm not making any further suggestions until I've thought about it more.

And again, sorry for the confusion caused by not realizing that the official borg package had significantly changed.

Last edited by Xyne (2021-07-07 01:46:36)

eschwartz · 2021-07-07 03:14:01

bulletmark wrote:

I agree with Xyne that borg-standalone-bin is a better name. After all, I am packaging a separate product produced by the borg project which they call Standalone Binary. It is independent to their Distribution Package as packaged by Arch.

"standalone binary" isn't the name of the software project. "borg-standalone" isn't the name of the software project.

"standalone" is the name of the *distribution policy*. It is essentially being used as a direct synonym of "-bin". Every single "upstream prebuilt binaries, doesn't require any dependencies, extract the tarball or decompress the gzipped binary to use" is "standalone". They are *all* traditionally called "foo-bin", not "foo-standalone", not the verbosely duplicate "foo-standalone-bin".

It doesn't matter whether that bin is created by statically compiling C library dependencies, or shipping rpath'ed binaries in /opt/foo/bin/ and /opt/foo/lib/, or (like here) using tools like pyinstaller that have a shim executable and extract a custom python interpreter into /tmp/ containing the actual program. None of these are new concepts in the AUR, but they are all still "foo-bin". It's a perfectly reasonable name.

bulletmark wrote:

I was aware of potential objections about the ARM package, that is why I described it above as a "hairy issue". I never appealed that deletion.

Sure, thanks for keeping the hairy issue under consideration. I'll try to elaborate on why I personally think it's a problem.

bulletmark wrote:

The upstream ARM packages can take a few days to a week or more. In fact, since taking up these packages, it has been me(!) every time who chases up the independent borg ARM maintainer to manually update those upstream packages.

This continues to feel like something so unofficial I kind of wonder why upstream borg even mentions it.

bulletmark wrote:

eschwartz proposes 4 "work-arounds" but none of them are appealing compared to my "work-around" which was to create a separate package. The simple fact is that Arch does allow AUR creators to specify foreign architectures in a PKGBUILD so therefore Arch does implicitly allow support of other architecture distros (Arch 32 and Arch ARM very notably). So is it really a problem if I make that support explicit by maintaining a separate package? As far as I can tell, the packaging guidelines do not explicitly prohibit it.

Quoting from my previous link https://wiki.archlinux.org/title/PKGBUILD#arch

For official repository and AUR packages, this means x86_64. Optionally, AUR packages may choose to additionally support other known working architectures.

Arch explicitly allows support of other architectures, but that explicit support specifically calls out the case of "hitching a ride" on an existing x86_64 package. I've previously told AUR package maintainers that in order to continue having their foo-bin packages in the AUR, they must add x86_64 support "even if you don't think x86_64 users will use this" (IIRC, one package in question was in the official repos but didn't compile on ALARM for mysterious reasons, so ALARM users needed to use the prebuilt bin but x86_64 users had the option to simply install the one in community). And they added x86_64 support to their packages, meaning that the package would no longer completely fail to install on the **officially supported AUR architecture**.

Maintaining *two* packages, one for x86_64 and one for ARM, feels a lot odder -- now it takes up twice as much space in the search results, and it won't be immediately clear which one is desired. Moreover, you're encoding the rightful value of the Architecture field inside the Name field instead, which is just technically wrong.

Allowing ARM to share the same pkgname on the AUR feels like a package maintenance hack, okay. But on the other hand, allowing borg-arm-bin to exist independently feels like a UX hack.

I have also just checked archive.org -- borg-bin used to have 13+ votes before it got deleted, the ARM package had one vote (and that one vote may have been yours, but archive.org doesn't say either way). So I wonder if it is all that practical to be concerned about it anyway?

bulletmark · 2021-07-07 03:14:20

OK, I capitulate. Let's leave them both deleted.

Here are the PKGBUILD for borg-standalone-bin and borg-standalone-arm-bin if anybody may want them.

Xyne · 2021-07-08 20:23:48

eschwartz wrote:

"standalone" is the name of the *distribution policy*. It is essentially being used as a direct synonym of "-bin". Every single "upstream prebuilt binaries, doesn't require any dependencies, extract the tarball or decompress the gzipped binary to use" is "standalone". They are *all* traditionally called "foo-bin", not "foo-standalone", not the verbosely duplicate "foo-standalone-bin".
It doesn't matter whether that bin is created by statically compiling C library dependencies, or shipping rpath'ed binaries in /opt/foo/bin/ and /opt/foo/lib/, or (like here) using tools like pyinstaller that have a shim executable and extract a custom python interpreter into /tmp/ containing the actual program. None of these are new concepts in the AUR, but they are all still "foo-bin". It's a perfectly reasonable name.

The -bin packages are built using pre-compiled binaries to skip the local compilation but there is an expectation that the resulting package be functionally equivalent to the standard version of the package. If the pre-compiled binaries are statically linked or embed what are normally external dependencies, there is a distinction that *may* be worth differentiating with the package name.

That is no longer the case with borg and the standalone version, but it was previously.

As for supporting other architectures in the AUR. -bin versions of official packages for other architectures should be brought up for discussion as a possible exception. Following the conclusion here, renaming the package is not the right solution and our current policy, if strictly applied, prohibits the -bin version of an official package even if it's only for other architectures. I think there is a net benefit to all Arch users in having a common AUR, with deference to x86_64 as the official supported architecture when conflicts arise.

borg-bin doesn't seem to be worth the fuss right now, but there should be a discussion about this in the future if it comes up again.

I'm going to re-delete borg-bin as the official borg package in our repos is now a standalone executable.

@bulletmark Thanks for the packages nevertheless and sorry about the confusion.

eschwartz · 2021-07-08 20:33:39

As I previously mentioned, borg-bin embeds its own python interpreter...

No exception to the policy needs to be made, anyway. Current policy permits "foo-bin" style packages which repackage upstream's official prebuilt x86_64 binaries. For any reason, including "the AUR maintainer would like to run the upstream binary in order to verify that a bug is reproducible on the upstream binary" (many upstream projects prefer bug reports that are verified to be reproducible using their prebuilt binaries).

There are many such packages on the AUR. And many of them are arch=('i686' 'x86_64' 'arm' 'armv6h' 'armv7h' 'aarch64')

Arch Linux

#26 2021-06-23 22:36:42

Re: alternatives to borg for backups

#27 2021-06-23 22:41:43

Re: alternatives to borg for backups

#28 2021-06-23 23:12:33

Re: alternatives to borg for backups

#29 2021-06-23 23:15:36

Re: alternatives to borg for backups

#30 2021-06-23 23:21:22

Re: alternatives to borg for backups

#31 2021-06-27 23:59:17

Re: alternatives to borg for backups

#32 2021-07-06 00:36:50

Re: alternatives to borg for backups

#33 2021-07-06 02:11:41

Re: alternatives to borg for backups

#34 2021-07-06 02:43:25

Re: alternatives to borg for backups

#35 2021-07-06 04:46:22

Re: alternatives to borg for backups

#36 2021-07-06 11:45:45

Re: alternatives to borg for backups

#37 2021-07-07 00:26:29

Re: alternatives to borg for backups

#38 2021-07-07 01:32:52

Re: alternatives to borg for backups

#39 2021-07-07 01:35:31

Re: alternatives to borg for backups

#40 2021-07-07 03:14:01

Re: alternatives to borg for backups

#41 2021-07-07 03:14:20

Re: alternatives to borg for backups

#42 2021-07-08 20:23:48

Re: alternatives to borg for backups

#43 2021-07-08 20:33:39

Re: alternatives to borg for backups

Board footer