[SOLVED] pacman problems in bootable rsync backup

NuSkool · 2016-11-24 21:05:55

I'm doing a weekly rsync backup of my current Arch system, to a different hdd. I set things up so I can boot the backup if needed. I used this wiki page as a source of info: https://wiki.archlinux.org/index.php/fu … with_rsync

The problem is within the backup, duplication of pacman (database?) stuff in /var/lib/pacman/local/.

I have a script to create the backups. As a solution to the above, thinking about modifying the script to delete /var/lib/pacman/local/ in backup just prior to running rsync.

Would it be better to delete /var/lib/pacman/?

Would this potentially create other problems with pacman and if so, anyone have a solution?

Last edited by NuSkool (2016-11-27 00:14:09)

ayekat · 2016-11-25 08:36:20

NuSkool wrote:

The problem is within the backup, duplication of pacman (database?) stuff in /var/lib/pacman/local/.

What do you mean with "duplication"?
If you don't want to backup the files in /var/lib/pacman/local, just tell rsync not to sync them (with the --exclude flag - it's mentioned in that wiki article you've linked).

Would it be better to delete /var/lib/pacman/?
Would this potentially create other problems with pacman and if so, anyone have a solution?

Yes, this would most certainly cause trouble: the directory contains everything pacman knows about your local system.

NuSkool · 2016-11-25 16:58:40

After rereading my op, I don't think I was very clear on what's actually going on. Sorry about that. The problem is if my main system gets updated prior to a backup, /var/lib/pacman/local in backup will contain various versions of the packages that were updated. I've fixed this in backup by manually getting rid of the older versions of the packages that have various versions, but this is a pita.

ayekat wrote:

What do you mean with "duplication"?

Duplication means creating more than one. For example, two instances of bash. This isn't exactly accurate though because the two instances of bash would not be identical. They would be different versions. Replacing the word "duplication" with "various versions of the same packages" would be accurate though.

If you don't want to backup the files in /var/lib/pacman/local, just tell rsync not to sync them (with the --exclude flag - it's mentioned in that wiki article you've linked).

Yea I already do that for many unwanted files and directories. I cant just skip backing up /var/lib/pacman/local in backup as doing so would result in certain breakage of pacman.

Would it be better to delete /var/lib/pacman/?
Would this potentially create other problems with pacman and if so, anyone have a solution?

If this quote included the other related info per below, the question still stands.

NuSkool wrote:

I have a script to create the backups. As a solution to the above, thinking about modifying the script to delete /var/lib/pacman/local/ in backup just prior to running rsync.
Would it be better to delete /var/lib/pacman/?
Would this potentially create other problems with pacman and if so, anyone have a solution?

So this would get rid of the directories and contents in backup, but they would be replaced by the current version of my main system being backed up, in the proceeding line of my script. /var/lib/pacman/ also contains info pacman uses if I'm not mistaken. That's why I was asking, between the two, what would be better.

I'm trying to figure out the cleanest solution for this. I don't know all the details and inner workings of pacman, so I thought I'd ask. If pacman does not keep track of things in ways other than the contents of /var/lib/pacman/, I would think the solution I propose would work, but may not be the best solution.

I could blindly experiment and come up with something that may eventually work, based on what I currently know. I have no doubt though, that there are people here who know enough about how pacman works, that they could answer this off the top of their head, knowing for sure the outcome.

Last edited by NuSkool (2016-11-25 17:48:04)

alphaniner · 2016-11-25 17:50:41

It sounds like you should use one of the various --delete options so that anything which exists on the destination but not on the source gets deleted. The result should be as though the destination was empty before rsync was run. I've never done a bootable backup like this, but it seems to me you'd want to do this for all "system" directories, except maybe /home.

NuSkool · 2016-11-25 18:11:50

You've brought up something I haven't considered alphaniner.

I already have root and home separated in my script for future plans. I'm thinking about possibly deleting everything under root, but that would seriously increase the amount of data being backed up. I plan to eventually back up to a separate system on my local network as soon as I get one of my pogoplugs setup to run Arch.

When you say delete all system directories, could you be more specific please? I understand what you're saying, just not aware of all the possibilities.

Last edited by NuSkool (2016-11-25 18:35:26)

Trilby · 2016-11-25 18:16:01

He didn't say delete all system directories. Following alphaniner's advice would not increase the ammount of data needing to be backed up. Please read his post again as he wrote it well - I don't really see how you interpreted it the way you did. One point to make explicit, he's referring to the '--delete' flag for rsync.

alphaniner · 2016-11-25 18:42:00

I'm thinking about possibly deleting everything under root, but that would seriously increase the amount of data being backed up.

The --delete flag (of rsync) and its variants have the same effect as that but without the downside. When used, rsync will delete anything on the destination that doesn't exist on the source. For example, src/ and dest/ are identical except that I've deleted colors/lapis256.vim from src/ and modified ftplugin/sh.vim on src/:

src/				dest/
src/syntax			dest/syntax
src/syntax/python.vim		dest/syntax/python.vim
src/syntax/pascal.vim           dest/syntax/pascal.vim
src/syntax/bethscript.vim	dest/syntax/bethscript.vim
src/indent			dest/indent
src/indent/python.vim		dest/indent/python.vim
src/ftplugin			dest/ftplugin
src/ftplugin/vim.vim		dest/ftplugin/vim.vim
src/ftplugin/sh.vim		dest/ftplugin/sh.vim
src/ftplugin/rust.vim		dest/ftplugin/rust.vim
src/ftplugin/python.vim		dest/ftplugin/python.vim
src/ftplugin/php.vim		dest/ftplugin/php.vim
src/ftplugin/pascal.vim		dest/ftplugin/pascal.vim
src/ftplugin/make.vim		dest/ftplugin/make.vim
src/ftplugin/html.vim		dest/ftplugin/html.vim
src/colors			dest/colors
                                dest/colors/lapis256.vim
src/colors/kolor.vim		dest/colors/kolor.vim
src/colors/color_demo.vim	dest/colors/color_demo.vim

$ diff -rq src dest
Only in dest/colors: lapis256.vim
Files src/ftplugin/sh.vim and dest/ftplugin/sh.vim differ

So when I run rsync:

$ rsync -avc --delete src/ dest/
sending incremental file list
deleting colors/lapis256.vim
colors/
ftplugin/sh.vim

sent 932 bytes  received 65 bytes  1,994.00 bytes/sec
total size is 59,516  speedup is 59.70
$ diff -rq src dest
$

ftplugin/sh.vim is copied to dest and colors/lapis256.vim is deleted from dest. Unmodified files aren't transferred of course. So the result is as though you had completely wiped dest before rsync'ing, but the only things deleted were files that didn't exist on src.

Last edited by alphaniner (2016-11-25 18:44:23)

NuSkool · 2016-11-25 18:49:05

Oh sorry about that. I was reading it as using one of my proposed delete methods rather than a delete option in rsync.

I was trying to explain that I can see other instances of duplication of stuff possibly creating issues, that I just now thought of based on his reply, and that I had not considered before.

What I was asking, "be more specific" about, was what other files may cause issues? The comment about deleting root was just me thinking out loud. In other words, I thought it through and realized it probably wasn't a good thing to do in this case.

Wow, sorry if I'm coming across like a do***bag. I kind of feel that way now. Really not my intention and possibly typing is not conveying my appreciation of getting help.

Trilby · 2016-11-25 18:51:19

I don't think you are coming across poorly NuSkool - but it just seemed like there was a miscommunication that might have left you missing important information.

NuSkool · 2016-11-25 19:10:31

OK, thanks all for providing info! I'm pretty sure everything I needed to know is right here.

Now I just need to spend some time to wrap my brain around everything and implement it.

EDIT: Read man rsync and wanted more info before proceeding with testing. Need to make sure I REALLY understand --delete, based on finding this. This tempts me to just delete the contents of the directory in backup manually with the script before rsync runs and deal with the increased (49.6 MB) data. I'll test --delete in vbox until I FULLY understand all the details and potential issues. Hate to break the system I'm backing up along with backup.

a search result wrote:

The --delete flag in particular is one I've seen screwed up many times. A lot of people use rsync to move files to low priority storage,
and in this case you want the files you're moving to still EXIST in the destination directory.
That's not what delete does: --delete makes sure that, when you delete a file from the source directory,
it is ALSO deleted from your destination directory, so your destination doesn't get full of junk...
Once saw a guy wipe out his backup by putting in a new drive, and not turning off his nightly rsync script.
Script saw that the source dir was now empty, and it deleted every file in the destination dir, so they'd match.

Last edited by NuSkool (2016-11-25 19:56:10)

ayekat · 2016-11-25 21:58:31

NuSkool wrote:

The problem is if my main system gets updated prior to a backup, /var/lib/pacman/local in backup will contain various versions of the packages that were updated. I've fixed this in backup by manually getting rid of the older versions of the packages that have various versions, but this is a pita.

I know this might now be a little off-topic, but doesn't /var/lib/pacman/local contain only the metadata for the currently installed packages? There shouldn't be multiple versions of the same package in there. This sounds more like /var/cache/pacman (which I personally do not backup, so that would still be a case for --exclude for me).

alphaniner · 2016-11-25 22:13:52

What's happening is that he does a backup, then updates his system which updates and "cleans out" only the running /var/lib/pacman/local. Then he does another backup to the same location, so the contents of /var/lib/pacman/local from previous backups never got cleaned out. At least, that was my understanding and why I recommended the --delete flag.

NuSkool · 2016-11-25 23:17:50

If I understand what I've read correctly, none of the --delete options for rsync would be capable of deleting anything from source, only destination. The --remove-source-files option would need to be used to delete source files.

It appears the words "source and destination" may have been confused by the submitter of the following in the rsync wiki page?

https://wiki.archlinux.org/index.php/full_system_backup_with_rsync wrote:

You may want to add rsync's --delete option if you are running this multiple times to the same backup folder. In this case make sure that the source path does not end with /*, or this option will only have effect on the files inside the subdirectories of the source directory, but it will have no effect on the files residing directly inside the source directory.

I ran the rsync --dry-run option to verify the delete option worked as I read, in the destination path, but I don't have /* at the end of either of my source paths.

NuSkool · 2016-11-25 23:39:29

alphaniner wrote:

What's happening is that he does a backup, then updates his system which updates and "cleans out" only the running /var/lib/pacman/local. Then he does another backup to the same location, so the contents of /var/lib/pacman/local from previous backups never got cleaned out. At least, that was my understanding and why I recommended the --delete flag.

That's what I understand to be going on and causing pacman errors when it's ran from backup.

ayekat, I believe you're also correct in /var/lib/pacman/local containing only metadata. Backup is getting out of whack with various versions of the same packages from pacman updates to the running system. As far as I know, the contents of /var/cache/pacman/pkg have no effect on pacman. I occasionally go in and manually clean out some the largest duplicates there, and have not had any errors show up using pacman afterwards.

ayekat · 2016-11-26 05:03:47

Ah, I totally missed the /var/lib/pacman in backup. I'm sorry.

/me should read stuff more carefully...

NuSkool · 2016-11-27 01:14:16

This is solved. To recap, I was getting, "error: duplicated database entry <packagename>" when running pacman in the "backed up" bootable system.

I was unsure if removing the older versions of packages in /var/lib/pacman/local would "fix" things correctly. I was initially thinking about deleting ALL the contents of <destination>/var/lib/pacman/local manually, in my backup script just prior to the rsync running. This may have worked, but would have been kind of a "sledge hammer" half ass fix, when what was needed was a "tuning fork" fix.

Alphaniner suggested to use the --delete option in rsync, which was definitely what was needed, and was the correct method for the way I'm doing backups.

Thanks Trilby for realizing I was out in left field with my thinking, based on the reply I wrote. You got me back on track.

The wiki initially scared me from using the --delete option (with incorrect info?). This prompted me to investigate further. I learned rsync --delete will not alter the source files under any circumstance. After doing a bit of reading, testing, and using the --dry-run and --delete options in rsync, the issue with pacman is now solved, as well as eliminating a lot of unwanted cruft building up in the weekly automated backups.

Thanks ayekat for discussing some things about pacman I was unsure of.

~~I believe~~ pacman keeps track of installed packages via the contents of /var/lib/pacman ~~Someone please correct me if I'm wrong on this.~~ I often see the term pacman "database" mentioned, ~~so I guess~~ the contents of /var/lib/pacman/sync is it. ~~these directories must be it?~~

Source: https://wiki.archlinux.org/index.php/pa … _structure

AND

The local database is not tarred @ /var/lib/pacman/local/

pacman -Dk = Check local package database is internally consistent.
Check all required files present and installed packages have required dependencies.
Check that files/packages do not conflict.
Check that multiple packages do not own same file.

# pacman -Dk
#

The sync database is tarred @ /var/lib/pacman/sync/

pacman -Dkk = Check sync databases to ensure specified dependencies available.

# pacman -Dkk
error: missing 'wine' dependency for 'playonlinux'
error: missing 'wine' dependency for 'wine-mono'
error: missing 'wine' dependency for 'winetricks'

The errors for -Dkk are caused by the data for packages playonlinux, wine-mono, and winetricks being present in /var/lib/pacman/sync/community.db, but that's the way it comes. If wine was available in one the repos I have enabled, -Dkk would not show the errors.

Last edited by NuSkool (2016-11-27 20:32:57)

Arch Linux

#1 2016-11-24 21:05:55

[SOLVED] pacman problems in bootable rsync backup

#2 2016-11-25 08:36:20

Re: [SOLVED] pacman problems in bootable rsync backup

#3 2016-11-25 16:58:40

Re: [SOLVED] pacman problems in bootable rsync backup

#4 2016-11-25 17:50:41

Re: [SOLVED] pacman problems in bootable rsync backup

#5 2016-11-25 18:11:50

Re: [SOLVED] pacman problems in bootable rsync backup

#6 2016-11-25 18:16:01

Re: [SOLVED] pacman problems in bootable rsync backup

#7 2016-11-25 18:42:00

Re: [SOLVED] pacman problems in bootable rsync backup

#8 2016-11-25 18:49:05

Re: [SOLVED] pacman problems in bootable rsync backup

#9 2016-11-25 18:51:19

Re: [SOLVED] pacman problems in bootable rsync backup

#10 2016-11-25 19:10:31

Re: [SOLVED] pacman problems in bootable rsync backup

#11 2016-11-25 21:58:31

Re: [SOLVED] pacman problems in bootable rsync backup

#12 2016-11-25 22:13:52

Re: [SOLVED] pacman problems in bootable rsync backup

#13 2016-11-25 23:17:50

Re: [SOLVED] pacman problems in bootable rsync backup

#14 2016-11-25 23:39:29

Re: [SOLVED] pacman problems in bootable rsync backup

#15 2016-11-26 05:03:47

Re: [SOLVED] pacman problems in bootable rsync backup

#16 2016-11-27 01:14:16

Re: [SOLVED] pacman problems in bootable rsync backup

Board footer