You are not logged in.

#1 2011-10-06 01:00:18

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

backpac: A package state snapshot and restore tool for Arch Linux

backpac:

A package state snapshot and restore tool for Arch Linux with config file save/restore support.

Summary & Features

It's a common method of setting up a single system: take some notes about what packages you've installed, what files you've modified.

Backpac creates those notes for you and helps back up important configuration files. Specifically, backpac does the following:

  • maintains a list of installed groups (based on 80% of group packages being installed)

  • maintains a list of packages (including official and aur packages, listed separately)

  • maintains a list of files (manually created)

  • backs up key config files as detailed in the files list you create

The package, group and files lists along with the snapshot config files allows system state to be easily committed to version control such as git.

Backpac can also use these lists to install packages and files. Essentially, then, backpac takes a snapshot of your system and can recreate that state from the files and lists it archives.

Use Cases
  • Ongoing system state backup to github

  • Quick install of new system from existing backpac config

  • Conform current system to given state in backpac config

Backpac is a very, very lightweight way of saving and restoring system state.
It's not intended for rolling out and maintaining multiple similar systems, it's designed to assist individual users in the maintainance of their own Arch Linux box.

Status

Alpha, release for testing among those interested. Passing all tests right now but will continue to rework and refine. Bug reports needed.

Why?

There are a lot of 'big-iron' solutions to maintaining, backing up and restoring system state. Setting these up for a single system or a handful of personal systems has always seemed like overkill.

There are also some existing pacman list making utilities around, but most of them seem to list either all packages or don't separate the official and aur packages the way I wanted. Some detect group install state, some don't. I wanted all these features in backpac.

Finally, whatever tool I use, I'd like it to be simple (c.f. the Arch Way). Lists that are produced should be human readable, human maintainable and not different from what I'm using in non-automated form. Backpac fulfills these requirements.

Regarding files, I wanted to be able to backup arbitrary system files to a git repository. Tools like etckeeper are interesting but non /etc files in that case aren't backed up (without some link trickery) and there isn't any automatic integration with pacman, so there is no current advantage to using a tool like that. I also like making an explicit list of files to snapshot.

Sample Output

This is the command line report. Additionally, backpac saves this information to the backpac groups, packages and files lists and the files snapshot directory.

$ backpac -Qf

backpac
----------------------------------------------------------------------------
(-b)    Backups ON; Files will be saved in place with backup suffix.
 -f     Force mode ON; No prompts presented (CAUTION).
(-F)    Full Force mode OFF; Prompt displayed before script runs.
(-g)    Suppress group check OFF; Groups will be checked for currency.
(-h)    Display option and usage summary.
(-p)    Default backpac: /home/es/.config/backpac/tau.
 -Q     Simple Query ON; Report shown; no changes made to system.
(-R)    Auto-Remove OFF; Remove/Uninstall action default to NO.
(-S)    System update OFF; No system files will be updated.
(-U)    backpac config update OFF; backpac files will not be updated.

Sourcing from backpac config directory: /home/es/.config/backpac/tau

Initializing.................Done


GROUPS
============================================================================
/home/es/.config/backpac/tau/groups

GROUPS UP TO DATE: group listed in backpac and >80% local install:
----------------------------------------------------------------------------
base base-devel xfce4 xorg xorg-apps xorg-drivers xorg-fonts

GROUP PACKAGES; MISSING?: group member packages not installed:
----------------------------------------------------------------------------
(base: nano)
(xfce4: thunar xfdesktop)


PACKAGES
============================================================================
/home/es/.config/backpac/tau/packages

PACKAGES UP TO DATE: packages listed in backpac also installed on system:
----------------------------------------------------------------------------
acpi acpid acpitool aif alsa-utils augeas cowsay cpufrequtils curl dialog
firefox gamin git ifplugd iw mesa mesa-demos mutt netcfg openssh rfkill
rsync rxvt-unicode sudo terminus-font vim wpa_actiond wpa_supplicant_gui
xmobar xorg-server-utils xorg-twm xorg-utils xorg-xclock xorg-xinit xterm
yacpi yajl youtube-dl zsh

AUR UP TO DATE: aur packages listed in backpac also installed on system:
----------------------------------------------------------------------------
flashplugin-beta freetype2-git-infinality git-annex haskell-json
package-query-git packer wpa_auto xmonad-contrib-darcs xmonad-darcs

AUR NOT IN backpac: installed aur packages not listed in backpac config:
----------------------------------------------------------------------------
yaourt-git


FILES
============================================================================
/home/es/.config/backpac/tau/files

MATCHES ON SYSTEM/CONFIG:
----------------------------------------------------------------------------
/boot/grub/menu.lst
/etc/acpi/handler.sh
/etc/rc.conf
/etc/rc.local

Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#2 2011-10-06 04:22:41

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

Brilliant! You just saved me hours of "work"! I hadn't yet found any of the configuration management tools out there that seemed to work just right for me.

1. Entering 'backpac --help' (yes, my python is showing) gave these errors and then installed packer all by itself. Wasn't quite expecting that!

20:19:46 $ backpac --help
/usr/share/backpac/lib/common: eval: line 277: unexpected EOF while looking for matching `''
/usr/share/backpac/lib/common: eval: line 278: syntax error: unexpected end of file
--2011-10-05 20:19:57--  https://aur.archlinux.org/packages/packer/packer.tar.gz
Resolving aur.archlinux.org... 208.92.232.29
........

2. Filenames in .config/backpac/$HOSTNAME/files that have spaces are not handled at all. I tried quoting the lines and using \ to escape the spaces, but neither worked right.

3. Are the filenames supposed to be printed out when copying the system files into the config directory (like "copied /etc/tsocks.conf; attributes set....." ? I got a big long list of those lines, which made it hard to tell if the script was working when they filled up my screen.

/etc/tsocks.conf
Copy file? (y/n==this) (Y/N==all) Y
(copied; attributes set from system file)
(copied; attributes set from system file)
.....

4. Is there a way to update the config on the system without also doing the package checks/updates or vice versa?

5. When updating system from backpac config, I got the following errors (for each file):

/home/firecat53/.config/ranger/bookmarks
Update file? (y/n==this) (Y/N==all) n
(skipped)
sed: -e expression #1, char 10: unknown option to `s'

6. When I changed one file on my system then ran backpac -S to restore it, it asked for my password before updating a file in my home directory. Should only need to elevate permissions for /etc directory, right?

7. I'd like to backup /etc/sudoers. Is there a way to ask for elevated privileges before backing up a file with root user read only permissions (with appropriate security warning that the file will be visible to all users, perhaps) ?

8. Another error I noticed during a backpac -S:

PACKAGES MISSING FROM GROUPS base-devel
The following packages are listed as group members in the
            repository but are not installed locally. Please choose whether
            to install each package to your system
egrep: Invalid range end

9. Is there a way to add an ignore list for files that are not installed that you don't want installed and don't want to be nagged about each time during a restore (like nano, vi (I just have vim installed), gcc (everyone with x86_64 has gcc-multilib instead), etc.) ?

Hope I didn't overwhelm you with little fixes!! This really is absolutely perfect for how I'd like to maintain my configs.

Thanks very much!!
Scott

PS - where are you at in Seattle? I live over on the Eastside in Sammamish :-)

Offline

#3 2011-10-06 04:52:46

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

Hey Sammamish! I really want to do an Arch meetup in this neck of the woods. Interested?

firecat53 wrote:

Brilliant! You just saved me hours of "work"! I hadn't yet found any of the configuration management tools out there that seemed to work just right for me.
1. Entering 'backpac --help' (yes, my python is showing) gave these errors and then installed packer all by itself. Wasn't quite expecting that!

Yeah, getopt/getopts. I went with getopts after getopt seemed to be unmaintained and flakey, so long options went out the window. I'll try to add at least --help back in (I did think about just stripping the leading dash and converting the long option to short, might go that route).

EDIT: done.

2. Filenames in .config/backpac/$HOSTNAME/files that have spaces are not handled at all. I tried quoting the lines and using \ to escape the spaces, but neither worked right.

They should work, but right now I've only been testing if they are formatted like this:

/file/path/with\ some\ spaces/in/it

i.e. no quotes, just escaped spaces. I can do some more work to handle quotes around them (which aren't necessary if you are escaping the spaces, of course). I do want to handle quoted paths with spaces. Added to the list.

If you can, try the "simple" escaped space paths as above... should work. If not, holler. Either way I'll work on the quoted forms and test further with other variations.

3. Are the filenames supposed to be printed out when copying the system files into the config directory (like "copied /etc/tsocks.conf; attributes set....." ? I got a big long list of those lines, which made it hard to tell if the script was working when they filled up my screen.

I intend to, just haven't added it. I'll do so later tonight.
EDIT: done.

4. Is there a way to update the config on the system without also doing the package checks/updates or vice versa?

Could be if you want it. I have thought that it would be a good idea as well. It's super easy to add options to this so I can do so.
EDIT: done.

5. When updating system from backpac config, I got the following errors (for each file):
/home/firecat53/.config/ranger/bookmarks
Update file? (y/n==this) (Y/N==all) n
(skipped)
sed: -e expression #1, char 10: unknown option to `s'

That's a very useful bug report. What's your sed --version ? I'll look into this here regardless of version, of course.

6. When I changed one file on my system then ran backpac -S to restore it, it asked for my password before updating a file in my home directory. Should only need to elevate permissions for /etc directory, right?

Yeah, it should be smarter about requiring passwords. I can improve that. Noted.
EDIT: done.

7. I'd like to backup /etc/sudoers. Is there a way to ask for elevated privileges before backing up a file with root user read only permissions (with appropriate security warning that the file will be visible to all users, perhaps) ?

Me too. I almost had a special warning in the readme about sudoers. There is another issue related to sudoers which is restore procedure. There should be a syntax check run against sudoers prior to restore. It's pretty much the only file I can think of that might need this special case, but it's a catastrophic failure if a bad file gets restored, so it seems like it might be worth putting a sudoers check in there.

I'll puzzle on this.

8. Another error I noticed during a backpac -S:

PACKAGES MISSING FROM GROUPS base-devel
The following packages are listed as group members in the
            repository but are not installed locally. Please choose whether
            to install each package to your system
egrep: Invalid range end

can you let me know your grep --version ? I don't think it's a grep version issue, but just to make sure (and so I can test if it is different). I'll see if I can reproduce a similar bug.

9. Is there a way to add an ignore list for files that are not installed that you don't want installed and don't want to be nagged about each time during a restore (like nano, vi (I just have vim installed), gcc (everyone with x86_64 has gcc-multilib instead), etc.) ?

Yes, absolutely. You mean the group packages, yes? You can skip the group package currency check with -g . If you mean other, arbitrary packages, let me know.

I thought about including an ignore list of packages at one point, but I think that was before I implemented the group currency check option (-g). I can still do so if there is a case for it.

Hope I didn't overwhelm you with little fixes!! This really is absolutely perfect for how I'd like to maintain my configs.

Not at all, it's excellent detailed feedback and much appreciated. I know the bugs are in there, they just weren't falling out when I shook the code over here.

It's pretty straightforward to modify at this point. I worked pretty hard to make it modular and robust enough to be maintainable on a structural level (despite these initial bugs, which will hopefully be resolved soon) and am also open to adding options and features if they fit in with the arch way, keeping things simple, etc.

Think about a meetup, it would be great and I know there are other arch users around here.

Last edited by altercation (2011-10-06 07:05:25)


Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#4 2011-10-06 05:36:06

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

Fix summary:

action confirmation strings now display the item name, so instead of (skipped) it now says (acpi skipped). Github updated so updating the package should get you the new code.

There are two new options -i and -a for skipping files and packages units.

--help now supported (any short option can be expressed as a long option now, -l and --long are equivalent).

smarter about file installation to system, now only prompts for password when required

Last edited by altercation (2011-10-06 07:06:10)


Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#5 2011-10-06 05:53:37

bernarcher
Forum Fellow
From: Germany
Registered: 2009-02-17
Posts: 2,281

Re: backpac: A package state snapshot and restore tool for Arch Linux

This is probably better placed in Community Contributions rather than Installation.
Moving there.


To know or not to know ...
... the questions remain forever.

Offline

#6 2011-10-06 08:02:39

Unia
Member
From: Stockholm, Sweden
Registered: 2010-03-30
Posts: 2,486
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

Didn't read everything, just overlooked it quickly, but isn't this essentially the same as Pacsync?


If you can't sit by a cozy fire with your code in hand enjoying its simplicity and clarity, it needs more work. --Carlos Torres

Offline

#7 2011-10-06 08:14:51

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

Unia wrote:

Didn't read everything, just overlooked it quickly, but isn't this essentially the same as Pacsync?

It encompasses Pacsync features but significantly extends beyond them. Pacsync addresses lists of official packages and group scanning and also conforms system state to list state like backpac. Backpac also handles AUR packages and arbitrary lists of files to backup/restore/maintain-state.

I don't know how flexible the Pacsync list structure is, but it's very simple in backpac (see the readme on the github page https://github.com/altercation/backpac ) and feels very arch to me.

I did try to look at most of the other options out there before working on backpac but I just didn't find anything that fit the bill.

Last edited by altercation (2011-10-06 08:15:07)


Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#8 2011-10-06 08:16:16

Unia
Member
From: Stockholm, Sweden
Registered: 2010-03-30
Posts: 2,486
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

Ok, cool smile Nice job then! Pacsync works for me tho, so unless I find a feature I miss I'll stick with that.


If you can't sit by a cozy fire with your code in hand enjoying its simplicity and clarity, it needs more work. --Carlos Torres

Offline

#9 2011-10-06 14:57:16

manolomartinez
Member
Registered: 2011-04-29
Posts: 66
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

It'd be nice to have in files lines such as

/home/me/.config/*
/home/me/.*rc

I see that backpac does interpret the wildcards, but only considers the first substitution of each.

Cheers,
Manolo

Offline

#10 2011-10-06 17:17:52

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

manolomartinez wrote:

It'd be nice to have in files lines such as

/home/me/.config/*
/home/me/.*rc

I see that backpac does interpret the wildcards, but only considers the first substitution of each.

Glad to hear that it's interpretting at least one wildcard... i hadn't done any testing there at all.

So this actually brings up two separate issues:

1. patterns
2. directories

Patterns create some more complexity in terms of the bi-directional nature of backpac. It's something that I can implement but I want to do it properly. Let's say we wildcard /etc/* . Then each time a new file is added to /etc/ , backpac would grab it. The problem there is that the permissions on all those will be different. It is very easy to screw up a system by writing back the backpac snapshot of /etc/ with identical permissions for each file.

Directories are similar. Most directories are going to have perms of 755. Most contents will by default have 644. If we backpac snapshot a directory, I will only have the perms for the directory itself unless I expand that list in the "files" list itself (a possibility).

In summary, I did think about these things and am still thinking about them. I'd like to support directories and wildcards as long as it doesn't increase the complexity for the end user and doesn't screw up the system. Let me think more about this. Ideas welcome.


Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#11 2011-10-06 18:25:39

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

firecat53 wrote:

installed packer all by itself. Wasn't quite expecting that!

I didn't address this issue in my previous comment but I really do need to change what happens there.

Originally I wanted to be able to use this script on a clean system with no install (maybe just wgeting the script). Since then I've broken out most of the key functions in the lib directory so the easiest use case is an AUR install of backpac right now. That means that I can more reasonably expect people to install an AUR helper prior to using the script.

I think the best option here is to:

1. read AUR_HELPER from user set environment variable or use default (currently packer, I don't have strong opinions about AUR helpers and any that supports pacman is fine with me). This is how backpac works right now.

2. if the AUR_HELPER isn't installed, abort with a notice to user that it needs to be installed first OR offer to install the AUR_HELPER at that time


Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#12 2011-10-06 21:43:29

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

I think your plan for handling an AUR_HELPER is good. If AUR_HELPER is defined by the user, then either you might need a list of major AUR helpers and their command line switches so you can pick the correct switch for what needs to be done (most use some variation of -S for installing, but not all), or have the user define the correct switch(es) somehow for their chosen AUR helper.

I also found directory tracking to be a weakness in other dotfile managers that I tried. I think you would definitely have to recursively list out the contents of a tracked directory and deal with each file individually. Wildcard support would be nice...I just haven't personally found a use case for it yet.

Edit: I just compiled the dev version and removed my comments for already fixed things...sorry!

Couple more minor bugs/fixes:

1. Still apparently didn't handle the escaped space for this item: (the file does exist on my system)
files:

....
/home/firecat53/.local/src/dotfiles.git/home/firecat53/.anki/decks/Building\ Construction.media
....
MISSING ON BOTH SYSTEM AND BACKPAC CONFIG:
--------------------------------------------------------------------------------
/home/firecat53/.local/src/dotfiles.git/home/firecat53/.anki/decks/Building\ 
Construction.media

2. Suggestion: you should make that awesome README into a man page!

3. Suggestion: add the word 'dotfile' into your description somewhere on this page, the github page, and in the package description so people looking for dotfile managers will find it. You could also consider modularizing the script into a dotfile manager and the package manager, so people on other distros could take advantage of your dotfile management scheme.

4. Suggestion: since -Q is a read-only operation, why not just make it run with -f automatically to avoid the prompt?

5. Another suggestion: any thought to providing some sort of 'scrub' function to remove private information from the stored files if desired? This would be cool for publishing public dotfiles to github. Perhaps a credentials file (I did this with python for my own configs). Probably detecting email addresses and passwords without a scrub file would be rather difficult because dotfiles come in so many flavors.

Question: am I missing an obvious option to remove a file from the files.d directory if I delete it from the files list? Or do I have to delete it manually? It might be helpful to add a section to the README on how to update and delete dotfiles from being tracked, and also a more detailed description of what the -b option does (and what is actually created when it's not used).

And finally, just out of curiosity, why did you choose to actually copy the files instead of symlink like so many other dotfile managers do?

Thanks!
Scott

edit: I think a Seattle area Arch meetup would be cool! Perhaps coffee someplace? Bellevue? U-district? Anyone else? BYOPOL (bring your own pimped out laptop)

Last edited by firecat53 (2011-10-06 22:26:23)

Offline

#13 2011-10-06 22:56:45

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

firecat53 wrote:

I think your plan for handling an AUR_HELPER is good. If AUR_HELPER is defined by the user, then either you might need a list of major AUR helpers and their command line switches so you can pick the correct switch for what needs to be done (most use some variation of -S for installing, but not all), or have the user define the correct switch(es) somehow for their chosen AUR helper.

That's a good idea. I'll add that to my AUR refactoring todo.

I also found directory tracking to be a weakness in other dotfile managers that I tried. I think you would definitely have to recursively list out the contents of a tracked directory and deal with each file individually. Wildcard support would be nice...I just haven't personally found a use case for it yet.

I've been thinking that I could just add the directory and scan through it for any non-default attribute files. If those are found then they get automatically added to the files list. That's pretty close to what etckeeper does.

Edit: I just compiled the dev version and removed my comments for already fixed things...sorry!

The master branch should have those fixes as well, but I didn't update the version number in the package build. I'll have to do that.

1. Still apparently didn't handle the escaped space for this item: (the file does exist on my system)

Ok, good to know. This wildcard directory business will require some new code and refactoring so I'll also rework my filenames handling.

2. Suggestion: you should make that awesome README into a man page!

I was working on one (the pkgbuild has a commented out line for the man page) but I had to leave it for later. Definitely want a man page. Once this stabilizes and I'm sure there aren't any big structural changes, I'll convert it to man format.

3. Suggestion: add the word 'dotfile' into your description somewhere on this page, the github page, and in the package description so people looking for dotfile managers will find it. You could also consider modularizing the script into a dotfile manager and the package manager, so people on other distros could take advantage of your dotfile management scheme.

I actually have a different script for dotfile management that doesn't touch packages, but there is definitely overlap with this one. That script isn't released yet, though, and if people find this useful for dotfile management that's great. I'll add that in.

4. Suggestion: since -Q is a read-only operation, why not just make it run with -f automatically to avoid the prompt?

Originally, running backpac without any command line options produced the Query output. I was concerned that since it is a utility that can potentially overwrite system files, it is important to give users a clear statement prior to execution about what will be done. Since the Query output is essentially the same as the Update and System reports in format and content, I wanted to be explicit about the Query being a passive no-change operation. The current command line options aren't set in stone though. If you feel strongly about it being different, let me know.

Long answer to a short question smile

5. Another suggestion: any thought to providing some sort of 'scrub' function to remove private information from the stored files if desired? This would be cool for publishing public dotfiles to github. Perhaps a credentials file (I did this with python for my own configs). Probably detecting email addresses and passwords without a scrub file would be rather difficult because dotfiles come in so many flavors.

Yes, absolutely. In fact, if you look at the lib/local file (pretty sure it's in both master and dev branches in this state) you'll see some references to a sanitize function. The idea there is that the user will list out bash associative arrays like this:

SANITIZE_WPA_=(
[FILE]='/etc/wpa_supplicant.conf'
[CMD]='sed s/expungepattern/sanitizedoutput/g'
)

Question: am I missing an obvious option to remove a file from the files.d directory if I delete it from the files list? Or do I have to delete it manually? It might be helpful to add a section to the README on how to update and delete dotfiles from being tracked, and also a more detailed description of what the -b option does (and what is actually created when it's not used).

You are only missing the function I didn't finish. There should be either dummy code or a TODO in the backpac main script referencing garbage collection, which isn't difficult but I just haven't finished it. The idea being another loop of "hey I found these old files in your files.d, mind if I delete them?" It's on my list and I'll try to get it in asap.

And finally, just out of curiosity, why did you choose to actually copy the files instead of symlink like so many other dotfile managers do?

git not following symlinks, hardlinks also out for permissions issues (git wouldn't be able to read the files, change them, etc.)

I definitely would prefer to not make an entire copy of the file, but I haven't come up with a better option. Shout with ideas, though. Also, if there is a way around the link issues I noted above, let me know. I don't see one but that doesn't mean it's not there.

edit: I think a Seattle area Arch meetup would be cool! Perhaps coffee someplace? Bellevue? U-district? Anyone else? BYOPOL (bring your own pimped out laptop)

A general meetup sounds good. I was also thinking it would be fun to do a mini archcon with some demos.


Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#14 2011-10-06 23:02:50

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

firecat53 wrote:

You could also consider modularizing the script into a dotfile manager and the package manager, so people on other distros could take advantage of your dotfile management scheme.

I've thought about that. At first I wanted to just reuse a different tool or git with a detached head but none of the other solutions I used worked the way I wanted, or they were too crazy complex and finicky so I ended up building the file management in to the utility. I am continuing to think about breaking it out.

Another option that I considered was to abstract the package scanning to support other non Arch package managers, but that would be a lot more work...


Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#15 2011-10-06 23:51:07

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

altercation wrote:

I've thought about that. At first I wanted to just reuse a different tool or git with a detached head but none of the other solutions I used worked the way I wanted, or they were too crazy complex and finicky so I ended up building the file management in to the utility. I am continuing to think about breaking it out.

Ha! You and me both!! I've tried everything google could feed me on dotfile management and nothing worked quite right. I was about 1/3 of the way through coding my own solution (python) when you popped up with your handy-dandy script smile

I'd suggest that it'd be much easier for the user to just list out 'bad' words/lines in a scrub/credentials file like:

password1
joe@joe.com
password2
fred@joe.com
mysecretlongpassphrase

Then just loop over each file and redact (#####) those words/lines anywhere they appear. The only issue with having a credentials or scrub file is the obvious security problem of having all your important info in one scrub file...wonder if that file could be encrypted and then un-encrypted by the script only when needed? Yes, more complicated...hmmm. Well, I guess publishing your configs is an optional step anyways, so there's the expectation that anyone needing a scrub file will also know how to protect it....or maybe not smile

Scott

Offline

#16 2011-10-07 00:12:08

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

firecat53 wrote:

Then just loop over each file and redact (#####) those words/lines anywhere they appear. The only issue with having a credentials or scrub file is the obvious security problem of having all your important info in one scrub file...wonder if that file could be encrypted and then un-encrypted by the script only when needed? Yes, more complicated...hmmm. Well, I guess publishing your configs is an optional step anyways, so there's the expectation that anyone needing a scrub file will also know how to protect it....or maybe not smile

I personally prefer to use sed/other regex based matching on a file by file basis, but I don't mind creating a scrub list. You could just add words to a scrub list array:

SCRUB_LIST=(
'frostedflakes'
'applejacks'
'captncrunch'
'user@someplace.org'
)

*Or* it could be a separate file. I'm trying to reduce the number of separate files where possible (i.e. the AUR list used to be separate but I consolidated it with packages) but if it's optional it would be acceptable.

One downside to the scrub list is that it would parse each file. Since I have to diff the contents to determine if we need to update (there are other options, but content diffing is pretty robust), I would have to sed through each file that needs to be checked, regardless of whether it contains scrub strings. Slows things down. I am probably only tracking a dozen system files myself, since I keep my home dir under various git repos, but if anyone is using this for a greater number of files, it will slow things down more significantly.

I don't want to do any security in this utility since it's outside my areas of non-expertise and wouldn't want to screw it up. Failure would result in Bad Things Happening.


Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#17 2011-10-14 11:53:55

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

FWIW, you can write:

Files_Differ ()
{
    local _filepath_1="${1}"
    local _filepath_2="${2}"
    $(diff -q "${_filepath_1}" "${_filepath_2}" &>/dev/null) && return 1 || return 0
}

as just:

Files_Differ ()
{
    local _filepath_1="${1}"
    local _filepath_2="${2}"
    diff -q "${_filepath_1}" "${_filepath_2}" &>/dev/null
}

in bash, the function exit code is that of the last command


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#18 2011-10-14 12:31:55

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

btw i see some bugs during my first run:

backpac                                                                                                                                                                                  127 ↵
/usr/share/backpac/lib/common: eval: line 277: unexpected EOF while looking for matching `''
/usr/share/backpac/lib/common: eval: line 278: syntax error: unexpected end of file
--2011-10-14 14:30:10--  https://aur.archlinux.org/packages/packer/packer.tar.gz
Resolving aur.archlinux.org... 208.92.232.29
Connecting to aur.archlinux.org|208.92.232.29|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 593 [application/x-tgz]
Saving to: `packer.tar.gz'

100%[==========================================================================================================================================================================>] 593         --.-K/s   in 0s      

2011-10-14 14:30:11 (5.59 MB/s) - `packer.tar.gz' saved [593/593]

packer/
packer/PKGBUILD
==> Determining latest git revision...
  -> Version found: 20111014
==> Making package: packer 20111014-1 (Fri Oct 14 14:30:11 CEST 2011)
==> Checking runtime dependencies...
==> Installing missing dependencies...

< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#19 2011-10-14 16:27:30

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

Small issues:

PACKAGES NOT INSTALLED: packages listed in backpac config but not installed:
--------------------------------------------------------------------------------
libwebkit3

However, it is installed:

09:18:42 $ pacman -Qs webkit3
local/libwebkit3 1.6.1-1
    An opensource web content engine (for GTK3)
$ backpac -Ub
.......
/home/firecat53/.config/dwb/default/bookmarks
(CURRENT IN backpac:600 firecat53 users) (NEW FROM SYSTEM:644 firecat53 users)
Update file & attributes? (y/n==this) (Y/N==all) y
/usr/share/backpac/lib/common: line 630: File_Install_Config: command not found
(/home/firecat53/.config/dwb/default/bookmarks update content & attributes
failed!)

Scott

edit:

One more error I get when running from cron:

/usr/bin/backpac: line 103: XDG_CONFIG_HOME: unbound variable

Last edited by firecat53 (2011-10-17 23:51:50)

Offline

#20 2011-11-14 09:59:29

bender02
Member
From: UK
Registered: 2007-02-04
Posts: 1,328

Re: backpac: A package state snapshot and restore tool for Arch Linux

Hi there!
I like your script quite a bit... it does what I used to do manually until now wink
There's one ?bug? I've had some gnome packages installed - not quite 80% of the gnome group (so it didn't ask me about that); but backpac didn't actually record *any* packages from that group as installed... they were just missing from the list.

Offline

#21 2011-11-14 18:59:29

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

bender02 wrote:

Hi there!
I like your script quite a bit... it does what I used to do manually until now wink
There's one ?bug? I've had some gnome packages installed - not quite 80% of the gnome group (so it didn't ask me about that); but backpac didn't actually record *any* packages from that group as installed... they were just missing from the list.

I was pretty caught up working on Arch EFI booting ( https://bbs.archlinux.org/viewtopic.php?pid=1016692 ) and haven't worked on backpac for a couple weeks but now that I have EFI working I'm in need of an updated backpac so will look into this.

I may ping you for some sample output from pacman to help debug.


Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#22 2011-12-21 18:14:41

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

hey did you see this?
https://github.com/devstructure/blueprint
philosophy and detailed documentation:  http://devstructure.github.com/blueprint/
it's like backpac, but more advanced(it finds stuff in /usr/local, modified configs, it can generate shellscripts/puppet template/chef recipes, etc), but minus the arch support smile
I packaged it in aur -- https://aur.archlinux.org/packages.php?ID=55179 but it would be cool if it would actually get support for pacman


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#23 2012-01-23 20:38:40

altercation
Member
From: Seattle
Registered: 2011-05-15
Posts: 136
Website

Re: backpac: A package state snapshot and restore tool for Arch Linux

Dieter@be wrote:

hey did you see this?
https://github.com/devstructure/blueprint
philosophy and detailed documentation:  http://devstructure.github.com/blueprint/
it's like backpac, but more advanced(it finds stuff in /usr/local, modified configs, it can generate shellscripts/puppet template/chef recipes, etc), but minus the arch support smile
I packaged it in aur -- https://aur.archlinux.org/packages.php?ID=55179 but it would be cool if it would actually get support for pacman

@Dieter@be, thanks for the pointer to that. I actually want to split out the file state snapshot functions from backpac so that it is only a pacman package summary/reinstall tool. If blueprint supported that it would be excellent.

Have you touched base with the blueprint dev on pacman support or patched?


Ethan Schoonover
Precision Colors - http://ethanschoonover.com/solarized

Offline

#24 2012-03-23 05:54:48

badwolf9
Member
Registered: 2012-02-16
Posts: 1

Re: backpac: A package state snapshot and restore tool for Arch Linux

Hi
Many thanks for this clever little program. I will follow its development from now on, having (almost) bricked my Arch install on a laptop with a rather too enthusiastic use of the testing repos.......
regards and keep up the good work
Martin

Offline

#25 2013-03-30 21:21:29

plenus
Member
Registered: 2012-04-19
Posts: 30

Re: backpac: A package state snapshot and restore tool for Arch Linux

Is this (hopefully) wonderful tool still maintained?

This seems to be exactly what I've always wanted. But when I've tried to compile from AUR (using pacaur and makepkg -s) it failed. The last AUR update was from 2011-10-05 23:34. The last update in github was from a year ago?

Well, I hope that is because all possible bugs and quirks were already ironed out! ;-)

update: the error I get running makepkg

$ ls
PKGBUILD

$ LANG=C makepkg -s
==> Determining latest git revision...
  -> Version found: 20130330
==> Making package: backpac 20130330-1 (Sat Mar 30 21:38:56 BRT 2013)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving Sources...
==> Extracting Sources...
==> Removing existing pkg/ directory...
==> Entering fakeroot environment...
==> Starting build()...
/home/plenus/projects/arch/backpac/PKGBUILD: line 15: cd: /home/plenus/projects/arch/backpac/src: No such file or directory
==> ERROR: A failure occurred in build().
    Aborting...

Last edited by plenus (2013-03-31 00:45:13)

Offline

Board footer

Powered by FluxBB