maint - automatic Arch system maintenance

mgdobachesky · 2018-06-11 02:23:52

Hello everyone,
Arch Linux can be a lot of work to maintain at times. I used to visit the maintenance wiki page (https://wiki.archlinux.org/index.php/System_maintenance) to look up a bunch of commands for execution every time I wanted to do any kind of system maintenance task. Now that is all fine and good, but one day I decided to streamline the whole process and just script all those commands. After that, I decided to add a bunch of features to make it even more convenient.

This project has made my Arch Linux system maintenance routine so much easier, and if it was able to help me out so much I figure that it could help some of you guys out as well.

Current Functionality:
- Get Arch Linux News
- Upgrade System (Following best practices)
- Clean Filesystem
- System Error Check
- Backup System
- Restore System

Here is a link to my GitLab repo housing the project: (check out the readme to get a bunch more information)
https://gitlab.com/mgdobachesky/ArchSystemMaintenance

Additionally, I have posted this package to the AUR under the name maint:
https://aur.archlinux.org/packages/maint/

It would be much appreciated if you guys would try it out and leave some feedback for potential improvements. (maybe even make some improvements yourself and follow up with a merge request on the GitLab repo!)

I really appreciate all of your feedback and contributions
- Mike

Last edited by mgdobachesky (2018-09-04 02:03:12)

jasonwryan · 2018-06-11 02:36:16

A couple of thoughts, based on a quick read.

If you are using bash, consider using [[ over [. And paccache is a more controlled approach than -Sc

Also Awk does a better job that multiple grep/tail/whatever pipes, eg: awk '/WARNING/' /var/log/pacman.log

Moving to Community Contributions

mgdobachesky · 2018-06-11 02:45:32

Thank you jasonwryan!
On my next batch of edits I will switch over to using double square brakets instead of single, switch over to paccache, and move towards Awk instead of using multiple pipes.
All excellent suggestions for making the project much cleaner!

- Mike

Last edited by mgdobachesky (2018-06-11 03:17:10)

eschwartz · 2018-06-11 02:59:01

You have a custom function with several conditionals in order to accept a numbered list choosing which actions to perform in a loop. The whole thing could be replaced by:

PS3='Action to take: '
select opt in "Arch Linux News" "Upgrade the System" "Clean the Filesystem" "Exit"; do
    case $REPLY in
        1) fetch_news;;
        2) system_upgrade;;
        3) system_clean;;
        4) break;;
        *) echo "Please choose an existing option";;
    esac
done

Last edited by eschwartz (2018-06-11 03:00:05)

mgdobachesky · 2018-06-11 03:16:09

Thank you Eschwartz!
That seems like a much more efficient way to go about implementing the UI loop.
I will certainly be adding your suggestions into the next batch of updates as well!

- Mike

Last edited by mgdobachesky (2018-06-11 03:17:55)

Alad · 2018-06-11 16:01:48

Some bikeshed assuming you want other people to use it:

https://gitlab.com/mgdobachesky/ArchSys … /run.sh#L5

Use paclog from pacutils to handle this more generally

https://gitlab.com/mgdobachesky/ArchSys … /run.sh#L7

Always quote variables in bash

https://gitlab.com/mgdobachesky/ArchSys … /run.sh#L8

Going by this segment the python script sets the exit code to the amount of news, but it always appears to be 1.

https://gitlab.com/mgdobachesky/ArchSys … run.sh#L28

Not everyone lives in US

https://gitlab.com/mgdobachesky/ArchSys … run.sh#L36

What does this do?

https://gitlab.com/mgdobachesky/ArchSys … run.sh#L57

This blindly merges changes from the AUR repo, which is a bad idea since a malicious or incompetent maintainer may take over a package

https://gitlab.com/mgdobachesky/ArchSys … run.sh#L72

You don't need to run pacman twice, just save the results (newline delimited) in an array.

mapfile -t orphans < <(pacman -Qtdq)
if [[ ${orphans[*]} ]]; then
  # print stuff 
  sudo pacman -Rns "${orphans[@]}"
fi

https://gitlab.com/mgdobachesky/ArchSys … run.sh#L82

First I'd use lower-case variables since none of these variables are exported. Otherwise this looks like an awfully complicated way to use grep:

grep -Fxvf <(aur_list) <(pacman -Qmq) | sudo pacman -Rns -

https://gitlab.com/mgdobachesky/ArchSys … un.sh#L112

Again paclog is your friend

https://gitlab.com/mgdobachesky/ArchSys … un.sh#L124

Use pacdiff, doesn't require sudo updatedb either

https://gitlab.com/mgdobachesky/ArchSys … un.sh#L137

That comment is generally wrong, see CleanMethod in pacman.conf

https://gitlab.com/mgdobachesky/ArchSys … un.sh#L143

Similar to the above on orphans you don't need to run find twice (in particular since links may have changed between both invocations)

https://gitlab.com/mgdobachesky/ArchSys … un.sh#L164

In most of the functions above that you ask if the user wants to remove something, here you run rmlint.sh without confirmation.

edit: I will add that most people will want pacutils installed anyway, due to pacrepairdb which is a tremendous help when something breaks the pacman db.

Last edited by Alad (2018-06-11 16:51:41)

Trilby · 2018-06-11 16:34:55

If you don't want to use paclog, at very least get rid of the useless use of cat. There is also no reason for the pipeline:

LAST_UPGRADE=$(sed -n '/pacman -Syu/h; ${x;s/.\([0-9-]*\).*/\1/p;}' /var/log/pacman.log)

mgdobachesky · 2018-06-11 23:09:30

Thank you Alad!
Your feedback covers issues that I had not even thought of, like creating mirrorlists outside of the US and switching over to cleaner tools like paclog and pacdiff.
I am really excited for my next opportunity to start implementing these (and other) improvements!

Also, you asked what the following code does:

aur_list() {
	LIST=""
	while IFS= read -r -d $'\0'; do
		for D in $REPLY/*/; do 
			if [ -d "$D" ]; then
				LIST="${LIST},${D}"
			fi
		done
	done < <(sudo find /home -name ".aur" -print0)

	echo "${LIST:1}"
}

I keep all of my AUR packages in a directory named .aur, so I call the function above whenever I want to do anything with them in the script. (It returns a comma-separated list of absolute paths)
I know this part is a bit specific, so if I can find a more standard way to handle AUR packages then I will switch over as soon as possible.

- Mike

Last edited by mgdobachesky (2018-06-11 23:18:46)

mgdobachesky · 2018-06-11 23:16:35

Thank you Trilby!
sed accomplishes the task of getting the last upgrade in a way that is so much cleaner than piping all those commands!
While I do plan on switching over to using paclog, your feedback has still given me insight as to of the better alternatives I can use in the future.

- Mike

EDIT: I ended using your sed command to avoid the pipeline afterall, atleast until I can figure out how to simplify things a bit more with paclog.

Last edited by mgdobachesky (2018-06-19 02:25:00)

Alad · 2018-06-12 10:36:17

I know this part is a bit specific, so if I can find a more standard way to handle AUR packages then I will switch over as soon as possible.

I'm not sure; pacaur and aurutils use AURDEST to set this directory so I guess you could use the same or a similar environment variable.

If not I'd consider following the XDG spec, e.g. XDG_DATA_HOME/aur or XDG_CACHE_HOME/aur, cf. https://wiki.archlinux.org/index.php/XD … irectories

Last edited by Alad (2018-06-12 10:36:59)

mgdobachesky · 2018-06-12 21:59:07

Okay awesome, thanks for the tip!
I will probably go with AURDEST, but I will look into the pros and cons of using that over an XDG variable.

- Mike

mgdobachesky · 2018-09-04 00:54:16

Got it

Thankyou,
- Mike

Last edited by mgdobachesky (2018-09-15 22:02:04)

jasonwryan · 2018-09-04 00:58:40

You can edit it: use the edit button on the first post.

mgdobachesky · 2018-09-04 02:04:06

I didn't realize I could do that.
Thanks for the help!

- Mike

Last edited by mgdobachesky (2018-09-15 22:03:03)

madddmike · 2020-03-01 18:12:00

Why can't I use this on my Raspberry Pi running Arch if it's just a bash script???

==> ERROR: maint is not available for the 'armv7h' architecture.

madddmike · 2020-03-01 18:13:56

Also trying to do an install from AUR, it gives me an error that "reflector" is required???

ewaller · 2020-03-01 18:49:27

maddmike, Unfortunately, we cannot help you here with Raspberry pi questions. As much as we love Arch Linux for ARM, it is a different distribution and is an architecture we do not support. You may want to ask on the Arch Linux ARM forums.

Before I send you off, the MAKEPKG probably specifies a x86_64 architecture. You could try to change the PKGBUILD to add your architecture and rebuild it. reflector must be a dependency that also needs to be installed.

Best of luck. Closing this old thread.

Edit: Re-opened thread after reconsideration after reading a compelling report.

Last edited by ewaller (2020-03-01 18:57:09)

eschwartz · 2020-03-01 20:08:21

Obviously reflector must be required for the feature which runs the reflector program and updates your mirrorlist. The OP may wish to add upstream support for failing graciously when it is not available, though.

As for the armv7h architecture thing, that's just flat out wrong, since it is a shellscript and thus the package can be built once and installed on any architecture (including any that archlinux.org might come to add in the future). So it should be arch=('any'), rather than listing x86_64 *or* your own architecture.

I guess *all* of us who commented almost two years ago only provided feedback on the script, and not the PKGBUILD, which in retrospect is ever so slightly embarrassing (for me at least).

mgdobachesky · 2020-03-28 02:09:21

Adding support for any architecture seems easy enough, as does including a check for the reflector package in the main script before actually using the reflector tool.
One thing that does not seem as straight forward, however, is requiring the reflector package as a dependency in the PKGBUILD while also including the ability to fail graciously if it is not available.
Most users will want to have reflector automatically installed as a depends so they can take advantage of the mirror-list updating feature while upgrading their systems, so it doesn't seem appropriate to move reflector over to an opt-depends.
However, if the user does not have the Community repository enabled then the reflector package will not be available, and it should not be included as a depends.
This might work with a conditional that checks /etc/pacman.conf before including reflector, as a workaround if there isn't already a better built-in way to handle this situation for the PKGBUILD.

Last edited by mgdobachesky (2020-03-28 02:10:01)

eschwartz · 2020-04-01 04:10:34

You could use the PKGBUILD feature depends_x86_64=('reflector'), this would make it only depend on reflector if built on an x86_64 system. Oddly enough, this applies even if the PKGBUILD specifies arch=('any'), though the truth is if you're going to have architecture-dependent dependencies I'd recommend you move back to arch=('i686' 'x86_64' 'armv7h' 'aarch64') and so on.

That being said, I think an optdepends is probably fair enough, since users will see

New optional dependencies for maint
    reflector: to maintain the pacman mirrorlist

and they should pay attention to such messages on general principle.

mgdobachesky · 2020-04-02 22:45:54

Thank you eschwartz. My guess as to of why maddmike was getting and error with the 'reflector' dependency was that he might have had the community repository disabled. Do you think it is possible that the package will not install because of his armv7h architecture? The pkgbuild for reflector seems to have arch=(any) specified, so how would this happen?

eschwartz · 2020-04-02 23:39:15

The Arch Linux ARM repositories don't have reflector at all, see their package search page: https://archlinuxarm.org/packages

It would need to be forked in order to make use of a different endpoint for scraping mirrors, and I think they use their pacman-mirrorlist package as the canonical source So there isn't much point to packaging it if it doesn't work.

Parabola, for example, runs a full archweb with the same API as www.archlinux.org, so they patch reflector with this very simple patch: https://git.parabola.nu/abslibre.git/tr … ding.patch

And they package reflector for their i686, x86_64, and armv7h repos, but in [libre] instead of [community].

EDIT: similarly, archlinux32 forked it and applied https://git.archlinux32.org/reflector32 … 18cce20160

Last edited by eschwartz (2020-04-02 23:46:25)

Xyne · 2020-04-03 00:34:09

eschwartz wrote:

It would need to be forked in order to make use of a different endpoint for scraping mirrors, and I think they use their pacman-mirrorlist package as the canonical source So there isn't much point to packaging it if it doesn't work.

There's no need to fork. Reflector now has a "--url" option that can be used to select a different json api. It still has to follow the same format though.

mgdobachesky · 2020-04-03 02:05:11

Okay great thanks for explaining that eschwartz, now I understand why the architecture would affect a users ability to install the reflector package. I did not realize how closely it integrates with the underlying architecture's repositories. With that, I definitely agree that it would make sense to add reflector as an optdepends (or depends_x86_64 at the very least). It might also make sense to add a requirement that the distribution in use is in fact Arch Linux.
Also, Xyne mentioned that there is a --url option for selecting different json APIs. This seems like it could be useful for dynamically changing the API based on the architecture in use, but wouldn't this only be useful if that architecture's repositories actually include the reflector package?

eschwartz · 2020-04-03 06:44:14

I don't see how the --url option helps, really, since other distributions will still want it to work out of the box, and that means patching the default. Or I suppose one could package an alternative /usr/bin script which sets an --url. Either way it would require modifying the reflector PKGBUILD somehow

Arch Linux

#1 2018-06-11 02:23:52

maint - automatic Arch system maintenance

#2 2018-06-11 02:36:16

Re: maint - automatic Arch system maintenance

#3 2018-06-11 02:45:32

Re: maint - automatic Arch system maintenance

#4 2018-06-11 02:59:01

Re: maint - automatic Arch system maintenance

#5 2018-06-11 03:16:09

Re: maint - automatic Arch system maintenance

#6 2018-06-11 16:01:48

Re: maint - automatic Arch system maintenance

#7 2018-06-11 16:34:55

Re: maint - automatic Arch system maintenance

#8 2018-06-11 23:09:30

Re: maint - automatic Arch system maintenance

#9 2018-06-11 23:16:35

Re: maint - automatic Arch system maintenance

#10 2018-06-12 10:36:17

Re: maint - automatic Arch system maintenance

#11 2018-06-12 21:59:07

Re: maint - automatic Arch system maintenance

#12 2018-09-04 00:54:16

Re: maint - automatic Arch system maintenance

#13 2018-09-04 00:58:40

Re: maint - automatic Arch system maintenance

#14 2018-09-04 02:04:06

Re: maint - automatic Arch system maintenance

#15 2020-03-01 18:12:00

Re: maint - automatic Arch system maintenance

#16 2020-03-01 18:13:56

Re: maint - automatic Arch system maintenance

#17 2020-03-01 18:49:27

Re: maint - automatic Arch system maintenance

#18 2020-03-01 20:08:21

Re: maint - automatic Arch system maintenance

#19 2020-03-28 02:09:21

Re: maint - automatic Arch system maintenance

#20 2020-04-01 04:10:34

Re: maint - automatic Arch system maintenance

#21 2020-04-02 22:45:54

Re: maint - automatic Arch system maintenance

#22 2020-04-02 23:39:15

Re: maint - automatic Arch system maintenance

#23 2020-04-03 00:34:09

Re: maint - automatic Arch system maintenance

#24 2020-04-03 02:05:11

Re: maint - automatic Arch system maintenance

#25 2020-04-03 06:44:14

Re: maint - automatic Arch system maintenance

Board footer