You are not logged in.

#1 2018-03-31 00:06:54

jernst
Member
From: Silicon Valley
Registered: 2014-03-04
Posts: 290
Website

Restart all updated services

When I pacman -Syu, some daemons and/or their .service files might have been updated. How do I find out which ones, so I can restart those?

Or even better, can I automate that somehow? It seems possible to watch for .service file changes via a Pacman hook, but that does not seem to allow me to watch for changes of the underlying executables. On the other hand, perhaps it would be a good idea to restart any service whose .service file is defined in a package that has been updated? That should be possible with a Pacman hook? (I have not done Pacman hooks before ...)

Thanks,



Johannes.

Offline

#2 2018-03-31 00:21:35

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: Restart all updated services

You don't need to restart anything, you can just use `systemctl daemon-reload`.

As for watching for changes to underlying executables, why would you need to?  If it is a running process, it will not stop functioning simply because the on-disk binary changed.  If/when that process restarts the new version will start, but why should there be a need to expedite that?


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#3 2018-03-31 00:27:26

jernst
Member
From: Silicon Valley
Registered: 2014-03-04
Posts: 290
Website

Re: Restart all updated services

Depends on the daemon IMHO whether if keeps functioninig. For example, if its directory layout changes that daemon would be out of luck.

Another example: some cipher scheme has been broken, the Apache config files disable the broken cipher in the new version, but because httpd.service is not restarted, Apache never picks it up, so that would be a security problem.

Offline

#4 2018-03-31 00:44:48

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 999

Re: Restart all updated services

jernst wrote:

When I pacman -Syu, some daemons and/or their .service files might have been updated. How do I find out which ones, so I can restart those?

Or even better, can I automate that somehow? It seems possible to watch for .service file changes via a Pacman hook, but that does not seem to allow me to watch for changes of the underlying executables. On the other hand, perhaps it would be a good idea to restart any service whose .service file is defined in a package that has been updated? That should be possible with a Pacman hook? (I have not done Pacman hooks before ...)

The only reliable way to do it is to reboot the system. Windows actually got it right, sSee also here https://fedoraproject.org/wiki/Features … temUpdates. In principle, making updates on a running system is always a risk due to possible incompatibilities of libraries, etc.


Arch Linux is more than just GNU/Linux -- it's an adventure
pkill -9 systemd

Offline

#5 2018-03-31 01:03:39

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: Restart all updated services

If you want apache to pick up the new configs `systemctl reload httpd` - but the previous version would continue running just fine.  You'd certainly know to reload apache as you would have had to merge in the .pacnew config(s) anyways for them to have any effect.

Leonid.I wrote:

The only reliable way to do it is to reboot the system.

No, that is not the only reliable way.

Leonid.I wrote:

In principle, making updates on a running system is always a risk due to possible incompatibilities of libraries, etc.

Do you have any concrete examples?  What exactly is this risk?  I have a server that does not get rebooted, but it gets regular updates.  No problem at all.  Rebooting is not the only reliable way to safely update - it may be the only reliable way to safely update if you never bother to learn otherways, but that's not saying much.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#6 2018-03-31 01:28:48

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 999

Re: Restart all updated services

Trilby wrote:
Leonid.I wrote:

The only reliable way to do it is to reboot the system.

No, that is not the only reliable way.

It is, unless you want to restart every running program on your system...

Trilby wrote:
Leonid.I wrote:

In principle, making updates on a running system is always a risk due to possible incompatibilities of libraries, etc.

Do you have any concrete examples?  What exactly is this risk?

Pls see that fedoraproject link... As for examples, I can only speak for us. Here (JILA, CU Boulder), we have a small group of machines running F26 which is updated nightly via cron. Well, ppl routinely complain about evince and gedits crashing in the mornings (they usually leave things running behind a screenlock). My machine runs Arch and I never saw such behavior, but my workstation is excluded from automatic updates. Moreover, it has ro /, and is updated rarely enough for the linux package to get pulled in...

Trilby wrote:

I have a server that does not get rebooted, but it gets regular updates.  No problem at all.

It is an artifact of olden days that ppl think of servers as something high-end. In reality, a server is usually a quite simple machine, software-wise. Of course, it must have reliable hardware, but software is usually well-tested and maintained. And there is simply not too much of it. On the other hand, workstations are WAY more complex in terms of software. To give you an example, my Arch workstations never have more than 400 packages. My Arch server (dnsmasq, samba, firewall, buildhost) is only about 150.

So, the fact that you have a server that never crashes really says nothing about your software management (my server runs Arch testing and I reboot it twice a year without any issues). If you tell me that you have such workstation with a bunch of software of questionable quality (a la GNOME, blender, etc), then I'll listen.

Trilby wrote:

Rebooting is not the only reliable way to safely update - it may be the only reliable way to safely update if you never bother to learn otherways, but that's not saying much.

Well, if I add a qualifier "updates to a general-purpose OS" would that help? tongue

Last edited by Leonid.I (2018-03-31 01:30:32)


Arch Linux is more than just GNU/Linux -- it's an adventure
pkill -9 systemd

Offline

#7 2018-03-31 01:29:37

amish
Member
Registered: 2014-05-10
Posts: 470

Re: Restart all updated services

Trilby wrote:

... but why should there be a need to expedite that?

Bug fixes? Security fixes/updates? New features?

Hence in my opinion one should always restart* immediately and verify everything is in order with new version.

* large package update = restart system
* smaller update = restart related service

Last edited by amish (2018-03-31 04:39:00)

Offline

#8 2018-03-31 01:33:57

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: Restart all updated services

Leonid.I wrote:

If you tell me that you have a workstation with a bunch of software of questinable quality, then I'll listen.

The same is true of my home computer.  I pretty much never shut it down or reboot.  But I don't use software of "questionable quality".  Why on earth would I?  And if by this you are implying that software that would require a reboot is of questionable quality, I'd agree.  That is a bug in said software: report it as a bug so it can get fixed.

Leonid.I wrote:

As for examples, I can only speak for us. Here (JILA, CU Boulder), we have a small group of machines running F26 which is updated nightly via cron.

You do realize this is the arch forums?  If you can only speak for Fedora, then don't speak here.  This question was about jernst's arch system(s), not Fedora.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#9 2018-03-31 01:59:04

jernst
Member
From: Silicon Valley
Registered: 2014-03-04
Posts: 290
Website

Re: Restart all updated services

1. I hear people saying "there is an algorithm for doing what I want to do", which is to be certain to get all the daemons running in the most recent version after an upgrade.
2. Some people say "the best known algorithm is to always reboot"
3. Some people say "there is a better algorithm that does (rarely) require a reboot".

Shall we try and define an algorithm that's better than "always reboot" and see how well we do? With "algorithm" I mean an unambigous set of instructions that either a person, or a computer, can execute that always produces the desired result, for any type of Arch system running any combination of daemon.

Trilby: you say you are using such an algorithm. Care to share what you do?

Here is my straw man:

Assumptions:
* all daemons are started and managed through systemd.
* all code on the machine comes from pacman packages

Then:
* record the set of all currently running daemons. Lets call that Set<D>
* for each d = member of Set<D>, determine the packages that contain d's service file and/or running executable. Let's called that Set<d,P,before>
* for each d = member of Set<D>. determine the recursive package dependency tree of all packages in Set<d,P,before>, and call it Set<d,PR,before> (which includes Set<d,P,before>)
* upgrade the system, and note the set of packages that were upgraded (Set<P,upgraded>) -- this set needs to include removed packages
* for each d = member of Set<D>, determine the packages that contain d's service file and/or running executables. Let's called that Set<d,P,after>
* for each d = member of Set<D>, determine the recursive package dependency tree of all packages in Set<d,P,after>, and call it Set<d,PR,after> (which includes Set<d,P,after>)
* for each d = member of Set<D>, intersect Set<d,P,upgraded> with the union of Set<d,PR,before> and Set<d,PR,after>
* Daemon d must be restarted iff the resulting set is not empty.

Chances are I got that wrong, but how is this for a start?

Offline

#10 2018-03-31 02:23:13

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: Restart all updated services

jernst wrote:

Trilby: you say you are using such an algorithm. Care to share what you do?

Generally, nothing.  If you are having trouble, describe it and we can ensure that it is avoided in the future.  If you are not having troube, don't create any.

But at most:

Trilby wrote:

You don't need to restart anything, you can just use `systemctl daemon-reload`.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#11 2018-03-31 02:28:14

jernst
Member
From: Silicon Valley
Registered: 2014-03-04
Posts: 290
Website

Re: Restart all updated services

Sorry, in my book that doesn't qualify as an algorithm that always produces the correct result :-) because that's what I'm looking for in this thread.

So we have two algorithms on the table, one known to be correct but with low availability, and one whose correctness has not been proven (aka needs bugfixing) but would be more efficient. I'd love some feedback, and I wonder whether anybody thinks it would be worthwhile implementing, e.g. as an (optional) pacman hook.

Offline

#12 2018-03-31 02:37:10

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: Restart all updated services

jernst wrote:

Sorry, in my book that doesn't qualify as an algorithm that always produces the correct result.

So you want an "algorithm" to solve a problem, but you cannot state what the problem is that needs to be solved?

Ok, then at least describe in what way daemon-reload fails to meet the unspecified criteria as well as rebooting does?


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#13 2018-03-31 02:42:54

jernst
Member
From: Silicon Valley
Registered: 2014-03-04
Posts: 290
Website

Re: Restart all updated services

Trilby, with all respect, we are not communicating. I stated the problem, I gave examples, and outlined a possible solution to the problem as stated. If neither you, nor anybody else, understands what I'm saying (for whatever reasons -- it could be, as you imply, that I make no sense) that's okay, I should get no responses on this thread because everybody has better things to do than commenting on something that is either incomprehensible or makes no sense.

I am, however, looking forward to comments from people who think that what I'm asking for makes sense.

Thank you for listening :-)

Offline

#14 2018-03-31 04:01:54

amish
Member
Registered: 2014-05-10
Posts: 470

Re: Restart all updated services

I think we can have a hook that considers different opinions mentioned above and allows them to set it as per their need.

For example: (rough idea only)

File: /usr/share/libalpm/hooks/systemd-restart.hook

[Trigger]
Type = Package
Operation = Upgrade
Target = *

[Action]
Description = Restart systemd services ...
When = PostTransaction
Exec = /usr/share/libalpm/scripts/systemd-restart-hook
NeedsTargets

File: /etc/conf.d/systemd-restart-hook

declare -A SERVICEMAP
#SERVICEMAP[squid]="squid.service"
#SERVICEMAP[apache]="httpd.service"
#SERVICEMAP[clamav]="clamav-daemon.service clamav-freshclam.service"

Anyone who wants certain services to be restarted on update, can set SERVICEMAP as above.

They can even add their own services and choices.

By default it will NOT restart anything (as SERVICEMAP is empty by default)


Now the heart of the hook:
File: /usr/share/libalpm/scripts/systemd-restart-hook

It will:
1) Do nothing if in chroot or equivalent
2) Read package name from STDIN (which will contain packages that were updated)
3) If there exists a "service map" for that "package", it will restart the related service

for example: (something like)
systemctl daemon-reload #will be run only once
systemctl try-restart ${SERVICEMAP[$pkgname]}

So for clamav it will automatically run:
systemctl try-restart clamav-daemon.service clamav-freshclam.service

try-restart will make sure that service is restarted only if running currently

This should keep both worlds happy!

Last edited by amish (2018-03-31 04:38:07)

Offline

#15 2018-03-31 04:07:20

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 999

Re: Restart all updated services

Trilby wrote:
Leonid.I wrote:

If you tell me that you have a workstation with a bunch of software of questinable quality, then I'll listen.

The same is true of my home computer.  I pretty much never shut it down or reboot.  But I don't use software of "questionable quality".  Why on earth would I?  And if by this you are implying that software that would require a reboot is of questionable quality, I'd agree.  That is a bug in said software: report it as a bug so it can get fixed.

By questionable quality I mean all of GNOME, KDE, and also things like Enlightenment and Mate smile Anyway, the particular issue that I always encountered is evince or firefox would crash or freeze when left open during an update. Of course, it's all case-by-case, i.e. depends on what kind of document you open, etc. The issue here is not a bug in software, but a change in a library.

Trilby wrote:
Leonid.I wrote:

As for examples, I can only speak for us. Here (JILA, CU Boulder), we have a small group of machines running F26 which is updated nightly via cron.

You do realize this is the arch forums?  If you can only speak for Fedora, then don't speak here.  This question was about jernst's arch system(s), not Fedora.

If you know the software that runs on your system, then you can always do things manually. I know *everything* about my Arch box, and so do you smile That's why our systems are solid. On the contrary, ppl here who use Fedora are generally clueless about their OS...


Arch Linux is more than just GNU/Linux -- it's an adventure
pkill -9 systemd

Offline

#16 2018-03-31 04:16:16

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 999

Re: Restart all updated services

jernst wrote:

Trilby, with all respect, we are not communicating. I stated the problem, I gave examples, and outlined a possible solution to the problem as stated. If neither you, nor anybody else, understands what I'm saying (for whatever reasons -- it could be, as you imply, that I make no sense) that's okay, I should get no responses on this thread because everybody has better things to do than commenting on something that is either incomprehensible or makes no sense.

I am, however, looking forward to comments from people who think that what I'm asking for makes sense.

Thank you for listening :-)

The problem that you are stating makes sense, but your question (as well as possible solution) DOES NOT because there is no *generic* algorithm except (1) reboot after each update and (2) run a stable distro like RHEL. "generic" here means "no prior knowledge about the system and assuming that you and other users of the system are idiots".

This means that if you want anything beyond (1) or (2), you need to tell us more about your particular setup: what daemons are running, how many users on average are logged in at any given time, etc.

Just to give you an example: sometimes, you need to manually update readline after bash (this happened few times in my memory). In between these two updates, is it OK for users to have broken bash?

Last edited by Leonid.I (2018-03-31 04:17:29)


Arch Linux is more than just GNU/Linux -- it's an adventure
pkill -9 systemd

Offline

#17 2018-03-31 08:07:34

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: Restart all updated services

Unless you know what you do and do it for a good reason, reboot or don't do anything. Period.

If a service tries to (re)load a kernel module and you had the kernel updated, this will fail.
If the service tries to restart a process in place (not kill/start) and the process or its libriaries are updated on disk, it will crash.

@Leonid.I, it is perfectly fine to update w/o reboot - this will not magically make processes crash.
What you may encounter is a process loading a plugin library and that (updated) plugin shares an object w/ the main process and got updated. Crash for sure. Or loading a symbol from a now updated library (which has not been in use before) - related to what also can happen is that a process got swapped out and its anon memory now mismatches its file backed memory. Crash for sure on re-swap.
You however do not have to reboot to deal with either of those. Kill/restarting the suspicious (updated) binary is completely sufficient.
A kernel update is a good reason to reboot (briefly after) - unless you want to use kexec - but that's about it.

Not sure what you mean by your readline/bash anecdote, though. Sounds like you're conducting partial updates, leaving the system in an incosistent state, but that is certainly not "fixed" by a reboot of the inconsistent system.

Offline

#18 2018-03-31 22:19:44

jernst
Member
From: Silicon Valley
Registered: 2014-03-04
Posts: 290
Website

Re: Restart all updated services

What if we restict this discussion to server-side systems where there are no user-initiated processes like Firefox, except for whatever the admin is running via ssh, and that should not be a problem. (Which happens to be my interest anyway) Does this make the problem easier? (I think it does)

@Leonid.I: Do you have any evidence for the thesis "there is no *generic* algorithm", or is it just that we don't know of one? It's also possible that the metadata available (e.g. .service files, pacman metadata) is insufficient for an algorithm to work. In which case I'd love to know what metadata is missing.

@amish: This is an interesting idea. It doesn't solve the entire problem, but it could certainly improve things. Come to think of it, if packages (daemons?) could declare when they want to be restarted (aka additional metadata) that might also improve things. E.g. "I'am httpd.service, please restart me if packge ssl, ... have been updated". That would require an integration between systemd and pacman, presumably, with all the downsides that would have. And additional metadata maintenance.

@seth: I've never used kexec myself, so I'd be okay with an unconditional reboot upon kernel update.

Offline

#19 2018-04-01 04:13:11

amish
Member
Registered: 2014-05-10
Posts: 470

Re: Restart all updated services

jernst wrote:

@amish: This is an interesting idea. It doesn't solve the entire problem, but it could certainly improve things. Come to think of it, if packages (daemons?) could declare when they want to be restarted (aka additional metadata) that might also improve things. E.g. "I'am httpd.service, please restart me if packge ssl, ... have been updated". That would require an integration between systemd and pacman, presumably, with all the downsides that would have. And additional metadata maintenance.

In my example this can be done by simply adding this to /etc/conf.d/systemd-restart-hook :
SERVICEMAP[openssl]="httpd.service"

So whenever openssl package is upgraded it will restart httpd.service


Script can be made smart enough NOT to restart httpd.service twice when both apache and openssl packages are updated together.

Script can also first check if service actually exists. (i.e. it should not do anything if openssl is upgraded but httpd.service does not exist i.e. when apache is not installed)

Last edited by amish (2018-04-01 04:15:49)

Offline

#20 2018-04-01 07:04:41

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: Restart all updated services

You do fundamentally not understand the complexity of the problem.
There is no generic way to know whether a service *should* be restarted (what is "should"? "should" the service be restarted if the update just adds a feature? do you "want" to use that feature? do you have to explicitly invoke it?)
There is no realisitic way to know whether a service *can* safely be restarted

The first problem is to understand what a process actually does, that is what scripts and binaries are invoked - you'd have to trace that down what implies a runtime analysis (for the invocation of further binaries can depend on all sorts of conditions: usually switches but in theory even the time)
The second problem is to figure whether the update touched anything involved in the above - in a relevant way, which is impossible w/o fundamental understanding of what was actually changed (you could at best hash all files in updated packages before and after the update and argue that any change is relevant, even if it actually just fixed a typo in a debug string)
The third problem is to predict whether the new binaries will still interact correctly w/ the running system, which is impossible w/o fundamental understanding of what was actually changed and how that interacts w/ running components (you could at best  "hope")

Also see https://en.wikipedia.org/wiki/Turing_completeness

If you just want to eg. restart apache in good faith whenever the apache package was updated: that's as trivial as it gets. But automatically knowing "what should and can be restarted when" is out of the cards. This is why in a professional context there're testing and productive systems. You test things on the testing system and if that doesn't break the system, you replay them on the productive system.

Offline

#21 2018-04-01 12:48:42

rdeckard
Wiki Maintainer
Registered: 2015-01-28
Posts: 137

Re: Restart all updated services

jernst wrote:

When I pacman -Syu, some daemons and/or their .service files might have been updated. How do I find out which ones, so I can restart those?

The following AUR package, overdue, notifies you when a running daemon references a library that has been updated:
https://github.com/tylerjl/overdue

Offline

#22 2018-04-01 21:25:02

jernst
Member
From: Silicon Valley
Registered: 2014-03-04
Posts: 290
Website

Re: Restart all updated services

@seth: I fail to see how this has anything to do with Turing completeness. It sounds to me that arguing that package management can also not be solved, for similar reasons. But I got your point of view, which is that you think it is impossible. I don't follow your (several) lines of reasoning, but thank you for laying them out.

@rdeckard: Interesting! That would certainly help reducing the guess work.

@amish: You have such a script, or is it just brainstorming?

Offline

#23 2018-04-01 23:20:51

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: Restart all updated services

jernst example where restarting is not the right thing to do postgresql major upgrade these always require the database be dumped then the binary upgraded then the database be recreated.
The arch service file infact stores the major version used to create the database and will fails if the package version mismatches.
second example cups recently had an update that changed the unprivileged user it ran some processes as and required manual intervention to change file ownerships just restarting the service would cause it to fail.
In the more general case on restart you run the risk of a config for the old version no longer being compatible with the new version and causing the restart to fail instead of having the old service still available
until system restart or administrator intervention.

Last edited by loqs (2018-04-01 23:21:24)

Offline

#24 2018-04-02 00:37:35

jernst
Member
From: Silicon Valley
Registered: 2014-03-04
Posts: 290
Website

Re: Restart all updated services

@loqs: good examples! I would be perfectly fine if the automation failed under some circumstances. Major package upgrades certainly qualify.

Like going from Apache 2.2 to 2.4 where almost everybody's config files had to be changed because some key words had stopped to be accepted.

On the other hand, it wouldn't be wrong to add a few more features into .service files so some stuff that isn't automated today could be automated. But that's an unrelated issue.

Offline

#25 2018-04-02 00:43:56

amish
Member
Registered: 2014-05-10
Posts: 470

Re: Restart all updated services

jernst wrote:

@amish: You have such a script, or is it just brainstorming?

No I dont have but script wouldnt be big. May be 20-30 lines.

I am just giving the idea. I do not intend to write a script because every 3-7 days there is kernel update so you anyway you end up restarting whole system.

Also for me Arch is a rolling release i.e. always up-to-date. Now I interpret always up-to-date as "programs / daemons" running should also should be up-to-date.

I do not believe in program / libraries installed on system being of new version but one still running in system is of old version. (where files / libraries do not exist anymore but their fd are still hanging around)

Last edited by amish (2018-04-02 00:51:01)

Offline

Board footer

Powered by FluxBB