How to improve resilience if there's a power outage?

james-h-arch · 2024-11-08 15:59:41

I've noticed when there's an unexpected power outage, it can be a bit of a coin flip on whether my Arch systems reboot cleanly.

They sometimes need some file system maintenance before they boot (ext4, fat16 for EFI).

I'd prefer the systems handle power outages more gracefully. In particular, they do whatever self-checking & self-repair might be required to allow reboot without manual intervention.

It's not something I know much about. Any pointers?

Thanks.

Last edited by james-h-arch (2024-11-08 17:49:37)

loqs · 2024-11-08 20:45:31

What currently happens? Are the file-system's being being fsck'd before mounting and failing the automatic fcsk and requiring manual intervention? Usually that happens when the repair can not be performed automatically and the maintainer has to make a decision about how to proceed.
Edit:
Have you investigated a UPS (Uninterruptible Power Supply)?

Last edited by loqs (2024-11-08 21:23:52)

james-h-arch · 2024-11-09 13:34:44

What currently happens? Are the file-system's being being fsck'd before mounting and failing the automatic fcsk and requiring manual intervention?

Unexpected power outage happens
Sometimes on reboot, system fails to load boot image from EFI partition
It's unclear exactly what caused the EFI partition to get confused
The workaround is to either boot from a USB stick or move the storage device to another machine. Then run Gparted > Check on the EFI boot & ext4 root partitions.
System then reboots normally & continues okay.

Have you investigated a UPS?

I'm aware of UPS and what they can do. I'm hoping to avoid needing these & instead use improved self-checking & self-repair system config.

loqs · 2024-11-09 13:49:37

james-h-arch wrote:

Unexpected power outage happens
Sometimes on reboot, system fails to load boot image from EFI partition

What fails to load the boot image from the ESP; the system's firmware, the bootloader or the kernel? You could only mount the ESP when performing kernel / bootloader updates which may help the occurrences of the issue.

james-h-arch · 2024-11-10 09:21:29

What fails to load the boot image from the ESP?

I'm not sure. How should I go about finding this out?

Thanks for the tip re. not mounting the ESP. It was being auto-mounted. I'll reconfigure so it isn't mounted anymore & see if this helps.

seth · 2024-11-10 10:12:05

How should I go about finding this out?

Start by describing the symptoms in detail, what do you see, what messages are printed etc.

Head_on_a_Stick · 2024-11-10 11:00:42

Use a copy-on-write filesystem to avoid power loss induced corruption. You might lose some data but it should never corrupt. I've battle tested btrfs literally hundreds of times and it always came back.

Btrfs is pretty much dead now though, and bcachefs is still pretty new but either will be more resiliant than ext4 for power loss.

loqs · 2024-11-10 14:40:11

Head_on_a_Stick wrote:

Btrfs is pretty much dead now though, and bcachefs is still pretty new but either will be more resiliant than ext4 for power loss.

Are you suggesting converting the ESP to btrfs or bcachefs?

Head_on_a_Stick · 2024-11-10 14:43:03

No, I was referring to the root filesystem. For the ESP I mount under /efi/ using systemd's automount to help minimise exposure.

james-h-arch · 2024-11-11 17:04:06

I've removed the ESP entry from /etc/fstab & rebooted. Will see what happens next.

cryptearth · 2024-11-11 17:33:30

james-h-arch wrote:

I've removed the ESP entry from /etc/fstab & rebooted. Will see what happens next.

I can tell you: next kernel update your system won't boot as without /boot mounted the new kernel image and initrd will get written to the /boot folder instead of the esp mounted there and at next reboot the bootloader will read the old kernel and initrd incompatible with the new kernel installed in the system landing you at a rescue shell
it's exactly the reason of so many topics about failure after update
if your esp mountpoint is /boot and hence it contains kernel and initrd - do yourself a favour and re-add it to your fstab or make sure it's properly picked up by systemd-boot auto-mount and spare us from yet another topic of "can't boot after update" cause by: user error

this whole topic is an XY problem:
your issue: unstable power
correct solution: use an ups
what you want to do (your own words): "try to avoid it"
how are we supposed to help if you're not willing to accept our answer?

Last edited by cryptearth (2024-11-11 17:38:56)

james-h-arch · 2024-11-11 17:50:01

Thanks, and I'm aware of the implications for when upgrading the kernel.

seth · 2024-11-11 20:30:39

You can add yourself a reminder: https://bbs.archlinux.org/viewtopic.php … 6#p1771996

Head_on_a_Stick · 2024-11-12 06:50:44

cryptearth wrote:

this whole topic is an XY problem:
your issue: unstable power
correct solution: use an ups

Not everybody is rich, d00d.

seth · 2024-11-12 07:21:36

You can get some < $100
If your power frequently drops out that's maybe a good investment because next to corrupted files during updates you'll lose any unsaved work and the pattern kills at least HDDs in no time.

kermit63 · 2024-11-12 07:48:36

seth wrote:

... and the pattern kills at least HDDs in no time.

Over the past decade, I had to replace 1 motherboard, 2 ram sticks, 3 power supplies, 2 HDDs and 1 SSD as a direct result of power outages/fluctuations. so yeah, a 100 USD investment is definitely worth it.

File system corruption is the least of your concern when it comes to power outages, but as seth mentioned, there is also a question of frequency. I live in a country where super typhoons happen regularly, so it is not uncommon to have 6-8 power outages annually, lasting for an hour to more than a day. With climate change, the frequency and severity of the storms have increased exponentially over the past few years.

cryptearth · 2024-11-12 18:01:07

Head_on_a_Stick wrote:

Not everybody is rich, d00d.

to keep a page long reply short:
there's a difference between being unable to afford and unwilling to spent - and to me this topic just screams the latter one

it doesn't has to be a brand new one three times the systems rating - it just has to be enough to be within spec on full load and grant enough time for a clean shutdown - and such system can be done even DIY
a 500w inverter is as cheap as 25 bucks + a car battery for 65 bucks - amazon even lists a 480w proper pc ups for just 60 bucks

Last edited by cryptearth (2024-11-12 18:07:56)

Head_on_a_Stick · 2024-11-13 06:44:06

And my point was that for some forum users "25 bucks" will represent a significant outlay and for those users switching to a more resilient filesystem is the best solution.

seth · 2024-11-13 07:26:21

kermit63 wrote:

File system corruption is the least of your concern when it comes to power outages

I think everyone here understands the concept of budgetary constraints and nobody can or wants to force the OP to overextend.

The point is that the current focus is to mitigate a mere nuisance and that frequent power outages tend to have more severe repercussions and to therefore maybe re-evaluate the ROI of an UPS in that scenario.
(And maybe to point out that these things can be somewhat affordable in case a brief research on UPS has confronted one w/ "industrial-grade" and their price tags…)

Edit: oh, and PSA:
If your backup strategy is "I sometimes copy important data onto a usb key" that's still *infinitely* better than having "concepts of a plan" of a solution for maybe later. Backup or otherwise.

Last edited by seth (2024-11-13 07:30:00)

cryptearth · 2024-11-13 07:30:21

well, I had it shortened to the above, but as you ask for it:
OP seems to be rich enough to own a computer with peripherals to use it, some kind of internet service and electricity to power all that - along with the usual living cost like food and some sort of rent for some kind of housing to live in
it's reasonable to doubt OP is really too poor to save up for an ups - otherwise the question comes up how they're able to afford all those other running costs

Awebb · 2024-11-13 08:05:19

There is no X-Y problem. It doesn't matter if OP doesn't want to buy one or just can't afford one. This thread was about improving resilience in case of a power outage and not about reducing the risk of a power outage. A UPS has been suggested and OP hasn't even replied to that. Maybe OP is already out there buying one. We don't know. However, in the meantime, having a more resilient setup is not a bad idea. An UPS is just one additional layer of defense.

james-h-arch · 2024-11-17 21:57:41

Thanks all. And yes, my topic was specifically on improving resilience in the event of power loss to the device, for whatever reason. The earlier answer from loqs covered this off nicely. Thanks again all.

Arch Linux

#1 2024-11-08 15:59:41

How to improve resilience if there's a power outage?

#2 2024-11-08 20:45:31

Re: How to improve resilience if there's a power outage?

#3 2024-11-09 13:34:44

Re: How to improve resilience if there's a power outage?

#4 2024-11-09 13:49:37

Re: How to improve resilience if there's a power outage?

#5 2024-11-10 09:21:29

Re: How to improve resilience if there's a power outage?

#6 2024-11-10 10:12:05

Re: How to improve resilience if there's a power outage?

#7 2024-11-10 11:00:42

Re: How to improve resilience if there's a power outage?

#8 2024-11-10 14:40:11

Re: How to improve resilience if there's a power outage?

#9 2024-11-10 14:43:03

Re: How to improve resilience if there's a power outage?

#10 2024-11-11 17:04:06

Re: How to improve resilience if there's a power outage?

#11 2024-11-11 17:33:30

Re: How to improve resilience if there's a power outage?

#12 2024-11-11 17:50:01

Re: How to improve resilience if there's a power outage?

#13 2024-11-11 20:30:39

Re: How to improve resilience if there's a power outage?

#14 2024-11-12 06:50:44

Re: How to improve resilience if there's a power outage?

#15 2024-11-12 07:21:36

Re: How to improve resilience if there's a power outage?

#16 2024-11-12 07:48:36

Re: How to improve resilience if there's a power outage?

#17 2024-11-12 18:01:07

Re: How to improve resilience if there's a power outage?

#18 2024-11-13 06:44:06

Re: How to improve resilience if there's a power outage?

#19 2024-11-13 07:26:21

Re: How to improve resilience if there's a power outage?

#20 2024-11-13 07:30:21

Re: How to improve resilience if there's a power outage?

#21 2024-11-13 08:05:19

Re: How to improve resilience if there's a power outage?

#22 2024-11-17 21:57:41

Re: How to improve resilience if there's a power outage?

Board footer