You are not logged in.

#1 2023-01-17 15:26:40

1ndeed
Member
Registered: 2018-09-28
Posts: 41

What's the best way to do multiple containers with systemd-nspawn?

Let's say I need 20 systemd-nspawn containers. Of which I choose arch linux as the container OS because I want to use pacman -Syu. A minimal installation is about 700MB. I can copy and paste for 19 times. Then set them up one by one.
That's clearly a problem. It's a waste of disk space, a waste of RAM for process and a waste of RAM for filesystem cache.

What if I use overlay mount?

Overlay is gooood. You know what, think bigger. Forget about 700MB minimal system. Let's install python, nginx, php, neovim... making it a 2GB golden base system image. Thus 20 different containers all use the same base system resource, the same python executable, etc...

But what if we upgrade the container? That's simple, we shutdown the 20 containers. Then fire up that base 2GB golden container, upgrade it. Thus the base system of 20 containers get upgraded.
But other parts still need upgrade. So I guess... manually upgrade all 20!!! of them one by one. We fire up the 20 containers. But wait, is it possible that after the first upgrade there is a conflict between the new base system and the old container overlays? Even if I only use the minimal 700MB one. There's still a chance that after upgrading the base system some conflict happens and boom end of the world.


So I guess you can use overlay to save resource. But you can't upgrade the lower layer. Which locks the containers to that one base system image. And if you upgrade the upper layer, over time different containers will replace everything in the lower layer to different locations on disk, which eventually would be just like you copy and paste one container 20 times. Resource wasted.

I also thought about zfs deduplication. Problem is that thing consumes too much ram to the point it's just unrealistic. And IOPS would suck, too. Also zfs deduplicate blocks on disk instead of ...what we want so of course it would be slow and unusable.

So what's the best way to do multiple containers with systemd-nspawn while saving resources, not being totally unusable and have the ability to upgrade elegantly?

Offline

#2 2023-01-17 16:15:46

twelveeighty
Member
From: Alberta, Canada
Registered: 2011-09-04
Posts: 1,096

Re: What's the best way to do multiple containers with systemd-nspawn?

Look into 'distroless containers', for example. There are drawbacks to using them, but in general, there's a trade-off between installation/building convenience and duplication between container images.

Offline

#3 2023-01-17 16:38:03

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: What's the best way to do multiple containers with systemd-nspawn?

You could use a date-based url from the arch linux archive mirror as your only mirror in pacman.conf.  Each time you fire up the base system to update, you'd set the most recent date available as the mirror, then update the base system.

Each of the 20 other containers, when booted up, would technically be in a partial upgrade state, so you'd need to complete their upgrade right away for any of the additional packages they had that were not on the base system.  This should be manageable as the base system would include everything needed for basic functionality - at the worst, the 20 overlays would have to be booted into single user mode to complete their update.

Then the overlay systems could continue installing additional packages as desired - they'd all just come from the archive mirror for the date of the last base system upgrade.

Essentially this would turn these into point-release systems, though you could advance that "release" at any time by updating the base system.

I suspect the only other safe option would be to not allow (system-wide) package installations on the 20 overlay containers but rather have then request a software addition to the base system which would be added at the next update.

Last edited by Trilby (2023-01-17 16:39:31)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#4 2023-01-17 17:10:20

1ndeed
Member
Registered: 2018-09-28
Posts: 41

Re: What's the best way to do multiple containers with systemd-nspawn?

twelveeighty wrote:

Look into 'distroless containers', for example. There are drawbacks to using them, but in general, there's a trade-off between installation/building convenience and duplication between container images.

Thanks for replying!

Given the complexity of the all the containers and the potential of missing new dependencies on the next rebuild. And the same resources, libraries they share. Even distroless containers have the benefit of small initial size, it's still not an option for me.

Offline

#5 2023-01-18 23:28:14

xerxes_
Member
Registered: 2018-04-29
Posts: 662

Re: What's the best way to do multiple containers with systemd-nspawn?

Maybe bind mount can be used for this 20 the same containers? And Alpine Linux is really small... or was when I recently check it.

Last edited by xerxes_ (2023-01-18 23:28:33)

Offline

Board footer

Powered by FluxBB