Upgrading Old Arch Image on VPS Hosting

docbrown5 · 2013-08-23 03:55:47

Greetings Arch Community. As I am sure you are all sick of new threads on upgrading Arch, I do believe my situation is relatively unique--based on some of the other threads I have read about the topic.

Situation

My host Gandi.net offers VPSs (virtual private servers) as a service. When one creates a VPS, one can choose from a limited list of GNU/Linux distributions. Of those, I chose "ArchLinux 64 bits" of course. It turns out, their static image of their Arch x86_64 distribution is at least older than 2013/06. As many of you are aware, the binaries moved to /usr/bin according to Allan in this news post. My situation turned out to be a little more complicated than simply following Allan's directions being that my hosting provider has customized their Arch images to work with their infrastructure and systems.

Environment

~# uname -r
3.2.50-xenU-8149-x86_64

~# pacman -Qqo /bin /sbin /usr/sbin | pacman -Qm -
error: cannot determine ownership of directory '/bin'
error: cannot determine ownership of directory '/sbin'
error: cannot determine ownership of directory '/usr/sbin'
dbus-core 1.6.8-1
gandi-hosting-agent r7010-1
gandi-hosting-vm r7152-1
initscripts 2012.10.1-1
mingetty 1.08-3

~# pacman -Qi pacman | grep Version
Version        : 4.0.3-3
~# pacman -Qi glibc | grep Version
Version        : 2.16.0-4
~# pacman -Qi filesystem | grep Version
Version        : 2012.10-1

I think that I have figured out how to take care of the hosting provider-specific packages--gandi-hosting-agent, gandi-hosting-vm, initscripts, mingetty, I hope--and I do believe most everything else (except for a few configuration files that the hosting provider's packages have meddled with) is vanilla. I do seem, however, to be caught in some kind of dependency hell.

Procedure

I am going to skip all of the stuff about moving my hosting provider's package members to /usr/bin. Also, a little bit of pacman conversation will be omitted for brevity.

Step 1: Update dbus to rid system of dbus-core

# Skipping the prompted offer to upgrade pacman first

~# pacman -S dbus
:: dbus and dbus-core are in conflict. Remove dbus-core? [y/N] y
userdel: user dbus is currently used by process 457
groupdel: cannot remove the primary group of user 'dbus'
error: command failed to execute correctly
(1/1) installing dbus
Optional dependencies for dbus
    libx11: dbus-launch support

# Success...I think

Please let me know if I was incorrect in doing this!

Step 2: Search for Stragglers

/bin# find /bin /sbin /usr/sbin -exec pacman -Qo -- {} + > /dev/null

Turns out all files in /bin, /sbin, and /usr/sbin belong to a package.

Step 3: Update System
Update system as per Allan's instructions.

~# pacman -Syu --ignore filesystem,bash
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
:: The following packages should be upgraded first :
    pacman
:: Do you want to cancel the current operation
:: and upgrade these packages now? [Y/n] n

Basically spams:

warning: cannot resolve "filesystem>=2013.01", a dependency of "glibc"
warning: ignoring package filesystem-2013.05-2

Right, so lets try to upgrade pacman then.

~# pacman -S pacman
resolving dependencies...
looking for inter-conflicts...

Targets (7): bash-4.2.045-4  filesystem-2013.05-2  glibc-2.18-2  libarchive-3.1.2-1
             linux-api-headers-3.10.6-1  lzo2-2.06-1  pacman-4.1.2-1

Total Download Size:    10.93 MiB
Total Installed Size:   49.64 MiB
Net Upgrade Size:       -2.69 MiB

Proceed with installation? [Y/n] Y

(7/7) checking package integrity                       [#############################] 100%
(7/7) loading package files                            [#############################] 100%
(7/7) checking for file conflicts                      [#############################] 100%
error: failed to commit transaction (conflicting files)
filesystem: /bin exists in filesystem
filesystem: /sbin exists in filesystem
filesystem: /usr/sbin exists in filesystem
Errors occurred, no packages were upgraded.

Okay...try to upgrade filesystem?

~# pacman -S filesystem
:: The following packages should be upgraded first :
    pacman
:: Do you want to cancel the current operation
:: and upgrade these packages now? [Y/n] n

resolving dependencies...
looking for inter-conflicts...

Targets (1): filesystem-2013.05-2

Total Installed Size:   0.01 MiB
Net Upgrade Size:       -0.09 MiB

# I'm pretty sure this type of math would cause a black hole or something...

Proceed with installation? [Y/n] Y

(1/1) checking package integrity                       [#############################] 100%
(1/1) loading package files                            [#############################] 100%
(1/1) checking for file conflicts                      [#############################] 100%
error: failed to commit transaction (conflicting files)
filesystem: /bin exists in filesystem
filesystem: /lib exists in filesystem
filesystem: /lib64 exists in filesystem
filesystem: /sbin exists in filesystem
filesystem: /usr/sbin exists in filesystem
Errors occurred, no packages were upgraded.

Maybe ignore filesystem errors...for upgrading filesystem...(just seems like a bad idea)

~# pacman -S filesystem --ignore filesystem
:: The following packages should be upgraded first :
    pacman
:: Do you want to cancel the current operation
:: and upgrade these packages now? [Y/n] n

:: filesystem is in IgnorePkg/IgnoreGroup. Install anyway? [Y/n] Y
resolving dependencies...
looking for inter-conflicts...

Targets (1): filesystem-2013.05-2

Total Installed Size:   0.01 MiB
Net Upgrade Size:       -0.09 MiB

Proceed with installation? [Y/n] Y
(1/1) checking package integrity                       [#############################] 100%
(1/1) loading package files                            [#############################] 100%
(1/1) checking for file conflicts                      [#############################] 100%
error: failed to commit transaction (conflicting files)
filesystem: /bin exists in filesystem
filesystem: /lib exists in filesystem
filesystem: /lib64 exists in filesystem
filesystem: /sbin exists in filesystem
filesystem: /usr/sbin exists in filesystem
Errors occurred, no packages were upgraded.

Sad day.

Conclusion
Well, it seems that we are where we started with Allan's news article with a few new filesystem errors. I do want to point out that I spent 3 solid days wrestling with this. I am literally wasting money with every hour my server is not operational; so I was hope someone might have a super clever solution that would cure my server woes. This is the first form that I am ever been a part of, and I have only started using Arch within the past 2 months (fear not, I did my time on the wiki); so please let me know if I am not doing something correctly or can do something better.

Aleksander · 2013-08-23 04:31:08

Did you try to do the full upgrade ignoring glibc along with pacman and filesystem? Don't upgrade glibc until you are ready to upgrade filesystem here.

docbrown5 · 2013-08-23 04:37:04

No packages were ignored during the attempt at a full system upgrade mentioned in step 3.

EDIT: Just tried ignoring different combinations of packages. Ignoring packages just starts a dependency cascade.

Last edited by docbrown5 (2013-08-23 04:44:17)

Aleksander · 2013-08-23 04:43:41

I see where you did

pacman -Syu --ignore filesystem,bash

Where it complains that it wants to update glibc as part of the system update but filesystem has been ignored.
The only way I was able to do that same upgrade was with

pacman -Syu --ignore glibc,filesystem,bash

and when it asks if you'd like to update pacman first tell it no, otherwise it will want to update filesystem and glibc

docbrown5 · 2013-08-23 04:47:03

Just tried that as well. Here's the output with a bunch of repeating errors cut out:

warning: ignoring package glibc-2.18-2
warning: cannot resolve "glibc>=2.17", a dependency of "gcc-libs"
:: The following packages cannot be upgraded due to unresolvable dependencies:
      automake  binutils  ca-certificates  cloog  coreutils  cryptsetup  curl  dbus-glib
      device-mapper  dirmngr  e2fsprogs  emacs-nox  fakeroot  gawk  gcc  gcc-libs  gettext
      glib2  gmp  gnupg  gpgme  grep  groff  gzip  iputils  isl  jfsutils  krb5  ldns
      less  libarchive  libldap  libpcap  libsasl  libssh2  libtool  libusbx  logrotate
      lvm2  man-db  mkinitcpio  mpfr  nmap  openssh  openssl  pacman  parted  pcre  perl
      pkg-config  pkgfile  python  python-dbus  python-lxml  python2  python2-dbus
      python2-lxml  reiserfsprogs  rsync  rsyslog  systemd  systemd-sysvcompat  texinfo
      util-linux  vim  vim-runtime  wget  xfsprogs

Scimmia · 2013-08-23 05:21:22

Not only is it before the /bin -> /usr/bin move, it's before the /lib symlink moved to the filesystem package (end of Jan 13), which is what is causing problems. brain0 gives you a *possible* solution at the end of this thread: https://bbs.archlinux.org/viewtopic.php?id=164491. Note that the ARM is no longer available, so that option is gone.

Aleksander · 2013-08-23 05:28:43

That really is an outdated system... I think almost a year old, not just earlier than 2013/06. Since you can't install from a newer image, possible solutions may be to either:

1) Add packages to the --ignore switch as they appear. I.E. gcc-libs would be next as it can't update until glibc does... this could be a long line of packages though; time consuming and lengthy process.

2) Look into pacman's other switches. --nodeps or -d will stop pacman from checking the version numbers on dependencies. That could have the potential to break the system, however, so I'd only try it if it was a last effort or you would be willing to start again from scratch should it fail.
It would have to be run with the ignore switch on filesystem first

pacman -Syud --ignore glibc,filesystem,bash

immediately followed by

pacman -Syu

to finish the update on filesystem and pacman.

There may be some other useful pacman switches though, I only use a few. If you can spare time to start again from scratch if it goes wrong, --nodeps (-Syud) could work and would be quicker if it does. Otherwise if that neither of those ideas work I don't know what else you could try. I've just never heard anything good about trying to update such old systems.

Aleksander · 2013-08-23 05:36:08

Scimmia wrote:

Not only is it before the /bin -> /usr/bin move, it's before the /lib symlink moved to the filesystem package (end of Jan 13), which is what is causing problems. brain0 gives you a *possible* solution at the end of this thread: https://bbs.archlinux.org/viewtopic.php?id=164491. Note that the ARM is no longer available, so that option is gone.

I knew I was forgetting about something... the /lib symlink gave me issues back in january and -d would almost certainly cause more problems here now that i think about it.

docbrown5 · 2013-08-23 13:09:19

@Scimmia and @Aleksander:

Aleksander's

pacman -Syud --ignore glibc,filesystem,bash

and brain0's

pacman -Su -d --ignore filesystem

suggestions don't seem too different to me. Either way, it leaves my system in such a state that I cannot even call ls or pacman. Because I cannot call any binaries I cannot fix initramfs. brain0's suggestion to call binaries using

~# /usr/lib/ld-2.18.so pacman -Su
pacman: error while loading shared libraries: pacman: cannot open shared object file

doesn't quite work either.

Last edited by docbrown5 (2013-08-23 19:59:46)

docbrown5 · 2013-08-23 15:42:39

Perhaps I am missing something important by not showing everyone the packages that I thought were my hosting provider's packages.

~# pacman -Qm
dbus-core 1.6.8-1
gandi-hosting-agent r7010-1
gandi-hosting-vm r7152-1
initscripts 2012.10.1-1
mingetty 1.08-3

I later found out that initscripts was replaced by sysemd. Although I am not sure how involved systemd is in upgrading the system, perhaps this could be a missing puzzle piece?

If there seriously isn't a way to upgrade Arch from a snapshot that is likely less than a year old--which I personally think is ridiculous if true--then perhaps there is another way... I can have my hosting provider spin up a second hard drive. Is there some way that I download a fresh ISO from the Arch website onto the second disk, chroot into it somehow, wipe original system disk, and literally fresh-install Arch?

Scimmia · 2013-08-23 17:14:05

You may find it ridiculous, but there is no official upgrade path. Arch is a fast paced system; if you don't keep up, you get left behind.

I wonder if you could do pacman -Su -d --ignore filesystem, then use a static version of ln to manually make the /lib symlink? There was a project to make a version of Arch that was entirely statically linked, the binary could probably be found there. After making the symlink, pacman should be functional again. Make sure that /bin and /sbin do not exist and that /usr/sbin is empty, then use --force to upgrade the filesystem package and the system *could* work again.

Edit: the glibc package already contains a statically linked version of ln: sln. Try this:

pacman -Syu -d --ignore filesystem
sln usr/lib /lib
ls -la /bin /sbin /usr/sbin
pacman -S --force filesystem

Making sure, like I said, that /bin and /sbin don't exist and that /usr/sbin is empty.

Last edited by Scimmia (2013-08-23 17:30:05)

docbrown5 · 2013-08-23 17:31:31

@Scimmia

I may be completely wrong, but I am pretty sure there is an official upgrade path: core. Also, as mentioned in the first post and title, I don't have complete control over the system; therefore, the fact that my hosting provider got "left behind" is completely out of my control. Insofar as your recommendation, it seems way too hacky to be appropriate, because I will be running this system as a server for real businesses.

---

I do find it ridiculous because Arch is a very popular and beautiful distribution of Linux. It should not be like M$ Windows where, after a gap, oops, you simply have to wipe the system and install the new version. Also, I thought that this kind of stuff what what pacman was "legendary" about: dependency resolution. If what you say is true about Arch, it has little chance of success as a professional server platform for multiple infrastructures, but especailly the cloud.

Last edited by docbrown5 (2013-08-23 18:50:32)

x33a · 2013-08-23 17:32:41

Scimmia wrote:

There was a project to make a version of Arch that was entirely statically linked, the binary could probably be found there.

I think it was named Starch.

Scimmia · 2013-08-23 17:34:18

LOL, and where has anyone involved ever stated that Arch wants to be "a professional server platform for multiple infrastructures"? They don't even purport themselves as a server OS, even discouraging it at times.

Insofar as your recommendation, it seems way too hacky to be appropriate, because I will be running this system as a server for real businesses.

I'm starting to think you don't understand what the problem even is. If you don't like my suggestion, don't take it, even though it's probably your best shot.

Last edited by Scimmia (2013-08-23 17:39:24)

x33a · 2013-08-23 17:35:36

docbrown5 wrote:

If what you say is true about Arch, it has little chance of success as a professional server platform for multiple infrastructures, but especailly the cloud.

Arch isn't really intended for the Servers. More so, production servers. Arch works well enough if you keep updating it regularly, but major changes such as encountered by you, can cause problem, especially if you try to tackle them months later.

In your case, you are not responsible, rather your hosting provider is. But, I always maintain, that Arch isn't the best server distro.

docbrown5 · 2013-08-23 17:57:12

@x33a
I don't completely agree, Arch isn't the most popular choice as a server distro (hopefully will be some day), but I have chosen to try and make it work.

@Scimmia:
I recommend that you read the About page for Arch Linux. Especially focus on the first sentence and the last paragraph. I'll quote it and bold the important parts for you convenience.

Arch Linux About page wrote:

Arch Linux is an independently developed, i686/x86-64 general purpose GNU/Linux distribution versatile enough to suit any role.

Arch Linux About page wrote:

To summarize: Arch Linux is a versatile, and simple distribution designed to fit the needs of the competent Linux® user. It is both powerful and easy to manage, making it an ideal distro for servers and workstations.

I hope this education on the mission of the Arch Linux community was helpful

---

Anyway, not trying to get too off-topic here. I think my latest query was relevant:

docbrown5 wrote:

Is there some way that I download a fresh ISO from the Arch website onto the second disk, chroot into it somehow, wipe original system disk, and literally fresh-install Arch?

Last edited by docbrown5 (2013-08-23 18:49:40)

Scimmia · 2013-08-23 17:59:47

Yeah, I'm done here.

WonderWoofy · 2013-08-23 19:12:39

@docbrown5, Scimmia was really trying to help you, and you just basically jerked your way out of any real support here. Yes, you can use Arch for a server, even a production server, but that means that you should probably have some pretty deep understanding of what is going on within your system. If you don't even know what it means to regenerate the initramfs, I honestly don't think that you have the knowledge (yet?) to even attempt to do such a thing. Besides, when you managed to break it, jusdging by your attitude in this thread, I can only assume that you would have deemed the breakage "ridiculous".

I think it is time for you to find a different distribution for your server. Might I suggest debian, it is an incredibly stable solution to what you are seeking.

docbrown5 · 2013-08-23 19:46:49

A thousand apologies to anyone who thought that I was being a jerk; that was certainly not my intent. I suppose that I read

Scimmia wrote:

You may find it ridiculous, but there is no official upgrade path. Arch is a fast paced system; if you don't keep up, you get left behind.

as misdirected chastisement (as I do not have full control over my system).

I was also slightly perturbed that he had not seemed to have read the rest of the thread. In post #9 I had tried the exact solution that Scimmia had proposed and stated that I could not use any of the binaries including ls and pacman (even after a few sln commands). The rest is history. And, no, I don't consider every breakage "ridiculous." I've never had a breakage to the point of needing to open a thread before.

I really appreciate you, WonderWoofy, for pointing out my perceived jerk-ness. This is exactly the situation that I was hoping to avoid--a situation that I have seen on so many other threads.

Jristz · 2013-08-23 21:19:10

~# pacman -Qm
dbus-core 1.6.8-1
gandi-hosting-agent r7010-1
gandi-hosting-vm r7152-1
initscripts 2012.10.1-1
mingetty 1.08-3

Try first rebuild gadi-hosting-{agent,vm} to use

--sbindir=/usr/bin --libdir=/usr/lib --libexecdir=/usr/lib/$pkgname

this move those binaries to /usr/bin and removing one problem

mingetty no longer in repos, so...or rebuild yourself to put binaries on /usr/bin or migratte to the standar getty provided by arch

initscript, so, this is complex whitout A.R.M.; maybe intall systemd whitout systemd-sysvcompat this going to replace udev so (but not sure if a good idea) ... another alternative is rebuild yourself initscripts to put binaries un /bin and add init=/usr/bin/init to the boot-append-line

dbus-core is provided by dbus

the only problems here is glibc that can have SERIOUS problem to others programs or worst case ever

Last edited by Jristz (2013-08-23 21:19:57)

docbrown5 · 2013-08-23 21:51:49

Thanks for the info, Jristz! Yeah, I solved the dbus-core thing. I also figured to moved my hosting provider's binary package members to /usr/bin and update package list, but never knew I could do it by rebuilding the packages. initscripts is going to take some work. I don't know if that is just simply outdated, or if it is customized by Gandi.net.

But, yeah, updating glibc, pacman, and filesystem is pretty confusing. I thought about moving all binaries into /usr/bin manually, maybe editing PATH to point to the binaries (so I can still use them), and then trying

pacman -S filesystem

progandy · 2013-08-23 22:10:17

docbrown5 wrote:

Thanks for the info, Jristz! Yeah, I solved the dbus-core thing. I also figured to moved my hosting provider's binary package members to /usr/bin and update package list, but never knew I could do it by rebuilding the packages. initscripts is going to take some work. I don't know if that is just simply outdated, or if it is customized by Gandi.net.
But, yeah, updating glibc, pacman, and filesystem is pretty confusing. I thought about moving all binaries into /usr/bin manually, maybe editing PATH to point to the binaries (so I can still use them), and then trying
pacman -S filesystem

I'm not sure if this still works with the current pacman, but I used the following dirty update path back then:
- use a busybox root shell
- move all binaries to /usr/bin (make sure nothing gets overwritten)
- create symlinks from bin, sbin, usr/sbin to usr/bin
- install filesystem (without force first, when only conflicts are those 3 symlinks, force it)
- continue upgrade normally.

Since you have the change from initscripts to systemd and you are using a remote connection, you'll have to make sure to enable all necessary services with the right configuration for ssh access before you reboot.

Last edited by progandy (2013-08-23 22:13:37)

dlandau · 2013-08-26 21:10:04

Hi all,

I had (solved!) the exact same problem as the original poster, that is, a Gandi VPS image with an old Arch. What follows is the steps I took to update from a clean image as given by Gandi.

Firstly, I don't really know yet what the two gandi-special packages do yet, so I backed them up just in case.

# cd ~
# for f in $(pacman -Ql gandi-hosting-agent | cut -d ' ' -f 2); do if [[ -f $f ]]; then mkdir -p root$(dirname $f); cp $f root$(dirname $f)/; fi; done 

# for f in $(pacman -Ql gandi-hosting-vm | cut -d ' ' -f 2); do if [[ -f $f ]]; then mkdir -p root$(dirname $f); cp $f root$(dirname $f)/; fi; done

Next, I removed the packages, since both of them had files in the wrong directories.

# pacman -R gandi-hosting-{vm,agent}

Update the package databases (but not the packages!)

# pacman -Sy

Install busybox as suggested by progandy

# pacman -S busybox
# chmod 4555 /usr/bin/busybox

Change working dir to file system root, while you can:

# cd /

Download all the packages to the cache, even the ones that you don't install straight away:

# pacman -Syuw

No for the leap into darkness, after this, practically none of the commands except for busybox will work. Answer no to updating pacman first and yes to replacing dbus-core with dbus.

# pacman -Syu --ignore bash,filesystem,glibc,gcc-libs,gcc,pacman,binutils,libtool

Install all the packages left except for filesystem by simply unpacking the archives.

# busybox tar xJf /var/cache/pacman/pkg/libtool-2.4.2-10-i686.pkg.tar.xz 
# busybox tar xJf /var/cache/pacman/pkg/binutils-2.23.2-3-i686.pkg.tar.xz 
# busybox tar xJf /var/cache/pacman/pkg/pacman-4.1.2-1-i686.pkg.tar.xz 
# busybox tar xJf /var/cache/pacman/pkg/gcc-4.8.1-3-i686.pkg.tar.xz 
# busybox tar xJf /var/cache/pacman/pkg/gcc-libs-4.8.1-3-i686.pkg.tar.xz 
# busybox tar xJf /var/cache/pacman/pkg/glibc-2.18-2-i686.pkg.tar.xz
# busybox tar xJf /var/cache/pacman/pkg/bash-4.2.045-4-i686.pkg.tar.xz

Remove the offending directories and unpack the filesystem package:

# busybox rm -r /bin/ /usr/sbin/ /sbin/
# busybox tar xJf /var/cache/pacman/pkg/filesystem-2013.05-2-i686.pkg.tar.xz

Now the system should already work somewhat. Let pacman retake ownership of the files of the system packages just unpacked with busybox:

# pacman -Su –force

Reinstall the files from the two Gandi packages:

# cd ~
# cp -r root/* /
# cp -r root/usr/sbin/* /usr/sbin/
# cp -r root/lib/* /lib/

Final cleanup, for some reason, I lost DNS lookup capabilities in all of this. To fix, I added Opennic projects nameservers to resolv.conf, any nameserver should probably do, or if you saved the original one, you could use that.

# echo nameserver 192.121.86.114 >>/etc/resolv.conf
# echo nameserver 192.121.121.14 >>/etc/resolv.conf

Final step, the process ate my user account, so I readded that, and changed the passwords of my normal user and that of root:

# useradd [username]
# passwd [username]
# passwd

And that's it! A fully updated Arch system. Simply got to love that the package format is really .tar.xz, which is workable even with busybox if the need really arises

WonderWoofy · 2013-08-26 23:35:49

dlandau wrote:

<SNIP>

This is quite interesting. I would be interested to hear how things go a few days down the road. Hopefully this will be a real solution for you and not bring about any unforeseen problems…

Good luck, and welcome to the Arch Linux forums!

Il Palazzo-sama · 2013-09-01 16:22:34

dlandau wrote:

Firstly, I don't really know yet what the two gandi-special packages do yet, so I backed them up just in case.

This could be an interesting read: English / Français.

Relevant excerpt for gandi-hosting-agent:

Gandi agent is used to setup the virtual machine according to customer information. In case of expert mode server, the setup of the local system is limited to setting the root password and creating the administrator user (as chosen by the customer) to avoid ssh-ing the server as root. […]
Once your expert server is setup after creation, you can remove gandi-hosting-agent packages. […]

Arch Linux

#1 2013-08-23 03:55:47