Fedora merging /usr/bin with /sbin and /bin

Allan · 2012-01-30 01:14:01

Jristz wrote:

I ask: ...devs-arch have pans to merge those directories too?

Some devs like the idea, some don't.... nothing has been discussed in detail. The transition is one that would be difficult (but not impossible) to do on a rolling release system.

karol · 2012-01-30 04:42:55

Allan kindly started a discussion about this issue on the ML: http://mailman.archlinux.org/pipermail/ … 22463.html

Last edited by karol (2012-01-30 04:43:25)

Allan · 2012-01-30 04:46:29

I specifically did not start a discussion about merging those folders...

karol · 2012-01-30 04:54:38

Allan wrote:

The transition is one that would be difficult (but not impossible) to do on a rolling release system.

I meant this ^^ issue (sorry for the ambiguity) ...

... but maybe your ML thread will answer this question anyway:

Jristz wrote:

I ask: ...devs-arch have pans to merge those directories too?

the.ridikulus.rat · 2012-01-30 06:34:32

Since it is virtually impossible to update all the packages having files in /{{s,}bin,lib} in a rolling release system, if this should be done in Arch, pacman should be instructed to copy all the files in /{{s,}bin,lib} to /usr/{{s,}bin,lib} even if the package simply contains /{{s,}bin,lib}/* . Maybe a flag/option can be added to pacman in this regard.

How about doing this from initramfs (like fedora dracut's usrmove module)? Is it better over using "pacman --root" option? Copy pacman into initramfs. Run pacman with the usrmove flag, in that case pacman moves /{{s,}bin,lib}/* to /usr/{{s,}bin,lib}/* and modifies the files in /var/lib/pacman/local accordingly with something like

/sbin/foo -> /usr/sbin/foo

How about this way?

Last edited by the.ridikulus.rat (2012-01-30 06:37:39)

the.ridikulus.rat · 2012-01-30 06:36:15

BTW what is the exact requirement for sbin dir, is it system (admin) binaries or static binaries. If its admin binaires, isn't that already satisfied by file (exec) permissions. If its static binaries, then the dir makes sense, but right now most of the binaries in sbin are not static execs.

Allan · 2012-01-30 06:44:38

the.ridikulus.rat wrote:

Since it is virtually impossible to update all the packages having files in /{{s,}bin,lib} in a rolling release system

Allan wrote:

difficult (but not impossible)

The main issue is things like shell scripts that use #!/bin/sh in them (in fact... pacman calls this directly too...). So the upgrade for bash that moves this file to /usr/bin/sh needs to occur at the same time as the update of the file system package that creates the symlinks.

So this can be done (and I have on a local chroot - but not on a real system), but it would take considerable planning. Looking at the Gentoo mailing list thread about this, they are thinking about the same problems.

the.ridikulus.rat · 2012-01-30 06:53:30

Why not use "#!/usr/bin/env sh" ?

I am not saying that this should be done as soon as possible. I am just asking whether the method I specified (involving pacman in initramfs) work? Even Fedora's move is 4-6 months away (i don't think its in rawhide yet).

Allan · 2012-01-30 09:11:20

It could work, but from what I understand would require user interaction beyond simply upgrading (as automatically adding something to run in the initramfs is not something we would do). But I would not do it via pacman. A shell script would be enough...

Leonid.I · 2012-01-30 20:12:46

So if shells are that important why not simply symlink /usr/bin/bash -> /bin/bash, then fix in-package scripts gradually as pks update, and in N months just move /bin/bash to /usr/bin/bash? This could be done inside respective packages... I think.

Last edited by Leonid.I (2012-01-30 20:14:29)

Allan · 2012-01-30 20:31:15

#!/bin/bash is used in scripts everywhere, not just in packages. That is not something we can even consider removing. So the upgrade of the filesystem package that would symlink /bin to /usr/bin needs to occur at the same time as the update that removes /bin/bash from the bash package.

Leonid.I · 2012-01-30 21:07:33

You are right, but I meant while "/usr/bin/bash" and "/bin/bash" exists, people (users, TUs and devs alike) can start changing scripts (which will now come with both shebangs). So the package to be changed is bash, not filesystem. In fact, the latter should be changed when /bin is almost empty, ideally containing only shells...

Jristz · 2012-01-30 22:52:44

the actual problem are the /bin/bash

but the others programs?
why not simply begin for the others programs and make the final step the transition to /bin/sh to /usr/bin/sh (or th /sur/bin/env sh)
this por one side delag this problem and make experience and news ideas for how implement this
or move the unescential for boot to /usr/bin and befor this begin w pacman filesystem and tose tipes of thinks

note: I read the reason for exist fo /usr/{{s,}bin,lib,lib64} and for me is a entire joke (specialy for this years)

More information from Fedora Wikia bout how manage this
https://fedoraproject.org/wiki/Features/UsrMove

EDIT: ER OpenSuSe .....want this to?
http://lists.opensuse.org/opensuse-fact … 01431.html
....
and in the Debian Mailisting speak bout the same proposal (and eventualy Ubuntu)
http://thread.gmane.org/gmane.linux.deb … ral/165785

Last edited by Jristz (2012-01-31 03:35:53)

Cows · 2012-01-31 07:18:31

Why not just make a copy of bash to the new location and have two copies of bash on the system until the transition is complete?

Bash is very small and I don't think the amount of space it takes will be an issue.

the.ridikulus.rat · 2012-01-31 08:17:11

Allan wrote:

It could work, but from what I understand would require user interaction beyond simply upgrading (as automatically adding something to run in the initramfs is not something we would do). But I would not do it via pacman. A shell script would be enough...

Is pacman ok with /bin, /sbin /lib being symlinks at present (not talking about future)? In that case isn't it just a matter of moving all the files from these dirs to the respective /usr/* dirs and creating the symlinks. This can be done manually by the user without issues. But automating this task without user intervention is tricky.

As mentioned in ML thread, any new software or update PKGBUILD(s) should configure with --bindir=/usr/bin --sbindir=/usr/sbin --libdir=/usr/lib .

Allan · 2012-01-31 10:53:29

the.ridikulus.rat wrote:

Is pacman ok with /bin, /sbin /lib being symlinks at present (not talking about future)? In that case isn't it just a matter of moving all the files from these dirs to the respective /usr/* dirs and creating the symlinks. This can be done manually by the user without issues. But automating this task without user intervention is tricky.

Be careful. Somethings in /bin or /lib have symlinks in /usr/bin or /usr/lib so you could either end up with a library or a symlink pointing at itself...

the.ridikulus.rat wrote:

As mentioned in ML thread, any new software or update PKGBUILD(s) should configure with --bindir=/usr/bin --sbindir=/usr/sbin --libdir=/usr/lib

So... --prefix=/usr

Anthony Bentley · 2012-01-31 16:20:49

Here’s an interesting justification/take on the split.

You know how Ken Thompson and Dennis Ritchie created Unix on a PDP-7 in 1969? Well around 1971 they upgraded to a PDP-11 with a pair of RK05 disk packs (1.5 megabytes each) for storage.
When the operating system grew too big to fit on the first RK05 disk pack (their root filesystem) they let it leak into the second one, which is where all the user home directories lived (which is why the mount was called /usr). They replicated all the OS directories under there (/bin, /sbin, /lib, /tmp...) and wrote files to those new directories because their original disk was out of space. When they got a third disk, they mounted it on /home and relocated all the user directories to there so the OS could consume all the space on both disks and grow to THREE WHOLE MEGABYTES (ooooh!).
Of course they made rules about "when the system first boots, it has to come up enough to be able to mount the second disk on /usr, so don't put things like the mount command /usr/bin or we'll have a chicken and egg problem bringing the system up." Fairly straightforward. Also fairly specific to v6 unix of 35 years ago.
The /bin vs /usr/bin split (and all the others) is an artifact of this, a 1970's implementation detail that got carried forward for decades by bureaucrats who never question why they're doing things. It stopped making any sense before Linux was ever invented, for multiple reasons:
Early system bringup is the provice of initrd and initramfs, which deals with the "this file is needed before that file" issues. We've already got a temporary system that boots the main system.
shared libraries (introduced by the Berkeley guys) prevent you from independently upgrading the /lib and /usr/bin parts. They two partitions have to match or they won't work. This wasn't the case in 1974, back then they had a certain level of independence because everything was statically linked.
Cheap retail hard drives passed the 100 megabyte mark around 1990, and partition resizing software showed up somewhere around there (partition magic 3.0 shipped in 1997).
Of course once the split existed, some people made other rules to justify it. Root was for the OS stuff you got from upstream and /usr was for your site-local files. Then / was for the stuff you got from AT&T and /usr was for the stuff that your distro like IBM AIX or Dec Ultrix or SGI Irix added to it, and /usr/local was for your specific installation's files. Then somebody decided /usr/local wasn't a good place to install new packages, so let's add /opt! I'm still waiting for /opt/local to show up...
Of course given 30 years to fester, this split made some interesting distro-specific rules show up and go away again, such as "/tmp is cleared between reboots but /usr/tmp isn't". (Of course on Ubuntu /usr/tmp doesn't exist and on Gentoo /usr/tmp is a symlink to /var/tmp which now has the "not cleared between reboots" rule. Yes all this predated tmpfs. It has to do with read-only root filesystems, /usr is always going to be read only in that case and /var is where your writable space is, / is _mostly_ read only except for bits of /etc which they tried to move to /var but really symlinking /etc to /var/etc happens more often than not...)
Standards bureaucracies like the Linux Foundation (which consumed the Free Standards Group in its' ever-growing accretion disk years ago) happily document and add to this sort of complexity without ever trying to understand why it was there in the first place. 'Ken and Dennis leaked their OS into the
equivalent of home because an RK05 disk pack on the PDP-11 was too small" goes whoosh over their heads.
I'm pretty sure the busybox install just puts binaries wherever other versions of those binaries have historically gone. There's no actual REASON for any of it anymore. Personally, I symlink /bin /sbin and /lib to their /usr equivalents on systems I put together. Embedded guys try to understand and
simplify...

Cows · 2012-02-02 02:27:01

Interesting stuff Anthony. I kept reading the mailing list and I learned a lot of info.

After analyzing the file system structure at the FHS website, and looking at Arch's currently layout, I came up with an improve layout (At least I believe it is easier, simpler, smaller, and better than what we currently have). The layout is for next generation computers, or current machines that can afford to make the switch (but don't have to/ convenient to though) GPT.

I emphasize GPT because you can then be unrestricted from the 4 prim or 3 prim/ 1 extended partition that MBR has. This gives you the benefit of maintaining the "read-only" /usr dir, but still put all relevant files in that dir to clean up things.

Using an initramfs is pretty much a standard nowadays for more advanced setups. But an initramfs is definitely something good since we can simplify a lot of the infrastructure.

This is the current layout:

/
/boot
/etc
/kernel
/kernel/proc
/kernel/sys
/mnt - All mountable drives (including /media stuff. So /mnt and /media are now merged)
/dev
/srv
/usr
/usr/bin
/usr/sbin
/usr/libs
/usr/libs/32 -- Any architecture libraries go inside a subdirectory of libs named after it's arch.
/usr/libs/64 -- ~
/usr/var
/usr/var/run
/usr/opt
/usr/home
/usr/home/root
/usr/include (or /usr/inc)
/usr/share (or /usr/shr)

That's the base layout, any dirs not mentioned are either considered to be the default dirs that are normally there, or will be implemented in a future "pondering" session (aka one of my "sit down and think" moments).

Last edited by Cows (2012-02-02 02:32:01)

nsb · 2012-02-02 06:47:58

I think, what was is intended by the guys of Fedora, is, that all your software (binaries, libraries) are in /usr. So that you only need to share one partition in a network, so that all computers have the same software.

Then /home is were your user data are, so this should probably not be mingled with /usr. And /var, /var/run are specific to one computer/host. So that should not be under /usr either.

taylorchu · 2012-02-02 16:32:13

My idea of layout based on cow's post

/
/boot
/etc
/kernel
  /kernel/proc
  /kernel/sys
/mnt - All mountable drives (including /media stuff. So /mnt and /media are now merged)
/dev
/srv
/home
  /home/root
  /home/{user}
/usr
  /usr/bin -- merge bin and sbin
  /usr/libs
    /usr/libs/32 -- Any architecture libraries go inside a subdirectory of libs named after it's arch.
    /usr/libs/64 -- ~
  /usr/var
    /usr/var/run
  /usr/opt
  /usr/include (or /usr/inc)
  /usr/share (or /usr/shr)

Stebalien · 2012-02-02 17:34:50

Moving /proc and /sys will break just about everything. Putting /root in /home would make a separate home partition impossible.

Cows · 2012-02-03 07:16:23

@nsb, stebalien

Putting the /home inside of /usr won't break anything. Just mount things sequentially. /usr is mounted, and then /usr/home is mounted as a separate partition inside of /usr. In btrfs, this could be a subvolume, in lvm it could be a separate logical volume. GPT solves the partition limitations.

Therefore since /usr and /usr/home are separate partitions, /usr can still keep it's "read-only" status.

As for /proc, /sys, I need more information on how they are used and prepared, but it should be possible to move those two partitions into /kernel. Doesn't seem like something hard to do.

Jristz · 2012-02-04 04:40:05

and for /home/root and /home/user
is factible maintain using a separate home...but become less kiss-eable(?)
the only requeriment is create the user and before this mount the filesystem (maintain a empty /home/user) in teh respective /home/user

in my opinion

/
/boot
/etc
/sys
/proc
/mnt --> /media #For compatibility...but if noone is break and all are rebuildable againt /mnt withing break or a Dificult tansisitio can ignore the symlinking
/dev
/srv
/home
  /home/root
  /home/users/{users} #for managin a separate /home...or a /home/users
/usr
  /usr/bin --> /bin
  /usr/sbin --> /sbin
  /usr/libs32 --> /lib 
  /usr/libs64  --> /lib64
  /usr/var
    /usr/var/run
  /usr
  /usr/include
  /usr/share
/opt

is most probable that FHS (the standarisation of fs-hierarchi...) merge the /{{s}bin,lib,lib64} into /usr

Last edited by Jristz (2012-02-04 06:19:27)

darkfeline · 2012-02-14 20:57:10

Maybe I'm overlooking something, but isn't it as simple as copying everything from /bin to /usr/bin and then symlinking /bin? (I'm tempted to try it right now after backing up everything). Ideally, everything should be fixed to use /usr/bin instead of /bin, but because /bin is symlinked, anything that depends on the old directory structure still works.

Allan · 2012-02-14 20:58:41

Won't work... we already have files in /bin with symlinks in /usr/bin. If pacman extracts the /usr/bin symlink last, you end up with a symlink pointing to itself...

Arch Linux

#26 2012-01-30 01:14:01

Re: Fedora merging /usr/bin with /sbin and /bin

#27 2012-01-30 04:42:55

Re: Fedora merging /usr/bin with /sbin and /bin

#28 2012-01-30 04:46:29

Re: Fedora merging /usr/bin with /sbin and /bin

#29 2012-01-30 04:54:38

Re: Fedora merging /usr/bin with /sbin and /bin

#30 2012-01-30 06:34:32

Re: Fedora merging /usr/bin with /sbin and /bin

#31 2012-01-30 06:36:15

Re: Fedora merging /usr/bin with /sbin and /bin

#32 2012-01-30 06:44:38

Re: Fedora merging /usr/bin with /sbin and /bin

#33 2012-01-30 06:53:30

Re: Fedora merging /usr/bin with /sbin and /bin

#34 2012-01-30 09:11:20

Re: Fedora merging /usr/bin with /sbin and /bin

#35 2012-01-30 20:12:46

Re: Fedora merging /usr/bin with /sbin and /bin

#36 2012-01-30 20:31:15

Re: Fedora merging /usr/bin with /sbin and /bin

#37 2012-01-30 21:07:33

Re: Fedora merging /usr/bin with /sbin and /bin

#38 2012-01-30 22:52:44

Re: Fedora merging /usr/bin with /sbin and /bin

#39 2012-01-31 07:18:31

Re: Fedora merging /usr/bin with /sbin and /bin

#40 2012-01-31 08:17:11

Re: Fedora merging /usr/bin with /sbin and /bin

#41 2012-01-31 10:53:29

Re: Fedora merging /usr/bin with /sbin and /bin

#42 2012-01-31 16:20:49

Re: Fedora merging /usr/bin with /sbin and /bin

#43 2012-02-02 02:27:01

Re: Fedora merging /usr/bin with /sbin and /bin

#44 2012-02-02 06:47:58

Re: Fedora merging /usr/bin with /sbin and /bin

#45 2012-02-02 16:32:13

Re: Fedora merging /usr/bin with /sbin and /bin

#46 2012-02-02 17:34:50

Re: Fedora merging /usr/bin with /sbin and /bin

#47 2012-02-03 07:16:23

Re: Fedora merging /usr/bin with /sbin and /bin

#48 2012-02-04 04:40:05

Re: Fedora merging /usr/bin with /sbin and /bin

#49 2012-02-14 20:57:10

Re: Fedora merging /usr/bin with /sbin and /bin

#50 2012-02-14 20:58:41

Re: Fedora merging /usr/bin with /sbin and /bin

Board footer