You are not logged in.

#1 2015-12-12 19:57:55

mr.MikyMaus
Member
From: disabled
Registered: 2006-03-31
Posts: 285

systemd's being "smart" - umounts removable drive just after mount

Hi,

for some reason, systemd unmounts a filesystem just after I mount it manually. The filesystem is on a removable drive that is in no way set up in the system (no fstab entry, never mounted via gvfs, or similar). It is a drive with few partitions with possibly broken filesystems (ntfs and fat) that is in need of a data recovery. Soft errors only, the drive is fine as far as dd/badblocks is concerned.

I was horrified just by finding out that systemd even "handles" these mounts even though I was under the impression that I was doing things manually and I was only doing things that I explicitly told the OS to do. Pretty important stuff in data forensics. But in the meantime, systemd was for God knows what reason doing it's broken magic right under my hands without telling me dip. This is why I left Windows, only to get it back in form of systemd trojan horse. Is there some kind of point I am missing in the KISS philosophy or systemd philosophy? I would really like to understand the idea behind this kind of behaviour.

Now, I mounted one the filesystems under /tmp/e/1 via "mount -o ro -t ntfs-3g /dev/sde1 /tmp/e/1". So far so good. But after a few minutes of rsyncin' I accidentally kicked the USB cable disconnecting the drive. After reconnecting the drive appeared as block device /dev/sdf instead of /dev/sde. Fine with me, but not-so-fine with systemd as I later found out. The hard way, of course. I mounted partitions sdf1 through sdf4 to already existing structure - /tmp/e/{1,2,3,4} respectively, pretty much the same way I did it the first time. But this time, the "1" directory was empty. No sdf1 in "/proc/mounts", no errors anywhere, dmesg or other. Only after third try I realised there might be something going on behind my back so I did query journald, just to get this:

Dec 12 20:16:46 gremlin ntfs-3g[19683]: Version 2015.3.14 external FUSE 29
Dec 12 20:16:46 gremlin ntfs-3g[19683]: Mounted /dev/sdf1 (Read-Write, label "", NTFS 3.1)
Dec 12 20:16:46 gremlin ntfs-3g[19683]: Cmdline options: ro
Dec 12 20:16:46 gremlin ntfs-3g[19683]: Mount options: ro,allow_other,nonempty,relatime,fsname=/dev/sdf1,blkdev,blksize=4096
Dec 12 20:16:46 gremlin ntfs-3g[19683]: Ownership and permissions disabled, configuration type 7
Dec 12 20:16:46 gremlin systemd[1]: tmp-e-1.mount: Unit is bound to inactive unit dev-sde1.device. Stopping, too.
Dec 12 20:16:46 gremlin systemd[1]: Unmounting /tmp/e/1...
Dec 12 20:16:46 gremlin ntfs-3g[19683]: Unmounting /dev/sdf1 ()
Dec 12 20:16:46 gremlin systemd[1]: Unmounted /tmp/e/1.
Dec 12 20:16:46 gremlin systemd[1]: tmp-e-1.mount: Unit entered failed state.

Why? I just... why? This isn't the first time systemd's black magic screwed up my mounts (topic #195387). Is there a way to make systemd behave? Or better, not to poke in stuff I want to do my way instead of Lennart's way?

-m.

Last edited by mr.MikyMaus (2015-12-12 23:05:42)


What happened to Arch's KISS? systemd sure is stupid but I must have missed the simple part ...

... and who is general Failure and why is he reading my harddisk?

Offline

#2 2015-12-12 20:08:19

Head_on_a_Stick
Member
From: London
Registered: 2014-02-20
Posts: 7,679
Website

Re: systemd's being "smart" - umounts removable drive just after mount

mr.MikyMaus wrote:

Is there a way to make systemd behave?

Mask the unit responsible for the device(s).

Last edited by Head_on_a_Stick (2015-12-12 20:08:47)

Offline

#3 2015-12-12 20:51:31

WorMzy
Forum Moderator
From: Scotland
Registered: 2010-06-16
Posts: 11,783
Website

Re: systemd's being "smart" - umounts removable drive just after mount

While I understand your frustration, your rant is unnecessary and adds nothing to your topic. Please remove it.


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#4 2015-12-12 23:07:21

mr.MikyMaus
Member
From: disabled
Registered: 2006-03-31
Posts: 285

Re: systemd's being "smart" - umounts removable drive just after mount

Head_on_a_Stick wrote:
mr.MikyMaus wrote:

Is there a way to make systemd behave?

Mask the unit responsible for the device(s).

There aren't any. The device is removable and this was the first (well second) time it was plugged in. Or is there some kind of wildcard masking option available, telling systemd to ignore "anything but"...? Can't find anything in the docs.


What happened to Arch's KISS? systemd sure is stupid but I must have missed the simple part ...

... and who is general Failure and why is he reading my harddisk?

Offline

#5 2015-12-12 23:14:28

Head_on_a_Stick
Member
From: London
Registered: 2014-02-20
Posts: 7,679
Website

Re: systemd's being "smart" - umounts removable drive just after mount

mr.MikyMaus wrote:

Or is there some kind of wildcard masking option available, telling systemd to ignore "anything but"...?

systemd creates units for each attached device and names them according to UUID

I can't check 'cos I'm in OpenBSD but you should connect the drive and look at the output of:

systemctl --all|grep dev-by-uuid # or similar

Offline

#6 2015-12-12 23:18:13

mr.MikyMaus
Member
From: disabled
Registered: 2006-03-31
Posts: 285

Re: systemd's being "smart" - umounts removable drive just after mount

Head_on_a_Stick wrote:
mr.MikyMaus wrote:

Or is there some kind of wildcard masking option available, telling systemd to ignore "anything but"...?

systemd creates units for each attached device and names them according to UUID

I can't check 'cos I'm in OpenBSD but you should connect the drive and look at the output of:

systemctl --all|grep dev-by-uuid # or similar

This I know but thanks for pointing it out, took me a while to figure it out and could be helpful for others.

However, this is not at all helpful for my particular problem as I would really like systemd not to do anything like this at all (i.e. not create any units without my prior authorisation), preferably without breaking the system. And this is a problem I have yet to figure out.

Also, on the topic, even if I accept the premise that it is ok for systemd to poke in stuff like this (which obviously I don't), there is still an objectively unwanted behaviour I have trouble understanding - the "auto-unmouting problem" described in the OP. Is this expected behaviour or should I fill a bug report to upstream and pray it's not designated as "feature"?

Last edited by mr.MikyMaus (2015-12-12 23:26:28)


What happened to Arch's KISS? systemd sure is stupid but I must have missed the simple part ...

... and who is general Failure and why is he reading my harddisk?

Offline

#7 2015-12-13 16:45:30

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,302

Re: systemd's being "smart" - umounts removable drive just after mount

Hello
I experienced something similar where I got a message like yours here:

Dec 12 20:16:46 gremlin systemd[1]: tmp-e-1.mount: Unit is bound to inactive unit dev-sde1.device. Stopping, too.
Dec 12 20:16:46 gremlin systemd[1]: Unmounting /tmp/e/1...
Dec 12 20:16:46 gremlin ntfs-3g[19683]: Unmounting /dev/sdf1 ()
Dec 12 20:16:46 gremlin systemd[1]: Unmounted /tmp/e/1.
Dec 12 20:16:46 gremlin systemd[1]: tmp-e-1.mount: Unit entered failed state.

When the drive was accidentally plugged off, the dev-sde1.device entered an inactive state and stayed bound to the tmp-e-1.mount generated by systemd.
And the tmp-e-1.mount blocked future mountings to the new dev-sdf1.device created when you plugged the disk back in.

I've seen this also here, and the inactive device unit cannot be removed, and the binding with the previous mount unit keeps existing, even after a logout/login.
This causes an immediate un-mounting by systemd after trying to mount again the file system.

I think there is a systemd malfunction there (maybe a bug), but it is difficult to identify clearly where the problem is located.
I will study this problem and if I find something useful, I'll post again here.

I think you could open a new bug in https://github.com/systemd/systemd/issues

Offline

#8 2015-12-14 16:12:56

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,302

Re: systemd's being "smart" - umounts removable drive just after mount

So systemd operates in background into mount actions on removable devices, creating device and mount units for the device and mount point;
and that can cause issues in some cases, even a blocking of subsequent mounts to a mount point which stays bound to an inactive device unit, without seemingly any way to unblock this state.
I searched a way to reduce systemd interference and tried a udev rule based on the systemd.device man page:

...
systemd will dynamically create device units for all kernel devices that are marked with the "systemd" udev tag
(by default all block and network devices, and a few others)
...

and the /usr/lib/udev/rules.d/99-systemd.rules file:

...
SUBSYSTEM=="block", TAG+="systemd"
...
# Ignore raid devices that are not yet assembled and started
SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", KERNEL=="md*", TEST!="md/array_state", ENV{SYSTEMD_READY}="0"
...

In the second rule shown above, appears the way for systemd to ignore a device through the SYSTEMD_READY variable.

Based on that, I tried to prevent systemd to automatically create a device unit for block sub system devices in /dev/sd[b-z]* used for removable devices.
First I created under root this /etc/udev/rules.d/990-systemd-nodevice.rules:

# Ignore USB removable block devices
SUBSYSTEM=="block", KERNEL=="sd[b-z]*", ENV{SYSTEMD_READY}="0"

then I ran these commands:

# udevadm control --reload
# systemctl daemon-reload

This seems to improve things for me: I can prevent a blocking of mount actions with inactive device units.

But I would need some confirmation from another use case, that this udev rule really improves things.
Could you try that please on your computer? or someone else interested...

Even if this doesn't entirely prevent systemd actions, if it at least prevents the immediate un-mountings from a blocked inactive device unit, it could be enough to improve things for a normal usage.

If I find something better, I will post again, or if someone wants to contribute...

Last edited by berbae (2015-12-14 16:15:41)

Offline

#9 2015-12-14 18:29:12

Archforum101
Member
Registered: 2015-12-11
Posts: 44

Re: systemd's being "smart" - umounts removable drive just after mount

Mostly just posting to say thanks for the interesting info about systemd HOAS  and berbae. OP, "I kicked out the usb cord", obviously that's systemd's fault huh ? Messing round, lotsa folks don't like systemd, I like the sucker well enough and concede that Lennart and team created a hell of a better init system than I'd ever be able to. Just keep doing what you're doing. Learning about systemd and ways to config it  to your liking.

There's always other init's to use. Wonder whatever happened to uselessd ? Then there's the systemd-shim, which am not sure but would imagine is available for Arch ? If you really dislike it enough and are willing to invest the time, go back to SysV or Openrc etc etc ?

It's cloudy/rainy here in the US midwest. DAMN U SYSTEMD, WHY ARE YOU DOING THIS TO ME !

More messing around, couldn't resist. smile

Last edited by Archforum101 (2015-12-14 18:50:16)

Offline

#10 2015-12-14 23:58:01

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 999

Re: systemd's being "smart" - umounts removable drive just after mount

@berbae:
Have you tried matching by ATTR{removable}=="1" and possibly dropping the match against sd[b-z]*? The latter is non-generic and needs to be adjusted based on the number of hard-drives in your machine... And yes, this is a systemd bug, but good luck reporting it -- IME the systemd people are not easy to deal with...

Archforum101 wrote:

Messing round, lotsa folks don't like systemd, I like the sucker well enough and concede that Lennart and team created a hell of a better init system than I'd ever be able to.

I agree with you on that, especially since you can't seem to make a grammatically complete sentence.

But in general, your comment is silly -- people shouldn't need to learn new things without a good reason, and there is none for systemd to create a bunch of defunct units. I have a whole list of similar systemd "features" and corresponding hacks necessary to have a working machine, but somehow I don't think that you are interested in it...

Last edited by Leonid.I (2015-12-14 23:59:51)


Arch Linux is more than just GNU/Linux -- it's an adventure
pkill -9 systemd

Offline

#11 2015-12-16 10:29:08

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,302

Re: systemd's being "smart" - umounts removable drive just after mount

Leonid.I wrote:

Have you tried matching by ATTR{removable}=="1"

Thanks for the suggestion, but this would work only for /dev/sdb type names
ATTRS{removable}=="1" would be necessary for /dev/sdb1 type names.
So two rules would be required.
It's not a big problem but I prefer the simplicity of sd[b-z]* adapted of course to the number of internal hard disks, which the user should know, and which doesn't change every day.

Offline

#12 2015-12-17 14:26:52

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,302

Re: systemd's being "smart" - umounts removable drive just after mount

After a closer examination of this issue, I'm sorry to say that my workaround given above is not effective.
So I removed the udev rule and I'm back to the default udev/systemd configuration.

The opening poster doesn't show any further visible interest in this matter, but as I am impacted by the same issue, I intend to continue posting here when I find new facts.

On my computer I got problems when a partition was uncleanly unmounted and also only with some USB memory sticks.

I remarked that the memory stick causing a problem shows this in journal (but may be unrelated):

déc. 17 14:24:51 arch64 kernel: sd 16:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA

but the main problem seems to come from (but it seems also not to be systematic...):

déc. 17 12:01:35 arch64 kernel: FAT-fs (sdb2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

Without these two messages I seem not to see any mount blocking with default configuration.

I found a way to unblock a mount point unit which stays bound to an inactive device unit:

# fsck /dev/sdb2
# systemctl daemon-reload
# systemctl reset-failed run-media-berbae-VIDEO2.mount

After that things work again, no more blocking.
So it's not clear where the problem comes from, and if there is a bug at all...

I continue studying this.

In fact the mount units created by systemd are only the content of /proc/self/mountinfo in a systemd form:

$ systemctl status run-media-berbae-VIDEO2.mount
● run-media-berbae-VIDEO2.mount - /run/media/berbae/VIDEO2
   Loaded: loaded (/proc/self/mountinfo)
   Active: active (mounted) since jeu. 2015-12-17 12:17:01 CET; 1min 18s ago
    Where: /run/media/berbae/VIDEO2
     What: /dev/sdb2

Last edited by berbae (2015-12-17 19:09:14)

Offline

#13 2015-12-20 15:23:27

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,302

Re: systemd's being "smart" - umounts removable drive just after mount

My feeling about this matter is that it comes mainly from systemd features which may not be desirable for mount actions which are not
configured in /etc/fstab (ie manual, through udisks or a program).
There are two features implied here:
1) systemd automatically creates mount units from /proc/self/mountinfo when a device is mounted at runtime.
'man systemd.mount':

Mount points created at runtime (independently of unit files or /etc/fstab) will be monitored by systemd and appear like any other mount unit in systemd.

This implies that systemd could interfere in mount/unmount actions after the mount unit is created.
2) when a unit enters a failed state, for whatever reason, systemd manages it following this feature:
'man systemctl' reset-failed action

When a unit fails in some way (i.e. process exiting with non-zero error code, terminating abnormally or timing out), it will automatically enter the "failed" state
and its exit code and status is recorded for introspection by the administrator until the service is restarted or reset...

This implies that the failed unit will stay in a failed state until an administrator action is done (systemctl restart or systemctl reset-failed).

In the case of mount units, a failed state can occur when an external device with a mounted file system (not in /etc/fstab) is
abnormally disconnected and systemd could not, for whatever reason, unmount the mount point
(because systemd feature in this case is to unmount a mount point when a device is abnormally disconnected).

After that when a new mount action is tried on the same mount point, systemd rejects that action and immediately unmounts the
file system just after the new mount action. Here is such a real case with udisks implied:

...
déc. 17 11:17:41 arch64 kernel: usb 2-9: USB disconnect, device number 2
déc. 17 11:17:41 arch64 udisksd[706]: Cleaning up mount point /run/media/berbae/VIDEO2 (device 8:18 no longer exist)
déc. 17 11:17:41 arch64 systemd[1]: Unmounting /run/media/berbae/VIDEO2...
déc. 17 11:17:41 arch64 umount[1114]: umount: /run/media/berbae/VIDEO2: not mounted
déc. 17 11:17:41 arch64 systemd[1]: run-media-berbae-VIDEO2.mount: Mount process exited, code=exited status=32
déc. 17 11:17:41 arch64 systemd[1]: Unmounted /run/media/berbae/VIDEO2.
déc. 17 11:17:41 arch64 systemd[1]: run-media-berbae-VIDEO2.mount: Unit entered failed state.
...
déc. 17 11:18:23 arch64 systemd[1]: run-media-berbae-VIDEO2.mount: Unit is bound to inactive unit dev-sdb2.device. Stopping, too.
déc. 17 11:18:23 arch64 systemd[1]: Unmounting /run/media/berbae/VIDEO2...
déc. 17 11:18:23 arch64 udisksd[706]: Mounted /dev/sdc2 at /run/media/berbae/VIDEO2 on behalf of uid 1000
déc. 17 11:18:23 arch64 systemd[1]: Unmounted /run/media/berbae/VIDEO2.
déc. 17 11:18:23 arch64 systemd[1]: run-media-berbae-VIDEO2.mount: Unit entered failed state.

In this case an abnormal disconnection occurs at 11:17:41 and udisks cleaned the mount point before systemd began to do the same;
so systemd failed its unmount action and the mount point unit entered a failed state;
At 11:18:23 udisks tried to remount the file system using the same mount point, and it succeeded but systemd immediately unmounted
it because the mount point unit was already in a failed state and systemd doesn't want that to stay untreated by an administrator.
So systemd will refuse any further mounting on the same mount point until for example:

# systemctl reset-failed run-media-berbae-VIDEO2.mount

is run to unblock the state.

All this stems from systemd features.
But in this case, it could be arguable to consider a failure to unmount an already unmounted mount point as an error which
justifies to place the mount point unit in a failed state and force an administrator action to unblock that state.

I doubt that systemd developpers will consider that as a bug and as an important matter to consider.
So I think we have to live with it...

Last edited by berbae (2015-12-20 21:59:41)

Offline

#14 2015-12-21 11:36:06

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,867

Re: systemd's being "smart" - umounts removable drive just after mount

An interesting read that sheds more light on systemd internal working.

Thanks for investigating, berbae.


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Online

#15 2016-01-16 22:42:56

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: systemd's being "smart" - umounts removable drive just after mount

Possibly related bugs:

https://github.com/systemd/systemd/issues/1741
https://github.com/systemd/systemd/issues/1656

I haven't found a bug for the udisks/2 project (which should be filed, probably)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#16 2016-07-15 13:53:20

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: systemd's being "smart" - umounts removable drive just after mount

It seems that disabling systemd-fstab-generator solves this for me, also for mounts not defined in /etc/fstab.

# mkdir /etc/systemd/system-generators
# cp -s /dev/null /etc/systemd/system-generators/systemd-fstab-generator

This implies fstab entries aren't mounted anymore, so you have to create a separate service. For example:

[Unit]
Description=Mount local filesystems
Documentation=man:mount(8) man:systemd.mount(5) man:bootup(7)
DefaultDependencies=No
After=local-fs-pre.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/mount -a -t nocifs,nosmbfs,nonfs

[Install]
WantedBy=local-fs.target

The mount units are still visible in "systemctl --type mount", but apparently unused. I'm not sure if this is due to the service above, or due to disabling systemd-fstab-generator. I'll post something to systemd-devel and report back.

Last edited by Alad (2016-07-15 14:34:52)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#17 2016-07-15 22:20:54

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: systemd's being "smart" - umounts removable drive just after mount

https://lists.freedesktop.org/archives/ … 37144.html

In other words, this issue should be fixed in systemd >220 for transient mounts (generated from mount e.a).


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

Board footer

Powered by FluxBB