systemd-journald: Monotonic clock jumped backwards

OpusOne · 2023-07-03 02:26:11

Hello,

I get the following entry in dmesg once at each boot:

systemd-journald[533]: /var/log/journal/8f67d1a6d6f44f84a3eea6e784985c42/user-1000.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.

What could be the cause of this? I understand that it's normally not something to worry about in general, but the fact it happens at each boot looks odd.

ewaller · 2023-07-03 02:34:09

My first guess is that your hardware real time clock has failed or is grossly wrong and is not being synchronized to Linux time.

What is the output of timedatectl ?

OpusOne · 2023-07-03 02:50:20

Thanks!

$ timedatectl
               Local time: lun. 2023-07-03 04:44:11 CEST
           Universal time: lun. 2023-07-03 02:44:11 UTC
                 RTC time: lun. 2023-07-03 02:44:11
                Time zone: Europe/Paris (CEST, +0200)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no

seth · 2023-07-03 06:41:21

Do you dual-boot windows?

ewaller · 2023-07-03 15:05:06

I looked at your timedatectl output and it isn't terrible. I don't know how long it took you to post, but your clock could be off as much as 6 minutes. My only comment, unrelated to your problem, is that you may want to use timedatectl to enable network time synchronization ( timedatectl set-ntp 1 )

Edit: I guess I would look at the end of the journal for the previous boot with journalctl -b-1 and compare the time stamps with those at the beginning of the current boot with journalctl -b. Pay particular attention to the time stamps around the line about monotonic time warning you found in dmesg when you find it in the journal.

Last edited by ewaller (2023-07-03 15:10:45)

OpusOne · 2023-07-03 19:54:27

- Dual boot: I have a Windows partition on a separate SSD, but haven't booted to it in several weeks. So it's not related to booting back and forth from Windows to Linux. Besides, I've enabled UTC time both for Windows and Linux to avoid time issues between both.
- I had ntpd running, but not systemd-timesyncd (which is enabled when you issue a ' timedatectl set-ntp 1'). ntpd on my machine takes a long time to synchronize (several minutes after boot), so the system clock remains unsynchronized for a few minutes.
- I understand that timedatectl doesn't detect ntpd so it states that the NTP service is inactive.
- I did compare timestamps from the previous boot with the current boot, and didn't notice anything abnormal. Weird.

The only thing is that systemd-timesyncd was not enabled, only ntpd.

What I did: I disabled ntpd, enabled systemd-timesyncd via ' timedatectl set-ntp 1'. The system clock gets synchronized much, much faster now than with ntpd. I haven't rebooted yet to see if that solves the monotonic clock jumping backwards issue at boot, I'll report back when I do.

seth · 2023-07-03 19:57:44

The system clock gets synchronized much, much faster now than with ntpd.

cat /etc/ntp.conf

https://wiki.archlinux.org/title/Networ … figuration

OpusOne · 2023-07-03 20:23:35

my /etc/ntp.conf file was like so:

# Please consider joining the pool:
#
#     http://www.pool.ntp.org/join.html
#
# For additional information see:
# - https://wiki.archlinux.org/index.php/Network_Time_Protocol_daemon
# - http://support.ntp.org/bin/view/Support/GettingStarted
# - the ntp.conf man page

# Associate to Arch's NTP pool
#server 0.arch.pool.ntp.org
#server 1.arch.pool.ntp.org
#server 2.arch.pool.ntp.org
#server 3.arch.pool.ntp.org

server 0.fr.pool.ntp.org iburst
server 1.fr.pool.ntp.org iburst
server 2.fr.pool.ntp.org iburst
server 3.fr.pool.ntp.org iburst

# By default, the server allows:
# - all queries from the local host
# - only time queries from remote hosts, protected by rate limiting and kod
restrict default kod limited nomodify nopeer noquery notrap
restrict 127.0.0.1
restrict ::1

# Location of drift file
driftfile /var/lib/ntp/ntp.drift

Last edited by OpusOne (2023-07-03 20:23:58)

seth · 2023-07-03 20:26:45

I guess you didn't configure timesyncd to use those ntp servers? (not sure whether it even supports iburst)

OpusOne · 2023-07-03 21:03:50

$ cat /etc/systemd/timesyncd.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the timesyncd.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# See timesyncd.conf(5) for details.

[Time]
NTP=0.fr.pool.ntp.org 1.fr.pool.ntp.org 2.fr.pool.ntp.org 3.fr.pool.ntp.org
FallbackNTP=0.arch.pool.ntp.org 1.arch.pool.ntp.org 2.arch.pool.ntp.org 3.arch.pool.ntp.org
#RootDistanceMaxSec=5
#PollIntervalMinSec=32
#PollIntervalMaxSec=2048
#ConnectionRetrySec=30
#SaveIntervalSec=60

Same NTP servers. Not sure timesyncd supports iburst either, at least one can't specify it in the conf file.
And it does use the first one in the NTP list, not any fallback one.

$  timedatectl timesync-status
       Server: 162.159.200.1 (0.fr.pool.ntp.org)
Poll interval: 34min 8s (min: 32s; max 34min 8s)
         Leap: normal
      Version: 4
      Stratum: 3
    Reference: A150804
    Precision: 1us (-25)
Root distance: 6.431ms (max: 5s)
       Offset: -895us
        Delay: 45.989ms
       Jitter: 989us
 Packet count: 17
    Frequency: -13,392ppm

OpusOne · 2023-07-04 02:56:30

OK, I actually have 3 machines with Arch on them including this one. The two others are completely different hardware and were installed a few years ago (and kept up to date).
I just checked, and turns out I have the same problem on the 3 machines now. I don't think I had noticed this in the 2 others before, must have appeared relatively recently.

Completely puzzled. Has something changed recently in systemd, or have I misconfigured all 3 machines and never noticed it until now?
What could be the misconfiguration that could explain it?
A hardware issue on all 3 machines at the same time looks pretty unlikely.

I noticed someone pointing out a similar issue yesterday on the Gentoo forum. No idea if it's related yet.

seth · 2023-07-04 06:17:26

Is this only (always) in the users/session journal?
What's the context? Any pattern?
Compare the system time to a wall clock when booting w/o any network access (or in the BIOS/UEFI)

frostschutz · 2023-07-04 07:46:09

I have the same issue. There are no weird timestamps in the logs, everything looks correct. The error message does not say how big a jump it supposedly detected. Also tried changing timedatectl settings (synchronize/ntp yes/no, local-rtc yes/no), it doesn't seem to change anything. There are no other ntp related services active. Although it's a dualboot system with Windows 11 on the side, the message always appears for Linux to Linux reboots, so it should not be Windows related.

I first noticed this issue after upgrading from Intel Haswell to AM4 / Ryzen platform. Updated BIOS for the heck of it, no change either. If the hardware clock is misbehaving I haven't noticed anything like that. The BIOS battery is definitely OK (the system was completely off the power grid for a week when I went on a trip recently, and no settings were lost).

There don't seem to be any ill effects, other than a bit too many logfile rotations. All in all I'm too lazy to debug it. I just put it down as one more mystery of systemd...

Lone_Wolf · 2023-07-04 11:08:26

I did notice the same issue but never looked into it until now.

I don't dual boot, boot to multi-user.target and use dhcpcd & ntp .
The first occurence of the monotonic clock jump message is

Apr 16 10:16:57 silverbolt systemd-journald[411]: /var/log/journal/23f343c183994ed18e475975f81b7ccb/system.journal: Monotonic clock jumped backwards
 relative to last journal entry, rotating

On april 15 I did a lot of aur building and ran pacman -Syu multiple times (to update from my local repo after each build) so it's hard for me to check what updates from core/extra/community/multilib took place that day.

I have checked wall clock with computer clock a few times after noticing in and not noticed any difference so if there is one it is less than a second.

Maybe other posters can verify whether that start date is correct and what was updated just before ?

The first entry I can find of it is from april 16 It looks like it started around april 15

Last edited by Lone_Wolf (2023-07-04 11:09:06)

ewaller · 2023-07-04 14:52:29

Well, I'll be....

I have it in my system as well. And it does not seem to be related to system start up. And they seem to come in bursts.

Jun 14 18:12:11 odin systemd-journald[239]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/user-1000.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 14 18:13:58 odin systemd-journald[240]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 14 18:14:10 odin systemd-journald[240]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/user-1000.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 14 18:31:04 odin systemd-journald[232]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 14 18:31:12 odin systemd-journald[232]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/user-1000.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 15 18:46:23 odin systemd-journald[233]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 15 18:47:02 odin systemd-journald[233]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/user-1000.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 16 10:32:28 odin systemd-journald[237]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
...

And, they seem to correlate with rotations in the systemd journal

-rw-r-----+ 1 root systemd-journal 4.0M Jun 14 18:13 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-0000000000294da1-0005fe20c2090fb2.journal'
-rw-r-----+ 1 root systemd-journal 4.3M Jun 14 18:14 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-0000000000295197-0005fe20c865c28f.journal'
-rw-r-----+ 1 root systemd-journal 4.0M Jun 14 18:31 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-000000000029568d-0005fe20c929d0ce.journal'
-rw-r-----+ 1 root systemd-journal 4.4M Jun 14 18:31 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-0000000000295990-0005fe21058da80f.journal'
-rw-r-----+ 1 root systemd-journal 4.5M Jun 15 18:46 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-0000000000295f35-0005fe21060ef841.journal'
-rw-r-----+ 1 root systemd-journal 4.4M Jun 15 18:47 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-0000000000296af7-0005fe355a2bda9b.journal'
-rw-r-----+ 1 root systemd-journal 4.7M Jun 16 10:32 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43 .....

I seem to have dad four in a row on June 14, then a delay of about a day until I had two more the next day, then nothing again until the 16th.
Between Jun 14 and yesterday (7/3) I have had 55 reports of the monotonic clock not being monotonic and a log rotation. None so far today.

I was going to take a stab at turning off NTP, but then I remembered that the OP had it turned off when they experienced this.
Looks like an annoying bug without serious consequences. Next question. Is it the kernel or systemd?

I updated my kernel to 6.4.1 yesterday, but journalctl -b still reports two hits with this new kernel.

Last edited by ewaller (2023-07-04 14:53:06)

ewaller · 2023-07-04 15:02:01

It would seem our friends over at Gentoo are seeing this as well...
https://forums.gentoo.org/viewtopic-p-8 … 825410d506

OpusOne · 2023-07-04 20:54:27

ewaller wrote:

It would seem our friends over at Gentoo are seeing this as well...
https://forums.gentoo.org/viewtopic-p-8 … 825410d506

Yes, mentioned that earlier. Didn't post the link as I wasn't sure if it was ok or against the forum's rules to link to another forum.
This was the one.

Not too surprised that others have the same issue, as the deeper I'm trying to debug it, and the least common part I see... except systemd itself.

On my 3 machines that exhibit the behavior:

- Seems to have appeared recently (but when exactly is hard to tell - could be linked to the release of systemd 253.5),
- All different hardware (one is a laptop, one a headless box with a mini-itx motherboard and one is my desktop workstation),
- All different generations of chipsets and CPUs, but all Intel CPUs,
- 2 are on the latest 'linux' kernel, one (the headless box) is on a 'rt-linux' kernel (6.3.3.15.realtime2-1-rt),
- 2 were using NTP, another timesyncd only, no difference,
- Apart from this journald oddity, the system time doesn't seem to exhibit any issues,
- RTC clocks seem to be fine,
- I get the messages both for system.journal and user-1000.journal

Note: it has just been reported on systemd's github: https://github.com/systemd/systemd/issues/28250
(that's not me)

Last edited by OpusOne (2023-07-04 21:06:45)

Arch Linux

#1 2023-07-03 02:26:11

systemd-journald: Monotonic clock jumped backwards

#2 2023-07-03 02:34:09

Re: systemd-journald: Monotonic clock jumped backwards

#3 2023-07-03 02:50:20

Re: systemd-journald: Monotonic clock jumped backwards

#4 2023-07-03 06:41:21

Re: systemd-journald: Monotonic clock jumped backwards

#5 2023-07-03 15:05:06

Re: systemd-journald: Monotonic clock jumped backwards

#6 2023-07-03 19:54:27

Re: systemd-journald: Monotonic clock jumped backwards

#7 2023-07-03 19:57:44

Re: systemd-journald: Monotonic clock jumped backwards

#8 2023-07-03 20:23:35

Re: systemd-journald: Monotonic clock jumped backwards

#9 2023-07-03 20:26:45

Re: systemd-journald: Monotonic clock jumped backwards

#10 2023-07-03 21:03:50

Re: systemd-journald: Monotonic clock jumped backwards

#11 2023-07-04 02:56:30

Re: systemd-journald: Monotonic clock jumped backwards

#12 2023-07-04 06:17:26

Re: systemd-journald: Monotonic clock jumped backwards

#13 2023-07-04 07:46:09

Re: systemd-journald: Monotonic clock jumped backwards

#14 2023-07-04 11:08:26

Re: systemd-journald: Monotonic clock jumped backwards

#15 2023-07-04 14:52:29

Re: systemd-journald: Monotonic clock jumped backwards

#16 2023-07-04 15:02:01

Re: systemd-journald: Monotonic clock jumped backwards

#17 2023-07-04 20:54:27

Re: systemd-journald: Monotonic clock jumped backwards

Board footer