You are not logged in.

#1 2023-07-03 02:26:11

OpusOne
Member
Registered: 2023-05-31
Posts: 80

systemd-journald: Monotonic clock jumped backwards

Hello,

I get the following entry in dmesg once at each boot:

systemd-journald[533]: /var/log/journal/8f67d1a6d6f44f84a3eea6e784985c42/user-1000.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.

What could be the cause of this? I understand that it's normally not something to worry about in general, but the fact it happens at each boot looks odd.

Offline

#2 2023-07-03 02:34:09

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,762

Re: systemd-journald: Monotonic clock jumped backwards

My first guess is that your hardware real time clock has failed or is grossly wrong and is not being synchronized to Linux time.

What is the output of timedatectl   ?


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#3 2023-07-03 02:50:20

OpusOne
Member
Registered: 2023-05-31
Posts: 80

Re: systemd-journald: Monotonic clock jumped backwards

Thanks!

$ timedatectl
               Local time: lun. 2023-07-03 04:44:11 CEST
           Universal time: lun. 2023-07-03 02:44:11 UTC
                 RTC time: lun. 2023-07-03 02:44:11
                Time zone: Europe/Paris (CEST, +0200)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no

Offline

#4 2023-07-03 06:41:21

seth
Member
Registered: 2012-09-03
Posts: 50,782

Re: systemd-journald: Monotonic clock jumped backwards

Do you dual-boot windows?

Offline

#5 2023-07-03 15:05:06

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,762

Re: systemd-journald: Monotonic clock jumped backwards

I looked at your timedatectl output and it isn't terrible.  I don't know how long it took you to post, but your clock could be off as much as 6 minutes.  My only comment, unrelated to your problem, is that you may want to use timedatectl to enable network time synchronization ( timedatectl set-ntp 1 )


Edit:  I guess I would look at the end of the journal for the previous boot with journalctl -b-1 and compare the time stamps with those at the beginning of the current boot with journalctl -b.  Pay particular attention to the time stamps around the line about monotonic time warning you found in dmesg when you find it in the journal.

Last edited by ewaller (2023-07-03 15:10:45)


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#6 2023-07-03 19:54:27

OpusOne
Member
Registered: 2023-05-31
Posts: 80

Re: systemd-journald: Monotonic clock jumped backwards

- Dual boot: I have a Windows partition on a separate SSD, but haven't booted to it in several weeks. So it's not related to booting back and forth from Windows to Linux. Besides, I've enabled UTC time both for Windows and Linux to avoid time issues between both.
- I had ntpd running, but not systemd-timesyncd (which is enabled when you issue a ' timedatectl set-ntp 1'). ntpd on my machine takes a long time to synchronize (several minutes after boot), so the system clock remains unsynchronized for a few minutes.
- I understand that timedatectl doesn't detect ntpd so it states that the NTP service is inactive.
- I did compare timestamps from the previous boot with the current boot, and didn't notice anything abnormal. Weird.

The only thing is that systemd-timesyncd was not enabled, only ntpd.

What I did: I disabled ntpd, enabled systemd-timesyncd via ' timedatectl set-ntp 1'. The system clock gets synchronized much, much faster now than with ntpd. I haven't rebooted yet to see if that solves the monotonic clock jumping backwards issue at boot, I'll report back when I do.

Offline

#7 2023-07-03 19:57:44

seth
Member
Registered: 2012-09-03
Posts: 50,782

Re: systemd-journald: Monotonic clock jumped backwards

The system clock gets synchronized much, much faster now than with ntpd.

cat /etc/ntp.conf

https://wiki.archlinux.org/title/Networ … figuration

Offline

#8 2023-07-03 20:23:35

OpusOne
Member
Registered: 2023-05-31
Posts: 80

Re: systemd-journald: Monotonic clock jumped backwards

my /etc/ntp.conf file was like so:

# Please consider joining the pool:
#
#     http://www.pool.ntp.org/join.html
#
# For additional information see:
# - https://wiki.archlinux.org/index.php/Network_Time_Protocol_daemon
# - http://support.ntp.org/bin/view/Support/GettingStarted
# - the ntp.conf man page

# Associate to Arch's NTP pool
#server 0.arch.pool.ntp.org
#server 1.arch.pool.ntp.org
#server 2.arch.pool.ntp.org
#server 3.arch.pool.ntp.org

server 0.fr.pool.ntp.org iburst
server 1.fr.pool.ntp.org iburst
server 2.fr.pool.ntp.org iburst
server 3.fr.pool.ntp.org iburst

# By default, the server allows:
# - all queries from the local host
# - only time queries from remote hosts, protected by rate limiting and kod
restrict default kod limited nomodify nopeer noquery notrap
restrict 127.0.0.1
restrict ::1

# Location of drift file
driftfile /var/lib/ntp/ntp.drift

Last edited by OpusOne (2023-07-03 20:23:58)

Offline

#9 2023-07-03 20:26:45

seth
Member
Registered: 2012-09-03
Posts: 50,782

Re: systemd-journald: Monotonic clock jumped backwards

I guess you didn't configure timesyncd to use those ntp servers? (not sure whether it even supports iburst)

Offline

#10 2023-07-03 21:03:50

OpusOne
Member
Registered: 2023-05-31
Posts: 80

Re: systemd-journald: Monotonic clock jumped backwards

$ cat /etc/systemd/timesyncd.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the timesyncd.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# See timesyncd.conf(5) for details.

[Time]
NTP=0.fr.pool.ntp.org 1.fr.pool.ntp.org 2.fr.pool.ntp.org 3.fr.pool.ntp.org
FallbackNTP=0.arch.pool.ntp.org 1.arch.pool.ntp.org 2.arch.pool.ntp.org 3.arch.pool.ntp.org
#RootDistanceMaxSec=5
#PollIntervalMinSec=32
#PollIntervalMaxSec=2048
#ConnectionRetrySec=30
#SaveIntervalSec=60

Same NTP servers. Not sure timesyncd supports iburst either, at least one can't specify it in the conf file.
And it does use the first one in the NTP list, not any fallback one.

$  timedatectl timesync-status
       Server: 162.159.200.1 (0.fr.pool.ntp.org)
Poll interval: 34min 8s (min: 32s; max 34min 8s)
         Leap: normal
      Version: 4
      Stratum: 3
    Reference: A150804
    Precision: 1us (-25)
Root distance: 6.431ms (max: 5s)
       Offset: -895us
        Delay: 45.989ms
       Jitter: 989us
 Packet count: 17
    Frequency: -13,392ppm

Offline

#11 2023-07-04 02:56:30

OpusOne
Member
Registered: 2023-05-31
Posts: 80

Re: systemd-journald: Monotonic clock jumped backwards

OK, I actually have  3 machines with Arch on them including this one. The two others are completely different hardware and were installed a few years ago (and kept up to date).
I just checked, and turns out I have the same problem on the 3 machines now. I don't think I had noticed this in the 2 others before, must have appeared relatively recently.

Completely puzzled. Has something changed recently in systemd, or have I misconfigured all 3 machines and never noticed it until now?
What could be the misconfiguration that could explain it?
A hardware issue on all 3 machines at the same time looks pretty unlikely.

I noticed someone pointing out a similar issue yesterday on the Gentoo forum. No idea if it's related yet.

Offline

#12 2023-07-04 06:17:26

seth
Member
Registered: 2012-09-03
Posts: 50,782

Re: systemd-journald: Monotonic clock jumped backwards

Is this only (always) in the users/session journal?
What's the context? Any pattern?
Compare the system time to a wall clock when booting w/o any network access (or in the BIOS/UEFI)

Offline

#13 2023-07-04 07:46:09

frostschutz
Member
Registered: 2013-11-15
Posts: 1,416

Re: systemd-journald: Monotonic clock jumped backwards

I have the same issue. There are no weird timestamps in the logs, everything looks correct. The error message does not say how big a jump it supposedly detected. Also tried changing timedatectl settings (synchronize/ntp yes/no, local-rtc yes/no), it doesn't seem to change anything. There are no other ntp related services active. Although it's a dualboot system with Windows 11 on the side, the message always appears for Linux to Linux reboots, so it should not be Windows related.

I first noticed this issue after upgrading from Intel Haswell to AM4 / Ryzen platform. Updated BIOS for the heck of it, no change either. If the hardware clock is misbehaving I haven't noticed anything like that. The BIOS battery is definitely OK (the system was completely off the power grid for a week when I went on a trip recently, and no settings were lost).

There don't seem to be any ill effects, other than a bit too many logfile rotations. All in all I'm too lazy to debug it. I just put it down as one more mystery of systemd... big_smile

Online

#14 2023-07-04 11:08:26

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,903

Re: systemd-journald: Monotonic clock jumped backwards

I did notice the same issue but never looked into it until now.

I don't dual boot, boot to multi-user.target and use dhcpcd & ntp .
The first occurence of the monotonic clock jump message is

Apr 16 10:16:57 silverbolt systemd-journald[411]: /var/log/journal/23f343c183994ed18e475975f81b7ccb/system.journal: Monotonic clock jumped backwards
 relative to last journal entry, rotating

On april 15 I did a lot of aur building and ran pacman -Syu multiple times (to update from my local repo after each build) so it's hard for me to check what updates from core/extra/community/multilib took place that day.

I have checked wall clock with computer clock a few times after noticing in and not noticed any difference  so if there is one it is less than a second.

Maybe other posters can verify whether that start date is correct and what was updated just before ?


The first entry I can find of it is from april 16 It looks like it started around april 15

Last edited by Lone_Wolf (2023-07-04 11:09:06)


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#15 2023-07-04 14:52:29

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,762

Re: systemd-journald: Monotonic clock jumped backwards

Well, I'll be....

I have it in my system as well.  And it does not seem to be related to system start up.  And they seem to come in bursts.

Jun 14 18:12:11 odin systemd-journald[239]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/user-1000.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 14 18:13:58 odin systemd-journald[240]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 14 18:14:10 odin systemd-journald[240]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/user-1000.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 14 18:31:04 odin systemd-journald[232]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 14 18:31:12 odin systemd-journald[232]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/user-1000.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 15 18:46:23 odin systemd-journald[233]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 15 18:47:02 odin systemd-journald[233]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/user-1000.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
Jun 16 10:32:28 odin systemd-journald[237]: /var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system.journal: Monotonic clock jumped backwards relative to last journal entry, rotating.
...

And, they seem to correlate with rotations in the systemd journal

-rw-r-----+ 1 root systemd-journal 4.0M Jun 14 18:13 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-0000000000294da1-0005fe20c2090fb2.journal'
-rw-r-----+ 1 root systemd-journal 4.3M Jun 14 18:14 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-0000000000295197-0005fe20c865c28f.journal'
-rw-r-----+ 1 root systemd-journal 4.0M Jun 14 18:31 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-000000000029568d-0005fe20c929d0ce.journal'
-rw-r-----+ 1 root systemd-journal 4.4M Jun 14 18:31 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-0000000000295990-0005fe21058da80f.journal'
-rw-r-----+ 1 root systemd-journal 4.5M Jun 15 18:46 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-0000000000295f35-0005fe21060ef841.journal'
-rw-r-----+ 1 root systemd-journal 4.4M Jun 15 18:47 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43d197ab4f488e9a71574e31c07e-0000000000296af7-0005fe355a2bda9b.journal'
-rw-r-----+ 1 root systemd-journal 4.7M Jun 16 10:32 '/var/log/journal/984b1e2fa88b442294f2433b2dde8d0d/system@519b43 .....

I seem to have dad four in a row on June 14, then a delay of about a day until I had two more the next day, then nothing again until the 16th.
Between Jun 14 and yesterday (7/3)  I have had 55 reports of the monotonic clock not being monotonic and a log rotation.  None so far today.

I was going to take a stab at turning off NTP, but then I remembered that the OP had it turned off when they experienced this.
Looks like an annoying bug without serious consequences.  Next question.  Is it the kernel or systemd?

I updated my kernel to 6.4.1 yesterday, but journalctl -b still reports two hits with this new kernel.

Last edited by ewaller (2023-07-04 14:53:06)


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#16 2023-07-04 15:02:01

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,762

Re: systemd-journald: Monotonic clock jumped backwards

It would seem our friends over at Gentoo are seeing this as well...
https://forums.gentoo.org/viewtopic-p-8 … 825410d506


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#17 2023-07-04 20:54:27

OpusOne
Member
Registered: 2023-05-31
Posts: 80

Re: systemd-journald: Monotonic clock jumped backwards

ewaller wrote:

It would seem our friends over at Gentoo are seeing this as well...
https://forums.gentoo.org/viewtopic-p-8 … 825410d506

Yes, mentioned that earlier. Didn't post the link as I wasn't sure if it was ok or against the forum's rules to link to another forum.
This was the one.

Not too surprised that others have the same issue, as the deeper I'm trying to debug it, and the least common part I see... except systemd itself.

On my 3 machines that exhibit the behavior:

- Seems to have appeared recently (but when exactly is hard to tell - could be linked to the release of systemd 253.5),
- All different hardware (one is a laptop, one a headless box with a mini-itx motherboard and one is my desktop workstation),
- All different generations of chipsets and CPUs, but all Intel CPUs,
- 2 are on the latest 'linux' kernel, one (the headless box) is on a 'rt-linux' kernel (6.3.3.15.realtime2-1-rt),
- 2 were using NTP, another timesyncd only, no difference,
- Apart from this journald oddity, the system time doesn't seem to exhibit any issues,
- RTC clocks seem to be fine,
- I get the messages both for system.journal and user-1000.journal

Note: it has just been reported on systemd's github: https://github.com/systemd/systemd/issues/28250
(that's not me)

Last edited by OpusOne (2023-07-04 21:06:45)

Offline

Board footer

Powered by FluxBB