You are not logged in.

#1 2021-01-15 11:58:01

freebench
Member
Registered: 2020-06-14
Posts: 14

[SOLVED] File System Read Only

Hi

Recently replaced a SSD disk and did a clean install.

After some events the apps fails with messages like File System is Read Only.
Reboot fails on fsck with proposal to fix it manually. (/dev/mapper/main-home: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.)
So I log in as root and issue this to fix it

fsck -y /dev/mapper/main-home

After a reboot it works for some time fine.

LVM on LUKS is used.
SSD TRIM is enabled.
Not all space available on disk is being used during partitioning the disk(on purpose).

Any ideas ?

uname -a
Linux rokz 5.10.6-arch1-1 #1 SMP PREEMPT Sat, 09 Jan 2021 18:22:35 +0000 x86_64 GNU/Linux
lsblk -f
NAME            FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINT
sda                                                                                              
├─sda1          vfat        FAT32          C095-85D0                               172.1M    33% /boot
└─sda2          crypto_LUKS 2              1e710aab-7ec0-4e5a-bc9b-a11283feef8d                  
  └─cryptlvm    LVM2_member LVM2 001       lRvl06-onXo-Qiei-rnaR-gfvY-INqW-uS7Zf4                
    ├─main-swap swap        1              49bc2ff5-571b-4386-b7b3-a2963f559091                  [SWAP]
    ├─main-root ext4        1.0            15e8df19-2225-4c0f-a6ed-0bc35ffd8219     81.7G    11% /
    └─main-home ext4        1.0            774a5e39-a703-49ad-9e16-662e4edaed4b    565.9G    50% /home
journalctl | grep fsck
Jan 15 00:45:10 rokz systemd[1]: systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0.service: Succeeded.
Jan 15 00:45:10 rokz audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 00:45:10 rokz systemd[1]: systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b.service: Succeeded.
Jan 15 00:45:10 rokz audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 00:45:10 rokz systemd[1]: Removed slice system-systemd\x2dfsck.slice.
Jan 15 00:45:54 rokz systemd[1]: Created slice system-systemd\x2dfsck.slice.
Jan 15 00:45:55 rokz systemd-fsck[458]: fsck.fat 4.1 (2017-01-24)
Jan 15 00:45:55 rokz systemd-fsck[458]: /dev/sda1: 365 files, 171754/524256 clusters
Jan 15 00:45:55 rokz audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 00:45:55 rokz systemd-fsck[491]: /dev/mapper/main-home: clean, 503053/84189184 files, 171204679/336728064 blocks
Jan 15 00:45:55 rokz audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 01:17:04 rokz systemd[1]: Created slice system-systemd\x2dfsck.slice.
Jan 15 01:17:05 rokz systemd-fsck[399]: fsck.fat 4.1 (2017-01-24)
Jan 15 01:17:05 rokz systemd-fsck[399]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Jan 15 01:17:05 rokz systemd-fsck[399]:  Automatically removing dirty bit.
Jan 15 01:17:05 rokz systemd-fsck[399]: Performing changes.
Jan 15 01:17:05 rokz systemd-fsck[399]: /dev/sda1: 365 files, 171721/524256 clusters
Jan 15 01:17:05 rokz audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 01:17:05 rokz systemd-fsck[426]: /dev/mapper/main-home: recovering journal
Jan 15 01:17:05 rokz systemd-fsck[426]: /dev/mapper/main-home: clean, 502905/84189184 files, 171201588/336728064 blocks
Jan 15 01:17:05 rokz audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 01:22:54 rokz systemd[1]: Created slice system-systemd\x2dfsck.slice.
Jan 15 01:22:55 rokz systemd-fsck[405]: fsck.fat 4.1 (2017-01-24)
Jan 15 01:22:55 rokz systemd-fsck[405]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Jan 15 01:22:55 rokz systemd-fsck[405]:  Automatically removing dirty bit.
Jan 15 01:22:55 rokz systemd-fsck[405]: Performing changes.
Jan 15 01:22:55 rokz systemd-fsck[405]: /dev/sda1: 365 files, 171721/524256 clusters
Jan 15 01:22:55 rokz audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 01:22:55 rokz systemd-fsck[435]: /dev/mapper/main-home: recovering journal
Jan 15 01:22:55 rokz systemd-fsck[435]: /dev/mapper/main-home: clean, 502911/84189184 files, 171199576/336728064 blocks
Jan 15 01:22:55 rokz audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 12:55:34 rokz systemd[1]: systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0.service: Succeeded.
Jan 15 12:55:34 rokz audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 12:55:35 rokz systemd[1]: systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b.service: Succeeded.
Jan 15 12:55:35 rokz audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 12:55:35 rokz systemd[1]: Removed slice system-systemd\x2dfsck.slice.
Jan 15 12:56:20 rokz systemd[1]: Created slice system-systemd\x2dfsck.slice.
Jan 15 12:56:20 rokz systemd-fsck[403]: fsck.fat 4.1 (2017-01-24)
Jan 15 12:56:20 rokz systemd-fsck[403]: /dev/sda1: 365 files, 171721/524256 clusters
Jan 15 12:56:20 rokz audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ins]Jan 15 12:56:21 rokz systemd-fsck[442]: /dev/mapper/main-home: recovering journal
Jan 15 12:56:22 rokz systemd-fsck[442]: /dev/mapper/main-home contains a file system with errors, check forced.
Jan 15 12:56:23 rokz systemd-fsck[442]: /dev/mapper/main-home: Deleted inode 69733087 has zero dtime.  FIXED.
Jan 15 12:56:23 rokz systemd-fsck[442]: /dev/mapper/main-home: Inodes that were part of a corrupted orphan linked list found.
Jan 15 12:56:23 rokz systemd-fsck[442]: /dev/mapper/main-home: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Jan 15 12:56:23 rokz systemd-fsck[442]:         (i.e., without -a or -p options)
Jan 15 12:56:23 rokz systemd-fsck[439]: fsck failed with exit status 4. [/ins]
Jan 15 12:56:23 rokz systemd-fsck[439]: Running request emergency.target/start/replace
Jan 15 12:56:23 rokz systemd[1]: systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b.service: Main process exited, code=exited, status=1/FAILURE
Jan 15 12:56:23 rokz systemd[1]: systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b.service: Failed with result 'exit-code'.
Jan 15 12:56:23 rokz audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jan 15 12:58:02 rokz systemd[1]: systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0.service: Succeeded.
Jan 15 12:58:02 rokz audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 12:58:02 rokz kernel: audit: type=1131 audit(1610708282.320:30): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 12:58:02 rokz systemd[1]: Removed slice system-systemd\x2dfsck.slice.
Jan 15 12:59:25 rokz systemd[1]: Created slice system-systemd\x2dfsck.slice.
Jan 15 12:59:26 rokz systemd-fsck[392]: fsck.fat 4.1 (2017-01-24)
Jan 15 12:59:26 rokz systemd-fsck[392]: /dev/sda1: 365 files, 171721/524256 clusters
Jan 15 12:59:26 rokz audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-C095\x2d85D0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 15 12:59:26 rokz systemd-fsck[415]: /dev/mapper/main-home: clean, 503469/84189184 files, 171549232/336728064 blocks
Jan 15 12:59:26 rokz audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck@dev-disk-by\x2duuid-774a5e39\x2da703\x2d49ad\x2d9e16\x2d662e4edaed4b comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
# smartctl -a /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.6-arch1-1] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     Disk 2TB
Serial Number:    1120112443237
LU WWN Device Id: 5 000000 000000000
Firmware Version: SN3186
User Capacity:    1,950,389,231,616 bytes [1.95 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Jan 15 13:39:52 2021 EET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(   33) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (   2) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x0031)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 19
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       99
 12 Power_Cycle_Count       0x0012   100   100   000    Old_age   Always       -       13
167 Unknown_Attribute       0x0022   100   100   000    Old_age   Always       -       0
168 Unknown_Attribute       0x0012   100   100   000    Old_age   Always       -       0
169 Unknown_Attribute       0x0013   100   100   010    Pre-fail  Always       -       7340060
171 Unknown_Attribute       0x0032   000   000   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   000   000   000    Old_age   Always       -       0
173 Unknown_Attribute       0x0012   200   200   000    Old_age   Always       -       4295884803
175 Program_Fail_Count_Chip 0x0022   100   100   010    Old_age   Always       -       0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033   100   100   000    Pre-fail  Always       -       6464
187 Reported_Uncorrect      0x0032   100   000   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0012   100   100   000    Old_age   Always       -       5
194 Temperature_Celsius     0x0022   040   040   000    Old_age   Always       -       40 (Min/Max 40/40)
206 Unknown_SSD_Attribute   0x0032   200   200   000    Old_age   Always       -       1
207 Unknown_SSD_Attribute   0x0032   200   200   000    Old_age   Always       -       14
208 Unknown_SSD_Attribute   0x0032   200   200   000    Old_age   Always       -       3
209 Unknown_SSD_Attribute   0x0032   200   200   000    Old_age   Always       -       1
210 Unknown_Attribute       0x0032   200   200   000    Old_age   Always       -       20
211 Unknown_Attribute       0x0032   200   200   000    Old_age   Always       -       13
231 Unknown_SSD_Attribute   0x0023   100   100   005    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       2392348544
234 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       749069564
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       1601041853
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       152261711
245 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       62456761
243 Unknown_Attribute       0x0032   050   050   000    Old_age   Always       -       39

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Conveyance offline  Completed without error       00%        93         -
# 2  Short offline       Completed without error       00%        93         -
# 3  Extended offline    Completed without error       00%        92         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
# smartctl -l error -l selftest /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.6-arch1-1] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Conveyance offline  Completed without error       00%        93         -
# 2  Short offline       Completed without error       00%        93         -
# 3  Extended offline    Completed without error       00%        92         
# parted
GNU Parted 3.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print free
Model: ATA Disk 2TB (scsi)
Disk /dev/sda: 1950GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
        17.4kB  1049kB  1031kB  Free Space
 1      1049kB  274MB   273MB   fat32              boot, esp
 2      274MB   1521GB  1521GB
        1521GB  1950GB  429GB   Free Space
# parted /dev/sda
GNU Parted 3.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) align-check optimal 1                                            
1 aligned
(parted) align-check optimal 2
2 aligned
# fdisk -l /dev/sda
Disk /dev/sda: 1.77 TiB, 1950389231616 bytes, 3809353968 sectors
Disk model: Disk 2TB 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 899B8776-8B5F-FE4F-B4E7-4F9070F6EBA5

Device      Start        End    Sectors  Size Type
/dev/sda1    2048     534527     532480  260M EFI System
/dev/sda2  534528 2971220240 2970685713  1.4T Linux filesystem

Last edited by freebench (2021-02-04 00:32:30)

Offline

#2 2021-01-15 16:06:45

seth
Member
Registered: 2012-09-03
Posts: 18,781

Offline

#3 2021-01-15 16:21:57

freebench
Member
Registered: 2020-06-14
Posts: 14

Re: [SOLVED] File System Read Only

yes, in fact the issue popped up before the TRIM was turned on

# hdparm -I /dev/sda | grep TRIM
	   *	Data Set Management TRIM supported (limit 8 blocks)
	   *	Deterministic read ZEROs after TRIM

here're some extras from the recent fs fail

Jan 15 15:53:42 rokz kernel: audit: type=1100 audit(1610718822.783:245): pid=6006 uid=1000 auid=1000 ses=2 msg='op=PAM:unix_chkpwd acct="admin" exe="/usr/bin/unix_chkpwd" hostname=? addr=? t
erminal=? res=success'
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#27 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=31s
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#27 CDB: Write(10) 2a 00 60 d0 9b d8 00 00 10 00
Jan 15 16:02:03 rokz kernel: blk_update_request: I/O error, dev sda, sector 1624284120 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#26 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=31s
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#26 CDB: Write(10) 2a 00 60 d0 9b c0 00 00 18 00
Jan 15 16:02:03 rokz kernel: blk_update_request: I/O error, dev sda, sector 1624284096 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#31 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=31s
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#31 CDB: Write(10) 2a 00 11 a7 ce a8 00 00 f8 00
Jan 15 16:02:03 rokz kernel: blk_update_request: I/O error, dev sda, sector 296210088 op 0x1:(WRITE) flags 0x800 phys_seg 31 prio class 0
Jan 15 16:02:03 rokz kernel: EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 10 writing to inode 69730976 starting block 2352085)
Jan 15 16:02:03 rokz kernel: Buffer I/O error on device dm-3, logical block 2352085
Jan 15 16:02:03 rokz kernel: Buffer I/O error on device dm-3, logical block 2352086
Jan 15 16:02:03 rokz kernel: Buffer I/O error on device dm-3, logical block 2352087
Jan 15 16:02:03 rokz kernel: Buffer I/O error on device dm-3, logical block 2352088
Jan 15 16:02:03 rokz kernel: Buffer I/O error on device dm-3, logical block 2352089
Jan 15 16:02:03 rokz kernel: Buffer I/O error on device dm-3, logical block 2352090
Jan 15 16:02:03 rokz kernel: Buffer I/O error on device dm-3, logical block 2352091
Jan 15 16:02:03 rokz kernel: Buffer I/O error on device dm-3, logical block 2352092
Jan 15 16:02:03 rokz kernel: Buffer I/O error on device dm-3, logical block 2352093
Jan 15 16:02:03 rokz kernel: Buffer I/O error on device dm-3, logical block 2352094
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#28 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=31s
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#28 CDB: Write(10) 2a 00 11 c2 4f 18 00 00 20 00
Jan 15 16:02:03 rokz kernel: blk_update_request: I/O error, dev sda, sector 297946904 op 0x1:(WRITE) flags 0x800 phys_seg 4 prio class 0
Jan 15 16:02:03 rokz kernel: EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 10 writing to inode 69730976 starting block 2569187)
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=30s
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#5 CDB: Write(10) 2a 00 a3 d4 df 78 00 00 08 00
Jan 15 16:02:03 rokz kernel: blk_update_request: I/O error, dev sda, sector 2748637048 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Jan 15 16:02:03 rokz kernel: EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 10 writing to inode 77464663 starting block 308905455)
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=30s
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#4 CDB: Write(10) 2a 00 9f 3d 5d a0 00 00 08 00
Jan 15 16:02:03 rokz kernel: blk_update_request: I/O error, dev sda, sector 2671599008 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Jan 15 16:02:03 rokz kernel: EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 10 writing to inode 69732594 starting block 299275700)
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=30s
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#3 CDB: Write(10) 2a 00 95 95 f1 b0 00 00 08 00
Jan 15 16:02:03 rokz kernel: blk_update_request: I/O error, dev sda, sector 2509631920 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Jan 15 16:02:03 rokz kernel: EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 10 writing to inode 69732597 starting block 279029814)
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=30s
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#2 CDB: Write(10) 2a 00 95 95 f1 90 00 00 08 00
Jan 15 16:02:03 rokz kernel: blk_update_request: I/O error, dev sda, sector 2509631888 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Jan 15 16:02:03 rokz kernel: EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 10 writing to inode 69732597 starting block 279029810)
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=30s
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#1 CDB: Write(10) 2a 00 52 99 40 60 00 00 08 00
Jan 15 16:02:03 rokz kernel: blk_update_request: I/O error, dev sda, sector 1385775200 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Jan 15 16:02:03 rokz kernel: EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 10 writing to inode 76554019 starting block 138547724)
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=30s
Jan 15 16:02:03 rokz kernel: sd 0:0:0:0: [sda] tag#0 CDB: Write(10) 2a 00 1c 25 fe d8 00 00 40 00
Jan 15 16:02:03 rokz kernel: blk_update_request: I/O error, dev sda, sector 472252120 op 0x1:(WRITE) flags 0x0 phys_seg 8 prio class 0
Jan 15 16:02:03 rokz kernel: EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 10 writing to inode 69732595 starting block 24357339)
Jan 15 16:02:03 rokz kernel: Aborting journal on device dm-3-8.
Jan 15 16:02:03 rokz kernel: EXT4-fs error (device dm-3) in ext4_reserve_inode_write:5749: Journal has aborted
Jan 15 16:02:03 rokz kernel: EXT4-fs error (device dm-3) in ext4_reserve_inode_write:5749: Journal has aborted
Jan 15 16:02:03 rokz kernel: EXT4-fs error (device dm-3) in ext4_reserve_inode_write:5749: Journal has aborted
Jan 15 16:02:03 rokz kernel: EXT4-fs error (device dm-3): ext4_dirty_inode:5951: inode #69732594: comm teams: mark_inode_dirty error
Jan 15 16:02:03 rokz kernel: EXT4-fs error (device dm-3) in ext4_dirty_inode:5953: Journal has aborted
Jan 15 16:02:03 rokz kernel: EXT4-fs error (device dm-3): ext4_journal_check_start:83: Detected aborted journal
Jan 15 16:02:03 rokz kernel: EXT4-fs (dm-3): Remounting filesystem read-only
Jan 15 16:02:03 rokz kernel: EXT4-fs error (device dm-3): ext4_dirty_inode:5951: inode #77464663: comm hexchat: mark_inode_dirty error
Jan 15 16:02:03 rokz kernel: EXT4-fs error (device dm-3): ext4_journal_check_start:83: Detected aborted journal
Jan 15 16:02:03 rokz kernel: EXT4-fs (dm-3): failed to convert unwritten extents to written extents -- potential data loss!  (inode 69733078, error -30)
Jan 15 16:02:03 rokz kernel: EXT4-fs (dm-3): failed to convert unwritten extents to written extents -- potential data loss!  (inode 69733005, error -30)
Jan 15 16:02:03 rokz kernel: EXT4-fs error (device dm-3): ext4_dirty_inode:5951: inode #69732595: comm teams: mark_inode_dirty error
Jan 15 16:02:03 rokz kernel: EXT4-fs error (device dm-3): ext4_journal_check_start:83: Detected aborted journal
Jan 15 16:03:54 rokz pulseaudio[778]: Failed to save default sink: Read-only file system
Jan 15 16:03:54 rokz pulseaudio[778]: Failed to save default source: Read-only file system

Also, after this happened last time I booted arch installation iso and run this

e2fsck -f /dev/sda

passed all the checks...

I have no idea but to think of some hardware issue.
It is a 2.5" disk in a laptop, maybe installed it wrongly

Last edited by freebench (2021-01-15 16:25:58)

Offline

#4 2021-01-15 16:53:11

seth
Member
Registered: 2012-09-03
Posts: 18,781

Re: [SOLVED] File System Read Only

IO errors, but no smart issues.
Cable/connector/seating or some powersaving feature?

Offline

#5 2021-01-19 11:26:03

freebench
Member
Registered: 2020-06-14
Posts: 14

Re: [SOLVED] File System Read Only

Spent some time to disassemble laptop and re-install the disk - plug it out and put it back.
Lasted for a day or so of normal work... Today got IO errors in the journal again sad and had to reboot in order to fix those via fsck.

Would it be a good idea to clean install arch? and probably without LVM... I've read in wiki that it could cause such issues, can't find that bit right now

Offline

#6 2021-01-19 15:04:17

seth
Member
Registered: 2012-09-03
Posts: 18,781

Re: [SOLVED] File System Read Only

seth wrote:

some powersaving feature?

The disk could fail to "spin" up after entering a powersaving mode. (And yes: no actual spinning ;-)

Offline

#7 2021-01-19 16:12:17

freebench
Member
Registered: 2020-06-14
Posts: 14

Re: [SOLVED] File System Read Only

The issue pops up during regular system use. No suspend-restore, or system goes idle events.
Laptop is on line power when this happens.
Power manager is the one that comes with xfce.

How "entering a powersaving mode" can be monitored ?

pacman -Q xfce4-power-manager
xfce4-power-manager 4.16.0-1
Jan 19 17:55:40 rokz kernel: sd 0:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=32s
Jan 19 17:55:40 rokz kernel: sd 0:0:0:0: [sda] tag#25 CDB: Write(10) 2a 00 60 d6 61 a0 00 00 48 00
Jan 19 17:55:40 rokz kernel: blk_update_request: I/O error, dev sda, sector 1624662432 op 0x1:(WRITE) flags 0x800 phys_seg 9 prio class 0
Jan 19 17:55:40 rokz kernel: Aborting journal on device dm-3-8.
Jan 19 17:55:40 rokz kernel: EXT4-fs error (device dm-3) in __ext4_new_inode:1130: Journal has aborted
Jan 19 17:55:40 rokz kernel: EXT4-fs error (device dm-3) in __ext4_new_inode:1130: Journal has aborted
Jan 19 17:55:40 rokz kernel: EXT4-fs error (device dm-3) in __ext4_new_inode:1130: Journal has aborted
Jan 19 17:55:40 rokz kernel: EXT4-fs (dm-3): Delayed block allocation failed for inode 69861439 at logical offset 0 with max blocks 3 with error 30
Jan 19 17:55:40 rokz kernel: EXT4-fs (dm-3): This should not happen!! Data will be lost
Jan 19 17:55:40 rokz kernel: EXT4-fs error (device dm-3) in ext4_writepages:2814: Journal has aborted
Jan 19 17:55:40 rokz kernel: EXT4-fs error (device dm-3) in ext4_reserve_inode_write:5749: Journal has aborted
Jan 19 17:55:40 rokz kernel: EXT4-fs (dm-3): Delayed block allocation failed for inode 69732978 at logical offset 0 with max blocks 1 with error 30
Jan 19 17:55:40 rokz kernel: EXT4-fs (dm-3): This should not happen!! Data will be lost
Jan 19 17:55:40 rokz kernel: EXT4-fs error (device dm-3) in ext4_writepages:2814: Journal has aborted
Jan 19 17:55:40 rokz kernel: EXT4-fs error (device dm-3): ext4_journal_check_start:83: Detected aborted journal
Jan 19 17:55:40 rokz kernel: EXT4-fs (dm-3): Remounting filesystem read-only
Jan 19 17:55:40 rokz kernel: EXT4-fs error (device dm-3): ext4_dirty_inode:5951: inode #69731827: comm worker: mark_inode_dirty error
Jan 19 17:55:40 rokz kernel: EXT4-fs error (device dm-3): ext4_journal_check_start:83: Detected aborted journal
Jan 19 17:55:40 rokz kernel: EXT4-fs (dm-3): failed to convert unwritten extents to written extents -- potential data loss!  (inode 77464755, error -30)
Jan 19 17:55:40 rokz kernel: EXT4-fs (dm-3): failed to convert unwritten extents to written extents -- potential data loss!  (inode 74711241, error -30)
Jan 19 17:55:40 rokz kernel: EXT4-fs (dm-3): failed to convert unwritten extents to written extents -- potential data loss!  (inode 69864582, error -30)
Jan 19 17:58:14 rokz gnome-keyring-daemon[777]: Gkm: couldn't create temporary file for: /home/admin/.local/share/keyrings/Default_keyring.keyring: Read-only file system
Jan 19 17:58:14 rokz gnome-keyring-daemon[777]: Gkm: couldn't create temporary file for: /home/admin/.local/share/keyrings/Default_keyring.keyring: Read-only file system

Offline

#8 2021-01-19 16:40:28

seth
Member
Registered: 2012-09-03
Posts: 18,781

Re: [SOLVED] File System Read Only

"man hdparm", see -s, -S and -y and -Y (latter to possibly trigger this)

Offline

#9 2021-01-20 02:54:33

freebench
Member
Registered: 2020-06-14
Posts: 14

Re: [SOLVED] File System Read Only

None of these commands to manipulate disk state performed manually have caused IO errors logged followed by files system switching to read only. Only -Y(sleep) logged something.

#standby

sudo hdparm -y /dev/sda

#sleep

sudo hdparm -Y /dev/sda
Jan 20 04:39:15 rokz kernel: ata1.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x6
Jan 20 04:39:15 rokz kernel: ata1.00: waking up from sleep
Jan 20 04:39:15 rokz kernel: ata1: hard resetting link
Jan 20 04:39:15 rokz kernel: ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jan 20 04:39:15 rokz kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) succeeded
Jan 20 04:39:15 rokz kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out
Jan 20 04:39:15 rokz kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) succeeded
Jan 20 04:39:15 rokz kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out
Jan 20 04:39:15 rokz kernel: ata1.00: configured for UDMA/133
Jan 20 04:39:15 rokz kernel: ata1: EH complete
sudo hdparm -S1 /dev/sda
/dev/sda:
 setting standby to 1 (5 seconds)
sudo hdparm -C /dev/sda
/dev/sda:
 drive state is:  active/idle

Last edited by freebench (2021-01-20 03:03:43)

Offline

#10 2021-01-21 20:10:04

crypticuser
Member
From: Kolkata
Registered: 2021-01-21
Posts: 3

Re: [SOLVED] File System Read Only

in order to remove the read only part... just change the ownership of the drive....

sudo chown USER:USER /dev/drive_name


Archie**

Offline

#11 2021-01-21 20:17:41

loqs
Member
Registered: 2014-03-06
Posts: 12,949

Re: [SOLVED] File System Read Only

crypticuser wrote:

in order to remove the read only part... just change the ownership of the drive....

sudo chown USER:USER /dev/drive_name

The drive is mounted read only because the kernel encountered errors and that triggers the default FAT mount option errors=remount-ro
/dev/ is a virtual filesystem so any change to it will not be permanent and will not affect the ownership of the mounted filesystem.  FAT ownership can be set using mount options.

Last edited by loqs (2021-01-21 20:18:03)

Offline

#12 2021-01-22 17:05:42

seth
Member
Registered: 2012-09-03
Posts: 18,781

Re: [SOLVED] File System Read Only

What device is this exactly? Have you googled it for known controller/firmware issues?
The IO error is on sda, I don't think it relates to LVM.
Were you able to re-create the errors w/ trimming disabled? (I understand that you had issues w/o trimming, but maybe those were unrelated, eg. from a hard reboot…)

Offline

#13 2021-02-04 00:31:29

freebench
Member
Registered: 2020-06-14
Posts: 14

Re: [SOLVED] File System Read Only

Seth wrote:

What device is this exactly?

Don't want to make anti-advertisement for a manufacturer.

Seth wrote:

Have you googled it for known controller/firmware issues?

Only read the feedback from the buyers of the same device. Seemed fine ...

Seth wrote:

The IO error is on sda, I don't think it relates to LVM.

Agreed on this. As now I've bought(+$25) similar device by the well known manufacturer, works fine out of the box with the same setup.

Seth wrote:

Were you able to re-create the errors w/ trimming disabled? (I understand that you had issues w/o trimming, but maybe those were unrelated, eg. from a hard reboot…)

Weren't able to test this as the "faulty" disk is returned to the seller. Haven't got a capacity to play around with it. Needed a working machine for work asap.

p.s. don't buy the cheapest disk. (at least you know what you're doing)
p.p.s. or i're really unlucky, which is quite possible.

Offline

Board footer

Powered by FluxBB