Disk R/W Speed Seems to Drop Randomly; System Lag

hoot_hoot · 2012-07-01 16:36:44

Hello all! FIrst post in Arch forum...!

Anyway, I switched from [insert random Linux distribution I was using at the time] to Arch because I love the rolling-release characteristic, the online resources, and the way one may build his system from the ground up.

However, one issue has been plaguing my system since my initial installation.

When I first installed, I did it under a fakeRAID setup per the wiki, because I'd been planning to do so for awhile and summer finally gave me the time to do it. I adapted to the arch environment and experimented with several window managers/de's before I settled on Gnome-Shell. However, one thing they all had in common was that the system lagged. In the beginning, I figured I would just bear through it, but it's become to noticable, that even a man page takes about 20 seconds to load.

I'm running a quad-core Phenom II at 3.2GHz, 12gb ram, and two 1tb drives. (n.b. they are different brand drives, but I had set them up in a RAID 1 anyway.)
It should not take 10 seconds for chromium to simply open a new blank tab. In addition, it would take about 5 minutes just for arch to boot.

I figured that it was the fakeraid that was bottle-necking my disk r/w, considering I'd read that bios-based raid was slow and I had two different brand harddrives, so I did away with it and reinstalled the arch system, planning to use some sort of rsync-based backup in the future. To my dismay, even after the installation, this lag lingers.

I thought it may have been one of my window managers or maybe my video driver, but that would not explain why the system boot normally hangs on "Triggering uevents", or why "man pacman" takes an excessive amount of time to load.

I know at this point you're probably thinking, "Dang, this guy must not have any patience to be able to wait [insert time it takes to open a chrome tab or man page, then add 10 seconds]," and I was just going to let it go, but then today (after I've done away with fakeraid and reinstalled) I pressed the power button and as soon as GRUB hit the screen, I experienced what things should be like. Arch flew threw the bootup, and it didn't take gdm ten seconds to load the password prompt after I pressed enter for my username. Gnome loaded like it was competent. My ecstasy was short lived -- I was about to install dropbox (random program which remains uninstalled), but when I hit Gnome's "Activities" button and then the icon for gnome-terminal, my dreams of a decently-fast system came crashing down. My system was at a standstill for a solid 15 seconds before I could do anything. The only other open window I had was Chromium, and the tab that had a loading page with the little loading circle had stopped, and I could no longer click on anything -- the moused moved, but I could switch to one of my ttys, nor could I go back into the "Activities" menu. I've come to despise the little red disk activity light that remains confidently on during these moments of terror. Once my system had recovered (its like it has a stroke or something), I needed to check the pacman command for additional -R options -- "man pacman". I entered this, and again, my system had another 15-second stroke. I come before you today to ask for suggestions.

I was recently introduced to minecraft (I know -- don't judge), and you can imagine how distressing it when about three times every 5 minutes, the system will freeze spontaneously before continuing about 5 seconds later.

Thank you for reading my potentially dramatic rant -- This experience of a bipolar system has left me quite frustrated. Even a liveCD can do many of these functions faster, and I'm desperate for a solution.

I know there's probably more information I can provide... Here's some:
My system is up to date.
I've had a feeling that I had below average disk r/w, but I have not done any type of benchmarking. (i.e., with the command hdparam, I think?)
I'll provide anything else per your request.

Any suggestions are very welcome, and thank you for your time.

UPDATE: I just rebooted again, and I'm experiencing those nice speeds again... I'm wondering if it's a period of time until it has a stroke. I'm writing this real time because I'm guessing it will happen soon... *begins performing normal tasks seeing if it will happen* Alright... Well it was always inconsistent. Things are running decently, but I'd still like to see if anyone has any ideas, because I'm 92% sure it will happen again.

Last edited by hoot_hoot (2012-07-01 17:32:32)

Lone_Wolf · 2012-07-02 18:02:15

This will likely be hard to troubleshoot, but let's make a start.

What is your partition lay-out and which filesystems do you use ?
post your rc.conf .
Since you use gnome, systemd will probably manage the daemons/services : Which daemons/services are started at boot ?

run htop or top to find out if there are processes hogging the cpu cores.

hoot_hoot · 2012-07-02 22:51:08

I figured; I didn't know where to start..

Here's my layout:
sda:
/boot -- 500MB
swap -- 5GB
/ -- 286GB
/home -- 286GB
/archive -- 309GB * A miscellaneous partition I made to keep important things on. I wanted a directory on root that was accessible by all for things worth keeping.

sdb: backup of /home, /, and /archive. More or less inactive, however. Each partition is slightly larger than the ones on sda.

I also have a smaller, third hard drive housing a windows installation.

rc.conf: Sorry, I don't know how to do quotes on this forum:

#
# /etc/rc.conf - Main Configuration for Arch Linux
#
# See 'man 5 rc.conf' for more details
#

# LOCALIZATION
# ------------
HARDWARECLOCK="UTC"
TIMEZONE="America/New_York"
KEYMAP="us"
CONSOLEFONT=
CONSOLEMAP=
LOCALE=
DAEMON_LOCALE="yes"
USECOLOR="yes"

# HARDWARE
# --------
MODULES=()
USEDMRAID="no"
USEBTRFS="no"
USELVM="no"

# NETWORKING
# ----------
HOSTNAME="arch"

interface=eth0
address=**.*.*.***
netmask=255.255.255.0
broadcast=**.*.*.255
gateway=1*.*.*.*

NETWORK_PERSIST="no"

# DAEMONS
# -------
#
DAEMONS=(syslog-ng network crond acpid dbus gdm alsa sshd httpd)

Those are the only extra daemons I believe I start at boot...
I've monitored those pretty closely, and nothing absorbs a particularly large amount of cpu usage during the times that the system hangs.

Lone_Wolf · 2012-07-03 12:42:22

The forum uses BBcode, see https://bbs.archlinux.org/help.php#bbcode for details.

your / is very large with 286 GB, but since you start httpd , i guess that space is necessary for the webserver files ?

what filesystem do you use on / and home ?

What is this system used for ?

please post the output of

$ free -l

both with webserver running and stopped .

Does stopping the webserver temporarily, make a difference in the lag ?

hoot_hoot · 2012-07-04 03:44:05

Thanks for the heads up with bbcode --

There was no particular reason that my / was that big; I used httpd for personal endeavors and launch the daemon at startup purely for convenience.

All of my linux filesystems are ext4.

This is my personal computer, and I use it for nearly all I do on the computer -- which includes some web and java development, music and other entertainment (more recently including minecraft), basic tasks like web browsing, etc, and I also use some cad on Windows.

With webserver running:

total       used       free     shared    buffers     cached
Mem:      12310864    1367840   10943024          0      39608     378608
Low:      12310864    1367840   10943024
High:            0          0          0
-/+ buffers/cache:     949624   11361240
Swap:      4996176          0    4996176

With webserver not running:

total       used       free     shared    buffers     cached
Mem:      12310864    1360156   10950708          0      39840     380340
Low:      12310864    1360156   10950708
High:            0          0          0
-/+ buffers/cache:     939976   11370888
Swap:      4996176          0    4996176

I'll get back to on stopping httpd; but I will say that I only recently added httpd to my daemons recently in light of some experimentation with a lamp server; this problem has been around since before httpd was a startup daemon. I'll remove it from the list and see if it makes any difference.

Also, if you're interested in another potential symptom; today I rebooted after making no changes to any system configuration, because the system was lagging again. It seems that there's a 50/50 chance that the system will run smoothly after boot or that it will lag like it does. At the point during boot when it says "Running uevents" and then proceeds to mount the root partition, it dropped me into a shell in which no commands worked (even though they were in the $PATH, e.g., it recognized "reboot" but when I entered it, nothing happened. I ended up resolving the issue after resetting the computer a few times with no avail by booting into the fallback kernel... Even though it acted like a normal boot, and ironically it was actually the good half of the probability; it was working quickly.

I initialized a smartctl long test on the drive with the root partition, and it's set to finish in about two hours; I'll post the results tomorrow.
n.b., I apologize for the delay in my responses, and really appreciate you taking a stab at this.

Lone_Wolf · 2012-07-04 19:31:18

Glad to help, and some delay in replies is no problem.

hoot_hoot wrote:

total       used       free     shared    buffers     cached
Mem:      12310864    1367840   10943024          0      39608     378608
Low:      12310864    1367840   10943024
High:            0          0          0
-/+ buffers/cache:     949624   11361240
Swap:      4996176          0    4996176

With webserver not running:

total       used       free     shared    buffers     cached
Mem:      12310864    1360156   10950708          0      39840     380340
Low:      12310864    1360156   10950708
High:            0          0          0
-/+ buffers/cache:     939976   11370888
Swap:      4996176          0    4996176

This shows that the webserver barely uses any memory, so i think we can remove that as possible cause.

You got plenty of free memory, and swap isn't used at all (common on systems with plenty of ram) , so swapping is not the problem.

Given the 50/50 success of running normally i'm beginning to think about a hardware problem.

Apart from the smartctl test, i suggest you run memtest .

ewaller · 2012-07-04 20:44:30

Have you checked your logs for I/O errors on your disk?
Does your drive support SMART ? If so, have you tried any SMART utilities to look at the health of the disk?

hoot_hoot · 2012-07-05 22:10:11

Here's the results from a short test of the smartctl:

smartctl --test=short /dev/sda
smartctl -a /dev/sda

smartctl 5.42 2011-10-20 r3458 [x86_64-linux-3.4.4-2-ARCH] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Blue Serial ATA
Device Model:     WDC WD10EALS-00Z8A0
Serial Number:    WD-WCATR0569697
LU WWN Device Id: 5 0014ee 25996420e
Firmware Version: 05.01D05
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Thu Jul  5 18:07:33 2012 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x85)	Offline data collection activity
					was aborted by an interrupting command from host.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(15360) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 178) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x3037)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   196   195   051    Pre-fail  Always       -       4454
  3 Spin_Up_Time            0x0027   179   172   021    Pre-fail  Always       -       4041
  4 Start_Stop_Count        0x0032   099   099   000    Old_age   Always       -       1692
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   095   095   000    Old_age   Always       -       3659
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1638
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       455
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1236
194 Temperature_Celsius     0x0022   110   107   000    Old_age   Always       -       37
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   198   198   000    Old_age   Always       -       353
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   059   000    Old_age   Offline      -       39

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3659         -
# 2  Extended offline    Completed: read failure       50%      3657         1110990248
# 3  Short offline       Completed without error       00%      3627         -
# 4  Short offline       Completed: read failure       80%      3301         1504712964
# 5  Extended offline    Completed: read failure       90%      3301         1149086751
# 6  Short offline       Completed: read failure       90%      3301         1153300035

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I'll reboot and run the memory test as well.

ewaller, where would those logs be? And yes, the smart results are above.

ewaller · 2012-07-06 02:23:45

They would be /var/log/everything.log or /var/log/system.log.But, based on your SMART output, I bet they are clean with respect to IO errors.

R00KIE · 2012-07-06 10:15:18

Based on this

197 Current_Pending_Sector  0x0032   198   198   000    Old_age   Always       -       353

I'd say there must be error reports of unreadable sectors and i/o errors in dmesg. Your could try to overwrite the whole disk and check if the Current_Pending_Sector attribute returns to zero. The best way would be to overwrite the whole disk (1) just to be sure but you may get away with it by filling your partition(s) with files full of zeros (2).

BIG WARNING: (1) WILL overwrite all your data.
(1) use as root: dd if=/dev/zero of=/dev/sda bs=10M
(2) you probably want to use this as root to make sure you really fill all the free space: dd if=/dev/zero of=/path/to/file bs=10M
where /path/to/file makes dd write inside every partition.

Judging by the smart self-test log I'd say the problem is caused by sectors allocated to either your /home and/or /archive. In any case if you have a spare disc you can use to backup everything I'd recommend method (1), that way you are sure you write to all sectors of the HD. An alternative that you may want to use to stress the drive a little bit is to use the badblocks program in write mode and let it do all the passes. At the end check for error reports from badblocks and also check the smart attributes.

ewaller · 2012-07-06 13:04:25

R00KIE wrote:

Based on this

197 Current_Pending_Sector  0x0032   198   198   000    Old_age   Always       -       353

Damm, I missed that

hoot_hoot · 2012-07-14 04:04:47

Thanks for the help, guys! I followed your advice, R00KIE, and running:

dd if=/dev/zero of=/dev/sda bs=10M

after backing up my stuff has left solved the problem. Thanks again!

I'm going to run a test on my second drive to see if it has any unreadable sectors, etc. and do the same.

Again, I apologize for the delay, I just got around to trying it today.

R00KIE · 2012-07-14 13:28:52

Don't forget to keep your backups up-to-date and keep an eye on the smart attributes for a while, pay special attention to the "Pre-fail" attributes, others you may want to keep an eye on are (in my opinion): "Reallocated_Event_Count", "Current_Pending_Sector" and "Offline_Uncorrectable", with the first being a bit worrisome(1) and the other two probably solvable by rewriting the affected sectors (or the whole disk for good measure ).

(1) Check [1] for a description of what each attribute means and decide for yourself which one(s) may be worth keeping an eye on. Also don't trust the smart parameters too much because sometimes they don't predict accurately enough when a drive is close to definitely failing.

[1] https://en.wikipedia.org/wiki/S.M.A.R.T.

Arch Linux

#1 2012-07-01 16:36:44

Disk R/W Speed Seems to Drop Randomly; System Lag

#2 2012-07-02 18:02:15

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#3 2012-07-02 22:51:08

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#4 2012-07-03 12:42:22

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#5 2012-07-04 03:44:05

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#6 2012-07-04 19:31:18

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#7 2012-07-04 20:44:30

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#8 2012-07-05 22:10:11

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#9 2012-07-06 02:23:45

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#10 2012-07-06 10:15:18

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#11 2012-07-06 13:04:25

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#12 2012-07-14 04:04:47

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

#13 2012-07-14 13:28:52

Re: Disk R/W Speed Seems to Drop Randomly; System Lag

Board footer