You are not logged in.
A mini-project created after getting too annoyed by KDE's performance when copying big files or doing any other I/O intensive stuff.
Its a small daemon written in bash that utilizes the kernel cfq scheduler and the ionice command to reprioritize processes. It will not make your system faster but more interactive. All processes prioritized by the daemon will run at the lowest I/O priority (not CPU, I/O = hdd's and optical drives for example), which allows other (foreground) processes to gain more of the precious I/O.
Result: The system is still usable when copying 10 big files at the same time
Features:
- small and simple
- processes are ioniced once only and then kept in a blacklist, which is checked and rebuilt periodically.
This ensures that no process will be ioniced more than once
Installation:
AUR-Package: ioreniced
Usage:
- check /etc/conf.d/ioreniced
- run /etc/rc.d/ioreniced start
- watch /var/log/ioreniced.log for debug output
The kernel cfq scheduler should be activated by default on your Arch system, you can check that with:
$ cat /sys/block/[sh]d[a-z]*/queue/scheduler
noop anticipatory deadline [cfq]
If it isnt enabled, add this parameter to your bootloader config: elevator=cfq
So far it got only tested on my system and it does a fine job. I would like to know how it works out for others and maybe improve it or fix bugs, so feedback is really appreciated
want a modular and tweaked KDE for arch? try kdemod
Offline
There's no check if it is already running. There's no documented option to disable logging.
Are there any advantages over ioniced other than being maintained, being simpler and having a more complete process list?
I applaud your effort, nevertheless.
Offline
There's no check if it is already running. There's no documented option to disable logging.
Will be fixed with the next update.
Are there any advantages over ioniced other than being maintained, being simpler and having a more complete process list?
I am the AUR maintainer of ioniced (that hasnt been updated since 2005), and from my experience it is a bit buggy and it renices only root processes. Its also written in perl (i prefer bash) and more of a quick hack, and i wanted something reliable that is easy to handle. There are also no config files in ioniced, you have to edit its source to add processes for example.
About the "more complete process" list: I've just added some defaults, its up to the user and what he wants to have in this list. Having the most common "i/o intensive" commands as a default is ok from my pov.
want a modular and tweaked KDE for arch? try kdemod
Offline
About the "more complete process" list: I've just added some defaults, its up to the user and what he wants to have in this list. Having the most common "i/o intensive" commands as a default is ok from my pov.
Yeah, of course I meant the default list - yours is more complete, which I regard as a good thing.
ioniced seemed to work okay for me (also for user processes), but so does ioreniced - gonna stick with your implementation.
Offline
Yeah, of course I meant the default list - yours is more complete, which I regard as a good thing.
Ah, totally misunderstood you there - the adventures of a non-native speaker
0.1.2 just got uploaded with a logging option and an improved initscript.
want a modular and tweaked KDE for arch? try kdemod
Offline
Thanks for this! I am using it on my Fedora 11 LiveUSB! It really helps keep the system from freezing when using yum/packagekit.
Offline
It keeps spiking my CPU every time it polls, and makes my CPU fan act like a ship out at sea, spinning up and down with every spike.
Last edited by DevoidOfWindows (2009-06-06 04:05:12)
Offline
That filtering loop causes a lot of processes to be spawned incessantly - I really don't like that. Perhaps there's a different way?
Offline
Here are my slightly modified files for Fedora (and probably other systems as well):
/etc/ioreniced.conf:
# the priority to be used:
# 1 -> real time -> DONT use this unless you know what you are doing
# 2 -> best effort -> default setting for all processes
# 3 -> idle -> lowest priority, shouldnt kill the system while
doing
# i/o-intensive stuff
IONICE_PRIO="3"
# all processes you want to run at the given i/o priority
# every "hit" in the process list will get the priority
# means adding "rar" here works for "rar" and "unrar"
IONICE_ME=('kio_file'
'find'
'cp'
'mv'
'mc'
'tar'
'updatedb'
'mandb'
'gzip'
'bzip2'
'rar'
'zip'
'7z'
'mkisofs'
'mksquashfs'
'gpk')
# how often should we check for new processes and apply some io niceness
# (value = seconds)
POLLING_INTERVAL="2"
# logging options
# (value = yes/no)
ENABLE_LOGGING="yes"
LOG_FILE="/var/log/ioreniced.log"
/etc/rc.d/init.d/ioreniced
#!/bin/bash
. /etc/rc.d/rc
. /etc/rc.d/init.d/functions
PATH=/sbin:/bin:/usr/sbin:/usr/bin
DESC="ioReniceD I/O priority Daemon"
NAME="ioreniced"
DAEMON="/sbin/$NAME"
PIDFILE="/var/run/ioreniced.pid"
case "$1" in
start)
stat_busy "Starting $DESC"
if [ -e "$PIDFILE" ]
then
stat_busy "$DESC already running (or zombie PID
file in /var/run/)"
stat_fail
else
$DAEMON &>/dev/null &
PID="$!"
sleep 1
if kill -0 $PID # pid check
then
add_daemon ioreniced
echo $PID > $PIDFILE
stat_done
else
stat_busy "$DESC isn't starting
up properly. Check your Config file."
stat_fail
fi
fi
;;
stop)
stat_busy "Stopping $DESC"
if [ -f $PIDFILE ]; then
kill -15 `cat $PIDFILE` &>/dev/null
rm $PIDFILE
else
stat_busy "$DESC not running"
stat_fail
exit
fi
if [ $? -gt 0 ]; then
stat_fail
else
rm_daemon ioreniced
stat_done
fi
;;
restart)
$0 stop
sleep 1
$0 start
;;
*)
echo "usage: $0 {start|stop|restart}"
esac
/usr/bin/ioreniced
#!/bin/bash
#
# ioReniceD 0.1.2
#
# GPL - jan.mette@berlin.de
#############################################################################
# SETUP
#############################################################################
# load config
source /etc/ioreniced.conf
# initialize pidlist for garbage collection
IONICED_LIST=""
# delete old logs
rm -rf ${LOG_FILE} &>/dev/null
# set traps
trap 'echo "${DATE} >>>> Reloading Config">>${LOG_FILE}; source /etc/conf.d/ioreniced' 1
trap 'rm -f /var/run/ioreniced.run; exit' 0
trap 'rm -f /var/run/ioreniced.run; exit' 3
trap 'rm -f /var/run/ioreniced.pid; exit' 15
#############################################################################
# MAIN LOOP
#############################################################################
while :
do
# update date var
DATE=`date '+%b%e %H:%M:%S'`
# create logfile if it doesnt exist
if [ "${ENABLE_LOGGING}" = "yes" ]
then
if [ ! -e "${LOG_FILE}" ]
then
touch ${LOG_FILE}
echo " ">>${LOG_FILE}
echo "-------------------------------------">>${LOG_FILE}
echo " ioReniceD Logfile ">>${LOG_FILE}
echo "-------------------------------------">>${LOG_FILE}
echo " ">>${LOG_FILE}
fi
fi
# garbage collection
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >>>> Running Garbage Collector">>${LOG_FILE}
fi
for pid2 in ${IONICED_LIST}
do
check_running=`ps -A | grep ${pid2}`
if [ -z "$check_running" ]
then
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >> PID ${pid2} not running anymore, removing from process list">>${LOG_FILE}
fi
IONICED_LIST=`echo ${IONICED_LIST} | sed "s/${pid2}//g"`
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >> PID LIST: ${IONICED_LIST}">>${LOG_FILE}
fi
fi
done
# walk through the list of processes we want to ionice
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >>>> Ionicing Processes">>${LOG_FILE}
fi
for item in `echo ${IONICE_ME[*]}`
do
# filter process list and remove stuff we dont want to ionice. just to be sure we dont hit anything cruicial ;) needs to be simplified
for pid in `ps -A | grep -e "${item}" | grep -v "acpi" | grep -v "getty" | grep -v "aio" | grep -v "ata" | grep -v "avahi" | grep -v "cdemud" | grep -v "console-kit" | grep -v "crond" | grep -v "cupsd" | grep -v "dbus" | grep -v "chroot" | grep -v "events" | grep -v "hald" | grep -v "audio0" | grep -v "init" | grep -v "irqbalance" | grep -v "kblockd" | grep -v "kcrypt" | grep -v "kded" | grep -v "kdeinit" | grep -v "kdm" | grep -v "kdmflush" | grep -v "kgameport" | grep -v "khelper" | grep -v "khubd" | grep -v "kjournal" | grep -v "klauncher" | grep -v "knotify4" | grep -v "kondemand" | grep -v "kpsmoused" | grep -v "kseriod" | grep -v "ksmserver" | grep -v "ksoftirqd" | grep -v "kstriped" | grep -v "ksuspend" | grep -v "kswapd0" | grep -v "kthreadd" | grep -v "kwrited" | grep -v "migration" | grep -v "pdflush" | grep -v "reiserfs" | grep -v "scsi" | grep -v "sshd" | grep -v "startkde" | grep -v "sync" | grep -v "syslog-ng" | grep -v "truecrypt" | grep -v "udevd" | grep -v "watchdog" | grep -v "work_on_cpu" | sed "s/[^ 0-9].*//g"`
do
# check if we already have ionized this process
PID_TMP=`echo ${IONICED_LIST} | grep -e "${pid}"`
if [ -z "${PID_TMP}" ]
then
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >> ionicing new process ${item} (PID ${pid}) with PRIO ${IONICE_PRIO}">>${LOG_FILE}
fi
/usr/bin/ionice -c${IONICE_PRIO} -p${pid} &>/dev/null
IONICED_LIST="${IONICED_LIST} ${pid}"
else
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >> process ${item} (PID ${pid}) already ioniced with PRIO ${IONICE_PRIO}, skipping">>${LOG_FILE}
fi
fi
done
done
# delete logfile it if its size is over 1mib
if [ -e "${LOG_FILE}" ]
then
LOGSIZE=$(stat -c%s "${LOG_FILE}")
if [ "${LOGSIZE}" -gt 1048576 ]
then
rm -rf ${LOG_FILE}
fi
fi
# marker
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >>>> ---------------------------------MARK">>${LOG_FILE}
fi
# polling interval
sleep ${POLLING_INTERVAL}
done
Overall changes: changed ioreniced.conf location in /usr/bin/ioreniced to the more appropriate /etc directory (Fedora doesn't include a conf.d directory). Put the daemon in /etc/rc.d/init.d, and changed the rc.conf and function files to the correct locations. Added GPK to the config, as that is a big eater of IO cycles.
Last edited by smartboyathome (2009-06-09 04:30:42)
Offline
It keeps spiking my CPU every time it polls, and makes my CPU fan act like a ship out at sea, spinning up and down with every spike.
CPU spikes 0.1.2
CPU spikes 0.1.3
Not perfect yet, but i'll see what i can do more
Perhaps i'll introduce a bruteforce mode, without any checking(s).
That filtering loop causes a lot of processes to be spawned incessantly - I really don't like that. Perhaps there's a different way?
Also fixed, now the maximum amount of concurrently spawned processes is 2.
want a modular and tweaked KDE for arch? try kdemod
Offline
yeah... thank you thank you thank you!!!
It work very fine
Offline
Really nice script there, funkyou
However, I'd have one feature request. Could it auto-detect that the laptop in on batery and skip the checks? It seems to wake up the processor quite often, as can be seen in powertop.
Offline
Thanks
However, I'd have one feature request. Could it auto-detect that the laptop in on batery and skip the checks? It seems to wake up the processor quite often, as can be seen in powertop.
Hmmm, is there a generic way to do that? From my research for the Chakra liveCD i know that there are several ways to detect a laptop and if its running on battery currently etc, but all of these ways are too big codewise to put them into this daemon imho.
Tried 0.1.4? It has a new brute force mode (without any checkings) that shouldnt push the cpu that much. Also try to play with the polling interval, maybe a higher value is better for you.
want a modular and tweaked KDE for arch? try kdemod
Offline
It works great! Thanks a lot!
With ioReniceD, even Pacman witha a /var on XFS (silly mistake) works like a dream
Offline
It uses a lot of CPU because it calls ps -A for every item in the list, instead use something like
config:
IONICE_ME='cp
mv'
at program start, if you want it to match the full process name (it appears as "(cp)" in /proc/#/stat)
IONICE_ME=$(echo "$IONICE_ME" | sed 's/.*/(&)/')
and for the loop:
for pid in $(grep -Fh "$IONICE_ME" /proc/[0-9]*/stat | cut -d' ' -f1); do
ionice -c ...
done
edit, or even: ionice -c 3 -p $(grep -Fh "$IONICE_ME" /proc/[0-9]*/stat | cut -d' ' -f1)
(actually that won't work, because it won't continue if one of them fails)
(ahh there is -t for that, so this will work)
ionice -c 3 -t -p $(grep -Fh "$IONICE_ME" /proc/[0-9]*/stat | cut -d' ' -f1)
Last edited by Procyon (2009-09-03 14:20:59)
Offline
Thanks. I dont have much time currently, or better: close to zero time, so my suggestion box only accepts patches at the moment
If you manage to get a patch ready, dont hesitate to post it here and i'll update the AUR package
want a modular and tweaked KDE for arch? try kdemod
Offline
It uses a lot of CPU because it calls ps -A for every item in the list, instead use something like
config:
IONICE_ME='cp
mv'at program start, if you want it to match the full process name (it appears as "(cp)" in /proc/#/stat)
IONICE_ME=$(echo "$IONICE_ME" | sed 's/.*/(&)/')and for the loop:
for pid in $(grep -Fh "$IONICE_ME" /proc/[0-9]*/stat | cut -d' ' -f1); do
ionice -c ...
doneedit, or even: ionice -c 3 -p $(grep -Fh "$IONICE_ME" /proc/[0-9]*/stat | cut -d' ' -f1)
(actually that won't work, because it won't continue if one of them fails)
(ahh there is -t for that, so this will work)
ionice -c 3 -t -p $(grep -Fh "$IONICE_ME" /proc/[0-9]*/stat | cut -d' ' -f1)
I realize this is fairly old by now, but could you please consider turning this into a working patch? Alternatively, do you have a working version that you could PM me so that I could submit a patch? It would be a very useful improvement for the script.
Offline
Hi again!
I have been playing around with various methods to get the PIDs and Procyon's suggestion to grep them from the /proc tree seems to be the most efficient solution. Judging from the sys time used to extract the PIDs, normal grep seems even more efficient than pgrep, which was specifically made for PID grepping :-D
However, I never grokked the way Procyon suggested we use $IONICE_ME with a list of processes in the grep statement. Can grep really match a list of words in one statement like that? So, I tried egrep instead with a little hack. When ioreniced starts, we build a regular expression that matches any of our process names that we specify in the configuration file (cp|mv|kio_file... and so on). We then run this expression on the stat files in /proc. It seems to be a very efficient solution, as visualized by my CPU usage plots in GkrellM:
Original ioreniced v0.1.4:
The ticks show to ioreniced waking up and running ps -A for every potential process in $IONICE_ME every two seconds
Modified ioreniced:
We see no ticks with the modified version, using the same scale.
The main question is how long expressions egrep can parse. It works fine with the standard configuration file, and I would guess it can do much longer.
Btw, the original version is too relaxed when parsing the output from ps to find matching pids. "cp" matches not only the process cp, but any process name that contains the string cp somewhere. As a side effect, this is also fixed in the modified version, which is coming in the mail funkyou :-)
Offline
If anyone else is interested in checking it out, this is the tweaked code for /sbin/ioreniced:
#!/bin/bash
#
# ioReniceD 0.1.4
#
# GPL - jan.mette@berlin.de
#############################################################################
# SETUP
#############################################################################
# load config
source /etc/conf.d/ioreniced
# initialize pidlist for garbage collection
if [ "${BEHAVIOR}" = "smart" ]
then
IONICED_LIST=""
fi
# Build egrep pattern by padding each item with parentheses: (item)
# These need to be pipe separated and backslashed to work as espected
# for egrep's pattern match and will look like this: \(item\)|
EGREP_PATTERN=""
for item in `echo ${IONICE_ME[*]}`
do EGREP_PATTERN="${EGREP_PATTERN}$(echo $item | sed 's/.*/\\(&\\)/')|"
done
# Remove trailing '|' pipe from the pattern
EGREP_PATTERN=$(echo ${EGREP_PATTERN%\|})
# delete old logs
rm -rf ${LOG_FILE} &>/dev/null
# set traps
trap 'echo "source /etc/conf.d/ioreniced' 1
trap 'rm -f /var/run/ioreniced.run; exit' 0
trap 'rm -f /var/run/ioreniced.run; exit' 3
trap 'rm -f /var/run/ioreniced.pid; exit' 15
#############################################################################
# MAIN LOOP
#############################################################################
while :
do
# update date var
DATE=`date '+%b%e %H:%M:%S'`
if [ "${BEHAVIOR}" = "smart" ]
then
# create logfile if it doesnt exist
if [ "${ENABLE_LOGGING}" = "yes" ]
then
if [ ! -e "${LOG_FILE}" ]
then
touch ${LOG_FILE}
echo " ">>${LOG_FILE}
echo "-------------------------------------">>${LOG_FILE}
echo " ioReniceD Logfile ">>${LOG_FILE}
echo "-------------------------------------">>${LOG_FILE}
echo " ">>${LOG_FILE}
fi
fi
# garbage collection
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >>>> Running Garbage Collector">>${LOG_FILE}
fi
for pid2 in ${IONICED_LIST}
do
check_running=`ps -A | grep ${pid2}`
if [ -z "$check_running" ]
then
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >> PID ${pid2} not running anymore, removing from process list">>${LOG_FILE}
fi
IONICED_LIST=`echo ${IONICED_LIST} | sed "s/${pid2}//g"`
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >> PID LIST: ${IONICED_LIST}">>${LOG_FILE}
fi
fi
done
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >>>> Ionicing Processes">>${LOG_FILE}
fi
# Use one round of egrep to filter through /proc/PID/stat to get pids of all processes we want to ionice
process_index=0
for pid in `egrep $EGREP_PATTERN /proc/[0-9]*/stat | cut -d' ' -f1 | cut -d':' -f2`
do
# check if we already have ionized this process
PID_TMP=`echo ${IONICED_LIST} | grep -e "${pid}"`
if [ -z "${PID_TMP}" ]
then
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >> ionicing new process ${IONICE_ME[$process_index]} (PID ${pid}) with PRIO ${IONICE_PRIO}">>${LOG_FILE}
fi
/usr/bin/ionice -c${IONICE_PRIO} -p${pid} &>/dev/null
IONICED_LIST="${IONICED_LIST} ${pid}"
else
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >> process ${IONICE_ME[$process_index]} (PID ${pid}) already ioniced with PRIO ${IONICE_PRIO}, skipping">>${LOG_FILE}
fi
fi
$process_index++
done
# delete logfile it if its size is over 1mib
if [ -e "${LOG_FILE}" ]
then
LOGSIZE=$(stat -c%s "${LOG_FILE}")
if [ "${LOGSIZE}" -gt 1048576 ]
then
rm -rf ${LOG_FILE}
fi
fi
# marker
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >>>> ---------------------------------MARK">>${LOG_FILE}
fi
fi
if [ "${BEHAVIOR}" = "brute" ]
then
# create logfile if it doesnt exist
if [ "${ENABLE_LOGGING}" = "yes" ]
then
if [ ! -e "${LOG_FILE}" ]
then
touch ${LOG_FILE}
echo " ">>${LOG_FILE}
echo "-------------------------------------">>${LOG_FILE}
echo " ioReniceD Logfile ">>${LOG_FILE}
echo "-------------------------------------">>${LOG_FILE}
echo " ">>${LOG_FILE}
fi
fi
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >>>> Ionicing Processes">>${LOG_FILE}
fi
# Use one round of egrep to filter through /proc/PID/stat to get pids of all processes we want to ionice
process_index=0
for pid in `egrep $EGREP_PATTERN /proc/[0-9]*/stat | cut -d' ' -f1 | cut -d':' -f2`
do
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >> ionicing process ${IONICE_ME[$process_index]} (PID ${pid}) with PRIO ${IONICE_PRIO}">>${LOG_FILE}
fi
/usr/bin/ionice -c${IONICE_PRIO} -p${pid} &>/dev/null
$process_index++
done
# delete logfile it if its size is over 1mib
if [ -e "${LOG_FILE}" ]
then
LOGSIZE=$(stat -c%s "${LOG_FILE}")
if [ "${LOGSIZE}" -gt 1048576 ]
then
rm -rf ${LOG_FILE}
fi
fi
# marker
if [ "${ENABLE_LOGGING}" = "yes" ]
then
echo "${DATE} >>>> ---------------------------------MARK">>${LOG_FILE}
fi
fi
# polling interval
sleep ${POLLING_INTERVAL}
done
Offline
Alas, don't expect a response from funkyou - he passed away.
Offline
Oops. Boy, do I feel stupid. I never interacted with him before but should have recognized his name from the sad news post on the front page since Spring. I am deeply sorry if my ignorance has hurt anyone.
Unless anyone else feels like stepping in, I can volunteer to maintain that package in AUR. How do I go about doing that?
Offline
Just click on "Adopt packages" on ioreniced's page in AUR.
Offline