schednice - verynice with ionice && schedtool support

Cloudef · 2011-12-07 16:05:25

Hello,

Did a fork of verynice that also handles ionice levels, and lets you set up CPU affinity masks for programs.
Found a need for this as running Ys Origins under wine caused small stops every few seconds.

Hopefully you find it useful as well.

Project was renamed to schednice, after rewriting bunch of original verynice's code to fit my needs better.
There is no currently PKGBUILD avaivable on AUR, since stuff is still experimental.

This means it can make your system responsivity worse, actually than better.
I've had few reports about freezing and momentular stops with verynice-fork, and they can affect schednice too.

Verynice fork with ionice && schedtool support.

NOTE: This daemon is still highly experimental!
      It may not give your system any better responsivity,
      or in contraction make it worse.

Ionices processes depending how they are classified.
   badexe    - idle
   goodexe   - realtime (with prioritory 4)

While leaving everything else (or immuned) to none.

Sets following scheduling modes depending how they are classified.
   goodexe   - SCHED_ISO (only BFS scheduler)
   badexe    - SCHED_IDLE

When process is considered high load by schednice:
   io    => idle
   sched => SCHED_IDLE

While leaving everything else (or immuned) to default.

Also intruducing new parameters, affinityexe, ioniceexe, schedexe.

As you know, goodexe already specifies default ionice mode and sched class.
If you specify that same applications using either schedexe or ioniceexe,
it will override these defaults, which can be useful.

For example to make firefox run only on CPU0:
affinityexe "firefox" 0x1

For example to make your porn crawler run with SCHED_IDLE mode:
schedexe "porn_crawl_v2" 5 0

NOTE: You should use SCHED_FIFO and SCHED_RR with care.
      BFS provides SCHED_ISO

For example to make your porn crawler to be ioniced to idle class:
ioniceexe "porn_crawl_v2" 3 0

Both, schedexe and ioniceexe takes two arguments.
However the last one is ignored unless the scheduling class needs a prioritory argument.

EXPERIMENTAL:
   Automatic scaling with ionice depending on load.
   Automatic scheduling mode switching depending on load.

DONE:
   Add ionice support.
   Add schedtool support.
   Add CPU affinity list for setting affinity masks using schedtool.
   Get rid of the generated configuration code.
   Add seperate ionice list with parameters.

Solaris specific code is dropped,

This fork is being developed on kernel with -ck patchset, so all testing is done with -ck kernel. However it should behave and work similar with vanilla kernel.

Reference for scheduling modes:
http://ck.wikia.com/wiki/SchedulingPolicies
man schedtool

This is fork of verynice
http://thermal.cnde.iastate.edu/~sdh4/verynice/
http://www.tam.cornell.edu/~sdh4/verynice/ (Old link?)

AUR: none, ATM

Experimental PKGBUILD:
http://cloudef.eu/armpit/schednice.tar.gz

Github:
https://github.com/Cloudef/schednice

verynice-fork AUR (old):
https://aur.archlinux.org/packages.php?ID=54680

Last edited by Cloudef (2012-03-07 08:22:51)

milomouse · 2011-12-07 23:04:11

I had used previous version and hacky script to mess with 'schedtool' and 'ionice' so this is just perfect! I uploaded a package to the AUR. If you want to adopt the package let me know and I'll disown it so you can go ahead and do that. Thanks for this.
https://aur.archlinux.org/packages.php?ID=54680

Cloudef · 2011-12-08 06:51:32

I've still got plans for adding custom ionice list similar to affinityexe, and maybe look if it's possible to calculate IO usage beetwen time differences. This would allow me to do automatic scaling depending on load with ionice, similar to verynice does for processes already.

Also am including this:
http://ck.wikia.com/wiki/SchedulingPolicies

Which will come with the custom ionice list update.
I'm wondering if I should do dynamic switching for the scheduling polices as well, or just assing them if they are defined in configuration file? eg. SCHED_RR for goodexe and SCHED_BATCH for hungryexe.

I will provide custom list for these too thought, schedexe which will let you set custom parameters.

milomouse wrote:

I had used previous version and hacky script to mess with 'schedtool' and 'ionice' so this is just perfect! I uploaded a package to the AUR. If you want to adopt the package let me know and I'll disown it so you can go ahead and do that. Thanks for this.
https://aur.archlinux.org/packages.php?ID=54680

Oh, din't expect AUR package so soon. Glad you like it, however. I did some makefile changes and included the init script from the verynice AUR package in git that might break the AUR package.

Thought looking at the PKGBUILD, you don't call make install at all, so it should still work just fine. So if that's the case then nothing needs to be changed. (I'll test and use it later too, whenever I get home , been on vim+ssh dose recently)

Cheers, and thanks

E: Oh also about the adoption. If you feel like you don't wan't to bother maintaining the package, sure I can adopt it.

Last edited by Cloudef (2011-12-08 08:00:02)

Cloudef · 2011-12-08 09:26:54

Added ioniceexe and schedexe lists for defining custom scheduling modes and ionice classes.
Also sets everything back to default, when verynice is quit.

I'll create seperate branch to experiement with the dynamic switching for ionice and schedtool next, so if it's not good or something goes terribly wrong. The current codebase stays on the master branch.

Last edited by Cloudef (2011-12-08 09:27:22)

Cloudef · 2011-12-16 18:18:05

Merged the dynamic branch to the master and added ioprio value for default goodexe ionice priority.
The dynamic scaling is by default turned off, but there is define to turn it on.

Cloudef · 2011-12-18 10:35:48

Some minor changes:
- Reduced default RR_PRIO value to 10, using 20 seemed bit too overkill.
- Added information about overriding the default parameters goodexe, batchexe and buddies can set for single program.
- Added ioniceexe "/usr/bin/Xorg" 0 0 to default configuration

Cloudef · 2012-01-04 20:07:38

Some changes:
- Fine tuned default parameters.
- Idlekarma, processes that use very few CPU or are considered Idle gets reniced to higher value.
- Configuration options for idlekarma.
- Dynamic SCHED scaling, processes that go above the level considered idle, gets schedtool'd with -D class. Other processes are -N
- Dynamic IO scaling, same as above.
- Some code fixes and more verbosing.

I did some testing today using htop and running few CPU heavy applications and the responsiveness with new changes resulted in quite good results.
If you notice any odd behavior, keep me notified.

--
Also I adopted the AUR package. Am using it now personally and fixed some issues with it.

Last edited by Cloudef (2012-01-04 20:09:33)

el mariachi · 2012-01-20 22:38:59

setting Luakit as goodexe makes everything a bit unstable, and if I open a facebook tab the whole computer freezes for a couple of seconds and the whole experience is veeeerrryyy slow. disabling verynice fixes this. is this supposed to happen?

Cloudef · 2012-01-29 18:17:42

el mariachi wrote:

setting Luakit as goodexe makes everything a bit unstable, and if I open a facebook tab the whole computer freezes for a couple of seconds and the whole experience is veeeerrryyy slow. disabling verynice fixes this. is this supposed to happen?

goodexe also sets the SCHED_RR which is only recommeded to audio/video applications really.
This is known to cause problems in eg. opera too. If you want to have goodexe flag on luakit, but disable SCHED_RR.

You could do this:

goodexe "luakit"
schedexe "luakit" 0 0 # Sets luakit's scheduling class to SCHED_NORMAL

I might still change the defaults in future.
Also need to write man pages in future.

EDIT:
I updated a code bit, I added limit for the dynamic idle scaling. So if you don't want all your processes that are considered to be idle (uses low CPU) to be niced far back as 19 (now 15 in the recent commit), you can change that.

The option for this is 'limit', merge the pacnew file

Last edited by Cloudef (2012-01-29 18:33:31)

Cloudef · 2012-02-14 14:03:11

Some changes:

- Make verynice detect threads/tasks
- SCHED_RR no longer set on goodexe flag, if you really want it use schedexe manually.
- Few fixes on dynamic scaling

el mariachi · 2012-03-06 20:53:41

Cloudef I noticed that a badexe is "schedtoold" does this mean I need schedtoold or ist schedtool okay? Noticed this now and was wondering...

Cloudef · 2012-03-07 08:14:18

el mariachi wrote:

Cloudef I noticed that a badexe is "schedtoold" does this mean I need schedtoold or ist schedtool okay? Noticed this now and was wondering...

Schetool'd, seems like i have typo there , btw. I rewrote lot's of verynice code and it's now https://github.com/Cloudef/schednice schednice,
I still don't have PKGBUILD on aur. I've noticed some users having freezing(momentual stopping) issues that I can't reproduce, but am trying to find out the cause, however these apply verynice-fork too.

If people are willing to test the current experimental PKGBUILD, it's here: http://cloudef.eu/armpit/schednice.tar.gz
Note: schednice doesn't need schedtool or ionice anymore, it uses kernel calls directly. It also doesn't conflict with verynice or verynice-fork, but it's not good idea to have them running at the same time though.

Also schednice uses SCHED_ISO for goodexe's which is kind of SCHED_RR, but less agressive and should not cause issues like with luakit before. However SCHED_ISO is BFS only scheduler policy.

Most of configuration options are stripped out, and there's pretty much only badexe, goodexe and some tunables (+ manual settings for scheduler policies and such through schedexe, ioniceexe and affinityexe)
And here is simple test to see if it works correctly on your system.

1. Open 2 terminals.
2. Make sure your shell has no scheduling policies by doing following
3. sudo schedtool -N $$; ionice -c0 -p $$
4. start stress with suitable options (I used, stress --cpu 1 --io 1 --vm 1 --vm-bytes 128M)
NOTE: ^ Don't go overkill with above, even schednice can't save you from frozen system.
If you are unsure, use --timeout 5s option to make it stop stressing after 5 seconds.
5. Start video, I used 1080p Hi10p episode of Break Blade, with mplayer and running it totally from CPU.
6. You'll video playback is probably choppy now. If it isn't play with stress variables.
7. Start schednice on another terminal.
8. After few seconds video playback should be smooth.

Of course you'll get more faster effect when you badexe the CPU stressing process. (In this case 'stress')

Last edited by Cloudef (2012-03-08 15:50:18)

el mariachi · 2012-03-07 21:24:37

Sounds good to me. is the config in /etc/schednice.conf now?
ill test it when i get home

Cloudef · 2012-03-08 15:50:04

el mariachi wrote:

Sounds good to me. is the config in /etc/schednice.conf now?
ill test it when i get home

Yes, configuration is /etc/schednice.conf now.

Cloudef · 2012-03-12 21:09:19

I've updated http://cloudef.eu/armpit/schednice.tar.gz to build from git.

Lockheed · 2012-04-30 21:34:23

Hey Cloudef. Have you done anything new on schednice? I am still using the last version from my old thread. Not sure what are the new changes and if it is worth updating it (why it is not in AUR yet?).

I have been also thinking whether such program has any point, as linux kernel should be doing the scheduling work by itself. Do you find it inefficient?

el mariachi · 2012-05-01 09:40:36

I stopped using it actually. My system got more unresponsive with it. Maybe it's the dynamic mechanism.. I simply couldn't make "make" be in batch mode and carry on.

Arch Linux

#1 2011-12-07 16:05:25

schednice - verynice with ionice && schedtool support

#2 2011-12-07 23:04:11

Re: schednice - verynice with ionice && schedtool support

#3 2011-12-08 06:51:32

Re: schednice - verynice with ionice && schedtool support

#4 2011-12-08 09:26:54

Re: schednice - verynice with ionice && schedtool support

#5 2011-12-16 18:18:05

Re: schednice - verynice with ionice && schedtool support

#6 2011-12-18 10:35:48

Re: schednice - verynice with ionice && schedtool support

#7 2012-01-04 20:07:38

Re: schednice - verynice with ionice && schedtool support

#8 2012-01-20 22:38:59

Re: schednice - verynice with ionice && schedtool support

#9 2012-01-29 18:17:42

Re: schednice - verynice with ionice && schedtool support

#10 2012-02-14 14:03:11

Re: schednice - verynice with ionice && schedtool support

#11 2012-03-06 20:53:41

Re: schednice - verynice with ionice && schedtool support

#12 2012-03-07 08:14:18

Re: schednice - verynice with ionice && schedtool support

#13 2012-03-07 21:24:37

Re: schednice - verynice with ionice && schedtool support

#14 2012-03-08 15:50:04

Re: schednice - verynice with ionice && schedtool support

#15 2012-03-12 21:09:19

Re: schednice - verynice with ionice && schedtool support

#16 2012-04-30 21:34:23

Re: schednice - verynice with ionice && schedtool support

#17 2012-05-01 09:40:36

Re: schednice - verynice with ionice && schedtool support

Board footer