You are not logged in.

#1 2012-06-13 11:54:25

wdirksen
Member
From: New Zealand
Registered: 2012-02-23
Posts: 105

Advice if Linux-RT could be helpful (or harmful for that matter)

Hello Archers,

I use Arch Linux for a media server and love it! It is used mostly as a MythTV backend, but also MiniDlna music server and as a file server for NFSv3 and Samba. It is not used as a desktop at all as it it located above the ceiling tiles. There are 3 PCI DVB-C basic tuners capturing the DTV signals for MythTV. This setup is stable and functions flawlessly as it should if I disable one of the cards so that the 2 working PCI DVB cards are on different distinct IRQ's. I can even record up to 6 programs at once this way when each card records multiple channels on a single stream.

The reason I'm looking into the Linux-RT kernel is because problems arise when recording with 3 tuners at the same time. 2 of the PCI slots are "hardwired" to the same IRQ so that 2 of the DVB cards must share this IRQ. Now the same setup will work with 3 tuners but sometimes cause the system to stall when trying to use 2 PCI cards on the same IRQ. It is an intermittend problem as it often does just work for some reason. The errors always mention something about the IRQ.

So, I read about the Linux-RT kernel working differently with IRQ's and that it has superior latency performance. Considering that solid DVB tuning is known to be picky when it comes to latency issues, I'm wondering if the IRQ dilemna is getting in the way of hardware throughput and then using the Linux-RT kernel could help here?

I will try it out when I get a chance but possibly I might score some good advice here.

Thanks

Last edited by wdirksen (2012-06-13 12:18:07)


Research | Trial | Make Mistakes | Ask questions | Learn | Repeat

Offline

#2 2012-06-13 12:18:02

brebs
Member
Registered: 2007-04-03
Posts: 3,742

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

Try the BFS kernel scheduler first, and try reducing its rr_interval parameter, e.g.:

echo 4 > /proc/sys/kernel/rr_interval

Offline

#3 2012-06-13 12:22:26

wdirksen
Member
From: New Zealand
Registered: 2012-02-23
Posts: 105

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

Thanks Brebs.

I forgot to mention that I have been using the Linux-Ck Corex kernel with this setup and it didn't seem to make a difference. But, I didn't know about the tweek you are suggesting. I always just put in elevator=bfq and let it go. I will try this.

Last edited by wdirksen (2012-06-13 12:29:48)


Research | Trial | Make Mistakes | Ask questions | Learn | Repeat

Offline

#4 2012-06-13 17:21:41

brebs
Member
Registered: 2007-04-03
Posts: 3,742

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

Can also try binding the IRQ to one CPU, e.g.:

echo 2 > /proc/irq/19/smp_affinity

Offline

#5 2012-06-14 08:09:24

wdirksen
Member
From: New Zealand
Registered: 2012-02-23
Posts: 105

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

Thanks again. I will try both options in the course of next week.


Research | Trial | Make Mistakes | Ask questions | Learn | Repeat

Offline

#6 2012-06-16 08:01:00

wdirksen
Member
From: New Zealand
Registered: 2012-02-23
Posts: 105

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

Hi Brabs,

I don't know if you are still plugged into this thread anymore, but I'm curious what I should do with Irqbalance. I do have it activated as it seems to help a little bit withour solving this. I am putting your advice into action now.


Research | Trial | Make Mistakes | Ask questions | Learn | Repeat

Offline

#7 2012-06-16 08:47:46

brebs
Member
Registered: 2007-04-03
Posts: 3,742

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

wdirksen wrote:

Hi Brabs,

Hi wdickson wink ,

My feeling about irqbalance is that it's not worth the overhead of it running, and that the kernel does it anyway. On yer average dual or quad-core, anyway.

Take a look:

cat /proc/interrupts

Check whether any of your devices can use MSI.

Offline

#8 2012-06-16 19:58:00

wdirksen
Member
From: New Zealand
Registered: 2012-02-23
Posts: 105

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

Ok so this MSI stuff is completely alien to me. So I'll leave the output of cat /proc/interrupts below. It does have some mention of MSI. saa7146 is the chip on the TT C-1501 DVB cards

BTW I misstated myself in the beginning. One of the PCI slots shares IRQ with video and one of the USB hubs, not with another PCI slot. Nonetheless, the DVB tuner that is in that particular PCI slot gets in trouble. The DVB cards are identical and if you swap them, the problem follows the slot, not card.

But the good news is, I did lower the rr-interval as you suggested and did the following from the Arch Pro Audio Wiki to raise the other latencies and lower the latency of the saa7146's as follows:

sudo setpci -v -d *:* latency_timer=b0
sudo setpci -v -s 07:00.0 latency_timer=ff
sudo setpci -v -s 07:01.0 latency_timer=ff
sudo setpci -v -s 07:02.0 latency_timer=ff

then ran it through a torture test of 9 recording concurrent programs and it held up over 30 minutes! It might be solved already. Thanks in all cases.

[mythtv@server ~]$ cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       
   0:        108          0          0          0   IO-APIC-edge      timer
   1:        928          0          0          0   IO-APIC-edge      i8042
   8:          1          0          0          0   IO-APIC-edge      rtc0
   9:          0          0          0          0   IO-APIC-fasteoi   acpi
  16:    3203174          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, saa7146 (2), nouveau
  17:        280          0          0          0   IO-APIC-fasteoi   firewire_ohci
  18:    3646192          0          0          0   IO-APIC-fasteoi   saa7146 (0)
  19:    4228719          0          0          0   IO-APIC-fasteoi   saa7146 (1)
  22:        386          0          0          0   IO-APIC-fasteoi   snd_hda_intel
  23:   17831875          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb6
  41:          1          0          0          0   PCI-MSI-edge      xhci_hcd
  42:          0          0          0          0   PCI-MSI-edge      xhci_hcd
  43:          0          0          0          0   PCI-MSI-edge      xhci_hcd
  44:          0          0          0          0   PCI-MSI-edge      xhci_hcd
  45:          0          0          0          0   PCI-MSI-edge      xhci_hcd
  46:          1          0          0          0   PCI-MSI-edge      xhci_hcd
  47:          0          0          0          0   PCI-MSI-edge      xhci_hcd
  48:          0          0          0          0   PCI-MSI-edge      xhci_hcd
  49:          0          0          0          0   PCI-MSI-edge      xhci_hcd
  50:          0          0          0          0   PCI-MSI-edge      xhci_hcd
  51:     252599          0          0          0   PCI-MSI-edge      ahci
  52:     112362          0          0          0   PCI-MSI-edge      ahci
  53:         11          0          0          0   PCI-MSI-edge      mei
  54:    2393762          0          0          0   PCI-MSI-edge      eth0
 NMI:       1412        715        164         39   Non-maskable interrupts
 LOC:    3547448    3056837    1056300     132906   Local timer interrupts
 SPU:          0          0          0          0   Spurious interrupts
 PMI:       1412        715        164         39   Performance monitoring interrupts
 IWI:          0          0          0          0   IRQ work interrupts
 RTR:          2          0          0          0   APIC ICR read retries
 RES:     177644     125519      51907      25997   Rescheduling interrupts
 CAL:        435        592        893        928   Function call interrupts
 TLB:       7847       2556       2274       1077   TLB shootdowns
 TRM:          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0   Machine check exceptions
 MCP:         61         61         61         61   Machine check polls
 ERR:          0
 MIS:          0

Last edited by wdirksen (2012-06-17 09:17:48)


Research | Trial | Make Mistakes | Ask questions | Learn | Repeat

Offline

#9 2012-06-16 23:56:46

brebs
Member
Registered: 2007-04-03
Posts: 3,742

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

Some of your interrupts are tied to CPU0, surprisingly. Not sure why.

I recommend moving IRQ 23 to a different CPU(s) - or yeah, try irqbalance.

Please use code tags, so we don't scratch our eyes out in pain:

[code]This is code[/code]

Offline

#10 2012-06-17 09:21:47

wdirksen
Member
From: New Zealand
Registered: 2012-02-23
Posts: 105

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

Hi Brebs,

I really appreciate this for the advice and the links as I'm learning alot here as well inproving the business at hand!!

I was always wondering how the code parts were boxed like that, now I know as I edited the above post. I will try to experiment further with your suggestions.


Research | Trial | Make Mistakes | Ask questions | Learn | Repeat

Offline

#11 2012-06-22 10:41:08

wdirksen
Member
From: New Zealand
Registered: 2012-02-23
Posts: 105

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

Just an update:

It turns out that the Linux-bfs kernel gives "smoothest" interaction when 3 DVB cards are tuning DTV simultaneously where the culprit seems to be the 3rd PCI slot which shares IRQ with 2 other devices. By smooth I mean less survivable delays and false starts when starting a new tuning job and certainly less non-survivable DVB crashes due to IRQ issues. The Linux-ck (BFS on or off) seems to help avoid the occaisional crash but operation is more bumpy and seems less stable. Because linux-bfs was built from scratch via AUR, I also tried the linux-ck built from AUR but it works the same as downloading from Graysky's CK repo. Lowering the rr timing as Brebs suggested makes this worse with linux-ck and seems to have no effect with linux-bfs. The setpci commands

sudo setpci -v -s 07:00.0 latency_timer=ff

certainly don't make a noticeable impact and at times seemed to have made it more bumpy but I don't know that for sure

For the fun of it I also tried the linux-rt and linux-rt-lts kernels with rtirq on and off but it turned out to be unusable for my system. The true DVB adaptors do work with these kernels, but the foldback simulated adaptors created by sasc-ng (which I must use) are not compatible with the RT kernels

Thanks again Brebs! It is better now and maybe someone will find this useful from a search

Last edited by wdirksen (2012-06-22 10:50:35)


Research | Trial | Make Mistakes | Ask questions | Learn | Repeat

Offline

#12 2012-06-22 10:59:48

brebs
Member
Registered: 2007-04-03
Posts: 3,742

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

You really should play with the CPU affinity of interrupts also. Having them *all* serviced by the one CPU core is very bad.

I know this by playing with *my* interrupts for Nvidia video and Intel HDA audio, in games.

Easy test:

echo 2 > /proc/irq/23/smp_affinity

Offline

#13 2012-06-23 22:11:07

wdirksen
Member
From: New Zealand
Registered: 2012-02-23
Posts: 105

Re: Advice if Linux-RT could be helpful (or harmful for that matter)

You know I tried that for the IRQ 23 as you suggested. Funny thing is that it refuses to go completely to CPU 1, 2 or 3. When you ask it to move to any other CPU, it splits itself between CPU 0 and 1 only. That didn't seem to make difference, good or bad. Interresting when I use irqbalance it also splits it up between CPU 0 and 1 but splits up lots of other IRQ's as well across the board and that seemed to make it worse.

This is really interesting as it seems that the 2 things that have seemed to help is using linux-bfs kernel over the linux-ck kernel, and turning off irqbalance. But I haven't played around with changing the CPU for the other IRQ's. Could be interesting.

Last edited by wdirksen (2012-06-23 22:15:50)


Research | Trial | Make Mistakes | Ask questions | Learn | Repeat

Offline

Board footer

Powered by FluxBB