You are not logged in.
Hello Archers,
I use Arch Linux for a media server and love it! It is used mostly as a MythTV backend, but also MiniDlna music server and as a file server for NFSv3 and Samba. It is not used as a desktop at all as it it located above the ceiling tiles. There are 3 PCI DVB-C basic tuners capturing the DTV signals for MythTV. This setup is stable and functions flawlessly as it should if I disable one of the cards so that the 2 working PCI DVB cards are on different distinct IRQ's. I can even record up to 6 programs at once this way when each card records multiple channels on a single stream.
The reason I'm looking into the Linux-RT kernel is because problems arise when recording with 3 tuners at the same time. 2 of the PCI slots are "hardwired" to the same IRQ so that 2 of the DVB cards must share this IRQ. Now the same setup will work with 3 tuners but sometimes cause the system to stall when trying to use 2 PCI cards on the same IRQ. It is an intermittend problem as it often does just work for some reason. The errors always mention something about the IRQ.
So, I read about the Linux-RT kernel working differently with IRQ's and that it has superior latency performance. Considering that solid DVB tuning is known to be picky when it comes to latency issues, I'm wondering if the IRQ dilemna is getting in the way of hardware throughput and then using the Linux-RT kernel could help here?
I will try it out when I get a chance but possibly I might score some good advice here.
Thanks
Last edited by wdirksen (2012-06-13 12:18:07)
Research | Trial | Make Mistakes | Ask questions | Learn | Repeat
Offline
Try the BFS kernel scheduler first, and try reducing its rr_interval parameter, e.g.:
echo 4 > /proc/sys/kernel/rr_interval
Offline
Thanks Brebs.
I forgot to mention that I have been using the Linux-Ck Corex kernel with this setup and it didn't seem to make a difference. But, I didn't know about the tweek you are suggesting. I always just put in elevator=bfq and let it go. I will try this.
Last edited by wdirksen (2012-06-13 12:29:48)
Research | Trial | Make Mistakes | Ask questions | Learn | Repeat
Offline
Can also try binding the IRQ to one CPU, e.g.:
echo 2 > /proc/irq/19/smp_affinity
Offline
Thanks again. I will try both options in the course of next week.
Research | Trial | Make Mistakes | Ask questions | Learn | Repeat
Offline
Hi Brabs,
I don't know if you are still plugged into this thread anymore, but I'm curious what I should do with Irqbalance. I do have it activated as it seems to help a little bit withour solving this. I am putting your advice into action now.
Research | Trial | Make Mistakes | Ask questions | Learn | Repeat
Offline
Hi Brabs,
Hi wdickson ,
My feeling about irqbalance is that it's not worth the overhead of it running, and that the kernel does it anyway. On yer average dual or quad-core, anyway.
Take a look:
cat /proc/interrupts
Check whether any of your devices can use MSI.
Offline
Ok so this MSI stuff is completely alien to me. So I'll leave the output of cat /proc/interrupts below. It does have some mention of MSI. saa7146 is the chip on the TT C-1501 DVB cards
BTW I misstated myself in the beginning. One of the PCI slots shares IRQ with video and one of the USB hubs, not with another PCI slot. Nonetheless, the DVB tuner that is in that particular PCI slot gets in trouble. The DVB cards are identical and if you swap them, the problem follows the slot, not card.
But the good news is, I did lower the rr-interval as you suggested and did the following from the Arch Pro Audio Wiki to raise the other latencies and lower the latency of the saa7146's as follows:
sudo setpci -v -d *:* latency_timer=b0
sudo setpci -v -s 07:00.0 latency_timer=ff
sudo setpci -v -s 07:01.0 latency_timer=ff
sudo setpci -v -s 07:02.0 latency_timer=ff
then ran it through a torture test of 9 recording concurrent programs and it held up over 30 minutes! It might be solved already. Thanks in all cases.
[mythtv@server ~]$ cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 108 0 0 0 IO-APIC-edge timer
1: 928 0 0 0 IO-APIC-edge i8042
8: 1 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 3203174 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, saa7146 (2), nouveau
17: 280 0 0 0 IO-APIC-fasteoi firewire_ohci
18: 3646192 0 0 0 IO-APIC-fasteoi saa7146 (0)
19: 4228719 0 0 0 IO-APIC-fasteoi saa7146 (1)
22: 386 0 0 0 IO-APIC-fasteoi snd_hda_intel
23: 17831875 0 0 0 IO-APIC-fasteoi ehci_hcd:usb6
41: 1 0 0 0 PCI-MSI-edge xhci_hcd
42: 0 0 0 0 PCI-MSI-edge xhci_hcd
43: 0 0 0 0 PCI-MSI-edge xhci_hcd
44: 0 0 0 0 PCI-MSI-edge xhci_hcd
45: 0 0 0 0 PCI-MSI-edge xhci_hcd
46: 1 0 0 0 PCI-MSI-edge xhci_hcd
47: 0 0 0 0 PCI-MSI-edge xhci_hcd
48: 0 0 0 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 252599 0 0 0 PCI-MSI-edge ahci
52: 112362 0 0 0 PCI-MSI-edge ahci
53: 11 0 0 0 PCI-MSI-edge mei
54: 2393762 0 0 0 PCI-MSI-edge eth0
NMI: 1412 715 164 39 Non-maskable interrupts
LOC: 3547448 3056837 1056300 132906 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 1412 715 164 39 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RTR: 2 0 0 0 APIC ICR read retries
RES: 177644 125519 51907 25997 Rescheduling interrupts
CAL: 435 592 893 928 Function call interrupts
TLB: 7847 2556 2274 1077 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 61 61 61 61 Machine check polls
ERR: 0
MIS: 0
Last edited by wdirksen (2012-06-17 09:17:48)
Research | Trial | Make Mistakes | Ask questions | Learn | Repeat
Offline
Some of your interrupts are tied to CPU0, surprisingly. Not sure why.
I recommend moving IRQ 23 to a different CPU(s) - or yeah, try irqbalance.
Please use code tags, so we don't scratch our eyes out in pain:
[code]This is code[/code]
Offline
Hi Brebs,
I really appreciate this for the advice and the links as I'm learning alot here as well inproving the business at hand!!
I was always wondering how the code parts were boxed like that, now I know as I edited the above post. I will try to experiment further with your suggestions.
Research | Trial | Make Mistakes | Ask questions | Learn | Repeat
Offline
Just an update:
It turns out that the Linux-bfs kernel gives "smoothest" interaction when 3 DVB cards are tuning DTV simultaneously where the culprit seems to be the 3rd PCI slot which shares IRQ with 2 other devices. By smooth I mean less survivable delays and false starts when starting a new tuning job and certainly less non-survivable DVB crashes due to IRQ issues. The Linux-ck (BFS on or off) seems to help avoid the occaisional crash but operation is more bumpy and seems less stable. Because linux-bfs was built from scratch via AUR, I also tried the linux-ck built from AUR but it works the same as downloading from Graysky's CK repo. Lowering the rr timing as Brebs suggested makes this worse with linux-ck and seems to have no effect with linux-bfs. The setpci commands
sudo setpci -v -s 07:00.0 latency_timer=ff
certainly don't make a noticeable impact and at times seemed to have made it more bumpy but I don't know that for sure
For the fun of it I also tried the linux-rt and linux-rt-lts kernels with rtirq on and off but it turned out to be unusable for my system. The true DVB adaptors do work with these kernels, but the foldback simulated adaptors created by sasc-ng (which I must use) are not compatible with the RT kernels
Thanks again Brebs! It is better now and maybe someone will find this useful from a search
Last edited by wdirksen (2012-06-22 10:50:35)
Research | Trial | Make Mistakes | Ask questions | Learn | Repeat
Offline
You really should play with the CPU affinity of interrupts also. Having them *all* serviced by the one CPU core is very bad.
I know this by playing with *my* interrupts for Nvidia video and Intel HDA audio, in games.
Easy test:
echo 2 > /proc/irq/23/smp_affinity
Offline
You know I tried that for the IRQ 23 as you suggested. Funny thing is that it refuses to go completely to CPU 1, 2 or 3. When you ask it to move to any other CPU, it splits itself between CPU 0 and 1 only. That didn't seem to make difference, good or bad. Interresting when I use irqbalance it also splits it up between CPU 0 and 1 but splits up lots of other IRQ's as well across the board and that seemed to make it worse.
This is really interesting as it seems that the 2 things that have seemed to help is using linux-bfs kernel over the linux-ck kernel, and turning off irqbalance. But I haven't played around with changing the CPU for the other IRQ's. Could be interesting.
Last edited by wdirksen (2012-06-23 22:15:50)
Research | Trial | Make Mistakes | Ask questions | Learn | Repeat
Offline