Crossfire Troubles

schmidtbag · 2013-05-17 22:45:22

I recently got myself a 2nd HD5750 to put into Crossfire. As a disclaimer, I do have the 12.4 Catalyst drivers, crossfire works in Windows, and 3D acceleration works fine in linux.

Anyways, I've run the following commands as root:
aticonfig --cfa --adapter=0,1
aticonfig --cf on --adapter=0,1
reboot

And after rebooting, I could confirm the following using "aticonfig --lsch":
Multiple GPU chain for adapter 0, status: Crossfire is enabled
0. 01:00.0 ATI Radeon HD 5700 Series
1. 08:00.0 ATI Radeon HD 5700 Series

Using "aticonfig --lscc" returns:
Master adapter: 0. 01:00.0 ATI Radeon HD 5700 Series
Candidates: 1. 08:00.0 ATI Radeon HD 5700 Series

I am a little curious about the term "Candidate", because shouldn't the 2nd card also be a master device? Or at least an active slave?

No programs (such as the Unigine Heaven benchmark) appear to use the 2nd GPU, and if I check on "Enable CrossfireX" in the CCC and reboot, it un-checks itself.

As a side note, if I run "aticonfig --adapter=0,1 --initial", crossfire fails completely and forces dual-monitor mode.

pam · 2013-05-18 04:28:39

run the administrator version of amd ccc and enable xfire there...OR do it manually..

I think after setting the chains(you have already done that) you need put it on..

aticonfig --adapter=0,1 --crossfire=on

reboot and xfire will come on... )

Last edited by pam (2013-05-18 04:29:54)

schmidtbag · 2013-05-18 04:42:15

The "aticonfig --cf on --adapter=0,1" command does the exact same thing, it's just abbreviated (besides, I already ran it the way you mentioned before with no luck). I ran everything as root and as non-root, doesn't seem to make a difference.

Seeing as you got crossfire working on your system (judging by your sig), would you mind telling me the results of your lsch and lscc? I think that might help as a diagnostic, since there seems to be very little documentation on this matter.

Also, would you mind showing me your xorg.conf? Perhaps that might help pinpoint the issue.

Last edited by schmidtbag (2013-05-18 04:45:48)

pam · 2013-05-18 04:51:16

aticonfig --lscs

the above command will show you clearly if crossfire is enabled....

watch your gpu load also...

watch -n10 aticonfig --adapter=all --odgt --odgc

the opengl heaven benchmark on linux wont use crossfire.

Last edited by pam (2013-05-18 04:51:59)

schmidtbag · 2013-05-18 15:13:19

aticonfig --lscs
Candidate Combination:
Master: 1:0:0
Slave: 8:0:0
CrossFire is disabled on current device
CrossFire Diagnostics:
CrossFire can work with P2P mapping through GART
Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA

lsch still says crossfire is enabled. Any ideas on what I can do at this point? I still think it might be my xorg.conf, so I'll keep tinkering with it.
I believe I do have IOMMU enabled in BIOS (if that makes a difference), and I tried adding a 2nd crossfire bridge which didn't do anything.

If Heaven doesn't use it, do you know of a strenuous test that does? I heard furmark is finally ported to linux, though it must be still a little too new since it isn't in the AUR yet.

lano1106 · 2013-05-19 02:23:55

Hi,

I am very interested into this thread as I am having, more or less, the same problem:

lano1106@whippet2 ~ $ aticonfig --lscc

Master adapter: 0. 01:00.0 AMD Radeon HD 7900 Series
Candidates: none
Master adapter: 1. 02:00.0 AMD Radeon HD 7900 Series
Candidates: none
lano1106@whippet2 ~ $ aticonfig --lsch

Multiple GPU chain for adapter 0, status: Crossfire is enabled
0. 01:00.0 AMD Radeon HD 7900 Series
1. 02:00.0 AMD Radeon HD 7900 Series
lano1106@whippet2 ~ $ aticonfig --adapter=0 --crossfire=on
Crossfire chain(s) enabled
CrossFire does not support on this platform
Warning: X needs to be restarted before CrossFire changes take effect.
lano1106@whippet2 ~ $ aticonfig --lscs
Candidate Combination:
Master: 0:0:0
Slave: 0:0:0
CrossFire is disabled on current device
CrossFire Diagnostics:
CrossFire can work with P2P mapping through GART
Candidate Combination:
Master: 0:0:0
Slave: 0:0:0
CrossFire is disabled on current device
CrossFire Diagnostics:
CrossFire can work with P2P mapping through GART

To me, it seems that as soon as you get:

CrossFire does not support on this platform

it is pretty much game over. I wish aticonfig could be more explicit about what is the problem. Something that I have found is:

lano1106@whippet2 ~ $ lsmod | grep fglrx
fglrx 5176689 101
amd_iommu_v2 7252 1 fglrx
button 4669 1 fglrx

amd_iommu_v2 is used by fglrx and

lano1106@whippet2 ~ $ dmesg | grep IOMMU
...
[ 47.937885] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com>
[ 47.937886] AMD IOMMUv2 functionality not available on this system

The problem is that I have an Intel platform. Does that mean that Crossfire only works if your on an AMD platform. Can someone confirm or deny that hypothesis?

schmidtbag · 2013-05-19 02:58:03

I don't think it matters if you have an intel based system, though considering how little documentation there is I could be completely wrong. I believe the Intel equivalent of AMD's IOMMU is called VT-d, which should be enabled in your bios. Does your dmesg message show anything else? Mine has a similar output except it also says:
[ 1.093822] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
I can't seem to get a straight answer on whether or not IOMMU is needed, because I thought crossfire was doable on lower-end systems without IOMMU. Just as an FYI, your 2nd GPU can't have an active display in your xorg.conf. I found that out the hard way.

Anyways, I tried adding the 2nd GPU to the xorg.conf while excluding anything display related and that didn't seem to help or worsen my problem.

I'm sure lano and I would appreciate a step-by-step guide on how to get crossfire working on modern GPUs with the modern catalyst on Arch systems. People seem to claim it's as easy and straight-forward as just running 2 or 3 aticonfig commands but there's obviously a lot more behind it than that. Also, does kernel version matter?

schmidtbag · 2013-05-19 03:12:31

Haha!! I fixed it! I went into BIOS and disabled IOMMU and suddenly crossfire started working fine! Credits go to:
http://forums.gentoo.org/viewtopic-t-91 … art-0.html

The interesting thing is the only time I would ever care to re-enable IOMMU is if I could actually redirect one of my GPUs to a virtual machine, in which case crossfire would not be of any use.

Oh and in case anyone cares, Heaven DOES use crossfire. Using Ultra settings, vysnc, and 2x AA I got about 6FPS-15FPS on 1 GPU and about 10FPS with 30FPS with 2 GPUs. I'm not really impressed with the performance considering the HD5750 isn't a bad card, but it is nice to see I am getting the near-doubled performance I paid for.

Last edited by schmidtbag (2013-05-19 03:17:23)

pam · 2013-05-19 03:38:29

schmidtbag, i have never faced any problems with crossfire on linux ever.
I did exactly what you did, Good you got it sorted out. Infact i never had to touch xorg.conf
since kernel 2.6.32....joys of using amd.

lano1106 try to run the commands schmidtbag started this topic with(intel/amd cpu is irrelevant)...
here is a simple howto:
http://forumubuntusoftware.info/viewtop … f=7&t=2336

..and you can even change fan speeds for your 79xx(a one line aticonfig command).....search this forum itself.

schmidtbag · 2013-05-19 03:47:06

pam, I apparently did everything right all along, it was just IOMMU that was in the way; my original xorg.conf would've sufficed. I'm not sure if any of your previous systems used IOMMU (the one in your sig doesn't have it AFAIK), but it does apparently make a difference in linux. Hopefully this is something AMD will fix in the future, but as I said before, if I want IOMMU enabled, I'm not going to expect to be using crossfire anyway.

I am finding the lack of crossfire-enabled software to be a bit disappointing, but I guess I should be thankful it works at all.

lano1106 · 2013-05-19 05:35:11

I have done some progress.

My problem was that when doing 'aticonfig --adapter=0,1 --initial'. I did not review the resulting conf file and this is where the problem was. In the Section "ServerLayout", both cards where displayed. I have removed on line for the second screen and that looks like:

Section "ServerLayout"
Identifier "aticonfig Layout"
Screen 0 "aticonfig-Screen[0]-0" 0 0
EndSection

Once this change done, I have been able to activate Crossfire but now, after I enter my DE (xfce4) the x server works fine until I attempt to open an application. Then the X server becomes unresponsive.

If I top from a remote ssh connection, I see X using 100% CPU. I have tried to kill it, attach to it with gdb. Nothing works. Even reboot will not work. I need a hard reset to recover.

I have attempted several combinations in regard to IOMMU.

With/without intel_iommu
IBM Calgary IOMMU have been compiled out of my kernel.

No IOMMU setup that I have tried has changed anything to the freeze result that I am experiencing.

BTW, these cards are RadeonSI.

Last edited by lano1106 (2013-05-19 05:43:04)

pam · 2013-05-19 05:45:55

WARNING: Do NOT touch xorg.conf file. There is no editing work required. If you do edit the file and you obtain 'good' results, its probably nothing and will end up with a broken desktop most times.

Just keep the installation of the drivers straightforward and the OS too. Reboot twice for changes to take place.

For graphics cards like nvidia/amd everything is straightforward, including crossfire/sli and multi-monitor setups.

Only for intel shared memory on rare occasions you may need to edit Xorg.

lano i dont know why you are using xfce. It will under utilize your graphics card. I'd suggest using mate/kde/gnome-shell.

Last edited by pam (2013-05-19 05:48:20)

schmidtbag · 2013-05-19 15:27:17

As I've said before, if you have displays/screens enabled for the 2nd GPU in your xorg.conf, it WILL break crossfire. I personally have found listing the 2nd GPU as a device in xorg.conf to just be redundant, but it doesn't break anything. Proprietary drivers still use and need the xorg.conf - the open source drivers are the ones where you don't want to be messing with it. When lano ran 'aticonfig --adapter=0,1 --initial', that actually prevented crossfire from being enabled, because you have to choose between either dual displays or combined GPU power, but not both.

XFCE is fine, it too has compositing effects (though that's not why I think its ok to use). If he wants the extra performance, XFCE is actually a better choice because 3D applications will get undivided attention from the GPUs, and XFCE uses less system resources than the other major DEs. It's been proven several times on phoronix.com that XFCE is generally (not always) the fastest. As of right now the only reason to switch away from XFCE is a way to test if that makes lano's system more stable, which I highly doubt.

Lano, for now, keep IOMMU off since that eliminates a possible cause of your problem. You can keep other virtualization instructions enabled, though, because they're not relevant. I'd suggest you try getting a "clean slate" so we all know exactly where you stand at this point. To do this:
aticonfig --adapters=0 --initial
aticonfig --adapters=all --cfd
aticonfig --adapters=0,1 --cfa
aticonfig --adapters=0,1 --cf=on
reboot
The first command intentionally ignores GPU 1, because in an X perspective, GPU 1 will just get in the way.

Last edited by schmidtbag (2013-05-19 15:28:07)

lano1106 · 2013-05-19 22:04:13

wow, that is really cool that you keep trying to help.

1.

Concerning xfce. That is a long story but, IMO simple is beautiful even ig KDE or Gnome do have more eye candies. In my Linux life, I have started with KDE and eventually switch to Gnome. With both experiences, I eventually ended up with a broken DE after a software update.

Also, after launching an opengl game in full screen, DE becomes absolutly non important. I have already used applications directly on X with no Window Manager at all!

What is really important to me is a DE that always work and is easy to fix when broken. xfce simplicity is contributing to reach my expectations. but hey, choices is what makes life beautiful isn't it?

2.

schmidtbag, ok. I'll reinitialize my xorg.conf and my /etc/ati/amdpcsdb file as you suggest. It could work. Just a small remark concerning

aticonfig --adapters=0,1 --cf=on

I believe that for the command to turn on cf, aticonfig expect only chain master adapter. In this case it would be

aticonfig --adapters=0 --cf=on

or else it does work but you get a warning that adapter 1 is not a master.

3.

I have few more thoughs concerning my freezes but I need to go prepare the supper. I'll come later with results and more comments.

Greetings.

schmidtbag · 2013-05-19 22:13:22

1. Agreed, XFCE is nice, I think it is the most ideal DE for just getting the job done and comes with all the features you really need. I personally prefer KDE (it has really improved a lot over the years). I however, find it very annoying when people call XFCE lightweight, because it isn't. It's blatantly middleweight. When you compare LXDE to XFCE on a 1GHz ARM platform, you'll see how much XFCE's extra libraries bog it down. However, LXDE is a little too lightweight for most people. It can be frustrating if you don't use it with limited purposes in mind.

2. I believe you're confusing cfa, not cf. I'm pretty sure you need to include both adapters when using --cf=on. Regardless, the warning is just simply a warning. While adding the ,1 might not help, it doesn't hurt. Every guide I've ever seen suggests to add it. I have crossfire working for me now and I included the ,1 for both commands. Considering your situation, it's better to take the safe way out.

lano1106 · 2013-05-20 04:45:09

whippet2 /tmp # aticonfig --adapter=0 --initial --output=./0.conf
Found fglrx primary device section
Using ./0.conf
Saving back-up to ./0.conf.fglrx-2
whippet2 /tmp # aticonfig --adapter=0,1 --initial --output=./1.conf
Found fglrx primary device section
Found fglrx primary device section
Using ./1.conf
Saving back-up to ./1.conf.fglrx-0
whippet2 /tmp # diff 0.conf 1.conf

So basically, I just removed any reference to adapter 1 from xorg.conf to see what it would happen. I have different outcome:

lano1106@whippet2 ~/bin $ aticonfig --lsch

Multiple GPU chain for adapter 0, status: Crossfire is enabled
0. 01:00.0 AMD Radeon HD 7900 Series
1. 02:00.0 AMD Radeon HD 7900 Series
lano1106@whippet2 ~/bin $ aticonfig --lscs
Candidate Combination:
Master: 1:0:0
Slave: 2:0:0
CrossFire is enabled on current device
CrossFire Diagnostics:
CrossFire can work with P2P mapping through GART
Dongle Capabilities: support PASSTHROUGH |INTERLINK_SW_AFR | INTERLINK_AUTO_AFR | INTERLINK_BLACKING | INTERLINK_SUPERAA

Same freeze :-(

Basically when a process freeze like, IMHO this is a deadlock inside the driver with interrupts disabled. It is possible to rebuild the fglrx to produce traces. I'll see what I can find.

pam · 2013-05-20 06:30:09

I believe that is AMD's job.

If you want to debug, sure, if your want to learn to write device drivers, i further encourage you to dig for problems.
But again, proprietary drivers are AMD's problem.

I think what you are saying is that the driver has yet to catch up, other than that i think its absolutely useless to debug(besides your personal interest).

Keep the OS install straightforward and make sure everthing with your hardware is setup correctly. Install the crossfire bridge and make sure you have enough watts on your PSU. Install the drivers next and enable crossfire with the aticonfig command, methodically( but not necessary).

OFFTopic: lano, you have given me enough incentive to take xfce more seriously. But i never hated xfce, there are no reasons. I will install it. I love LXDE, though.

Last edited by pam (2013-05-20 06:53:57)

schmidtbag · 2013-05-20 13:17:04

pam might be right, I got the impression that even with the catalyst drivers, the 7900 has been a little flakey. I guess if you really want to prove us wrong, install something like open suse on a separate hard drive and see if you're able to get crossfire working there. If you use an environment AMD is focused on, you might have better luck finding out exactly what's wrong. AMD sponsors suse, which is why I mention that. While you use Suse, you should also gnome or kde instead of xfce, since that MIGHT be a factor.

Off Topic:
I didn't take xfce seriously until about a year ago. The first time I used it was about 5 years ago, and I really hated it back then. It was equally as bloated as gnome 2 but lacked a lot of features and was uglier. Today, I'd say it's the most perfectly balanced DE there is, in a good way. It is, IMO, the best laptop DE because it's cleaner and generally faster than other DEs while not lacking core features like LXDE. I'd say XFCE is linux's last hope of having a reliable user-friendly interface, at least until KDE gets more bugs and optimizations out of the way.

Last edited by schmidtbag (2013-05-20 13:19:20)

lano1106 · 2013-05-20 19:42:10

PAM:

You're totally right. It should be AMD job to fix it but will they? To my knowledge, there is even not a webpage or an email address where you can contact AMD to report problems. I might try search for an AMD email in the open source Radeon driver to see if he could not hook me with one of his pal working on catalyst.

but as long as I have ideas of things to try to fix it, I'll try them. This is the major appeal of using open source software. You have the power to fix it yourself.

From AMD Catalyst 13.4:

[366895]: Fix hard hang when startx with CrossFire enabled for NI Caicos Pro or SI Cape Verde Pro

Cape Verde Pro is a 7750. Looks like the same problem is present for 7970 and AMD haven't caught it.

schmidtbag:

Very good suggestion. I might not follow it to the point of installing another distribution but since the freeze is a symptom of a deadlock inside the driver, I'll see what are the kernel parameters used by suse and/or Ubuntu. I run a customized linux-ck kernel and I am a low latency maniac. I have CONFIG_PREEMPT=y and HZ=1000. If there is a potential race condition that leads to a dead lock, you bet that I am going to get it.

I do not know about Suse but last time I checked Ubuntu settings, it was something like CONFIG_PREEMPT_VOLUNTARY and HZ=250

Last edited by lano1106 (2013-05-20 19:43:50)

schmidtbag · 2013-05-20 19:50:49

Like I said, AMD sponsors Suse so you should probably try that first. Ubuntu is focused on compatibility but I think Suse should take a higher priority. You could also try installing a demo of Windows 7 or 8 as a way to prove if whether or not this is a hardware problem. I know installing something else might not be appealing to you (even if just for testing purposes) but there could be a much deeper problem than you're aware of.

As a side note, are you able to enable displays on both GPUs in linux?

lano1106 · 2013-05-20 21:31:32

It is not hardware. Crossfire on Windows7 has been tested.

I have never tried to enable displays on the second GPU. The principal usage I do of this card is an opencl application. For that, the second GPU is working very well.

pam · 2013-05-21 03:50:40

I am absolutely sure amd will fix it in the next release.
Open source drivers are not that powerful and lack features.

Happy Hacking lano1160 !!

schmidtbag · 2013-05-21 21:19:09

hah looks like one of your problems should be fixed. I just got an update for catalyst and after looking at the aur page for it, I saw you were mentioned. Makes me wonder if I can enable IOMMU. Is crossfire working at all for you at this point?

lano1106 · 2013-05-22 00:50:35

unfortunately. It doesn't fix my crossfire problem. This is something that I have stumbled on while searching for a solution. really it is a small improvement but not the hoped crossfire nirvarna!

schmidtbag · 2013-05-22 00:52:00

That's too bad. I wonder if it fixes my IOMMU problem, but as of right now I don't care enough to mess with that.

Arch Linux

#1 2013-05-17 22:45:22

Crossfire Troubles

#2 2013-05-18 04:28:39

Re: Crossfire Troubles

#3 2013-05-18 04:42:15

Re: Crossfire Troubles

#4 2013-05-18 04:51:16

Re: Crossfire Troubles

#5 2013-05-18 15:13:19

Re: Crossfire Troubles

#6 2013-05-19 02:23:55

Re: Crossfire Troubles

#7 2013-05-19 02:58:03

Re: Crossfire Troubles

#8 2013-05-19 03:12:31

Re: Crossfire Troubles

#9 2013-05-19 03:38:29

Re: Crossfire Troubles

#10 2013-05-19 03:47:06

Re: Crossfire Troubles

#11 2013-05-19 05:35:11

Re: Crossfire Troubles

#12 2013-05-19 05:45:55

Re: Crossfire Troubles

#13 2013-05-19 15:27:17

Re: Crossfire Troubles

#14 2013-05-19 22:04:13

Re: Crossfire Troubles

#15 2013-05-19 22:13:22

Re: Crossfire Troubles

#16 2013-05-20 04:45:09

Re: Crossfire Troubles

#17 2013-05-20 06:30:09

Re: Crossfire Troubles

#18 2013-05-20 13:17:04

Re: Crossfire Troubles

#19 2013-05-20 19:42:10

Re: Crossfire Troubles

#20 2013-05-20 19:50:49

Re: Crossfire Troubles

#21 2013-05-20 21:31:32

Re: Crossfire Troubles

#22 2013-05-21 03:50:40

Re: Crossfire Troubles

#23 2013-05-21 21:19:09

Re: Crossfire Troubles

#24 2013-05-22 00:50:35

Re: Crossfire Troubles

#25 2013-05-22 00:52:00

Re: Crossfire Troubles

Board footer