Arch Linux

newbie115

Member

Registered: 2014-04-13

Posts: 28

[SOLVED] Completely power off a PCIe device

Hi.

I've been researching this subject for several days now and haven't found a definite answer.
My PCs components are MSI Z170a Gaming M7, Intel i7-6700K and Radeon R9 280x.
I'm using a VGA passthrough for the Radeon R9 280x. Now if the card is not being used, the vfio-pci driver switches its state to D3hot. But is there a way of turning the card completely off?
I started thinking that maybe if I switch off the PCIe port into which the card is inserted, the motherboard will stop feeding it power but that doesn't seem to be the case. I tried removing all devices from the port using /sys/bus/pci/devices/.../remove but the port refused to go to sleep. So I forced the D3 state on the PCIe bus to which my R9 280x was connected by a setpci command (according to Intel's datasheets at offset 0x84 two least significant bits control the state) but aside from not being able to switch my card to D0 in such state nothing changed. The fans were still spinning, the card was still outputting heat so it was still using electricity. I'm wondering how it's done so that when the entire computer is suspended the card is completely off. Is switching off a PCIe port only possible when the entire motherboard is off? I hope not...

Last edited by newbie115 (2017-03-05 14:45:56)

blahhumbug

Member

Registered: 2016-10-08

Posts: 64

Re: [SOLVED] Completely power off a PCIe device

This is a very hardware specific question, and you haven't mentioned which CPU and/or chipset you are using.

I've seen very few root-ports that support completely powering down having once been powered on.  Sometimes there are bios options to shut off a specific root-port completely, but I don't believe this is what you want as you'd have to reboot to again use any attached card.

In my experience, most Intel server root-ports support D3-hot, but not D3-cold.  I'm less familiar with client-end CPUs and their feature set though.

newbie115

Member

Registered: 2014-04-13

Posts: 28

Re: [SOLVED] Completely power off a PCIe device

Right, totally forgot about that part which now that you mention it seems pretty important. I've edited my original question to add my PCs specs. My question was fuelled by curiosity, I'm really glad there's someone out there who can explain this to me.

So how is suspending the entire PC any different from suspending only the PCIe root-port? I thought it was done sequentially, as in one by one the PCIe devices are put into D3hot, then the power to the ports switches off, then some other component and finally the CPU. Or is it done in such way that all devices are put into D3hot and then the chipset decides to cut power to everything at once effectively putting the whole PC into D3cold? I understand that this would mean there is no way of suspending the PCIe root-port without suspending the entire computer.
Also why do some devices report D3coldsupported as true (for instance my R9 280x) if I can't force them into D3cold? Am I misinterpreting the D3cold state as the complete-power-off state?
Please, sprinkle me with wisdom from your mighty brain.

Last edited by newbie115 (2017-02-05 21:07:15)

blahhumbug

Member

Registered: 2016-10-08

Posts: 64

Re: [SOLVED] Completely power off a PCIe device

I don't believe that the entire chipset gets powered down in the D3hot -> S3 flow, because this would break USB-wake or wake-on-lan flows. I'm fairly certain that there is hardware out there that supports putting individual devices in D3cold while the system remains in S0, and some quick searches show that there is OS support for that.  But, in my experience, Intel root-ports have not supported this.

However, just because the root-ports don't support going to D3cold while in S0 doesn't mean that the PCIe endpoint can't.   I took a quick look into the PCIe spec and I don't see any obvious spec requirement that the root-port has to go to d3cold for the card itself to go to d3cold.  But I think in practice, the intel chipsets will still be supplying power to the endpoint, so it might be up to the Radeon hardware and driver to internally gate the power, and I don't know if they do that.

Last edited by blahhumbug (2017-02-05 21:25:12)

mich41

Member

Registered: 2012-06-22

Posts: 796

Re: [SOLVED] Completely power off a PCIe device

Suspend is done by shutting down the whole PSU except for small standby part which supplies 5V through separate cables to devices that want to be active in standby (power button , sometimes USB and network).

You say that putting the GPU in D3 and powering down its PCIe connection doesn't cause it to power down so probably shutting power to it is the only way. I've never heard of desktop motherboards having per-port power switches (PCIe power pins are wired directly to ATX yellow cable), but the GPUs I owned never seemed to work without power on their auxiliary plugs - maybe a switch there would do the trick. Not sure if a desktop chipset would notice when the device is brought back, though.

newbie115

Member

Registered: 2014-04-13

Posts: 28

Re: [SOLVED] Completely power off a PCIe device

Thanks for your informative posts.

@blahhumbug I thought about some sort of internal power switch in the GPU however I don't think there is one. I booted up to win10 with official AMD drivers and disabled the whole card in the device manager while having an LCD connected to my i7's IGP. Nothing. Then I disabled the whole PCIe port and still the fan was working. So if there is an internal switch it its control is not included in the AMD's drivers. I'm guessing it just doesn't exist and even if it did there's nearly 0% chance of finding the way to operate it through trial and error.

@mich41 That explains a lot, thanks. I've also noticed that the GPU does not start without the auxiliary power plugs. So I'm thinking that if I take the plugs out after the PC has booted, the chipset will have no problem with recognizing the GPU when they're reinserted. I'm just a bit worried about current spikes when switching the power on, so I'll try to make a soft start circuit maybe? I'll get back to you then.

EDIT: I don't want to bump the thread, so this is to anyone who is visiting this thread: I've recently had a bit of free time and made a power keying circuit with a soft start. The effect is that the GPU disappears from the lspci list when the power is turned off but does not reappear when power is back on. So this method is a dead end. It was quite informative though, so there's that.

Last edited by newbie115 (2017-03-05 14:45:40)

stedaniels

Member

From: UK

Registered: 2013-11-21

Posts: 3

Website

Re: [SOLVED] Completely power off a PCIe device

@newbie115

Did you try to power cycle the card and rescan the PCI bus once you'd reapplied power?

E.g.:

echo "1" > /sys/bus/pci/devices/DDDD\:BB\:DD.F//remove
sleep 1
echo "1" > /sys/bus/pci/rescan

Source: StackExchange answer to "How to Reset/Cycle Power to a PCIe Device?"