You are not logged in.

#5826 2015-08-05 22:58:22

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@FlatronEZ, for Code 43, look here : http://vfio.blogspot.fr/2015/05/vfio-gp … first.html, and search for :

     <kvm>
      <hidden state='on'/>
    </kvm>

and hypervclock.

Offline

#5827 2015-08-06 00:19:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

WorMzy wrote:
FlatronEZ wrote:

Hello,

I am new to this community and hope I don't do anything completely wrong here wink
First my setup:

Ubuntu 15.04
3.19.0-26-generic #27-Ubuntu SMP Tue Jul 28 18:27:31 UTC 2015
QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3)

i7 4790k
Nvidia GTX 750 Ti (Palit) (in main PCIe slot)
MB: Gigabyte Z97X Gaming 7

Please ask on the Ubuntu support forums, we only support Arch here.

https://wiki.archlinux.org/index.php/Fo … pport_ONLY

You realize that question had no dependency whatsoever on the distro, right?  And banned?  For that?  Was that Duelist trolling or was that really ban-worthy?  Not gaining any friends here.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5828 2015-08-06 00:26:10

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 23,835
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
WorMzy wrote:
FlatronEZ wrote:

Hello,

I am new to this community and hope I don't do anything completely wrong here wink
First my setup:

Ubuntu 15.04
3.19.0-26-generic #27-Ubuntu SMP Tue Jul 28 18:27:31 UTC 2015
QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3)

i7 4790k
Nvidia GTX 750 Ti (Palit) (in main PCIe slot)
MB: Gigabyte Z97X Gaming 7

Please ask on the Ubuntu support forums, we only support Arch here.

https://wiki.archlinux.org/index.php/Fo … pport_ONLY

You realize that question had no dependency whatsoever on the distro, right?  And banned?  For that?  Was that Duelist trolling or was that really ban-worthy?  Not gaining any friends here.

Banned because the mod made it clear that these boards are for the support of Arch, not Ubuntu: Flatron ignored that polite pointer to our Etiquette.
Duelist was banned for provoking the staff and then creating three more accounts to continue to post; again, flouting the rules.

As for making friends, that isn't high on my list of priorities. Joining this community and participating here is contingent on following the rules. If members can't--or won't--abide by those rules then the thread will be closed.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Online

#5829 2015-08-06 02:00:41

Ecliptual
Guest

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:

If members can't--or won't--abide by those rules then the thread will be closed.

Alright then. If this endangers or jeopardizes the whole thread here - okay. I will leave, if my presence is so strictly not allowed here. I know you can track my activity there easily.

But there will be other users coming in from the... google(or other search engines) results. And not everyone will be using arch.
Sadly, that's how it is happening now.
I respect your actions. I see you all really have gathered and discussed the problems. You have the final word after all.
Do you have the technical possibility to put an advice somewhere on this thread's page, where everyone will see it, that non-arch distributions support is not provided?
That would be very helpful for those, who may accidentally come here to ask.

P.S.
Disband this account, again. That's a nice enough nickname to be used by someone else.

Last edited by Ecliptual (2015-08-06 02:01:37)

#5830 2015-08-06 02:14:07

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 23,835
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ecliptual wrote:
jasonwryan wrote:

If members can't--or won't--abide by those rules then the thread will be closed.

Alright then. If this endangers or jeopardizes the whole thread here - okay. I will leave, if my presence is so strictly not allowed here. I know you can track my activity there easily.

This isn't about you. If you had adopted this sort of helpful approach earlier, much of this drama could have been avoided. I'll delete this account, but if you are prepared to abide by the rules, your original account can be unbanned. Email me if you want this to happen.

Do you have the technical possibility to put an advice somewhere on this thread's page, where everyone will see it, that non-arch distributions support is not provided?
That would be very helpful for those, who may accidentally come here to ask.

There is no point pasting the Etiquette into a 200 page thread. Everyone here is expected to have read it and to agree to follow it.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Online

#5831 2015-08-06 07:34:01

mauorrizze
Member
Registered: 2014-08-22
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:

If you had adopted this sort of helpful approach earlier,

Just my quick thoughts... Duellist has been helping in this thread for several months or even a year and invested some time in this, probably comparable to the commitment of your moderators. To my knowledge he started being a little more offensive after being told that this help was unwanted by the authorities.

jasonwryan wrote:

much of this drama could have been avoided.

Most if not all the drama could have been avoided if you wouln't have let this thread grow to one of the most resourceful places about this topic (including questions and help from non-archers) and start enforcing your arch-only-rule delayed by two years.

jasonwryan wrote:

Everyone here is expected to have read it and to agree to follow it.

Of course.

Offline

#5832 2015-08-06 07:46:41

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 23,835
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mauorrizze wrote:
jasonwryan wrote:

much of this drama could have been avoided.

Most if not all the drama could have been avoided if you wouln't have let this thread grow to one of the most resourceful places about this topic (including questions and help from non-archers) and start enforcing your arch-only-rule delayed by two years.

I'm really getting tired of this. The fact that no-one from the staff has paid any attention to this thread for two years is completely irrelevant: that is not a license to believe that you are free to behave as you will and have earned some sort of special dispensation from the rules others have to abide by.

The boards are busy and the mod team is small. The only reason I came in here was because some muppet using Ubuntu posted in one of the high volume boards and I mistakenly merged it here before I realised it had nothing to do with Arch.

So, let me be very clear, for the last time. There is nothing about this thread or the people that post in it that is special: you aren't exempt from the rules that govern these boards and they will be applied equally to all. If you have a problem with that, one of the mod team can show you the door.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Online

#5833 2015-08-06 08:10:02

morphological
Member
Registered: 2015-08-01
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using the i915 and arbiter patch method.
I am trying to enable HyperV as suggested by nbhs: http://blog.wikichoon.com/2014/07/enabl … h-kvm.html

However, I cannot seem to find the xml config file for my virtual machine.
/etc/libvirt/qemu  only contains a folder named networks, with xml files specific to that.

I also could not extract the xml file from my existing virtual machine.
Some sites are discouraging Nvidia card owners from applying these hyper-v settings, as they could cause instability. Are there any Nvidia owners here who have tried it?

I set it up using this this script:

 
#!/bin/bash

configfile=/etc/vfio-pci.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=4 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/media/jackson/2cf57f0f/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-usb -usbdevice host:045e:0745 \
-boot menu=on

exit 0

Last edited by morphological (2015-08-06 10:26:25)

Offline

#5834 2015-08-06 11:15:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morphological wrote:

Using the i915 and arbiter patch method.
I am trying to enable HyperV as suggested by nbhs: http://blog.wikichoon.com/2014/07/enabl … h-kvm.html

However, I cannot seem to find the xml config file for my virtual machine.
/etc/libvirt/qemu  only contains a folder named networks, with xml files specific to that.

I also could not extract the xml file from my existing virtual machine.
Some sites are discouraging Nvidia card owners from applying these hyper-v settings, as they could cause instability. Are there any Nvidia owners here who have tried it?

I set it up using this this script:

 
#!/bin/bash

configfile=/etc/vfio-pci.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=4 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/media/jackson/2cf57f0f/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-usb -usbdevice host:045e:0745 \
-boot menu=on

exit 0

They do not cause instabilities. The nvidia drivers shut themselves off when they see you're trying to do the passthrough thing with a geforce card.
That guide tells you to add

hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time  

to your -cpu flag in commandline.
So it'll be like this:

-cpu kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time 

For additional info, read aw's blog


Now for the distro battery:
Seems like it's really against the local law to ask for support of other distributives(even though most of the questions are distro-agnostic), so i've come up with a shaky compromise.
Statistics(that big spreadsheet) shows that there's a majority of arch users, and then come debian, ubuntu, gentoo, fedora and mint.
My plan is simple: i'll create an universal and fresh op-post and a thread on each distribution's board. So when people come from google(or other search engines), they will be directed to their distro's forum into an appropriate thread.
I do have doubts about this working, but well, what else can i do.
Now just stand by while i'm trying to write this all up.

EDIT:
OP-post alpha(or beta). Not even close to finish. Feel free to comment. I knew i didn't include the link...

Last edited by Duelist (2015-08-06 16:24:26)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5835 2015-08-06 12:58:27

morphological
Member
Registered: 2015-08-01
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you very much for your quick reply!

I believe I am missing the necessary packages to use this -cpu option from the commandline.
Is this a command specific to qemu or virsh etc.?
On google I only see people using it in conjunction with their VM scripts, not from the commandline.

-cpu kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time 

As in terms of the other issue.
In order to be able to participate, I made a fresh install of arch and got the passthrough working.
Then I applied everything I had learned to my Lubuntu installation.:)

It's amazing how knowledgeable everybody is here, and I think its only fair, that if you are consulting an arch thread, that you also actually use the environment it is intended for.

Last edited by morphological (2015-08-06 12:58:45)

Offline

#5836 2015-08-06 16:27:07

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morphological wrote:

Thank you very much for your quick reply!

I believe I am missing the necessary packages to use this -cpu option from the commandline.
Is this a command specific to qemu or virsh etc.?

relaxed/hv_relaxed. Available in libvirt 1.0.0+ (commit) and qemu 1.1+ (commit). This bit disables a Windows sanity check that commonly results in a BSOD when the VM is running on a heavily loaded host (example bugs here, here, and here). Sounds similar to the Linux kernel option no_timer_check, which is automatically enabled when Linux is running on KVM.
    vapic/hv_vapic. Available in libvirt 1.1.0+ (commit) and qemu 1.1+ (commit).
    spinlocks/hv_spinlocks. Available in libvirt 1.1.0+ (commit) and qemu 1.1+ (commit)
    hypervclock/hv_time. Available in libvirt 1.2.2+ (commit) and qemu 2.0+ (commit). Sounds similar to kvmclock, a paravirt time source which is used when Linux is running on KVM.

Note the QEMU versions. We had 1.6.2 in fedora 20, and that was literally two years ago. I honestly believe every distro now has a recent version of QEMU.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5837 2015-08-06 17:08:43

avi9526
Member
Registered: 2015-05-15
Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is that possible to unbind the one and only GPU (radeon) from host OS, then use it for guest and after guest shutdown bind back to host (and maybe restart user session)?

Offline

#5838 2015-08-06 18:02:23

MrChefur
Member
Registered: 2015-08-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't get the kernel to patch with the i915 patch, should I be using the older version of the kernel?
Currently I'm using kernel 4.1.4.
I run the command

patch -p1 i915_2.patch

and it returns

(Stripping trailing CRs from patch; use --binary to disable.)
patching file drivers/gpu/drm/i915/i915_dma.c
Reversed (or previously applied) patch detected! Skipping patch.
2 out of 2 hunks ignored -- savings rejects to file drivers/gpu/drm/i915/i915_dma.c.rej
(Stripping trailing CRs from patch; use --binary to disable.)
can't find file to patch at input line 56
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------------------------
|diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
|index ec82f6b..f3908f6 00644
|--- a/drivers/gpu/drm/i915/i915_drv.h
|+++ b/drivers/gpu/drm/i915/i915_drv.h
-----------------------------------------
File to patch:

I'm using the patch from Here

EDIT (again):
I redownloaded the kernel and patch, extracted the kernel and applied the patch using the same command as before and now it's sitting on a blank line with the cursor just refreshing (I forget the term). Is it actually installing?
It's been going for half an hour now...

EDIT (again again):
I would really appreciate any help I can get here. The terminal doesn't do anything (I left it for 8+ hours on a high end system and nothing happened).
I read all the documentation I can find and I'm still lost.

Last edited by MrChefur (2015-08-07 06:23:40)

Offline

#5839 2015-08-06 18:38:26

4kGamer
Member
Registered: 2014-10-29
Posts: 85

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't believe what I am reading here ...are you kidding me? Do you have ANY idea how valuable aw and nbhs and others have been for numerous people worldwide??? I have seen this very thread being linked in various other forums as the number one source for getting this complicated matter done. I certainly wouldn't have done it without them. No chance.

Please Jasonwryan Go AWAY and never come back here.

Last edited by 4kGamer (2015-08-06 18:51:38)

Offline

#5840 2015-08-06 19:06:13

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 23,835
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

I can't believe what I am reading here ...are you kidding me? Do you have ANY idea how valuable aw and nbhs and others have been for numerous people worldwide??? I have seen this very thread being linked in various other forums as the number one source for getting this complicated matter done. I certainly wouldn't have done it without them. No chance.

No-one cares what you think: https://wiki.archlinux.org/index.php/Fo … _the_staff


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Online

#5841 2015-08-06 20:38:34

c00ter
Member
From: Alaskan in Washington State
Registered: 2014-08-28
Posts: 184

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:
nbhs wrote:
jasonwryan wrote:

Please ask on your distro's boards, we only support Arch: https://wiki.archlinux.org/index.php/Fo … pport_ONLY

What? Why? we've been using this forum for 2 years now for everything and everyone related to gpu passthrough, so why now? make no sense

We have the policy for a reason. This thread is no different to the rest of the boards; so, yes, it does make sense, it is entirely consistent with the overall approach.

Hence me old ma's saying: "A foolish consistency is the hobgoblin of little minds."


UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things. -- Doug Gwyn

Offline

#5842 2015-08-06 20:49:41

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 23,835
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone else looking for a ban?


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Online

#5843 2015-08-06 22:05:58

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:

Anyone else looking for a ban?

Yes sir, give me one please.

I don't see any harm in what we are doing in here helping other people, "distro agnostic". Linux and open source is also about being open minded, imho. What I am reading here, is totally unbelievable. KVM VGA Passthrough is a topic many of us are comfortable with (or even more for a few of us), if we can't share that knowledge on those boards, then there is no point being here.

Last edited by Nesousx (2015-08-06 22:13:24)

Offline

#5844 2015-08-06 22:10:56

mauorrizze
Member
Registered: 2014-08-22
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

avi9526 wrote:

Is that possible to unbind the one and only GPU (radeon) from host OS, then use it for guest and after guest shutdown bind back to host (and maybe restart user session)?

That's very unlikely, graphic cards are a lot more complicated than USB devices or some controllers. The card you wanna pass through must not have been initialized by the Mainboards BIOS/UEFI before, so it's even hard to get a headless host setup working. My advise: add a cheap graphics card, insert it in the 1x oder 4x PCIe slot, not the faster ones connected to the CPU, to avoid the need of an ACS patch. Then convince your BIOS to boot with the small GPU and you can pass the other one using aw's guide.

Offline

#5845 2015-08-06 22:14:04

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 23,835
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

I don't see any harm in what we are doing in here...

None of the people moaning here contribute outside this thread or, I would guess, to Arch at all, so--to be clear--your thoughts about how the boards are run are irrelevant.


The next person that posts a whine in here about our policy can thank themselves for having the thread closed. You want to share knowledge about KVM passthrough, then stick to that.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Online

#5846 2015-08-07 09:10:12

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 108

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:

Anyone else looking for a ban?

Yes, me too please. I do use Arch as my favourite distro, but I also happen to use other distros as well, and I am not going to abide by the order to discriminate their users. You are welcome to interpret this as a declaration of disobedience and ban me to pre-empt any post of mine which might (or might not) benefit users of other distros. Have it your way, if that is what you want.

Offline

#5847 2015-08-07 09:11:39

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 23,835
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
jasonwryan wrote:

Anyone else looking for a ban?

Yes, me too please. I do use Arch as my favourite distro, but I also happen to use other distros as well, and I am not going to abide by the order to discriminate their users. You are welcome to interpret this as a declaration of disobedience and ban me to pre-empt any post of mine which might (or might not) benefit users of other distros. Have it your way, if that is what you want.

jasonwryan wrote:

The next person that posts a whine in here about our policy can thank themselves for having the thread closed. You want to share knowledge about KVM passthrough, then stick to that.

And... we have a winner!



Closing.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Online

#5848 2015-08-25 04:32:16

fukawi2
Forum Moderator
From: .vic.au
Registered: 2007-09-28
Posts: 5,441
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After extensive discussion, the moderation team have decided to leave this thread in a publicly accessible area of the forums for the sake of posterity (no pun intended) as an archival reference of information. It will however remain locked.

The philosophy of the Arch Linux forums is that they are to be the canonical reference for users of Arch Linux. This is the primary reason we try to limit discussion to pure Arch Linux, we don't really want to provide ambiguity by saying do it this way for Arch, that way for Manjaro, and another for Fedora.

It is important to note that we as the moderation team do not really have any animosity towards other distributions; they exist for reasons that are perfectly valid. We don't care what distribution someone chooses to use, and we don't mind people using other distributions using or participating on the forums. It is primarily when requests for help with other distributions pop up that we are compelled to shut them down. We had a long debate at one point about permitting the support of Arch Arm as many of us use it. In the end, we decided to not create an exception.

It was not initially considered that moving the thread would break links from third party sites. We have to caution, however, that third party tutorials are a pain in that they tend to create a great deal of support issues. As such, we are not huge fans. What happens is tutorials tend not to be updated over time and become stale. People follow those tutorials, find that they no longer function, then come to the forums for support without having looked at newer reference material. For Arch, the reference material is largely contained in our wiki.

We really would like this to come to closure in a way to make everyone happy. In an ideal world, we would like to see this information captured and be given a fitting place on the web with its own wiki and forum. The admin and moderator team will be cooperative in allowing this data to be captured and archived to anyone wishing to undertake such a task. We also wish to assure that the data is in no danger of being deliberately destroyed.

MOD ACTION: moved from TGN to Kernel and Hardware subforum, leaving locked.

Offline

Board footer

Powered by FluxBB