You are not logged in.

#1 2021-02-23 22:17:27

yinger
Member
Registered: 2021-02-23
Posts: 3

[SOLVED] pci passthrough mess up

Following this: https://wiki.archlinux.org/index.php/PC … _device_ID
Cannot get VM working and looked back and had some discrepancies for outputs. If you need any other files, please ask me. I just thought these were relevant.
x470 gaming plus mobo. Updated to latest bios, enabled SVM and IOMMU. System has amd cpu, 32gb mismatched ram, a 1070 (one I want to pass) and a 750ti for my host.

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=190fe6bd-8314-46f2-9ae8-ecf6b9a1c79f rw iommu=1 amd_iommu=on vfio-pci.ids=10de:1b81,10de:10f0 loglevel=3 cryptdevice=/dev/lvm/lvroot:root root=/dev/mapper/root 1
[    0.125177] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=190fe6bd-8314-46f2-9ae8-ecf6b9a1c79f rw iommu=1 amd_iommu=on vfio-pci.ids=10de:1b81,10de:10f0 loglevel=3 cryptdevice=/dev/lvm/lvroot:root root=/dev/mapper/root 1
[    0.414746] iommu: Default domain type: Translated 
[    0.557616] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.557676] pci 0000:00:01.0: Adding to iommu group 0
[    0.557685] pci 0000:00:01.1: Adding to iommu group 1
[    0.557697] pci 0000:00:01.3: Adding to iommu group 2
[    0.557709] pci 0000:00:02.0: Adding to iommu group 3
[    0.557722] pci 0000:00:03.0: Adding to iommu group 4
[    0.557730] pci 0000:00:03.1: Adding to iommu group 5
[    0.557739] pci 0000:00:03.2: Adding to iommu group 6
[    0.557751] pci 0000:00:04.0: Adding to iommu group 7
[    0.557763] pci 0000:00:07.0: Adding to iommu group 8
[    0.557773] pci 0000:00:07.1: Adding to iommu group 9
[    0.557786] pci 0000:00:08.0: Adding to iommu group 10
[    0.557794] pci 0000:00:08.1: Adding to iommu group 11
[    0.557810] pci 0000:00:14.0: Adding to iommu group 12
[    0.557818] pci 0000:00:14.3: Adding to iommu group 12
[    0.557856] pci 0000:00:18.0: Adding to iommu group 13
[    0.557865] pci 0000:00:18.1: Adding to iommu group 13
[    0.557874] pci 0000:00:18.2: Adding to iommu group 13
[    0.557884] pci 0000:00:18.3: Adding to iommu group 13
[    0.557892] pci 0000:00:18.4: Adding to iommu group 13
[    0.557900] pci 0000:00:18.5: Adding to iommu group 13
[    0.557908] pci 0000:00:18.6: Adding to iommu group 13
[    0.557916] pci 0000:00:18.7: Adding to iommu group 13
[    0.557926] pci 0000:01:00.0: Adding to iommu group 14
[    0.557981] pci 0000:03:00.0: Adding to iommu group 15
[    0.557991] pci 0000:03:00.1: Adding to iommu group 15
[    0.558001] pci 0000:03:00.2: Adding to iommu group 15
[    0.558005] pci 0000:20:00.0: Adding to iommu group 15
[    0.558007] pci 0000:20:01.0: Adding to iommu group 15
[    0.558011] pci 0000:20:02.0: Adding to iommu group 15
[    0.558014] pci 0000:20:03.0: Adding to iommu group 15
[    0.558017] pci 0000:20:04.0: Adding to iommu group 15
[    0.558020] pci 0000:20:08.0: Adding to iommu group 15
[    0.558024] pci 0000:22:00.0: Adding to iommu group 15
[    0.558027] pci 0000:23:00.0: Adding to iommu group 15
[    0.558031] pci 0000:26:00.0: Adding to iommu group 15
[    0.558047] pci 0000:27:00.0: Adding to iommu group 16
[    0.558059] pci 0000:27:00.1: Adding to iommu group 16
[    0.558075] pci 0000:28:00.0: Adding to iommu group 17
[    0.558086] pci 0000:28:00.1: Adding to iommu group 17
[    0.558095] pci 0000:29:00.0: Adding to iommu group 18
[    0.558105] pci 0000:29:00.2: Adding to iommu group 19
[    0.558115] pci 0000:29:00.3: Adding to iommu group 20
[    0.558125] pci 0000:2a:00.0: Adding to iommu group 21
[    0.558134] pci 0000:2a:00.2: Adding to iommu group 22
[    0.558143] pci 0000:2a:00.3: Adding to iommu group 23
[    0.562303] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.563443] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    0.577968] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
#!/bin/bash
shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

outputs me:

IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 1:
	00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 2:
	00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 3:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 5:
	00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 6:
	00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 7:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 8:
	00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 9:
	00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 10:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 11:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 12:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 14:
	01:00.0 Non-Volatile memory controller [0108]: Realtek Semiconductor Co., Ltd. Device [10ec:5760] (rev 01)
IOMMU Group 15:
	03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d0] (rev 01)
	03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
	03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
	20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	20:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	22:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
	23:00.0 Network controller [0280]: Broadcom Inc. and subsidiaries BCM4352 802.11ac Wireless Network Adapter [14e4:43b1] (rev 03)
	26:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
IOMMU Group 16:
	27:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2)
	27:00.1 Audio device [0403]: NVIDIA Corporation GM107 High Definition Audio Controller [GeForce 940MX] [10de:0fbc] (rev a1)
IOMMU Group 17:
	28:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
	28:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 18:
	29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU Group 19:
	29:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 20:
	29:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU Group 21:
	2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
IOMMU Group 22:
	2a:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 23:
	2a:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]

My grub looks like:

# GRUB boot loader configuration

GRUB_DISABLE_SUBMENU=y
GRUB_DEFAULT=saved
GRUB_SAVEDEFAULT=true
GRUB_TIMEOUT=3
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT="iommu=1 amd_iommu=on vfio-pci.ids=10de:1b81,10de:10f0 loglevel=3 cryptdevice=/dev/lvm/lvroot:root root=/dev/mapper/root 1"
GRUB_CMDLINE_LINUX=""

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
#GRUB_ENABLE_CRYPTODISK=y

# Set to 'countdown' or 'hidden' to change timeout behavior,
# press ESC key to display menu.
GRUB_TIMEOUT_STYLE=menu

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
#GRUB_COLOR_NORMAL="light-blue/black"
#GRUB_COLOR_HIGHLIGHT="light-cyan/blue"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
GRUB_BACKGROUND="/boot/grub/fortnite.jpg"
#GRUB_THEME="/path/to/gfxtheme"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

# Uncomment to make GRUB remember the last selection. This requires
# setting 'GRUB_DEFAULT=saved' above.
#GRUB_SAVEDEFAULT="true"

^^( ignore my cringe boot splash image ;) )
/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1b81,10de:10f0

/etc/mkinitcpio.conf

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
MODULES=( vfio_pci vfio vfio_iommu_type1 vfio_virqfd nvme )

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=()

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS="base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 filesystems fsck"
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS="base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 filesystems fsck"
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS="base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 filesystems fsck"
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS="base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 filesystems fsck"
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS="base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 filesystems fsck"
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.
HOOKS="base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 filesystems fsck"

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
#COMPRESSION="zstd"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

"dmesg | grep -i vfio" produces me:

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=190fe6bd-8314-46f2-9ae8-ecf6b9a1c79f rw iommu=1 amd_iommu=on vfio-pci.ids=10de:1b81,10de:10f0 loglevel=3 cryptdevice=/dev/lvm/lvroot:root root=/dev/mapper/root 1
[    0.125177] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=190fe6bd-8314-46f2-9ae8-ecf6b9a1c79f rw iommu=1 amd_iommu=on vfio-pci.ids=10de:1b81,10de:10f0 loglevel=3 cryptdevice=/dev/lvm/lvroot:root root=/dev/mapper/root 1
[    5.747042] VFIO - User Level meta-driver version: 0.3

now on the manual, they have the following output:

[    0.329224] VFIO - User Level meta-driver version: 0.3
[    0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000
[    0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000
[    2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)

they have an "enabling device line". I do not.
"lspci -nnk -d 10de:1b81" produces me:

28:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
	Subsystem: eVga.com. Corp. Device [3842:5173]
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia

"lspci -nnk -d 10de:10f0" produces me:

28:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
	Subsystem: eVga.com. Corp. Device [3842:5173]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

the outputs in the manual have different kernel driver in use for each the card and audio:

06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau nvidia
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

Currently, my vm doesn't output anything. I've been working on this crap since mid day and is currently 10 pm for me. Been on it about 7 hours and have waited for parts such as the 750ti, displayport cable, etc to deliver for around 10 days. Any help is appreciated as my current oracle virtualbox setup for school programs I don't like isn't much of a solution.

edit:
Also, not sure of the relevance but currently using the linux-mainline kernel if that has any impact. This is the first time I have ever posted here so if I am not doing stuff properly, please just point them out so I can improve.

edit1:
I've done my pacman -Syu before starting too. Using nvidia kernel from the aur (I think it's 460 or something now).

edit2:
my /etc/X11/xorg.conf:

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
    Option         "Xinerama" "0"
EndSection

Section "Files"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/psaux"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "dvdxvcx"
    HorizSync       30.0 - 85.0
    VertRefresh     40.0 - 75.0
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce GTX 750 Ti"
    BusID          "PCI:39:0:0"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "Coolbits" "12"
    Option         "Stereo" "0"
    Option         "nvidiaXineramaInfoOrder" "DFP-0"
    Option         "metamodes" "1920x1080_75 +0+0"
    Option         "SLI" "Off"
    Option         "MultiGPU" "Off"
    Option         "BaseMosaic" "off"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

edit3:
got it setup now!! Took some effort and basically just cleared up everything and started from scratch. Works now. Aaahhh! So happy. Removed the spice crap mainly which is what I think might have furked everything. Just trying to figure out sharing usb keyboard and mouse rather than just letting the vm hijack it.

Last edited by yinger (2021-02-25 18:28:22)

Offline

#2 2021-02-24 15:39:52

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [SOLVED] pci passthrough mess up

Glad you figured things out, please prepend [SOLVED} to the thread title (edit first post)


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

Board footer

Powered by FluxBB