Arch Linux

Espionage724

Member

From: Pennsylvania, United States

Registered: 2013-09-12

Posts: 20

Website

Have Hardware IOMMU; Kernel Still Does Software IO TLB?

I use a MSI 990FXA-GD80V2 motherboard (V13.6B1 BIOS), a FX-8350, and a Radeon HD 7850 (open-source radeon driver). IOMMU is enabled in BIOS (64MB). dmesg says:

[    0.963439] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    0.963440] AMD-Vi: Interrupt remapping enabled
[    0.969714] AMD-Vi: Lazy IO/TLB flushing enabled
[    1.036747] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.036749] software IO TLB [mem 0xbd498000-0xc1498000] (64MB) mapped at [ffff8800bd498000-ffff8800c1497fff]

From my understanding, shouldn't PCI-DMA be mentioning something about GART instead of software since I have hardware IOMMU?

Using iommu=force doesn't change anything. Using iommu=pt iommu=1 adds a new line to dmesg and drops the Lazy IO/TLB flushing, but PCI-DMA still states software:

[    0.963468] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    0.963469] AMD-Vi: Interrupt remapping enabled
[    0.963600] AMD-Vi: Initialized for Passthrough Mode
[    1.030702] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.030704] software IO TLB [mem 0xbd198000-0xc1198000] (64MB) mapped at [ffff8800bd198000-ffff8800c1197fff]

Disabling IOMMU in BIOS also gives the same messages as above (AMD-Vi still finds an IOMMU, remaps, and PCI-DMA still uses software), but some AGP aperture message is also shown early on (something about taking 64MB of RAM).

---

RoE | Wiki - My Linux set-up tweaks and notes!

azat

Member

Registered: 2015-05-10

Posts: 1

Re: Have Hardware IOMMU; Kernel Still Does Software IO TLB?

If my understanding of sources is correct then ( you could also ask your question at iommu list -- https://lists.linuxfoundation.org/mailm … info/iommu ):

- your hw iommu must be in use, regardless swiotlb (According to execution chain and that fact that swiotlb will be enabled if you have more than 4GB of memory, I guess that this is your case)

arch/x86/include/asm/iommu_table.h:

/*                          
 * History lesson:          
 * The execution chain of IOMMUs in 2.6.36 looks as so:
 *                          
 *            [xen-swiotlb]
 *                 |        
 *         +----[swiotlb *]--+
 *        /         |         \
 *       /          |          \
 *    [GART]     [Calgary]  [Intel VT-d]
 *     /                    
 *    /                     
 * [AMD-Vi]                 
 *                          
 * *: if SWIOTLB detected 'iommu=soft'/'swiotlb=force' it would skip
 * over the rest of IOMMUs and unconditionally initialize the SWIOTLB.
 * Also it would surreptitiously initialize set the swiotlb=1 if there were
 * more than 4GB and if the user did not pass in 'iommu=off'. The swiotlb
 * flag would be turned off by all IOMMUs except the Calgary one.

- you don't need iommu=pt

drivers/iommu/amd_iommu.c:

static unsigned device_dma_ops_init(void)
{
    ...
  for_each_pci_dev(pdev) {
    ...
    if (!dev_data->passthrough)
      pdev->dev.archdata.dma_ops = &amd_iommu_dma_ops;
    else
      pdev->dev.archdata.dma_ops = &nommu_dma_ops;
  }
  ...
}

UPD:

According to you messages that tree of execution order is in reverse order, but this doesn't affect the result, since amd iommu driver installs per-device dma_ops, and get_dma_ops() prefers it:

arch/x86/include/asm/dma-mapping.h:

static inline struct dma_map_ops *get_dma_ops(struct device *dev)
{                    
/** defined for X86_64 */
#ifndef CONFIG_X86_DEV_DMA_OPS
        return dma_ops;
#else                
        if (unlikely(!dev) || !dev->archdata.dma_ops)                                                                                                                                                                                                                             
                return dma_ops;
        else         
                return dev->archdata.dma_ops;
#endif               
}

P.S. sources from 64fb1d0e975e92e012802d371e417266d6531676 -- https://git.kernel.org/cgit/linux/kerne … 66d6531676

Last edited by azat (2015-05-10 22:25:53)