Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to access PCIe configuration space? (ECAM)

In order to access PCI Configuration Space, I/O port address 0xCF8, 0xCFC is used according to various articles.

mov eax, dword 0x80000000

mov dx, word 0x0CF8
out dx, eax
mov dx, word 0x0CFC
in eax, dx

The value of eax in the above code was 0x12378086, which means vendor ID = 0x8086 and device ID = 0x1237.

Here's the question.

Q1. I've seen this method only works for PCI Configuration Space. Then, what is the other method to access PCIe Configuration Space?

"This extended configuration space cannot be accessed using the legacy PCI method (through ports 0xCF8 and 0xCFC)"

https://wiki.osdev.org/PCI_Express

But some other articles say this legacy method is compatible with PCIe Configuration Space as well.

This is confusing.

Q2. If the legacy 0xCF8, 0xCFC works with PCIe Configuration Space as well, NASM assembly source code in detail (not about linux) will be truly appreciated, because I've seen plenty of ECAM(Enhanced Configuration Access Mechanism) articles but all of them were about conceptual content.

Hardware Spec:

Motherboard : P775TM1
Processor : Intel i7 8700K
like image 643
YoonSeok OH Avatar asked Aug 12 '19 07:08

YoonSeok OH


Video Answer


2 Answers

Q1. I've seen this method only works for PCI Configuration Space. Then, what is the other method to access PCIe Configuration Space?

For 80x86 PCs there are actually 3 mechanisms to access PCI configuration space. The legacy mechanism you've mentioned (using IO ports 0x0CF8 and 0x0xCFC) is "mechanism #1". There's another legacy mechanism called "mechanism #2" which also used 0x0CF8 and 0x0xCFC; but it wasn't used by many chipsets and (for modern computers) can be considered obsolete.

Originally the size of PCI configuration space for each PCI "function" was 256 bytes; and for both of the legacy mechanisms you can only access 256 bytes. When they invented PCI-E they added a third mechanism (the memory mapped ECAM) and they also increased the size PCI configuration space for each PCI-E "function" to 4096 bytes. The legacy "mechanism #1" still works, but is still only able to access the first 256 bytes (out of the 4096 bytes that a PCI-E "function" can have). Fortunately, hardware manufacturers are smart enough to ensure that the stuff an older operating system would need to access is within the first 256 bytes, so older operating systems that don't support PCI-E still work (using "mechanism #1"), and the remainder of the 4096 bytes (that can't be accessed without ECAM) is mostly used for things like PCI-E extended capabilities.

Note: It's possible to have a "PCI-E to PCI conventional" bridge, with PCI conventional devices behind the bridge. In this case the PCI conventional devices/functions will only provide 256 bytes of PCI configuration space, even though it's still using ECAM.

To use ECAM; you'll need to use ACPI's "index of tables" (RSDT or XSDT) to find a table called the "MCFG". Sadly (unless it's changed since I went looking last) this table isn't described in the ACPI specs where it belongs; and is instead described in a PCI spec that's locked behind a "maliciously expensive" PCI SIG paywall. Hopefully you can find a reliable third-party description somewhere.

In general; the MCFG table consists of entries describing the address range to use for a range of bus numbers; and there may be multiple different address ranges for multiple separate ranges of bus numbers. The idea would be to use the bus number of the device to find the address of the right area; then combine "address_of_area + ((bus - first_bus_for_area) << 20) | (device << 15) | (function << 12)" to find the address for the start of a function's PCI configuration space. Once that's found you can read/write to offset/s within that physical page to access the corresponding offsets in the function's PCI configuration space.

like image 64
Brendan Avatar answered Oct 31 '22 21:10

Brendan


You read the quote

This extended configuration space cannot be accessed using the legacy PCI method (through ports 0xCF8 and 0xCFC)

out of context.
Here is the quote in context (emphasis mine):

The PCI Express bus extends the Configuration Space from 256 bytes to 4096 bytes. This extended configuration space cannot be accessed using the legacy PCI method (through ports 0xCF8 and 0xCFC).

The author was talking about the part of the PCIe configuration space that starts at 0x100.

In the beginning there was a configuration space, for each PCI device function, of 256 bytes.
This space was accessed using the PCI legacy mechanism (we can ignore the fact there were two mechanisms) at ports 0xcf8 and 0xcfc.

The PCIe extended this space from 256 bytes to 4KiB and introduced a new mechanism to access the configuration space (all of it).

So, to recap:

  • There is a single PCI configuration space of 4KiB. It is divided into a PCI 3.0 Compatible region (from 0x000 to 0x0ff) and PCIe extended configuration region (from 0x100 to 0xfff).
  • There are two mechanism to access the PCI configuration space. One is the legacy mechanism at 0xcf8/0xcfc the other one is a memory mapped area.
  • The Legacy mechanism can only access the compatibility region (the first 256 bytes).
  • The ECAM can access all of the space.

Quoting the PCIe specification:

PCI Express extends the Configuration Space to 4096 bytes per Function as compared to 256 bytes allowed by PCI Local Bus Specification.

PCI Express Configuration Space is divided into a PCI 3.0 compatible region, which consists of the first 256 bytes of a Function’s Configuration Space, and a PCI Express Extended Configuration Space which consists of the remaining Configuration Space (see Figure 7-3).

The PCI 3.0 compatible Configuration Space can be accessed using either the mechanism defined in the PCI Local Bus Specification [NdR: The legacy configuration mechanism] or the PCI Express Enhanced Configuration Access Mechanism (ECAM) described later in this section.

Accesses made using either access mechanism are equivalent. The PCI Express Extended Configuration Space can only be accessed by using the ECAM.


Its very (very) likely that Intel's CPUs will support the legacy PCI configuration mechanism for many years to come.
Internally, the part of the uncore that generates PCI configuration transactions (i.e. the System Agent/UBox) is already using only PCIe configuration transactions (i.e. the same MMCFG type that is generated by the ECAM) but the legacy software interface was not removed.

Since the PCIe root complex is within the CPU, the CPU is the only concern when it comes to legacy PCI software compatibility (legacy PCI is need a PCIe to PCI bridge that will probably expose a configuration mechanism).

In short, you can safely use the legacy PCI mechanism to access the first 256 bytes (per function) of the PCIe configuration space.
Actually, unless Intel finds a new way to configure the uncore devices, the legacy mechanism will never go away because it is required to configure the ECAM itself.


The legacy mechanism is straight forward to use, you already posted some code using it. I'm not sure what else is needed.

You can use it like this:

%define CFG(bus, dev, fun, reg) (0x80000000 | (bus << 16) | (dev << 11) | (fun << 8) | reg)

%macro cfg_arm 4
  mov dx, 0cf8h
  mov eax, CFG(%1, %2, %3, %4)
  out dx, eax
%endmacro

%macro cfg_read 4
  cfg_arm %1, %2, %3, %4
  mov dx, 0cfch
  in eax, dx
%endmacro

%macro cfg_write 5
  cfg_arm %1, %2, %3, %4
  mov dx, 0cfch
  mov eax, %5
  out dx, eax
%endmacro

 cfg_read 0, 0, 0, 0   ;eax <- VID:DID of dev 0, fun 0 on bus 0

THIS CODE IS UNTESTED

If you were referring to the content of the configuration space (i.e. what to set), that's too broad.
You can read the datasheet of the device of interest, they usually document even the standard registers defined in the PCI specification.
Alternatively, you can read the PCI specification itself.

If you were asking how to use the ECAM, read Brendan's answer.
The only thing I can add is that, for your CPU, you can find the base of the ECAM by reading the register PCIEXBAR (offset 60h) from the (legacy) PCI configuration space of the iMC of the CPU (bus 0, dev 0, fun 0).
Something like this:

cfg_read 0, 0, 0, 60h       ;Low 32-bit
mov ebx, eax                
cfg_read 0, 0, 0, 64h       ;high 32-bit
shl rax, 32
or rax, rbx                 ;RAX = ptr to ECAM area 

The firmware has already configured everything to use this area correctly.

like image 20
Margaret Bloom Avatar answered Oct 31 '22 20:10

Margaret Bloom