PCIe DMA Model

This page explains how axi-pcie-core moves data between host memory and the FPGA application over PCIe. The central entity is AxiPcieDma, which wraps surf’s AxiStreamDmaV2 engine and presents per-lane AXI-Stream interfaces to the application.

Inbound and Outbound FIFOs

Inbound (IB) refers to data flowing from the host into the FPGA application. Outbound (OB) refers to data flowing from the FPGA application back to the host.

AxiPcieDma instantiates one AxiStreamDmaV2 lane for each DMA channel. Each lane contains a dedicated IB FIFO (host → FPGA application) and a dedicated OB FIFO (FPGA application → host). The number of lanes is set by the DMA_SIZE_G generic (maximum 8), which is bounded by the AxiPcieCrossbar slave-port budget: the crossbar provides 10 AXI4 slave ports: one descriptor port, up to eight DMA lane ports, and one user general-purpose port.

Descriptor Rings

Each AxiStreamDmaV2 lane uses a descriptor ring: a host-resident circular buffer whose entries each point to a host DMA buffer (physical address + byte count). The DMA engine fetches descriptors from the ring over PCIe using the descriptor AXI4 slave port on AxiPcieCrossbar. After the transfer completes, the engine writes a completion status word back to the descriptor entry, signalling the driver that the buffer is ready.

The descriptor address space is at most 40 bits wide, even though the internal AXI4 bus uses 64-bit addresses (ADDR_WIDTH_C = 64 in AXI_PCIE_CONFIG_C). This 40-bit limit is an architectural constraint of AxiStreamDmaV2; software must ensure descriptor ring memory is allocated within the low 1 TB of host physical address space.

Back-Pressure via tReady

tReady is the AXI-Stream handshake signal asserted by a slave to accept a data beat from its upstream master. When a downstream application sink de-asserts tReady it stalls AxiStreamDmaV2 IB delivery on that lane, which in turn stalls AxiPcieDma, and ultimately stalls PCIe completion scheduling for that DMA channel. Conversely, when an application source has no data to send it simply de-asserts tValid; this produces no OB beats and does not affect other lanes.

This is the canonical SLAC AXI-Stream back-pressure model: the FPGA application must keep its IB sink draining fast enough to absorb the expected PCIe throughput, or it must accept that the IB FIFO will fill and PCIe transfers will stall on that lane. Per-lane IB/OB traffic monitors (accessible at BAR0 offsets 0x0006_0000 and 0x0006_8000) can be used to observe stall conditions.

DMA IRQ Flow

When AxiStreamDmaV2 completes one or more transfers it asserts the level-sensitive dmaIrq output of AxiPcieDma. How that level reaches the host depends on the PCIe PHY family:

• UltraScale+ boards (U50, U55C, U200, U250, U280, C1100, VCU128, KCU116, XUP-VV8): the PHY wrapper instantiates AxiPcieUltrascalePlusIrqFsm, which drives the PHY’s usr_irq_req / usr_irq_ack handshake. The FSM raises usrIrqReq when dmaIrq asserts, waits for the first usrIrqAck, then holds the request until dmaIrq clears (or a 250 000 000 cycle timeout, about 1 s at 250 MHz) and drops it. For legacy INTx, the PHY returns a second ack when it sends Deassert_INTA, so the FSM waits for that ack before returning to idle. For MSI or MSI-X there is no deassert message and no second ack, so the FSM returns to idle directly. The FSM selects the handshake at run time from its msiEnable / msixEnable inputs, which the U55C, C1100, and XUP-VV8 wrappers drive from the PHY’s msi_enable status output. Wrappers that leave these inputs unconnected always use the INTx handshake.

• UltraScale boards (KCU1500, KCU105, PC821, AlphaData KU3, PgpCard G4): dmaIrq drives the PHY’s intx_msi_request input directly, and the PCIe IP sends INTx or MSI according to what the host enabled.

The host-side datadev kernel driver (aes-stream-drivers) services the interrupt: it reads the completion status words from the descriptor ring, recycles completed descriptors, and wakes the userspace readers (for example rogue.hardware.axi.AxiStreamDma) waiting on the device node.

End-to-End Data-Flow Diagram

The diagram below traces both the IB path (host → FPGA application) and the OB path (FPGA application → host), including the descriptor and IRQ paths:

Host (DMA buffers + descriptor ring in host physical memory)
  |
  |  PCIe lanes (Gen3 x16 or Gen4 x8, board-dependent)
  |
  v
PCIe PHY (per-board .dcp wrapper, 250 MHz recovered clock)
  |
  |  AXI4 (256-bit data, 64-bit addr, ID_BITS=4)
  |
  v
AxiPcieCrossbar  (DMA_SIZE_G + 2 slaves -> 1 master -> PCIe PHY)
  |
  +---> AxiPcieReg  (register path -> BAR0 AXI-Lite slaves)
  |
  +---> AxiPcieDma  (AxiStreamDmaV2 engine, DMA_SIZE_G lanes)
           |
           |  IB path (host -> FPGA application)
           +--- IB FIFO 0 --> appAxisMasters[0]  -> application lane 0
           +--- IB FIFO 1 --> appAxisMasters[1]  -> application lane 1
           +--- ...
           +--- IB FIFO N --> appAxisMasters[N]  -> application lane N
           |
           |  OB path (FPGA application -> host)
           +--- OB FIFO 0 <-- appAxisSlaves[0]   <- application lane 0
           +--- OB FIFO 1 <-- appAxisSlaves[1]   <- application lane 1
           +--- ...
           +--- OB FIFO N <-- appAxisSlaves[N]   <- application lane N
           |
           |  Descriptor path (one AXI4 slave port on AxiPcieCrossbar)
           +--- descriptor read/write <-> host descriptor ring (<=40-bit addr)
           |
           |  IRQ path
           +--- dmaIrq -> AxiPcieUltrascalePlusIrqFsm -> usrIrqReq (UltraScale+)
           |    dmaIrq -> intx_msi_request              (UltraScale)
                                                              |
                                                              v
                                                 PCIe PHY -> host INTx / MSI / MSI-X