DMA under the PCI express protocol?

pci express dma

Dear all,

I bought the Spartan-3 Starter Kit from the xilinx company. And I know how to realize the PIO transfer using the logic-core. But I don't how to realize the DMA tranfer?
Can anybody tell me how to realize the DMA tranfer? Any information is appreciated.

pcie dma

Can anybody help me?

How can the PC system know the PCI express device become the "master"? And how can the correct "salve" get the tranfer data?

Added after 4 hours 32 minutes:

nobody can give me some tips?

pci express and dma

Can anybody give me some suggestion? Anybody do some project about the DMA on the PCI or PCI express? Thanks.

cfg_interrupt_rdy_n

Is anyboday do some projects about the PCI express protocol? Give me some suggestion, plz.

pci express protocol

There are no masters or slaves with PCIe. PCIe is a point to point protocol, where transmits and receives can occur at the same time.

"DMA" occurs when the downstream device transmits read or write cycles to the upstream port, i.e. without initiation from the host CPU.

Usually there is a "DMA" queue structure in host memory and the associated registers in the downstream device to manage the "DMA" transfers.

xiliinx legacy interrupt msi

Thanks sbob.

Can you give me some more details about that? I surf on the internet, but I find few can be helpful.

pci express dma transfer

What do you want to know?

Are you familiar at all with the PCIe spec? How about PCI?

doing dma with pci express

I've read the PCI express spec. And I want to know how to add the DMA controller to the protocol.

I know serveral registers must be realized in the FPGA, including address, length, and status. But I don't know how the FPGA send out the DMA REQUEST and how it know it can use the bus.

I am now using the logic core of the xilinx company to realize the PCI express protocol. But the core has no the DMA control part, so I am confused.

pci express msi legacy interrupt

OK. To initiate a memory read, you transmit a memory read TLP (of size X at address Y) and wait for the completion with data to come back. To initiate a memory write you send a memory write TLP with the data. There are all kinds of wonderful things to watch out for, like don't send more than the max. payload size, don't cross a 4K address boundry, etc.

PCIe is a bi-directional "always ready" bus. There is no need to request the bus and wait for the grant like PCI. The bus is not shared, there is no arbiter. You write your transaction to the TLN bus of the Xilinx core and wait for the completion.

To add "DMA" to the Xilinx PIO example, you need to modify the RX_ENGINE.v to respond to Completion With Data TLPs and TX_ENGINE.v to send out Memory Write32 or Memory Write64 TLPs. Pay attention to the byte ordering when sending more than 4 bytes of data.

If you want information on decent DMA queue structures and implementations, I suggest ripping-off the Intel Ethernet controller style of DMA. They have queue start and length registers to define the queue, and queue head and tail registers to manage the queue.

I've completed products that implement "DMA" with the Xilinx PCIe core already, so if your company is in a tough spot, I can consult to help you out if required.

xilinx pcie core

sbob, thanks a lot.

I know the PCIe is a bi-directional "always ready" bus, but the memory is shared by all the devices. How can the PCIe device write the memory as possible as it can? I am confused. Whether the "Root" can help to do that?

Added after 8 minutes:

By the way, Can I understand the DMA write steps as following:
1. Memory write or IO write address, length and so on registers.
2. Memory write or IO write "run" register to make the "DMA write handle" module to work.
3. When the DMA write work is done, the interrupt is set up to inform the logic core to send out the interrupt information. And the driver receive the interrupt and do some memory copy.

Is my understanding right?

I am using the PIPE Endpoint 1-lane Core v1.3. In the core's configuration interface, there are two signals: cfg_interrupt_n, cfg_intr_rdy_n. I think, when the cfg_interrupt_n is low, the core will send out the correct interrupt and when the "message" is sent out, the core will set the cfg_intr_rdy_n to low? But how can the core know which interrupt(INTa, INTb, INTc, INTd) will be triggered?

inta_n

Hi,

For Intel chipset implementations the root complex and the memory controller are in the same chip. The arbitration for the memory bus is outside of the scope of PCIe. The root complex can queue up the requests from the many downstream devices. The memory access latency may vary on PCIe depending on other PCIe devices and CPU accesses to the memory.

test for pcie legacy interrupt

Ok, I now understand the DMA transfer process. Thanks a lot, sbob.

xilinx pci express interrupt

sbob, I've realized the DMA controller to read/write information from/to physical memory. But I faced another problem: I can't control the interrupt as I want.
I found the interrupt control information of the logic core can only be found in the pcie_pipe_ug167.pdf.
Can you explain how the cfg_interrupt_rdy_n and cfg_interrupt_n signal work?

pcie dma controller structure

anybody knows the issues about the DMA interrupt under the PCI express?

trn_clk

Hi,

Yeah, Xilinx screwed up in their implementation of the interrupt function. You need to know if the the core has been configured to use legacy interrupts (INTA) or MSI interrupts and the only way to know that is from the configuation bits, but they don't give you that one, so you have to read it. I put the following code in to constantly read the bits, since I don't know when it might be changed or configured:

//need to continuously read the MSI control register to get the interrupt
//mode - either MSI or INTA
reg cfg_rd_en_n;
reg [9:0] cfg_dwaddr;
reg cfg_msi_enable;

always @( posedge trn_clk or negedge trn_reset_n) begin
if (!trn_reset_n) begin
cfg_rd_en_n <= 1'b1;
cfg_dwaddr[9:0] <= 10'h0;
cfg_msi_enable <= 1'b0;
end
else if (!cfg_rd_wr_done_n) begin
cfg_rd_en_n <= 1'b1;
cfg_msi_enable <= cfg_do[16];
end
else begin
cfg_rd_en_n <= 1'b0;
cfg_dwaddr[9:0] <= 10'h12;
end
end

You also need to assert/deassert cfg_interrupt_n correctly based on which interrupt mode you're in. I used the following code to do it:

//use enhanced 64-bit interface interrupt rules:
// 1) assert cfg_interrupt_n only if cfg_interrupt_rdy_n is asserted
// 2) deassert cfg_interrupt_n only if cfg_interrupt_rdy_n is asserted in INTA mode
always @(posedge clk or negedge rst_n) begin
if (!rst_n)
interrupt_sync <= 1'b0;
else begin
if (interrupt & !cfg_interrupt_rdy_n)
interrupt_sync <= 1'b1;
else if (!interrupt & !cfg_interrupt_rdy_n & interrupt_sync & !cfg_msi_enable) //INTA_N int
interrupt_sync <= 1'b0;
else if (!interrupt & interrupt_sync & cfg_msi_enable) //MSI int
interrupt_sync <= 1'b0;
end
end

// pulse cfg_interrupt_n once in MSI mode...
always @(posedge clk or negedge rst_n) begin
if (!rst_n)
interrupt_sync_delay <= 1'b0;
else
interrupt_sync_delay <= interrupt_sync;
end
assign interrupt_sync_edge = interrupt_sync & (interrupt_sync ^ interrupt_sync_delay);

// and keep cfg_interrupt_n asserted until interrupt is serviced in INTA_N mode
assign cfg_interrupt_n = cfg_msi_enable ? ~interrupt_sync_edge : ~interrupt_sync;


Sorry, but the stupid reply formatter takes out all the extra spaces and tabs...


Sbob

pcie fpga dma

Thanks a lot, sbob. Now I can trigger the interrupt.

But I want to know how you control the "interrupt" signal? I use the FSM to make the signal asserted for a while. But so, I can only interrupt for just one or two times. Whether the signal is asserted for a long time?

Added after 5 hours 45 minutes:

sbob, did you meet the problem?

pci express dma latency

I have an interrupt mask register and an interrupt status register. I have several interrupt sources that can each generate an interrupt pulse. When the corresponding interrupt mask bit is set, the interrupt pulses set individual bits in the interrupt status register that stay set until software clears them (usually in the interrupt service routine). My "interrupt" signal is asserted if the interrupt mask bit is set and the interrupt status bit is set. In summary, my interrupt signal stays asserted until it is cleared by software in the interrupt status routine.

dma express

Hi sbob,
I saw your implementation of interrupt generation.
i have done a similar thing but when i run my program in a loop for testing the number of interrupts generated the number is always more than expected.

**broken link removed**
https://obrazki.elektroda.pl/40_1170653834.png
the waveform has been captured from chipscope

when i do a memory write to a particular location it sets gen_int signals and it is deasseted when i do a memory write to another particular location by the ISR. as you may infer from the image provided above the ISR is called twice

Have you faced a similar kind of problem if so kindly tell me what has to be done to coorect my design.

Thanks in advance

pcie dma

Can you post the ISR? I think the code costs too much time during ISR. When interrupt occurs, you must firstly clear the interrupt as soon as possible.

Another possible reason: Do you share the interrupt with other device?

xilinx pci-express dma

for my test purpose the ISR is simply doing a memory write to a particular location.

irqreturn_t ccp_intr_handler(

IN int irq,

IN void *dev_id,

IN struct pt_regs *regs)
{
unsigned long lock_flags ;

volatile unsigned int *ptr ;
int i ;
CCP_DEV *dev = (CCP_DEV *)dev_id ;

//#ifdef DEBUG_INFO

printk("I am in ccp_intr_handler\n") ;

//#endif

ptr = (volatile unsigned int *)dev->bar_virt_address[0] ;

spin_lock_irqsave(&dev->irq_lock, lock_flags) ;

*(ptr + 0xE) = 0xFF ;

spin_unlock_irqrestore(&dev->irq_lock, lock_flags) ;

return IRQ_HANDLED ;
}

the ISR is shared but it is shared with usb controller and presently i have no usb connected with the system.