How to connect multiple FPGAs?

Hi, I'm totally new to FPGA. Currently, I'm building FPGA boards using Altera Max 10 (10M50DAF484C8G).

I want to connect 3 boards so the data from board 1 goes to board 2. Then, it's combined with data from baord 2 and goes to baord 3. So in the end, I'll have 3 data from 3 boards and send it to another board.

What I'm facing now is, what protocol can be used to do this? I've done some reading and found out these protocols:

1. SPI: I've read somewhere that SPI is not recommended to be used outside the PCB or it will be very slow.
2. SATA: I think Max 10 doesn't support SATA transceiver, I don't understand.
3. PCIe: I also thin Max 10 doesn't support this protocol.
4. Ethernet: It is doable, but I want to use other protocols, if it's possible.
5. LVDS: I'm not sure how this works.
6. HSMC: I'm not sure also how this works with FPGA.

So, which one is the best?
Or maybe you have other recommendations?
And how can I use it?
What components do I need (controller IC, etc)?
Or can I directly put connectors on the board (with direct conenction to FPGA pins)?

I hope I can find answers. Thank you.

ewo you must present the data rate first. Without this no recommendation can be made I am afraid. Depending on this data rate, whether it is a continuous stream of data or something is only transmitted occasionally (which is still related to the data rate as well), it will not be possible to make a recommendation. When making boards communicate with each other which are directly connected to each other and not through a computer network like LAN, you need not use Ethernet as it will be an overkill.

Please note that LVDS is a just low voltage differential signalling. It may describe the voltage threshold levels but it is not a protocol. A protocol describes in detail what steps are taken when data is exchanged and how the data is put together for transmition e.g in Ethernet we have packets so we take the data and put more bits with the data that give ability for error correction, source and destination of data. Now all these extra bits do not provide more data, but still provide information about the data so the communication may succeed.

The HSMC is High Speed Mezaine Card interface. This is also not a protocol. It is merely name of a connector used on the Altera development boards. Basically like you can have a male/female pin header/socket to connect one board to another, the HSMC does the same thing. The only difference between the everyday 0.1" header/sockets and the HSMC is that the HSMC can work at VERY high frequencies without the signal being distorted.

At high frequencies which is usually taken to be 10s of MHz (it really depends on the signal path length vs signal rise/fall time but lets not make this complicated) we get what are called as high speed effects that can cause signal distortion. Under these circumstances the signal travel path must be modelled as a "transmission line" and the signal path with source and destination modelled as a distributed element circuit. OK, if this paragraph does not make sense it is ok. Just remember that HSMC is used in the same way as any other connector we use to connect two cards together so they can communicate with each other.

Hi, I'm totally new to FPGA

Click to expand...

For new drivers, I suggest - drive slowly.
Implement SPI master and slave cores on both devices (using your favorite HDL) and scale up from there.
As matrixofdynamism mentioned - LVDS is just a physical standard - communication protocol definition is out the LVDS scope.
So your SPI can use any electrical standard supported by your FPGA's - LVDS included.

Hi, thank you for replying. Right now maybe I'm focusing on LVDS. So, can it be used to transfer data between boards? The data rate is continuous stream. If I want to use LVDS, how can I use it to transfer data? Do I need additional transceiver and receiver chips on the boards? Can I use any type of connectors?

So, if I want to use SPI, I don't need special additional chips right? Just program it using HDL and use LVDS pins, right?

ewo, since we do not know the details of what you are trying to achieve i.e datarate, what the data is i.e video/audio or something less demanding, don't know the final aim of this design, the advice that can be given is limited.

Just transferring data between the two devices can be done easily by using SPI. Yes, it shall be SPI core written in VHDL and then synthesized and implemented into the CPLD. I am not sure what brand CPLDs you are using, I am familiar with Altera tool suite. Just use a compatible IO standard on the receiver and transmitter i.e the voltage level on both ends should be compatible in terms of what voltage is considered logic low and what voltage is considered logic high and also the current drive stength.

psst. if you use the "default" settings on both CPLDs which I assume are the same device type, it will be ok to make them communicate as the logic high and logic low voltage level shall be the same on both sides and they are probably (yes I am assuming here) powered to the same voltage. You do not even need to change it to use LVDS unless you specifically want differential signalling. Also, remember that you can also implement parallel data transfer, you know, where we have 8 bits or 16 bits being transferred at once and use control signals to mark the point in time when the data can be latched. However, for starters, using SPI is better. Especially since you should have a core already existing to implement SPI in your design written by the CPLD manufacturer or someone on the internet. Also, if you are using something like Qsys (applies to Altera Suite only), achieving what you want can be done without bothering about writing HDL. I will not go into details of Qsys since I think you really need to use HDL as this looks like a school project.

Actually, I want to send measurement data from sensors (temperature, current, voltage, etc). So, the data is continuous stream, about 100Mbps (not demanding, if it's fast enough, it will be just ok). Can SPI provide this speed? The distance between boards is about 15cm.

So, if I use SPI, I just need SPI core written in VHDL which is available on verilog I think.

This is the design guideline for SPI from Altera. https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/an/an485.pdf

It is said that I need a microprocessor to control the master. Do I need additional mp on board or just directly use SPI core?

Sorry for some basic questions since I want to get it right without confusion.

It is said that I need a microprocessor to control the master. Do I need additional mp on board or just directly use SPI core?

Click to expand...

Have a uP makes config of the registers for the SPI/I2C masters easier (the firmware for the uP can be easily written in C). That means you have more control and can flexibly communicate with your sensors.
If you have the logic blocks (a state-machine) that can configure the masters then you don't need a uP.

Also a lot depends on the fact what you want to do with the data output from the master. In most cases it is used by th uP for further analysis.

Sorry for some basic questions since I want to get it right without confusion.

Click to expand...

In that case, it is best to provide the complete scenario and use-case, rather than providing incremental information.

ewo said:

Actually, I want to send measurement data from sensors (temperature, current, voltage, etc). So, the data is continuous stream, about 100Mbps (not demanding, if it's fast enough, it will be just ok). Can SPI provide this speed? The distance between boards is about 15cm.

Click to expand...

Not sure if anyone "reads" posts by the OP sometimes....

SPI isn't a suitable 100Mbps data streaming protocol. It's really a protocol meant to allow access to memory mapped addresses via a very small number of signal lines. Due to the overhead of the address cycle you'll be running the interface with a clock rate that is greater than 100 MHz depending on the number of addresses bits you use. It's also not a very good protocol for running at even 100 MHz as it effectively has to meet timing at 200 MHz due to the opposite edge clocking. You certainly won't be able to use most of the SPI designs that exist as they typically overclock (sample) the interface to make it single clock edge synchronous.

Either use an existing board to board protocol from the Altera megacore library or make a simple homegrown one that sends a source synchronous clock, data, and a start of packet strobe (if you have known length packets) or a data valid (for variable length packets). The data can then be made serial, or a multi-bit bus to reduce the clock frequency. As your design is sensors I'm assuming that you won't need any backpressure for the streaming data (if you need this I would add a serially encoded backpressure channel as it doesn't need to be that fast).

Regardless of what you finally implement. You should first create a block diagram with signal rates (and other requirements) between blocks and then use that as a basis of what proposed interface buses would meet those criteria. This is part of what is called architectural design (which is what most people seem to like to skip over) and is more important than the HDL coding of that design. Without a good architectural design you could be setting yourself up for failure (e.g. using SPI as a 100Mbps data streaming protocol).

Hi,

I totally agree with ads-ee.

First I have to say 100Mbps sounds for me far too high for your task. (Sensor measurement, temperature ...).
If you really need this speed, you may have problems elsewhere, too. Like timing of ADC sampling, sensor communication, and so on. Then the FPGA to FPGA communication will be a minor problem.

The next is the "continous datastream". Are you sure you want a continous datastream? What happens when the communication to a signle sensor fails. You should decide how to handle this.
For sure it is possible in an contionous stream. Maybe it´s worth to think about sending in frames or packets.
In eiher case you need some kind of frame_sync. To be able to recognize wich data on the line belongs to wich sensor.

If you really need that high speed, then I recommend to use LVDS.
LVDS describes the physical interface (two wire differential signaling, curents, voltage levels, speed...), but you need some protocol to recover your data from the serial datastream.
Especially for LVDS signalling there are circuits with "SERDES" protocol. There are ready to use "serializer" and "deserializer" circuits.
If I remeber right, then some/many FPGAs have built in SERDES hardware. Usually only one twisted pair (per direction) is necessary from one FPGA to the other. It´s possible to transmit data rates in the range of Gbps.

Klaus

ewo said:

Actually, I want to send measurement data from sensors (temperature, current, voltage, etc). So, the data is continuous stream, about 100Mbps (not demanding, if it's fast enough, it will be just ok).

Click to expand...

You haven't given any reason why any interface protocol is needed. Wanting to get measurement data from one board to another implies you need wire, nothing more. No SPI, no Ethernet, no protocol at all because there is no definition of what is going on. Perhaps you mean that the raw measurement data goes into board 1, gets processed (i.e. changed) and the processed data is then sent to board 2...or not. Maybe all three boards need access to the raw measurement data in order to do their thing...or not.

On the assumption that there is a real need for the data to flow through each of the boards, here are some additional questions to consider that will help anyone that is trying to help get you an answer:
- Are these board to board interconnects only point-to-point or are the loads potentially distributed?
- How is the system configured from a ground perspective? In other words, what does each board use as a ground reference? Any chance of large ground differences?
- Could these three boards be mounted in a backplane instead? Alternatively, could one of the three boards also act as a backplane to interconnect with the other two boards?
- If the boards truly are all physically separate and can't be combined in a backplane, are the board interfaces strictly internal and not really of importance to the end customer? Since you have no pre-conceived notion of what protocol to use, this implies that no protocol is actually required to be used, you're just trying to get signals from here to there.
- What are some of the physical constraints on the board to board interconnect? Can a flat ribbon cable be used? Do you have to minimize the physical size of the connection?
- Will the connection be mechanically stressed in the system say by bending and unbending for example?
- What exactly is the format of the input measurement data? Analog? Digital? If digital, what format? Any reason why that digital format can't be used for the board to board interfaces?

Like I said, so far you have only given requirements that imply you need wire to connect the boards, nothing else. A 100 Mbps interface can easily be a parallel byte wide 12.5 MHz interface which would likely be the least costly approach. Or it could be a custom serial SERDES connection for slightly more money. Or it could be many other things that will likely be even higher cost approaches. You'll need to provide some answers to the above to give an informed opinion.

Kevin Jennings

ewo, as others have mentioned the more information you provide the easier it will be to provide answers that best match your situation. Since the complete architecture of your design is not completely clear i.e what end result you are trying to achieve, the data rate you have given certainly seems quite high.

- - - Updated - - -

By the way ewo, I have not mentioned this yet but will do so now. In Altera design suite they have a softcore called Nios II basically it is a processor written in VHDL by Altera (and encrypted) that we can customize to a large extent and then implement it into Altera FPGAs and CPLDs. The "economic" version of the core which uses the least amount of logic resources can fit into smaller logic devices also but that will still required 100s of logic blocks. This softcore can then be programmed via the Eclipse IDE. Basically we write the C program in Eclipse IDE (which is different from Altera Quartus II used to write and compile VHDL) and then compile this C program into machine code and then program the softcore inside the FPGA/CPLD to run it. In this way, you need not connect an external microcontroller/microprocessor to the FPGA/CPLD to use the SPI in that FPGA/CPLD, should you wish to use a microcontroller/microprocessor to send the data to the SPI core. That being said, please do not go down this path unless this is required by your project. At this time all requirements of your design are not clear to us. If this is a school project, you may be required to write your own VHDL SPI core, don't know yet,.

In my opinion, if your data is just from sensors (which usually have very low update rate, sometimes even once in few seconds) you can use just a Clock, Dout, Dvalid to send serially( Set valid high when data is valid, 1 data/clk). Receiver logic is also easy to implement (when valid is high, read serial data bit by bit and convert to parallel bits.

For pretty low speeds, you do not worry about about any protocol overhead. Plus, this makes it a simple fit on your Max 10 FPGA, eats least resource as well. You can use LVDS as IO standard for this. I hope this should be sufficient for several MHz without any issue. My opinion is "Try to make your design as simple as possible"

Hi guys! Thanks for answering. To be honest, this is a project assigned by my 'boss'. He decided that he want to use LVDS. Formerly, we have tried to use direct connection (wire) but it was unsuccesfull.

So, can you help me how to use LVDS? I've read from this guidelines: https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/hb/max-10/ug_m10_lvds.pdf

Are LVDS transmitter/receiver different with SERDES? I think based on the guidelines, MAX 10 ahs built in SERDES hardware, am I right?

So, how can I use this LVDS? There are some I/O standard for LVDS like true LVDS, emulated LVDS, etc. Some of them need external termination.
What is it? What does it do?
If I use different cable impedance Z0, how can i choose the right value of the resistors?


Should I use RJ45 connector or can I use other connector?

I read here: https://www.altera.com/en_US/pdfs/literature/wp/wp_lvdsboard.pdf

It says that I need CAT3/CAT5 cable or ribbon cable. The distance between boards is about 15-20 cm. What connector do you suggest?

For K-J, I'll try to answer those questions after this post.

- - - Updated - - -

- Are these board to board interconnects only point-to-point or are the loads potentially distributed?
Not sure what you mean.

- How is the system configured from a ground perspective? In other words, what does each board use as a ground reference? Any chance of large ground differences?
I think it has different GND since the boards are physically separated.

- Could these three boards be mounted in a backplane instead? Alternatively, could one of the three boards also act as a backplane to interconnect with the other two boards?
No it cannot

- If the boards truly are all physically separate and can't be combined in a backplane, are the board interfaces strictly internal and not really of importance to the end customer? Since you have no pre-conceived notion of what protocol to use, this implies that no protocol is actually required to be used, you're just trying to get signals from here to there.
It’s a prototype, so I can try to use any way to get signals from here to there. But I want to use LVDS.

- What are some of the physical constraints on the board to board interconnect? Can a flat ribbon cable be used? Do you have to minimize the physical size of the connection?
There are no constraints. The distance is about 15-20 cm. I don’t know if I want to use RJ45 or ribbon cable. See my previous post for the LVDS guidelines.

- Will the connection be mechanically stressed in the system say by bending and unbending for example?
I think yes, but not much. Most time it’s left in fixed position.

- What exactly is the format of the input measurement data? Analog? Digital? If digital, what format? Any reason why that digital format can't be used for the board to board interfaces?
Analog sensor—ADC—Max 10. Each board has the same sensors.

Hi,

LVDS <--> SERDES:
LVDS is the hardware, the physical layer.
SERDES is the protocol.

--> read about the two topics.

Often SERDES is used with LVDS.

Many FPGAs have built in LVDS drivers. And many FPGAs have built in hardware to decode the SERDES protocol. (clock recovery, PLL, shif register...)
Read if your FPGA supports LVDs and read if your FPGA supports SERDES.

If bot is true, then the wiring is easy: Just connect one LVDS wire pair from one FPGA (output) to the other FPGA (input). No additional hardware needed.
The is rest FPGA configuring /programming.

****

Cable impedance.
How long is the cable, Or is it just wiring on a PCB? Then follow the LVDS guidelines.
(Indeed if you run 100Mbits/s for 150mm on the PCB then you don´t need to take much care about line impedance and termination. But I recommend to follow wiring rules)

****

Klaus

I think mx Max 10 supports SERDES. This I got from the guidelines:

The MAX 10 devices use registers and logic in the core fabric to implement LVDS input and output
interfaces.
• For LVDS transmitters and receivers, MAX 10 devices use the the double data rate I/O (DDIO)
registers that reside in the I/O elements (IOE). This architecture improves performance with regards to
the receiver input skew margin (RSKM) or transmitter channel-to-channel skew (TCCS).
• For the LVDS serializer/deserializer (SERDES), MAX 10 devices use logic elements (LE) registers.

I need to connect the FPGA using wire with distance about 15 to 20 cm. The guidelines says that there are some standard I/O such as emualated LVDS that needs to use external termination. However, for True LVDS, it doesn't say that I need to add additional termination. So, I'll just need to connect the True LVDS pins of the 1st and 2nd FPGAs using ribbon wire, am I correct?

Altera MAX10 is essentially Cyclone III/IV technology and has no differential termination for LVDS receivers, review MAX10 handbook, chapter LVDS Receiver I/O Termination Schemes. Means you need external 100 ohm parallel termination on the receiver side.

If a single differential pair is used for the link, the receiver must implement clock recovery (CDR) or a similar synchronization scheme. Altera low cost FPGAs families Cyclone and MAX10 have no clock recovery hardware. A kind of soft CDR can be realized with PLL dynamic phase shift, but there's no Altera IP available for it and I won't suggest it as a FPGA beginner project. A UART like asynchronous receiver scheme with data oversampling (at least factor 4) could be used to synchronize the receiver.

Altera LVDS communication examples are expecting a separate clock channel along with the data, driving the LVDS receiver PLL.

For 10 to 20 cm, a common ground for both boards should be possible. If you are operating in very noisy environment, the common mode rejection of LVDS receivers might be insufficient. Additional common mode filters can help. Ultimately you'll use transformer isolation similar to ethernet, but it requires a DC free bit stream which can be generated e.g. by 8b/10b encoding, or more simply by Manchester encoding.

Don't use a ribbon cable for LVDS unless it's a 100 ohm twisted pair ribbon cable (typically expensive). If you must use a regular flat ribbon cable you absolutely must separate pairs of wires with grounds between each pair, that might work for short runs. A balanced CAT5 cable has the 100 ohm differential impedance and will work well for even longer runs than your 15-20cm.

As there are 4 pairs in a CAT5 you could use 1 for a source synchronous clock and the other three for data or even just two to make it easier. That will cut your transmission rate to a little over 50 MHz instead of 100 MHz. If you use all three your data rate will only be about 33 MHz.

ewo said:

Hi guys! Thanks for answering. To be honest, this is a project assigned by my 'boss'. He decided that he want to use LVDS. Formerly, we have tried to use direct connection (wire) but it was unsuccesfull.

Click to expand...

Understanding why a simple wire connection was not successful would probably be the first order of business.

So, how can I use this LVDS? There are some I/O standard for LVDS like true LVDS, emulated LVDS, etc. Some of them need external termination. What is it? What does it do? If I use different cable impedance Z0, how can i choose the right value of the resistors?

Click to expand...

Every signal you want to transmit will be a differential pair. You will use 100 ohm termination across the differential pair

Should I use RJ45 connector or can I use other connector?

Click to expand...

Since you say there are no constraints, you can use a flat ribbon cable (or flex circuit) connector. So far, you haven't given any reason why a simple parallel interface protocol would not work. Until you get the protocol nailed down, there is no point in discussing connectors.

For K-J, I'll try to answer those questions after this post.
- Are these board to board interconnects only point-to-point or are the loads potentially distributed?
Not sure what you mean.

Click to expand...

Point to point means there is one source and one load like this...
Sensor --->Board1--->Board2...
Distributed would look something like this...
Sensor --+--> Board1
|
+--> Board2

So which do you have?

- How is the system configured from a ground perspective? In other words, what does each board use as a ground reference? Any chance of large ground differences?
I think it has different GND since the boards are physically separated.

Click to expand...

Best to get that figured out really quickly then. If the grounds are unrelated, then you'll have to isolate each board from the actual connection. Typically this would be done with either a transformer coupling (like Ethernet) or an optoisolator. If you do need this type of connection, then you'll want to drastically limit the number of board to board connections which will dictate using some form of serial interface. If not, then simple parallel is still in the running.

- If the boards truly are all physically separate and can't be combined in a backplane, are the board interfaces strictly internal and not really of importance to the end customer? Since you have no pre-conceived notion of what protocol to use, this implies that no protocol is actually required to be used, you're just trying to get signals from here to there.
It’s a prototype, so I can try to use any way to get signals from here to there. But I want to use LVDS.

Click to expand...

If the grounds are unrelated you can't use LVDS directly, you'll need to go through a transformer or optoisolator. If the grounds are related and can be guaranteed to always be within a volt or so of each other then LVDS will work.

Before you choose a solution, you have to understand your problem. Simply wanting to use LVDS because the boss said to is not a justifiable reason by itself. Based on the grounding system that is (or is not) in place between the various boards will dictate whether or not LVDS could be used as the signalling mechanism or not. It doesn't work the other way around that you choose LVDS regardless of the grounding.

- What are some of the physical constraints on the board to board interconnect? Can a flat ribbon cable be used? Do you have to minimize the physical size of the connection?
There are no constraints. The distance is about 15-20 cm. I don’t know if I want to use RJ45 or ribbon cable. See my previous post for the LVDS guidelines.

Click to expand...

Simple flat ribbon cable (or flex circuit) with a parallel data protocol (i.e. clock, data and data valid) is the simplest way to go for the protocol. Assuming that the grounds are within range to allow for LVDS signalling, then use that as the electrical signalling mechanism.

- What exactly is the format of the input measurement data? Analog? Digital? If digital, what format? Any reason why that digital format can't be used for the board to board interfaces?
Analog sensor—ADC—Max 10. Each board has the same sensors.

Click to expand...

Not quite sure what this is supposed to mean.
- Is the ADC with the sensor or with the board? I'm assuming that the Max 10 is on the board.
- What do you mean 'Each board has the same sensors'?
- What is being communicated between the boards? The raw analog sensor data in parallel to each board? The digitized sensor data in parallel to each board? Something else?

I'll also note that the grounding of the system has to take into account the analog sensor itself.

Kevin Jennings

Multiple boards inside an instrument enclosure can be usually operated with common signal ground (provided it's not a power electronics device). But as K-J mentioned, e.g. noise suppression for analog signal connections can require separated grounds. You'll know the other signal and power wiring and then decide about appropriate I/O standards for fast logic signals.

Presuming common ground is possible, standard ribbon cable should work with a GND DATA+ DATA- GND wire sequence. It's not exactly 100 ohms but near enough to work for 0.5 or 1m with 100 MBPS digital data.