Most of the signals are differential and running at 640MHz. Could you please tell me how to do this impedance matching or where I can learn it.
Then you're design is going to be high speed and you will need to make sure the transmission lines all have good signal integrity.
You'll have to decide what trace widths you will be using the stackup of the board, I'm sure others here will pipe up about doing the board in two layers at home with no ground planes....don't listen to them ;-).
If you want this to work without a lot of problems, then use a multi-layer board with ground planes built by a good fab house. Sandwich most if not all of the high speed signals between two ground layers (stripline), route different differential pairs 2-3x the spacing between the differential pairs. Stackups like top-gnd-sig1-sig2-gnd-bot would work, where the sig1 layer is routed primarily orthogonal to the sig2 layer. what you don't want is any parallel runs on adjacent layers that are within say 2-3x of the trace width, the more spacing the better.
Most PCB layout tools and some online tools have a way to put in the trace widths, stackup, and such and will calculate the trace impedance. I don't do any board design or layout work, but I know that there are tools available to help with this.
As you haven't done any of this before, you might want to read whatever you can find on signal-integrity and board design online. This generally isn't the kind of design you start off with as the first foray into making a PCB. At least in your case it seems to be just a passive bridge board, so what's important is mostly controlled impedance and signal integrity.