Video processing using FPGA and GPU

So, I have this really abstract idea, any tips or ideas will be much appreciate.

I have an input video stream, I want to process it and throw it out as fast as possible, I don't believe a regular PC can do it because of the overhead of the CPU (getting the video stream and throw it into to memory).

I thought of the following.

I will acquire the video stream using USB3, connected to a FPGA (Spartan 6), I have seen a board which does it by opalkelly

https://www.opalkelly.com/products/xem6310/

That is how I plan to dump the the data into the memory.

Then I'll have a GPU which will use this memory for video processing, I thought about CUDE or such.

Finally throwing the data out using Ethernet or any other connection.

The problems I see here are the interface between the GPU and the memory, I'm not sure exactly how this two are going to be connected.
Another one is "throwing" the data out, the GPU may process the data faster the I can output it.
I bet there's a ton of other problems.

I'm new to video processing, I do have much experience in FPGA due.

As I said, any help will be much appreciate.

Thanks a lot.