For now,you should think first about the algorithm without the seven segment display. Think of it as a C code. you have 4 cases : A>B , B>A, A=B, otherwise. You need to write an if statement for each case. you should write something like this :
if A>B then
output <= "1010";
elsif B>A then
.
. and so on . tell me if you need more help coz I don't know what's your knowledge in VHDL.
That's generally a good suggestion. The present problem has however one case, that can't be represented by a circuit, because it's not synthesizable. In other words, the problem refers to simulation and behavioral code only. (In my opinion, it isn't a well considered exercise)