I am not really able to comment on what may happen in the case of an infinite load, but I think you mean a load that tends to infinity and I think you are on the right track. A current source is a quantified and limited thing.
When you think of a voltage source you should remember that voltage and current are reciprocal quantities.
A zener diode is a good voltage source when fed with a suitable current. We could use a voltage follower to get quite large currents, and whatever resistance we put in the load would receive the same voltage, subject to the limits of the design.