How would you amplify the signal from a comparator?

I have my comparator based capacitive sensor working.

As it is it gives me a differential output voltage of 0.1V to 0.5V roughly.

Presumably the output current is quite high because an LED is developing enough power to be visibly illuminated, though not at full illumination.

I would like to amplify that signal to provide a voltage differential of 0.1V to 5V roughly.

I created a common emitter amplifier: Zin=50k, Zout=2k, Voltage gain = 20, Vcc=6V. The output of the comparator goes to the base of the transistor and the collector connect to the LED.
But this doesn't really work because, with DC coupling and both ends the quiescent current just illuminates the LED permanently and, with AC coupling at both ends, the LED is initially off but then when I touch the sensor the LED seems to become illuminated permanently.

How would you go about properly amplifying the comparator signal? Directly coupled opamp?

I cannot see the schematic of your comparator. Typical TTL comparators offer a logic-level output, 0-5V, so no amplifiers are needed. For your lower output voltages, connect a TTL comparator to get 0-5V output, or a CMOS comparator to get 0-15V. No amplifiers are needed and circuits are very stable.