Just a suggestion.
Un-balanced ? do mean that the voltages
May not be the same on each phase due to loading.
As there is not neutral point then you can either create one
By using 3 resistors and use the resistor connection junction
as the reference point for your measurements.
The instrumentation should not draw any significant current
To up set the false reference point.
If the phases needs to be completely isolated then
a voltage rectifier across each phase could be used.
It would require multiplex ADC input (or 3 ADC), to acquire
Each phase peak voltage to calculate RMS
Value using the micro computer.
You can find many procession full wave and half wave AC to DC
Rectifiers circuits design suggestions on the WWW, but if it is for power applications, Whereby the phase voltage is much larger than a diode Voltage drop, then maybe a simple diode peak voltage rectifier Would suffice.
To maintain the phase isolation small transformers (one connected to each phase) Could be used before the rectifiers Or an isolated front end instrumentation amps could be used.