Hi,
we can only guess, as long as you don´t provide the related schematic.
So should I modify my PCB and try again?
PCB layout has also a big influence on ADC reading precision.
Maybe power supply.
My recommendation:
Find out where the error comes from, then change this detail.
So should I modify my PCB and try again?
If you have too much time, you are free to use the trial-and-error-method.
****
give informative data. You gave:
* ATMEGA8: the controller itself won´t craete this problem
* 12V relay. Relay voltage is not important. The relay itself will not cause your problem
* the same is with your LM326-2.5 reference.
--> no useful information.
****
Hysteresis:
--> wikipedia ia always a good source for informations:
https://en.wikipedia.org/wiki/Hysteresis
In short: It is not a good idea to switch ON/OFF any device at a fixed threshold.
Instead you could switch on your relay at voltages above 12.1V and switch it off at voltages below 11.9V.
The difference voltage between ON and OFF = 12.1V -11.9V = 0.2V is called hysteresis.
Klaus