Woking out transition thresholds

[Aussie Susan]

Context: My home automation system has a couple of monitors for my gas and water consumption. The meters are mechanical and have a magnet on the least significant 'digit' of the dials. I have a 'Hall Effect' transistor that measures the magnetic field from each of these meters every few seconds. In this way I wont to detect the number of complete rotations of these display wheel and there by work out my gas and water consumption over days/weeks/months.
The monitoring electronics measures the voltage from the Hall Effect transistor. To detect the turns, I have a 'Schmitt Trigger' system that turns on and off at predefined voltage thresholds - the lower one turns 'off' and the upper one turns 'on'.
My problem is to best calculate the 'on' and 'off' thresholds.
I suspect that this is complicated by some environmental factors (temperature????) as the initial thresholds that I worked out seem to drift over time (months).
I have access to the individual measurements that are in the home automation system database going back about 6 months (at least).
Plotting out the voltage for the last couple of days for the gas I get:


The gas hot water pilot is continuously using a little gas and there is a reasonable 'difference' when the magnet comes close to the transistor. Therefore I'm reasonably confident about setting an 'on' threshold at about 3700 and an 'off' threshold at about '3500'.

However for the water usage I get:

where the pauses show where no water is being used.

Given the noise on the measurements and the nature of the underlying 'signal' I'm looking for suggestions as to how best determine the thresholds for the water monitor.

I might be able to make this a dynamic system in the future but for now just being able to run a program every day/week/month (until I see how stable the threshold values really are) is fine.

Susan

 

[Aussie Susan]

Here is a bit more information:
The water meter is the one liked to in post #6 which can measure up to 66L/min or about 1 L/s. The least significant wheel wheel turns once for 10 L and therefore I should be getting a complete 'cycle' at most every 10s.
The water company supplies water via a 20mm diameter pipe. they don't specify a maximum delivered flow rate but I assume that it is less than that max for the meter.

The circuit that I'm using is:

The Hall Effect transistor is attached to the main circuit board by a 4 core cable (1 left unused) that is about 20cm long. The power is from a 5V SMPS connected to the 'TP1' screw terminal.
The ESP32S3 is running the ESPHome software.

It probably doesn't mean anything to most of you but the Home Assistant yaml code that generates the software is:

external_components:
  - source:
      type: local
      path: /config/My_Components
    components: [my_pulse_counter]

sensor:
  - platform: adc
    pin: GPIO2
    name: "Water Meter Reading"
    update_interval: 1s
    raw: True
    device_class: water
    id: "water_meter_reading"
    attenuation: 12db

binary_sensor:
  - platform: analog_threshold
    name: "Water Meter Threshold"
    id: water_meter_threshold
    sensor_id: water_meter_reading
    threshold:
      upper: 2450
      lower: 2300


my_pulse_counter:
    id: "water_pulse_count"
    sensor_id: water_meter_threshold
    name: "Water Pulse Count"
#    filters:
#      - multiply: 5

The ADC reads the data from the 'A0' pin every second, passes it to the binary sensor that acts as the Schmitt Trigger. The output of that goes to a pieve of code I wrote that is based on the 'ESPHome Pulse Counter' module but takes it input form the binary sensor and not a GPIO port.

The following is a sample taken from the last hour and a bit:

The red line is from the ADC output of the device. The green line is my external software 'Schmitt Trigger' and it shows 7 'pulses equivalent to 70L

This is equivalent from the Home Assistant software:

Again to top trace is from the ADC and the bottom from the pulse counter.
Susan

 

[KlausST]

Hi,

Power supply:
SMPS output may be noisy. I see no decoupling in direction to the sensor. I highly recommend at least a series R and a C in parallel to the sensor - close to the sensor.
... Or: why not using the ESP32-module 3.3V output? Still a decoupling of the sensor power supply won´t hurt.

Signal:
As already written: IMHO every ADC input should be designed to satisfy nyquist. Here a simple RC (C close to the ADC) should be highly beneficial.
Sampling rate:
I´d use one with an integer multiple of your mains frequency.
let´s say 16 x 50Hz = 800Hz. Just add the numbers up and output them ever 800th cycle (= 1s).

Signal waveform:
I expect a highly distorted sine wave. And to get a reliable signal maybe a hysteresis is a good way to go.
Like:
* if (averaged_ADC_value > 2500) --> dig_output = 1
* if (averaged_ADC_value < 1500) --> dig_output = 0
For sure you should adjust the thresholds.

Water flow rate:
I don´t know length nor pressure, but I wouldn´t be surprised if the flow rate exceeds 500l/min.
(I´ve just seen a table where it says: ca 500l/min for a 1" pipe, 50m length, 6 bar pressure)

Klaus

 

[Aussie Susan]

@KlausST - several points from your last post:
- the UGN3503 requires a supply voltage of between 4.5V and 6V (as per the data sheet). I take the point about decoupling the transistor itself though
- re the ADC sampling frequency - see below
- the threshold values are exactly the reason I put this question in the 'Mathematics and Physics' forum - I think I can sort out the rest but I was looking for ways to best calculate the thresholds other than by eye (as I'm doing now). @Tony has given me some good ideas (after translating from a circuit to C code)
- water flow rate: I have just tried to fill a 9L can from a tap that is right next to the water meter at 'full blast'. The spike just after 10:25 is about 14 seconds wide (I actually used a stop watch that showed 12.28s but that could be a little off.) That is about 44l/s as a rough calculation (given the various inaccuracies in the measurement system) but is under the maximum flow rate possible for the meter so the assumptions about 10L in 10 s to give 1 pulse are probably valid for the underlying signal.

The other 'high flow' items in the house are the 6/3L tanks for the toilets. A quick measurement there shows a 6L fill in about 24 seconds.

As I read it, the sampling rate is more than adequate to approximate what I expect to see as the fastest flow rate for the house.

Also it shows something that I should have looked at earlier: there is not much noise on the overall signal but that it is somewhat 'impulsive' in nature when it does occur. (It looks to be in the 10 to 20mV range in general.)

Overall I'm happy to leave this for now and play a bit with the maths from here on.

Thank you all for the assistant and suggestions - always appreciated.

Susan

 

[D.A.(Tony)Stewart]

Nice transcoding my envelope detector susan. Results look better and testing shows you are close to my magic number of SNR >=10

The noise seems to be asymmetric ( more negative ) which tells me that it’s getting clipped from something like common mode offset so hardware Improvements are needed to avoid threshold skew going negative.

The oversampling that Klaus suggested would also help but will include some negative drift after averaging negative spikes.

 

[KlausST]

Hi,

The oversampling that Klaus suggested would also help but will include some negative drift after averaging negative spikes.

true somehow. but the overall SNR (which refers to signal quality) will be improved, because the "uncertainty" will be reduced. The error amplitide gets smaller.

***

first: you are free to work as you like.
But I highly recommend to take care about the noise (frequencies) first.

Signal quality wise these noise spikes (mainly in negative direction) is what kills the overall performance.

You say: regarding sampling frequency you are an a safe side for the flow signal. Yes, true.
I understand that in your thinking you focus on the sensor signal. YOU can focus, but your circuit does not. It can not destinguis between sensor signal and introduced (however) noise and other errors.
And thus - as long as you don´t design your circuit "to focus" on the right signal, you are surely NOT on the safe side regarding noise frequency.

Technically I see two options to achive a "suitable" sampling rate:
* either use higher sampling rate to care for the noise frequencies and do the filtering on the digital side
* or use an appropriate anti aliasing filter in front of the ADC.
(usually I use both)

As already said: the noise and it´s frequecies get "aliased". And in your case these alias frequencies are not predictable and the amplitide of these alias frequencies are not predictable, too. In worst case this may lead to random failure in the field. Random in the meaning: some boards may fial, others not, but also random in time. Some boards may fail every 5 mintes, others may fail maybe once a year.
I personally simply don´t understand why some people take days to evalute a "workaround" while the solution may be as simple as soldering a 2.2uF (ceramics) capacitor in parallel to R2. (Schematic of post#21). "I don´t understand" is not meant that I get angry ... I just don´t get the reason for "why" or "why not" into my brain cells.

****

If I´m not mistaken you are into radio communication. If so, then you know how to build an antenna. And every 20cm wire acts like an antenna - wanted or not.
20cm is quite a suitable antenna for nowadays WiFi, cell phones or RF remote control applications. So your 20cm wire WILL pick up these frequencies (some better than others). So if one of these HF sources is nearby exist ... you automatically get these "irritations".
But it´s not only this high frequencies, it may also be a refrigerator nearby switching ON/OFF, an electronic controlled lamp, a PWM controlled LED stripe, the soldering iron, an ESD event, the (common mode) noise of your SMPS, ....
No capacitor means no defined cut off frequency, means all these frequencies ... the whole bandwidth from F_sample/2 up to unknown WILL create alias frequencies that can NOT be canceled out on the digitial side.

The suggested 2.2uF will limit this "to be aliased band" from F_sample/2 to about 10 Hz. So it also attenuates low frequency signals like mains hum. A huge improvement. Any capacitor from 0.1uF to 10uF should work quite well.
A simple ceramics capacitor connected as close as possible to the ADC input. Nothing fancy. No rocket science. Nothing pricy.

Indeed In the past decades of electronics design, I think I never used an ADC without analog filter (in simplest case just a C) just to ensure that I only read what I want to read.

.. all just a well meant recommendation.

Klaus

 

[D.A.(Tony)Stewart]

Hi,


true somehow. but the overall SNR (which refers to signal quality) will be improved, because the "uncertainty" will be reduced. The error amplitide gets smaller.

***

first: you are free to work as you like.
But I highly recommend to take care about the noise (frequencies) first.

Signal quality wise these noise spikes (mainly in negative direction) is what kills the overall performance.

You say: regarding sampling frequency you are an a safe side for the flow signal. Yes, true.
I understand that in your thinking you focus on the sensor signal. YOU can focus, but your circuit does not. It can not destinguis between sensor signal and introduced (however) noise and other errors.
And thus - as long as you don´t design your circuit "to focus" on the right signal, you are surely NOT on the safe side regarding noise frequency.

Technically I see two options to achive a "suitable" sampling rate:
* either use higher sampling rate to care for the noise frequencies and do the filtering on the digital side
* or use an appropriate anti aliasing filter in front of the ADC.
(usually I use both)

As already said: the noise and it´s frequecies get "aliased". And in your case these alias frequencies are not predictable and the amplitide of these alias frequencies are not predictable, too. In worst case this may lead to random failure in the field. Random in the meaning: some boards may fial, others not, but also random in time. Some boards may fail every 5 mintes, others may fail maybe once a year.
I personally simply don´t understand why some people take days to evalute a "workaround" while the solution may be as simple as soldering a 2.2uF (ceramics) capacitor in parallel to R2. (Schematic of post#21). "I don´t understand" is not meant that I get angry ... I just don´t get the reason for "why" or "why not" into my brain cells.

****

If I´m not mistaken you are into radio communication. If so, then you know how to build an antenna. And every 20cm wire acts like an antenna - wanted or not.
20cm is quite a suitable antenna for nowadays WiFi, cell phones or RF remote control applications. So your 20cm wire WILL pick up these frequencies (some better than others). So if one of these HF sources is nearby exist ... you automatically get these "irritations".
But it´s not only this high frequencies, it may also be a refrigerator nearby switching ON/OFF, an electronic controlled lamp, a PWM controlled LED stripe, the soldering iron, an ESD event, the (common mode) noise of your SMPS, ....
No capacitor means no defined cut off frequency, means all these frequencies ... the whole bandwidth from F_sample/2 up to unknown WILL create alias frequencies that can NOT be canceled out on the digitial side.

The suggested 2.2uF will limit this "to be aliased band" from F_sample/2 to about 10 Hz. So it also attenuates low frequency signals like mains hum. A huge improvement. Any capacitor from 0.1uF to 10uF should work quite well.
A simple ceramics capacitor connected as close as possible to the ADC input. Nothing fancy. No rocket science. Nothing pricy.

Indeed In the past decades of electronics design, I think I never used an ADC without analog filter (in simplest case just a C) just to ensure that I only read what I want to read.

.. all just a well meant recommendation.

Klaus

I was an expert in EMI since '77 and have found many sources in the home besides grid e-fields 50V/m (50/60 Hz) which causes hum and can overdrive the common mode range of an analog input even if the differential is small.

One that is very common is thermal dry switches like those in the lab in the late '70s using Weller stations and now with coffee makers and a hot plate. These will give random spikes every few to 10 minutes. depending on the amount of coffee and insulation to the pot. But your plots are every 2 to 4 seconds which could be aliasing the 1 Hz crystal controlled sample rate software with the 50 Hz stable CM noise or diff mode (DM) noise , maybe .... In any case the peak noise is more than 10 % asymmetric of the total peak to peak signal which means the mean threshold will be in error by that amount, but then there is margin loss created by hysteresis if using 50% vs 33%.

I'm off to play Pickleball and now moved my Tower from WIN 11 to Linux Mint Cinnamon. and liking it. So far pretty painless with some input from Grok3 and very fast install.

 

[D.A.(Tony)Stewart]

Back.... the bottom line is you must use analog filter ferrite CM chokes and/or filter caps to suppress the low freq impulses seen in your plots that occurs due to noise BW higher than the Nyquist limit of 0.5 Hz when sampling at a 1 Hz rate. With more details on your circuit and I/O cables, if you wish, we can advise better. Sometimes it is conducted CM from SMPS floating DC supply and other times radiated on the cables and often both.
So it is not what it looks like (~ 0.3(?) Hz spikes ) rather it is undersampled and/or unfiltered aliasing noise.

This is why Klaus indicated you need to sample much faster than the noise then average and decimate to 1 Hz and/or put a big cap on the signal.

 

[Aussie Susan]

After going all the tasks to get ready for Easter I had a chance to do a bit more playing.

Thinking about the setup, and not to discount all of the above comments about noise from various sources, I realised that I was getting about the same noise from both the gas meter reader and the water meter reader. They were about 30cm from each other and shared the power supply so therefore I would expect any external interference to be the same in both signals. Post #1 shows noise on both signals and about the same level. However when I zoomed in on the signals I found:

As you can see the 'noise' occurs mainly at different times. There might be some coincident noise pulses - those going up do seem to coincide and aer therefore possibly from the same source.

Next I used another ESP32S3 that I had and set it up with some fixed resistors in a 2:1 divider and connected that to the ADC port and created a new ESPHome device to read that port. The input voltage (according to my DVM) was 1.633V (to 1.634V). The result was:

I therefore conclude that there is a fair amount of noise from the way I have the ADC set up.

The ESPHome software for the ADC does have a few options and playing with those I can get the signal to be:

With the same hardware in the same location, that gives the correct output of 1.63V +/- 0.01V.

Finally I gradually applied the changes to the water meter and got with no water running:

There is a bigger variation but smaller than before.

I then ran water (flushed the toilet on full flush so about 6L) at about 15:45 and 15:47 to see the approximate range of the signal, adjusted the threshold values (to high 1.8, low 1.7) (top trace of "On" an d"Off") and then flushed twice more (15:53 and 15:55) and can see the pulse counter trigger on the 15:55 flush as the signal had a positive edge.

Therefore I am concluding that much of the noise came from the incorrect set up of the ADC. There may still be some externally generated noise (the Hall Effect transistor data sheet does say that it is 'low noise' (90uV) and 'extremely sensitive' (typ 1.3mV/G)) but what I have now is certainly better than what I did have and 'good enough' (at least for now - many more projects on my 'to do' list!).

I'll look at some of the additional circuitry if/when I get to Version 2.0.

@KlausST - You got me : VK3ANZ is my callsign.

Thanks again to everyone - I have learned a lot from some very knowledgeable people.

Susan

 

[KlausST]

Hi,

I see: using a simple capacitor at the ADC input is no option for you.... Not even worth mentioning it.

Klaus

 

[Aussie Susan]

@KlausST - it would be except that the physical sensors were put in just over a year ago. Since then my physical ability has reached the state that I can't easily access them or do anything with them so I'm forced to do what I can and that is on the software side.
Susan

 

[D.A.(Tony)Stewart]

I appreciate your efforts . I wonder how many if any switched regulators are used which create some grief with common mode then when lines get unbalanced and more grief with multiple grounds. Since ground by definition is 0V there can only be one central ground ( unless heavily distributed) THe water meter is no doubt protected by earth (PE) But SMPS will inject ground noise to divert it from the line, so caution is needed when more than one SMPS is used. It would help to know if this is the case. I often use wide-float braid wire for lower inductance or coax. to distribute 0V.
ADC's are often single-ended and thus lack the high CM rejection of a differential Op Amp or Instrument Amp. (INA)

HTH
Tony

So it is not easy to put any Caps on the ADC input. (try 0.1uF ~ 1uF)
I might also use a high inductance Balun or 1;1 transformer wired as a Balun (=) to improve noise rejection. Our crawl space is packed too .

Happy Easter.