Woking out transition thresholds

[Aussie Susan]

Context: My home automation system has a couple of monitors for my gas and water consumption. The meters are mechanical and have a magnet on the least significant 'digit' of the dials. I have a 'Hall Effect' transistor that measures the magnetic field from each of these meters every few seconds. In this way I wont to detect the number of complete rotations of these display wheel and there by work out my gas and water consumption over days/weeks/months.
The monitoring electronics measures the voltage from the Hall Effect transistor. To detect the turns, I have a 'Schmitt Trigger' system that turns on and off at predefined voltage thresholds - the lower one turns 'off' and the upper one turns 'on'.
My problem is to best calculate the 'on' and 'off' thresholds.
I suspect that this is complicated by some environmental factors (temperature????) as the initial thresholds that I worked out seem to drift over time (months).
I have access to the individual measurements that are in the home automation system database going back about 6 months (at least).
Plotting out the voltage for the last couple of days for the gas I get:


The gas hot water pilot is continuously using a little gas and there is a reasonable 'difference' when the magnet comes close to the transistor. Therefore I'm reasonably confident about setting an 'on' threshold at about 3700 and an 'off' threshold at about '3500'.

However for the water usage I get:

where the pauses show where no water is being used.

Given the noise on the measurements and the nature of the underlying 'signal' I'm looking for suggestions as to how best determine the thresholds for the water monitor.

I might be able to make this a dynamic system in the future but for now just being able to run a program every day/week/month (until I see how stable the threshold values really are) is fine.

Susan

 

[betwixt]

Just to confirm Susan, is the water metering using the same magnetic pulse system as the other meter and you are using a Hall sensor in the same way?

One clearly shows a distinct on/off action while the other has many levels. I suspect that is caused by slower passing of the magnet and hence it stopping at differing distances from the sensor. Perhaps the solution is to use 'delta' measurements rather than absolute ones so you no longer have a high and low threshold but one depending on how much change since the previous reading. Can you plot that as a graph using the same data as you showed before to see what it looks like?

I'm interested in this because one of the design jobs I will have soon is to measure water flow from a small pumping station (~ 20,000 litres per day) and send the readings to a monitoring system about 0.5Km away. It's like your problem but scaled up 100 times! I've been thinking of using ultrasonics in view of the large flow, the existing system to be replaced uses magnets on gear wheels but a reed switch to count rotations and it isn't reliable long term.

Brian.

 

[KlausST]

The signals showing bad signals with unclear threshods.

Maybe a wiring problem, maybe a schematic problem (0.6V true signal on a 3.5V biased signal) like a wrongly calculated resistor.
Maybe too much noise, missing filters...


Klaus

 

[betwixt]

I think this may be a Hall sensor with analog output Klaus. These meters work mechanically and usually have a mechanical counter on them but driven through a gearing mechanism. They glue a small magnet to one of the gears and make provision in the casing for either a reed switch or Hall sensor. Each turn of the gear wheel is intended to produce one pulse from which the flow can be calculated. When a Hall device is used, the output will depend on how close the magnet stops relative to the sensor so in theory any level could be produced. I think the steps in the graph are just indications of where it stopped each time water flow stopped.

My reason for looking at the amount of change rather than absolute measurement is it could stop at a low level and then stop again at an even lower level without a peak between them. Any change, up or down would still indicate a flow. Theoretically, it could stop with the sensor in the exact same position every time but that would be unlikely. A graph of absolute change "ABS(new reading - old reading)" should see if that method is feasible.

Brian.

 

[D.A.(Tony)Stewart]

Your S/N ratio is only 4 to 1 which leads to unacceptable error rates even at ideal thresholds.

You need at least 10:1 (20 dB random SNR gives 1e-14 BER or so 1e-45 if using ideal integrator discriminators) ) by reducing the noise with shielding ( STP wire with a CM choke) Then a LP filter to suppress the crosstalk as long as the supply is clean.

When I was in the AMR business, our magnet sensors were more like SNR= 20:1. If the noise is random that's a very low error rate but if you have relay flyback spikes, those must be eliminated.

Next you must ensure you have at least 5 times the Nyquist rate to get enough of the full scale swing with sampling of a sine wave. That means at least 10 samples per cycle. The Nyquist rate does not guarantee full swing, only that it can resolve with infinite SNR a maximum frequency with 2 random phase point in a cycle.

When these tasks are successful then using a Shmuck trigger will be reliable at the mean with hysteresis always greater than the noise Vpp. If this still cannot be achieved, then increasing Hall sensitivity might be possible adding a weak external magnet on the other side of the Hall sensor to bias the magnetic field where it might be more sensitive or moving the sensor for a stronger response.

AMR:

Automatic meter reading (AMR) is the technology of automatically collecting consumption, diagnostic, and status data from water meter or energy metering devices (gas, electric) and transferring that data to a central database for billing, troubleshooting, and analyzing.

At Iris Systems circa mid-'90's in Winnipeg, MB, we had a 928 MHz wireless network from the old mechanical meter to the head office utility Unix databases for Gas , Water & Elect. So we had 15 H/W designers and 15 software designers and I was Ops Mgr. for quick turn SMD prototypes, Master Registry, QA, Purchasing, Stores and Testing. This biggest problem was getting utilities to order more than 100 units so after 7 yrs, the patents and all were sold to Itron, who is the ,market leader who used a method of replacing all the meters instead of retrofit the old ones but then their broadband wireless failed so they still require a human to walk door to door to scan the meter for data, but then we can look at the plots online by hour, day, week, month or year.

 

[Aussie Susan]

@betwixt - yes: both systems use the same hall effect transistor and ESP32 MCU. Also your description to @KlausST is exactly right. The water meter is an Elster Volumetric type (https://www.mwatechnology.com/products/honeywell-elster-v100-cold-water-meters/).
I have attached a different plot of the data - the top red line is the raw data and the bottom blue data is the 'difference' you wanted to see. (I don't think it helps much TBH). The values are from midnight yesterday to now but are representative of the data.

@D.A.(Tony)Stewart - yes I know the noise is high and I should be trying to reduce that - perhaps a low pass filter etc.. As @betwixt mentions above, the wheel turns only when the water is flowing which is rather intermittent. Also using a hall effect transistor means that it can stop 'part way around'. However the key point is that I only need to count when it gets to the 'top' which indicates that the least significant wheel has turned to a given point - the next time it gets there means that the specified amount of water has flowed through the meter. Given that, I think the difference between the 'top' and the 'bottom' values (despite the current noise level) should be sufficient for me to find suitable trigger points for the Schmitt Trigger part of the program.

I have been thinking about this as well. Now that I have got my iMac reading the database from the home assistant system, I can start to perform some more sophisticated analyses (python, numpy, matplotlib - which I used for the attached plots rather than the previous ones which came directly from the home assistant package - and pandas etc.) That is the basis for my question: what sort of analysis might help.

Thanks for the assistance.

Susan

(PS: Sorry about the delay in responding - its school holidays here and my daughter was sick today, so grandparent duties took precedence! Also I hope for more time shortly to try other analysis methods.)

 

[betwixt]

Sorry for the extra work - my thoughts clearly didn't help.
I'm not sure the graph is showing what I intended though. Periods where the red line is almost flat should show as little change (=0) on the blue line. It looks almost like it shows the background noise instead of filtering it out. The blue line should only be positive and rise when there is a change in the red line.

The only other thing I can suggest given the almost random possibilities of measurement is to find the peak and back off some degree, perhaps 10%, and repeat the peak check continuously so it tracks any changes caused by drift or environmental influences. I would assume a peak would occur fairly frequently, specially during hours when water is being used.

Brian.

 

[D.A.(Tony)Stewart]

Susan my comments on SNR requirements were for binary systems. So reducing the cause of the noise is paramount (crosstalk?, ground shift? from shared current grounds , supply ripple? unshielded vs STP wire?

After filtering to match your spectrum is ideal once you avoid the noise within your signal spectrum (like a spike).
If 10 Hz use a 10 Hz Bessel LPF ( one quad Op Amp and a few discretes ) or a fancy FIR S/W filter. Over-filtering can cause errors too.

Although I cannot tell how much of your noise is not random, but there are some spikes that will cause errors.
If lucky the noise is outside your sensor signal BW.

 

[D.A.(Tony)Stewart]

This simulates a 10 Hz LPF which you can scale down by increasing C or R or both Bessel-type minimum group delay.

The wire plot shows the filter input, with a higher white noise than the triangle signal.

 

[Aussie Susan]

@betwixt - Never apologise for making reasonable requests and suggestions. I was always needing to access the database from my iMac (as hinted at in my OP) and learning about the Python libraries is always a good idea.

Just for a laugh, I extracted a short sample from this morning when I tapped a bit of water and then took a shower - see below.
The noise seems a lot lower - which is something I suspected. Also the underlying signal frequency is a lot lower than the noise which is why (I think) the 'difference' plot does not really show the transitions - there are multiple noise oscillations as the signal rises and falls.

More playing later today (I hope). I want to try taking the FFT of he signal to see how wide spread the 'noise' is and also to see if that lets me calculate the cutoff frequency for a LPF. (Also to just learn....)

Susan

 

[D.A.(Tony)Stewart]

So far so good , it seems hysteresis of 1/3 of the mean range would be a good choice for discrimination.

 

[FvM]

As long as you don't want more than count full pulses, thresholds at e.g. 1/4 and 3/4 of sensor output range do the trick.

A problem arises if DC bias is unstable, temperature dependent or just unknown. Measuring unit could try to identify voltage swing automatically, but only if a minimal change rate per day is achieved.

I wonder what's causing noise, if it's actual hall sensor output or picked-up hum and RF interferences due to unsuitable measurement circuit.

We didn't yet see measurements clarifying if filtering can reduce effective noise level.

 

[betwixt]

For clarification, is the latest red graph showing the voltage from the sensor or the number of pulses over a time period?
I'm trying to establish whether the 'noise' is really random variations in signal or whether it is showing the wheel stopping in the same place repeatedly. The unknown in this is how the flow is detected, does the mechanism, which I assume is an impeller, have any preferential stopping points (cogging for example) or can it stop freely at any angle. How many pulses per liter does the unit produce? For example does the graph show three rotations of the sensor wheel or are some hidden in the sampling period.

A side thought, assuming your water usage never varies much from average, can you use a monostable (in software or hardware) to block any signal faster than your maximum usage.

Brian.

 

[D.A.(Tony)Stewart]

Beware that Hall Sensor amplitudes can drift
If the mean or gain drifts….
Too large a hysteresis gap might cause a false negative (missed events)
Too small a hysteresis gap might cause a false positive ( invalid event counts)

There are algorithms or filters that involve previous cycle time and rate change, (PID corrections) but the proportional threshold I had plotted is ideal for random drift and CMOS Schmitt Triggers are close to that at 1/3 hysteresis of the valid range which also have some random drift and thermal drift.

 

[Aussie Susan]

is the latest red graph showing the voltage from the sensor

This. From what I can tell the noise is random and from some as yet unknown source - something for me to find out sometime. I seem to get similar levels of noise from the gas meter monitor that uses the same hardware, circuit and software, but I suspect that the hall effect transistor can get a bit closer to the magnet in the gas meter and so the 'top' to 'bottom' difference is more marked and therefore easier to work out the thresholds.

Also the wheel can stop at any point in the cycle (see pots #1 and #6). One turn of the least significant wheel represents 10 litres. (I assume that the water company hopes that whatever mechanism is used to measure the water flow - impeller or something??? - does not clog as that would mea I get free water.)

Most of the time we are not using water but there are times when the water flows for as while (showers, watering the garden, dish washer etc.). In those cases, there can be fairly rapid transitions (see the 8 to 10AM period below, and just before 4PM).

Beware that Hall Sensor amplitudes can drift

And that is why I'm looking to see if there is some automated way of adjusting the thresholds. However manual checking is OK for now while I see if this is the case and if so, by how much - and this could take many months if I consider the seasonal temperature variations.

Yesterday I put into my test software a moving average filter and the 'Schmitt trigger' to give the information below. The software also counts the number of 0 to 1 transitions which in my case would give me an approximation of the water usage (in 10L increments). One thing I noticed is that the count varies depending on the length of the moving average and so I've taken the count from the meter this morning and will do the same tomorrow (at around the same time of day) and then see what level of smoothing approximates the actual water usage.

Susan

Red is the raw signal (mV), blue is the smoothed version and green are the detected transitions

 

[D.A.(Tony)Stewart]

The transitions are too close for me to read your error margin.

I don’t know the low pass filter method you are using but could be improved.

A rolling average can be offset by long delays in consumption. An envelope detection filter that stores average +/- max with a bit more denoising would be better. This would ignore the idle states in between the envelope.

 

[Aussie Susan]

OK - I've compared the actual meter reading to the samples from the hall effect transistor

There is fairly close agreement in that I actually used 511L and the above shows 49 transitions (or 490L - rounded to the nearest 10).
However even a small change in the length of the moving average changes that quite a bit: the above is 21 samples long but 20 samples long gives 52 transitions and 24 gives 48 transitions.

I've blown up the section around 2PM (14-00) (21 samples in the moving average):

So you can see that there are some transitions that are being missed - and why the shorter moving average picks them up perhaps too well.

So, @Tony - what is the envelope detection filter you were mentioning. (My signal analysis skills are not good but I'm wanting to learn.)

Susan

 

[KlausST]

Hi,

If it was my task .. I´d go the scientific approach.

At first I need the hardware/ wiring setup, like this (wihtout knowing if this is realisitc:

Bare Hall sensor --> 100 unshielded twisted pair wire --> ADC

Then I´d need the maximum speed of change, like one full cycle in 0.5 seconds, plus the waveform (or at least a brief description)
Then I´d need the ADC sampling rate

*****
If sampling rate really is "every few seconds" then for sure I´d install an analog filter that satisfies nyquist. Because: if there is unknown noise, then (without the nyquist filter) you get alias frequencies. And you have zero chance on the digitial side to remove these "alias frequencies" because you can´t differentiate them from the true signals.

My approach would be to use an analog filter --> ADC (with higher sampling rate) --> adjustable (maybe adaptive) digital filter.

Klaus

 

[D.A.(Tony)Stewart]

Envelope detection analyzes the average +/- peak to determine the midpoint. But your noise levels can cause errors such as these narrow spikes which are either undersampled to reach full scale or has noise exceding threshold.

 

[D.A.(Tony)Stewart]

I'd like to know if you can improve SNR of the signal going to the ADC.
What is the max flow rate if 10 litre/cycle and your sample rate. It should be about 20 x the sensor frequency to avoid missing the peak amplitude with light filtering that prevents aliasing with 8 bits and prevents attenuation of the max data cycle rate. Then you can apply digital filtering (FIR if you want) without averaging which can reduce peak-peak signal amplitude. After you can the data. or compress data as required.


>> What is the envelope detection filter you were mentioning...

This is a H/W method of using +/- envelope tracking with peak and decay times matching buffered and used to track the mean level without the problem of averaging delays. Then the output comparator uses that which I simulated with Noise and a wandering baseline of 10 mHz or 0.01 Hz.

This simulation assumed the max data rate from the water meter was 1 cycle per second so it can be scaled to match yours.

The detection method can also be done in S/W.

It is much easier to eliminate the noise with a good cable and Hall sensor type/location to get >= 10:1 signal/noise ratio (SNR) especially when there is spectral overlap.

Here I use an FM sweep signal or a noise data signal to show results. Top right is data+ 0.01Hz noise + white noise
The amplifiers used a ~1 Hz Bessel LPF, then a + peak and decay, -ve peak and decay detector then the average is buffered then inverted again then the filtered input and envelope mean into the comparator. Averaging is a poor method of filtering if it attenuates the data amplitude and the noise. That's why the oversampling method is used to prevent aliasing with decimation.

Filter response test