Show by Label

Friday, October 16, 2020

Monitoring, measuring & logging a GPSDO

This is actually a continuation of another project, a high precision 10MHz GPS disciplined oscillator (GPSDO), also described on my blog here

Everything that I discuss here will eventually be posted in a dedicated Github project, available here.

This is very much a work in progress, so keep in mind that things will change, so please come back often to keep track of the developments.


Monitoring, measuring and logging

During the development of my GPSDO's, I actually built three complete units, I wanted to monitor, measure and log the results so I could analyze the data later, and in different ways. Most of the changes take place over a long period of time, mostly several days, so it's easier to look at changes when you can graph the data. I have been doing that for many months now, but I wanted more.

My additional design goals

1. I wanted to move away from the RPi's currently sitting on top of my GPSDO's, and being fed by an additional 5V wall-wart supply. 

2. When I wanted to look at the NEO performance with u-center, or reprogram them, I needed to open-up the enclosure and add a serial to USB connector and cable. Or leave that cable connected through a slit in the enclosure. This is cumbersome, so I also wanted the RPi to have access to the NEO serial data, as a minimum just listening, but with the option to program it through the RPi as well, although this is way more complicated to realize. 

3. What is also cumbersome at the moment, is that I have to open-up the enclosure to de-solder/solder the 10uF capacitor on the reset pin of the Arduino, to avoid a general reset when you plug the USB cable in, just to reprogram the firmware. Even if I added a jumper, which I should have done, I don't want to open up the enclosure. It really disturbs the performance of the GPSDO. A simple on-off switch on the front- or back-panel could do that job.

4. I found a high precision counter project for the 10MHz, and I wanted to integrated that into the GPSDO.

5. With the additional information of the 10MHz counter and the NMEA data, I also wanted to add a very small OLED display to the front-panel, so I can show the 10MHz count, the number of satellites in view and maybe the DAC or temperatures, whatever...

6.  I'm also considering adding two more SMA connectors for the 1PPS and 10MHz outputs, but then isolated with a transformer. I have noticed losses of lock when I fiddled with the BNC cables connected to the current outputs. These outputs are DC coupled to the circuitry, and the whole GPSDO is floating from earth ground. To counter that from happening at the moment, I always have one BNC cable between the GPSDO and the external trigger input from my scope, which is earth grounded.

To put everything inside the GPSDO enclosure, I needed a smaller foot print for the RPi itself, so I choose to use the model Zero-W. The "W"stands for wireless, which is what I needed to get access to the internet and add email functionality. I also wanted to power the Zero with the already available 5V supply inside the GPSDO. I hope that the aluminum enclosure has enough openings for the WiFi to work.

To put everything inside the enclosure, I'm designing a small additional PCB that will house the counter hardware, a connector to fit the RPi Zero and the isolation circuitry for the additional 1PPS and 10MHz outputs. 


Bit-banging my way out of a predicament

Because I wanted to monitor the Lars' report, the NMEA messages from the NEO, and Yannick's 10MHz counter, I needed two extra serial ports, something the RPi family does not have. Luckily, there is the equivalent of a SoftSerial that the Arduino has, by bit-banging the GPIO pins of the RPi. That library (pigpio) took me quite some time to figure out, because the bit-banging function does not support a "readline" equivalent. It actually reads the serial data in bytearray chunks, so you need to piece together these chunks in order to make a complete sentence. 

This was rather easy to do for the counter, because it sends out the serial data in a fixed length format. However, things changed dramatically when I tried to collect the NMEA data from the NEO. After some additional programming and testing, I also managed to figure that out. Next up is using the bit-banging software for the Lars' report, which should be a lot easier now that I know how to work with the bit-banging. After that I will start the process of combining it all, and will then show the results soon.
When everything is programmed and working, I will make the decision which of the serial inputs will be going to the "real" UART. This will probably be decided on the processor load, but more importantly, the severity of possible errors, which currently would vote for the counter report.
 
Below I will describe the various project parts.
 

Monitoring the GPSDO status report

Since the beginning, I used a Raspberry Pi Model 2B (the "classic" first model, because I have several of them) to capture the output of the status report from the firmware of the GPSDO and add a timestamp in front of every line. After capturing that data into a file, an email program sends that just before the end of a day to my PC for further processing in Excel.

I used two separate Python scripts to do that. They are on the Github site.

Because I will most likely need to move this serial monitoring away from the "real" UART and use bit-banging, the serial monitoring code needs to undergo some serious treatment. I have not yet started that.


Measuring the accuracy of the GPSDO 10MHz output

To really measure the accuracy of the 10Mhz, you need a counter with a precision that can show at least 5-6 decimal digits of the 10MHz frequency. A normal counter will need a gate time that is measured in weeks, so the only viable way to measure this frequency is to use a good reciprocal counter. These instruments are normally way beyond the pocket-money most DIY hobbyists can spare. I know I can't.
 
So the only cheap method available to me is to use a counter that can measure down to 3-4 decimal digits.
While I was looking for counters that can be used on GPSDO's, I came across an interesting GPSDO project, that uses an integrated  counter to measure the output. 

 
I found this so intriguing that I ordered the parts to build this counter. Unfortunately, the maximum gate was "only" 1,000 seconds, giving you a mere 3 digits or 1pps. I wanted more, so I contacted the author, Yannick Turcotte, and described my problem. He was very quick in providing me with a test version that could go as high as 10,000 seconds, giving us another digit of precision.
 
Here is my first attempt with what we called the 10K version. I feed it with the 1PPS of my Bliley GPSDO, and also use the 10MHz output to serve as the clock for the Arduino. The connector you see on top is part of the programming interface.
 

 
Even with a 10,000 second gate time, the resolution is "only" 100microHerz. ( 10,000s * 10MHz = 10,000,000.0000 +/- .0001) My Bliley is better as you can see in the TimeLab frequency difference view (note that this is with a TC of 500 which is not the optimal setting):
 
 

 
Timelab reports a frequency of 9,999,999.999 991 948 at 10,000 seconds, so the Bliley is accurate to 5 decimal digits, one more than the counter can measure. If you realize that 1,000 seconds is 2.77 hours, it's clear that another digit of resolution would mean 100,000 seconds or almost 28 hours. Not very practical. The solution of course is to use a good reciprocal counter, but as I mentioned earlier, that is way outside my amount of pocket money, and even more so beyond my skills to build one. 

The caveat of using this counter technique with such a long gate time is that you only measure the average frequency during this gate period. However, in this particular case, with this application, this is not such a big deal, because that's what a GPSDO is supposed to provide anyway. Long term stability and precision.

In any case, just displaying this result, that is only refreshed every 2.7 hrs, is nice but not very useful. Especially when your GPSDO has another digit of precision higher. However, if you can log the results and graph them, you can see the drift over time, or the GPSDO getting "out of specification". Maybe this is just a gadget, I'm going to add it anyway and see what benefits or insights it will bring.

After a lot of testing and fiddling (due to the bit-banging requirement)  from my side, Yannick has now produced a counter version especially made for this project, that has a remotely programmable gate time of 1,000 seconds to measure with 3 decimal digits, or 10,000 seconds to get 4 decimal digits. It also sends the results to the serial output of the Arduino so it can be captured and manipulated by the RPi.  Yannick also made a special 10s gate time version that is essential for testing.
 
The source for this special version is available here : https://github.com/YannickTurcotte/Lars-DIY-GPSDO

 

Raspberry Pi serial port caveat

Most of the time I spent testing the counter was by getting a reliable serial connection between the Arduino and the RPi Model Zero-W. This was by using the standard TxD and RxD pins on the connector. After days of struggling, I found that the occasional dropped data and garbage I received was caused by a not-so-well-known change in the firmware for the Model 3, 4 and the Zero-W I'm using. 
 
The Raspberry Pi models only have one "real" UART. The Raspberry Pi Foundation made a decision a while ago to replaced the reliable UART for what they call a "mini-UART", to provide Bluetooth support using the "real"one, but neglected to communicate this well enough. Note that the Model 2B that I was already using to monitor the GPSDO's, did not have this swap, so when I moved the applications to the Zero-W, I was totally unaware of this change and fell into this nasty trap. The justification from the Foundation for this change was that the serial UART was mainly for console use and therefore less critical. Yeah, right.
 
So using the "mini-UART" turned out to occasionally produce unreliable results (dropped data or garbage). This was most likely caused by the non-deterministic nature of Linux, so when it did some heavy housekeeping, the mini UART lost data . When you're waiting for one message every 2.7 hours, you have to capture it reliably, or wait for the next one. It took me several days to hunt down the problem and then find the available solution, that is, if you knew what to look for. Grrr.... By no means the first time the Foundation caused issues like this.
 
How to switch from the "mini-UART" as the default back to the "real" UART.
First, enable the serial port with sudo raspi-config.
For the RPi 3, 4 and Zero-W, you need to swap the mini-UART with the TL011, the real one.
Edit the boot configuration: 
    sudo nano /boot/config.txt
At the end of the file, add:
     dtoverlay=disable-bt
Save the file and close the editor.
You now need to disable the modem software:
    sudo systemctl disable hciuart
Reboot the RPi.
This allows you to use the following serial device again : /dev/ttyAMA0

Logging the NMEA sentence from the GPS module

Because of the limited number of serial UART's available on the RPi family, I needed to find a kind of SoftwareSerial that would let me use any of the available GPIO ports of the RPi. There are three libraries available that I know of, but I selected to use the pigpio library from Joan, who has been very active on the RPi front right from the start. He or she, is very skilled, active and helpful, so this library is where I started the adventure with. This library has support for bit-banging of the GPIO pins to act as serial TxD and RxD ports.

As I mentioned earlier, the pigpio bit-banging software reads the serial data in irregular chunks in the form of Python bytearrays, which you need to piece together again. Sounds trivial, but was not so easy to do to make it work with different data from different sources.

After some inspiration and a lot of testing, I figured out a way to do this reliably, and the script now reads the data of the counter and the NMEA sentences without problems for more than a day. Good enough for me.

The code is on the Github.

If you can find a more reliable or a more clever way of doing this, please chime in and post your solution.

 

The OLED display

The little 0.9" diagonal display will easily find a place on the front-panel. I already worked out how to program it. The script is on the Github. 

 

The hardware circuitry

 The schematics and design is still a work in progress, but it shows what I'm planning to do and already built on perf. board as a prototype.
 

 This part of the circuit is heavily based on Yannick's version, I just stripped away the non-essential parts. There is only one LED left, this is the counting LED that I will put on the PCB, it's there to see if the counter is counting.
The inputs for the counter are the 10MHz and the 1PPS coming from the GPSDO. The 12V supply comes from the same supply that is  powering the GPSDO.

Above is the RPi portion of the circuit. The tiny 0.91" display is an i2c version, so it only needs the minimum amount of connections. There is a library available that is simple to use.
 
The RPi will read the serial data from the counter, the NEO and the Arduino Nano of the GPSDO. At this moment, only listening. The NEO produces 3V3 level signals, but  the Nano and  the counter are at 5V levels so I use a 1K series resistor or a level convertor to keep the RPi, which is at 3V3 levels, safe. These resistors keep the maximum current going in to the GPIO ports in check. (5V-3.3V / 1,000 Ohm = 1.7mA)
 
There is already a USB connector on the front-panel of the GPSDO to get access to the Arduino Nano so it does not need a TxD connection. Same with the NEO, although those connections are still inside the GPSDO enclosure. If you need to re-program the NEO, you still have to open-up the enclosure. However, this is rare if you did it right during the first tests.
 
The RPi can select the 1,000 or 10,000 second gate time for the counter so it's under program control. It can also program the counter control button, and even reset the counter so it knows when the long counting period starts. This allows me to put a remaining number of minutes until the next update on the screen.

The prototype on perf. board has been done and has been tested for a few weeks now, with good results.
 
I also decided to put the fan driver circuit on the final version of this board so you don't need a separate PCB.
 
Note:
Be aware that the ATMEGA will be "back-powered" by the 1PPS and 10MHz signal levels, so you need to program the extended fuse with 0xFD, for 2.7V, to set a lower brown-out level, otherwise the ATMEGA may not reset properly.
 

Putting it all together

I'm first going to produce a prototype on perf board that will go inside the Bliley enclosure. With PuTTY I can get access to the RPi and work on the programs without disturbing the GPSDO. Once that works to my satisfaction, I will work on the PCB layout and also work out where and how I'm going to place it inside the enclosure, and how and where to mount the OLED screen.  

For the prototyping stage, I already mounted the 0.9" display on the front-panel, and created holes for the isolated 1PPS and 10MHz SMA connectors. The toggle switch for the capacitor on the reset pin of the Arduino took it's place in one of the ventilation holes. I had at least one too many ventilation holes for the Bliley, because it does not get as hot as the Trimble, meaning that the fan was running on the lowest RPM when the room temperature got low during the night. I would like to keep the fan regulated at all times. With the additional heat that the counter/RPi board will generate, I will need to re-visit this anyway. 

Below are pictures of the carries board with the counter circuit and the isolation transformers. This is just a prototype I quickly put together to see if it all works, and fits. The carrier board will slide up-side down in the slot on the top cover of the enclosure. The RPi will fit on top of the connector, and will also hang up-side down. The green "counting" LED from the counter you see in the bottom-right corner will show through one of the ventilation holes.

 

I didn't need access to all the GPIO pins of the Zero, so I just used a 2x12 connector I had plenty of.



I have added two SMA connectors for the isolated outputs, and added a switch that can be used when programming the Nano. I also added the 0.9"OLED display. Eventually, when everything is really, really finished, I will also make a new front-panel. When working with the fan regulation, I already butchered a front-panel and drilled some 6mm holes here and there. One of them is now used for the programming switch. For this additional requirements, I added room for the display and for two more SMA connectors.

 

Below is how you can see the position of the boards relative to the front-panel. The carrier board will slide into the rails of the top cover of the enclosure.

Major setback!

I build the prototype but the wireless connection of the Raspberry Pi Zero-W is disturbed in a big way I first thought was caused by the 10MHz on the carrier board. The 10MHz is needed as a clock for the counter, and I also need it there to provide the floating outputs. Even when the board is still outside of the enclosure, the wireless connection is terrible and unusable. After some more fiddling, I found that it's not just the 10MHz feed, because I disconnected it and even removed the counter chip. The Zero wireless connection is still hopeless, even without the cover of the enclosure installed, which will make it worse. The antenna for the wireless connection on the Zero is just a small track on the PCB. That antenna must be disturbed by the signals coming from the GPSDO. With this setup, I have no room for a USB wireless dongle, because the isolation of the OCXO is in the way. It would probably draw too much power anyway.

If it's not the 10MHz, the other source could be the 5V ocxo supply. As a test, I fed the carrier board with my bench supply, and bingo, that seems to work better. I also see no issues when running ifconfig and iwconfig on the RPi, so I may be on to something.

There is a lot of ripple and more than 1V of high frequency noise riding on the ocxo 5V supply. I tried to filter this, but no big success, with the enclosure open, this stuff is seemingly coming from everywhere. I am now going to try a dedicated and separate LM7805. It must be mounted on the enclosure because of the heat developed by the current for the RPi, and also the 12 to 5V voltage drop. I will need to try that first and see if it's acceptable. 

 I tried this morning by giving the board a separate 5V supply with extra filtering but no dice. It's definite, the Raspberry Pi and the GPSDO are not playing ball together. The RPi even disturbed the Arduino Nano. Looks like the two camps are bitter enemies within the same enclosure.

Back to the drawing board

I need to find another solution.

Swapping the Raspberry Pi to another processor will not help I think. As soon as you add the wireless capability, I'm pretty sure we'll run into the same problem. Besides, maybe it's better to not have a wireless transmitter blasting inside the GPSDO box at all.

I do not want to add software (in assembler, yikes) to the counter code and make it drive the OLED display. Besides, it would not allow me the freedom to display what I want easily. I will also need an i2r library to drive the display as well. I've already decided that I'm not going in that direction. That means that I need to remove the display from the GPSDO box and move it to wherever the RPi and the counter find a new home.

The only viable alternative I see at the moment is to use another enclosure and put the counter, the display and the RPi in it. I'm considering just adding a small board inside the GPSDO with just the hardware to provide the floating outputs.

Because I'm going to use a separate enclosure, I need some inputs from the GPSDO to make it function. The required signals are: 10MHz and 1PPS for the counter, TxD from the Nano and TxD from the Neo for the RPi monitoring. 

The 10MHz and 1PPs could come from the SMA or BNC outputs already on the front-panel of the GPSDO. I will feed these to the front-panel of the new box. The Nano TxD, RxD and Gnd signals are already available on the front-panel of the GPSDO, but the NEO signals are only available on a header inside the GPSDO enclosure. I would like to move all the serial connections to the back of the GPSDO enclosure, and do the same for the new enclosure, to keep these wires out of the way and short. I'm considering using a flat cable and 2.54mm connectors. I'm also considering powering the new enclosure with the same 12V power I already use for the GPSDO. This 12V comes from a home-built UPS so everything will be on all the time, even when there is a power failure.

Splitting things up in two parts

I stripped most of the carrier board wiring, and only left the insulated output components on it. This board now sits into the GPSDO, in the intended slot on the top cover. I covered the hole of the display on the front-panel with tape, and added a 5 pin connector to the back of the GPSDO to route the NEO TxD, RxD and the Nano Txd and Gnd to the new enclosure. I found a suitable enclosure locally and ordered two of them.

The new enclosure has to be plastic to allow the wireless connection to function. Because of the plastic, I can't use a regular 7805, because with the amount of current for the RPi, it needs cooling. I will have to use a Buck switching regulator. Because of the required Buck regulator, the ones I ordered turned out to be too small. I overlooked the large size mounting holes that eat-up a lot of real-estate, so I needed to order larger ones.

The additional board for the GPSDO

Basically, this board is the old one I used for the version 1, but I stripped most of the components. It now only has the section called "insulated outputs"  If I will design a PCB, I'll also add the fan controller on it.


With these modifications installed, I turned on the GPSDO's and let them settle again.
 

First results with the prototype

I built the first prototype on proto-board (see above) and initially worked on getting the software up and running. I then moved the whole contraption in position on top and next to my Bliley GPSDO, with the OLED display literally hanging in free air, and tried it all since then. In this version, I do not have the inverting buffers in the 10MHz and 1PPS signal lines going to the counter. This also means that the counter is triggering on the wrong edge of the 1PPS signal.

Here are some results:


This is just a daily sample of the number of satellites in view by the M8T NEO. I've seen some dips down to 8, but mostly, it is very stable around 12. Plenty for a good fix I think. There is no correlation between the fewer number of satellites and the aberrations I see in the TIC and DAC reports on the GPSDO. What is very interesting is that every day for several weeks, this picture looks the same, with the glitches in exactly the same spots. This must be due to the constellation coverage of the sky above me, so a simple one-time look at u-center is not sufficient to determine the true coverage.


2020-11-15 00:00:09,058 INFO     gate    10000s    counter    10000000.0000
2020-11-15 02:46:53,055 INFO     gate    10000s    counter    10000000.0000
2020-11-15 05:33:37,049 INFO     gate    10000s    counter    10000000.0000
2020-11-15 08:20:21,055 INFO     gate    10000s    counter    10000000.0000
2020-11-15 11:07:05,053 INFO     gate    10000s    counter    10000000.0000
2020-11-15 13:53:49,055 INFO     gate    10000s    counter    10000000.0000
2020-11-15 16:40:33,056 INFO     gate    10000s    counter    10000000.0000
2020-11-15 19:27:17,054 INFO     gate    10000s    counter    10000000.0000
2020-11-15 22:14:01,061 INFO     gate    10000s    counter    10000000.0000

Above is a daily log of the counter measuring the Bliley GPSDO, set at the maximum gate period of 10,000 seconds or, 2,77 hours. It can only provide 4 decimal digits while the Bliley is another 2 digits more accurate/stable, see below.


As you can see, the Trimble may show other counter results. I'll be able to test that when the PCB come in and after I've modified the Trimble GPSDO to provide the NEO serial data on the back-panel.

 

The Counter/Logger Version 2


For the new standalone Counter & Logger, I stripped and modified the earlier design to turn it into a stand-alone instrument. In the meantime, Yannick and I settled on a version of the Counter that has now been running for several weeks, with all the bells & whistle's I wanted. In that final version, there is no longer support for the mode button, so that's gone.

After ordering two different enclosures, I settled on the Hammond 1593L. It's very small, so I needed to go to mostly SMD parts to make it all fit. After a few design alterations, I settled on the above schematics, and designed a PCB for it.
 
The RPi Zero will be mounted on the carrier board with a 2x12 connector, because I have many of them. The original RPi connector is 2x13, so if you have one of those, it will fit, just cut the two extra pins. 
 
I ordered the PCB's and when they came in, I built two of them with parts. Unfortunately, while testing the power distribution without the logic chips and the RPi in the socket, I noticed a major screw-up. for the RPi socket, I used a custom version I created a long time ago, that was for a UPS board sitting upside-down on top of an RPi Model 1,2 or 3. The connector I designed used a pin-out that was a mirror-image because it was supposed to sit on top of the RPi. The connector for the counter board however has the RPi plug in to the carrier PCB. So the lower row of pins were reversed with the upper row, making the PCB's useless. Bummer!

Below are the revised versions.





These boards are now in production and next week I should have them. If they work this time, I will put the Gerber files on my Github. I needed to order five, and will build two myself. One is already reserved by another user, so I will have two more available that I will sell at cost + postage. Use the comment section below to show your interest.
 
In the mean-time, I worked on the front-panel and the back-panel design. Because the enclosure is pretty small, there is not a whole lot of room. I used my DipTrace PCB layout program to design precise layouts I can print them on paper, and attach them to the plastic panels with double-sided sticky tape in order to drill the holes and file the opening for the display and the serial connector on the back.

The front-panel will have the window for the OLED display. I will need to use double-sided sticky tape to fix the display because there are no mounting facilities. The PCB size of the OLED display is the larger (keep-out) rectangle, and the visible part the smaller one.
To the left and right are the holes for the SMA connectors for the 10MHz and 1PPS signals, and in the middle the 3 mm counting LED hole.

The rectangle on the back-panel is for the protruding serial connections going to the GPSDO. To the right is the DC chassis part for the 12V DC.


 

Still more to come... 

Github site with all files is here: https://github.com/paulvee/GPSDO-Monitoring