Cloud-native ramblings: Adding Redis

Many people think I am “just” building a paludarium… But they don’t see the complex world of automation behind it. In fact, there is a full blown application running the paludarium, build in a limited-but-functional cloud-native architecture.

This architecture is limited… Because you can have multiple instances of your microservices… But in the end there is only one piece of hardware to control. A single relais, a single led light, a single pump, a single valve and a single level sensor. And this is a problem; what if multiple microservices all talk to the one hardware platform at the same time? I needed a way to put a lock on who can send commands, and Redis came to the rescue.

The limitations of having “one hardware”

What if you have all kinds of microservices and functions that all want to talk to a single hardware platform? There are several functions and services that want to talk to the hardware:

• lightFunction: controls any light source from a jungle LUX value and time to actual led lighting
• airFunction: controls anything air related. Air heater, Air misters, Air fans.
• rainFunction: controls anything rain related, mainly the rain pump but also the osmosis valves.
• thunderFunction: controls the lights, rain, and audio channel for rumbles and flashes.
• waterService: monitors and controls anything (osmosis) water related.
• hwMonitor: monitors the hardware, so the Raspberry Pi but also reads the fan rpm to make sure it still operates.

And I probably forgot a couple…

First of all, there is a single microservice and it alone controls the hardware (actually sending commands and receiving commands using a serial port). But that microservice has an API. And API “everyone” can talk to. How to make sure there is only one conversation going on at a time? The hard way would be to feature a locking mechanism in the API to the hardware controller (called hwService). The easier way… And as this is a hobby… Was to include Redis and build a locking value in its store.

The quickest Redis implementation ever? 😉

I decided I needed a locking mechanism between all these microservices. And I found it in Redis. I logged into the Raspberry Pi, hit “sudo pip3 install redis” and played away. Redis on Python is SO simple!

Next I looked up the sendlib_hw.py library. It contains all commands to actually talk to the hardware (things like setDO, walkPWM, getPWM, setVALUE, getVALUE). I added two functions to it, HWlock(string) and HWunlock. HW lock sets a locking value in Redis to the string passed to it, while HWunlock() simply fills the locking value with the string “unlocked”. The HWlock() function will actively wait for any pending lock to be removed, so it waits for the value to become “unlocked” before locking it for itself.

This worked remarkably well! In the logging I can regularly see lines saying something like “waiting for the lock to be dropped by “waterService” so I can lock it for “lightFunction”, and then it resumes.

This has made communications with the hardware SO much more reliable! With this assurance I am now ready to start to fill the paludarium with life 🙂

Arduino crashing and once again Redis to the rescue!

I had something nasty happening the other day. As I have always seen the Arduino on the controller as the more stable system compared to the Raspberry Pi, I never thought that the Rapsberry Pi would have to restart the Arduino… But in the world of embedded, anything you don’t expect WILL happen.

I came downstairs one morning and the paludarium was still in “night” mode, even though the sun should have risen already. As it turned out, the Raspberry Pi was flooding the log with errors talking to the Arduino (which I call the Artemis). The control light on Artemis, normally fading in and out slowly to prove its livelyness had stopped. The Arduino had crashed.

Why is a crashed Arduino a problem? Well, as the paludarium sits underneath the sewer level in my mancave, the excess water needs to be pumped out. If the Arduino crashes, it may miss the “sump level sensor” coming on, not start the drain pump and cause the system to overflow into the mancave. NOT GOOD.

So how to solve? In the end I came up with another use for Redis: keeping a global counter for hardware control errors. Every time a control command to the hardware works, I reset this counter in Redis. On every failure, I increase the counter.

If the counter reaches 100, something is REALLY wrong; it means the Raspberry Pi has tried to fire 100 commands to the Artemis, and none of them got a valid reply back. Time for drastic action.

Using the Raspberry Pi to reboot the Arduino-based Artemis

I was already designing a reset circuit in my head. A circuit that the Rapsberry Pi could assert, and that would cause the RESET line on the Artemis to be briefly activated. Then it dawned on me: Whenever I reprogram (“flash”) Artemis with new firmware, the programmer (I use platformio) performs a reset… But how? As it turns out, the Mega2560-based Arduino has it’s reset connected to the serial DTR pin. I could simply write a tiny piece of code to wiggle the DTR pin:

!/usr/bin/python3

import serial
import glob
import time

----------------------------------
W A R N I N G
-----------------------------------
Running this script will result in the Arduino-based Artemis to HARD RESET.
This means that all outputs will be killed, lights go off etc.
Artemis should restart and resume its comms automatically, but the PaluPi should shoot all output states new.

Disable DTR on serial as this will reset the Arduino when receiving serial comms
path = glob.glob('/dev/ttyUSB*')[0]
serialport = serial.Serial(path, 9600, timeout=2)
serialport.setDTR(False) #Disable DTR so it will go LOW. Arduino will RESET as it thinks it is being flashed
time.sleep(0.02)
serialport.setDTR(True) #Enable DTR so it will not stay LOW. Arduino will come out of RESET and restart!
serialport.close()

I ran the code, and POOF the Arduino performed a hard reset! It started running again and all became well after the system started to program the outputs again.

Still do not know why the Arduino crashed. It has an active watchdog and the watchdog was still being kicked (as the Arduino did not reset). Most timings seemed to be running (including the logic to start the drain pump on an activating sewer level sensor). Hopefully I won’t be seeing this again, I do look through the logging from time to time to see if the Raspberry actually ever performs this RESET to the Arduino. So far the Arduino has been rock stable (as it has been for many months of testing).