An Accidental Data Platform



Albeit a simple hobby project in the beginning, the design has proven flexible enough & the implementation stable enough to enable the addition of many bits & bobs over the years, accidently creating an end-to-end platform of sorts. The aim here is to capture all those bits in one place & discuss them more thoroughly.


The primary "data bus" is redis pub/sub while implementation languages vary depending on the platform & requirements. Both Raspberry Pi and Arduino devices feature significantly, and in the fully-deployed system under discussion rely on the Bosch BME680 for temperature/humidity/pressure sensing & the Sensirion SPS30 for particulate matter (air quality) sensing.

The Bits & Bobs

Arranged roughly in order of where each appears in the pipeline, from sources to sinks. Maturity varies widely.


A sensor management & data collection/emission suite for Raspberry Pi devices written in Python 2.7 that acts as the primary data source for the platform.


Azure Sphere-based pipeline monitoring devices that inspired the creation of & consume YARL (below).


Yet Another Redis Library, this one in POSIX-compliant C.


Arduino live data display devices that inspired the creation of & consume Arduino-Redis (below).


A Redis client library for Arduino devices written in C++11.


A rudimentary Redis-to-HTTP proxy, written in Go, that enables the Weather Dashboard (below).


A live dashboard of weather & air quality data built with vanilla Javascript/HTML/CSS (and a bit of help from Plotly). Features user-configurable plots with both historic and live data.

Air-quality index dashboard

Enabled by ThinkSpeak and the powerful MatLab integration within, this dashboard features a live air-quality index value based on data from the pipeline, as defined by the US EPA. The MatLab source is available here.

In Detail

The design, build & deployment of an RPJiOS device for air quality and T/H/P data gathering is detailed in full in this Hackster.io build log. As noted within, it has long been my intention to document & discuss the software pipeline in its entirety as well; the hope is that this site is forward progress in that direction. More to come!

Something Entirely Different

Have a look at Fauna - the full source of which is available here - a game (of sorts) that was an unintended but welcome respite during the COVID-19 quarantine(s). Not yet compelete & and unlikely to ever be, it has been a thouroughly enjoyable endeavour nonetheless.

Fauna also resulted in the creation of a 2d sparse bitmaps library for node.js, published in the hope that may help other developers who face similar challenges.