victron-esp32-ble-mqtt/README.md

438 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<div align="center">
# 🔋 Victron SmartSolar BLE → MQTT
**"A Raspberry Pi 4 drawing 4W to read one MPPT is an absurd tax on a solar battery."**
ESP32 firmware that reads a **Victron SmartSolar MPPT** over Bluetooth LE, decrypts the
advertisement, and publishes it to MQTT with full Home Assistant auto-discovery — at
**~0.15W** instead of ~4W.
[![Licence: MIT](https://img.shields.io/badge/licence-MIT-blue.svg)](#licence)
[![Board](https://img.shields.io/badge/board-ESP32%20CH340-e7352c.svg)](#hardware)
[![Built with](https://img.shields.io/badge/built%20with-PlatformIO-f5822a.svg?logo=platformio&logoColor=white)](#4-build-and-upload)
[![Crypto](https://img.shields.io/badge/crypto-AES--128--CTR-6f42c1.svg)](#how-it-fits-together)
[![MQTT](https://img.shields.io/badge/MQTT-auto--discovery-660066.svg?logo=mqtt&logoColor=white)](#mqtt-topics)
[![Home Assistant](https://img.shields.io/badge/Home%20Assistant-zero%20YAML-41bdf5.svg?logo=homeassistant&logoColor=white)](#home-assistant)
[![Idle draw](https://img.shields.io/badge/idle%20draw-~0.15W-brightgreen.svg)](#power-consumption)
[![Buy me a coffee](https://img.shields.io/badge/buy%20me%20a%20coffee-ffdd00?logo=buymeacoffee&logoColor=black)](https://www.buymeacoffee.com/MrSossidge)
</div>
---
## The problem
Victron's `victron-ble` Python library is excellent, and running it on a Raspberry Pi 4 works
perfectly — until you notice the Pi is idling at 3–5W purely to relay a few numbers off a
charge controller. On an off-grid system that's around **40Wh a night**, or 3–4Ah at 12V:
the monitoring was costing more than the thing it was monitoring.
This firmware does the same job on an ESP32 at roughly a fortieth of the power, and behaves
better in Home Assistant while it's there.
## What you get
| | |
|---|---|
| 📶 **Reads the advertisement** | Passive BLE scan of the SmartSolar MPPT's broadcasts — no pairing, no connection, lower draw. |
| 🔐 **Real decryption** | AES-128-CTR exactly as the Victron BLE protocol specifies, matching the reference Python implementation bit for bit. |
| 📡 **MQTT every 10s** | One JSON payload with every value, plus a retained online/offline LWT topic and a `diag` topic with uptime, RSSI, heap and reconnect counters. |
| 🏠 **Auto-discovery** | Appears in Home Assistant as a **Victron SmartSolar MPPT** device with entities grouped. No YAML. |
| 🧵 **Two cores** | BLE scanning pinned to core 1, WiFi/MQTT to core 0 via FreeRTOS. Coexistence is handled by a 50% BLE scan duty cycle, not by the core split. |
| 🔁 **Self-healing** | Non-blocking reconnects, forced MQTT reconnect after 5 minutes without a successful publish, and a reboot after 60 minutes |
| 🔌 **External load too** | The MPPT's load output current is published as amps, and multiplied by battery voltage to give watts — so the dashboard shows draw as well as charge. |
| 📊 **Dashboard included** | A ready-made Mushroom + Mini Graph dashboard card and optional template sensors. |
| 🪫 **0.15W idle** | The entire point. |
## How it fits together
```mermaid
flowchart LR
mppt["☀️ Victron SmartSolar MPPT<br/><i>BLE advertisement, encrypted</i>"]
subgraph esp["⚡ ESP32 · dual core"]
core1["Core 1 · NimBLE<br/><i>passive scan</i>"]
dec["victron_ble.cpp<br/><i>AES-128-CTR + bit unpack</i>"]
core0["Core 0 · WiFi + MQTT<br/><i>PubSubClient</i>"]
core1 --> dec --> core0
end
broker["Mosquitto<br/><i>solar/mppt/#</i>"]
ha["Home Assistant<br/><i>auto-discovered</i>"]
mppt -.->|"~every few seconds"| core1
core0 -->|"every 10s"| broker --> ha
style core1 fill:#e7352c,stroke:#e7352c,color:#fff
style dec fill:#6f42c1,stroke:#6f42c1,color:#fff
style core0 fill:#334155,stroke:#334155,color:#fff
style broker fill:#660066,stroke:#660066,color:#fff
style ha fill:#41bdf5,stroke:#41bdf5,color:#fff
```
SmartSolar MPPTs broadcast encrypted BLE advertisements every few seconds. Decryption uses
AES-128-CTR with your device's 32-character hex key, and a 16-bit nonce from the
advertisement used as the little-endian initial value of a 128-bit counter. The plaintext is
a bit-packed structure read LSB-first.
---
## Hardware
- An ESP32 dev board — tested on the **ESP32 CH340 38-pin**
- Within BLE range of the SmartSolar MPPT (~10m)
- A WiFi network
> The ESP32-C3 SuperMini also works, but needs the BOOT button held during flashing. The
> classic CH340 board is the easier one to live with.
## Quick start
### 1. Clone
```bash
git clone https://github.com/MrSossidge/victron-esp32-ble-mqtt.git
cd victron-esp32-ble-mqtt
```
### 2. Install PlatformIO
Use either the VS Code extension (handy for interactive work) or the CLI in a
Python virtual environment:
- **VS Code** — install [VS Code](https://code.visualstudio.com) and the
**PlatformIO IDE** extension.
- **CLI** — PlatformIO needs a Python environment:
```bash
sudo apt install python3-venv # Debian/Ubuntu, only if missing
python3 -m venv ~/.venvs/pio
source ~/.venvs/pio/bin/activate
python -m pip install --upgrade pip platformio
pio --version
```
The venv isolates the `pio` CLI only; PlatformIO still downloads the ESP32
toolchain and libraries into `~/.platformio` (a few hundred MB on the first
build).
### 3. Configure
```bash
cp include/example.config.h include/config.h
```
```cpp
#define WIFI_SSID "your_wifi_ssid"
#define WIFI_PASSWORD "your_wifi_password"
#define MQTT_HOST "192.168.1.x" // your MQTT broker IP
#define VICTRON_MAC "xx:xx:xx:xx:xx:xx" // lowercase, colon-separated
#define VICTRON_ENC_KEY "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" // 32 hex chars
```
`config.h` is gitignored — your key does not end up in a commit.
Optional settings in the same file:
```cpp
// 1 = disable WiFi modem sleep (default, recommended)
#define WIFI_DISABLE_POWER_SAVE 1
// 1 = publish Home Assistant discovery, 0 = data/status/diag only
#define HA_DISCOVERY_ENABLED 1
#define HA_DISCOVERY_PREFIX "homeassistant"
#define HA_DEVICE_ID "victron_smartsolar_garage"
#define HA_UNIQUE_PREFIX "victron_mppt"
#define HA_DEVICE_NAME "Victron SmartSolar MPPT"
```
If you run a second MPPT, give it its own `HA_DEVICE_ID` **and** `HA_UNIQUE_PREFIX`
so its entities and discovery topics do not collide with the first one.
<details>
<summary><b>Finding your MAC address and encryption key</b></summary>
1. Open the **Victron Connect** app
2. Connect to your SmartSolar MPPT
3. Tap the **⋮ menu** → **Product Info**
4. The MAC address is at the top of that page
5. Scroll to **Instant readout via Bluetooth** and tap **SHOW** for the encryption key
</details>
### 4. Build and upload
With the venv active (or call `~/.venvs/pio/bin/pio` directly):
```bash
pio run # compile only
pio run -t upload # compile + flash
pio device monitor -b 115200 # serial log at 115200 baud
pio run -t upload -t monitor # flash, then open the monitor
```
In VS Code the PlatformIO status bar does the same:
| Action | Shortcut |
|---|---|
| Build | `Ctrl+Alt+B` |
| Upload | `Ctrl+Alt+U` |
| Serial monitor | `Ctrl+Alt+S` |
#### Build output
Everything is written to the hidden `.pio/build/esp32dev/` directory:
```
.pio/build/esp32dev/
├── firmware.bin # application image (the file you flash / ship via OTA)
├── bootloader.bin # ESP32 bootloader
├── partitions.bin # partition table
├── firmware.elf # ELF with debug symbols
├── firmware.map # linker map
└── src/ # compiled objects for main.cpp and victron_ble.cpp
```
- `.pio/` is gitignored, so build artefacts are never committed.
- `pio run -t upload` flashes all three images automatically at the usual
offsets: `bootloader.bin` at `0x1000`, `partitions.bin` at `0x8000`,
`firmware.bin` at `0x10000`.
- For OTA updates or a manual `esptool.py` write, `firmware.bin` is the image
you need.
- `firmware.elf` is used to decode crash backtraces, e.g.
`xtensa-esp32-elf-addr2line -pfiaC -e .pio/build/esp32dev/firmware.elf <addr>`,
or with PlatformIO's exception decoder in the serial monitor.
#### Build notes
- `platformio.ini` pins `espressif32@6.13.0` (Arduino core 2.0.17). It is the
last platform compatible with NimBLE-Arduino 1.4.x — do not bump it without
migrating to NimBLE-Arduino 2.x.
- Dependencies are NimBLE-Arduino 1.4.x, PubSubClient 2.8.x and ArduinoJson
6.21.x, resolved by PlatformIO from `lib_deps`.
- A clean build takes a few minutes and needs network access; rebuilds only
compile changed files.
- Reference sizes with the default 4 MB partition table: about **77 % flash**
and **17 % RAM** used. Disabling `HA_DISCOVERY_ENABLED` frees ~4.6 KB of flash.
- Clean rebuild: `pio run -t clean`.
#### Upload troubleshooting
- `fatal error: config.h: No such file or directory` — run step 3 first.
- Upload times out at 921600 baud — lower `upload_speed` in `platformio.ini`
to `460800` (or `115200`).
- `Permission denied: /dev/ttyUSB0` — `sudo usermod -aG dialout $USER`, then
log out and back in. On Windows, install the CH340 driver.
- Board not detected — hold **BOOT** while the upload starts (needed on some
ESP32-C3 SuperMini boards).
#### Manual flashing with esptool
`pio run -t upload` drives esptool internally. For recovery, a board without
PlatformIO, or a machine that only has the pre-built images, flash the three
partitions yourself.
```bash
pip install esptool # esptool 5.x installs the command as `esptool`
```
Port names: Linux `/dev/ttyUSB0`, macOS `/dev/cu.usbserial-XXXX`, Windows `COM5`.
Linux / macOS:
```bash
esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 460800 write_flash \
0x1000 .pio/build/esp32dev/bootloader.bin \
0x8000 .pio/build/esp32dev/partitions.bin \
0x10000 .pio/build/esp32dev/firmware.bin
```
Windows PowerShell (backtick continues the line):
```powershell
esptool.py --chip esp32 --port COM5 --baud 460800 write_flash `
0x1000 .pio\build\esp32dev\bootloader.bin `
0x8000 .pio\build\esp32dev\partitions.bin `
0x10000 .pio\build\esp32dev\firmware.bin
```
Erase the whole flash first if the board is in a bad state or has a different
partition table:
```bash
esptool.py --chip esp32 --port /dev/ttyUSB0 erase_flash
```
Optional: merge everything into one factory image (handy for archiving or
sharing a known-good build):
```bash
esptool.py --chip esp32 merge_bin \
--flash_mode dio --flash_freq 40m --flash_size 4MB \
-o factory.bin \
0x1000 .pio/build/esp32dev/bootloader.bin \
0x8000 .pio/build/esp32dev/partitions.bin \
0x10000 .pio/build/esp32dev/firmware.bin
esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 460800 write_flash 0x0 factory.bin
```
Notes:
- These offsets assume the default PlatformIO `esp32dev` 4 MB layout. If you
changed the partition table, flash `partitions.bin` and `firmware.bin` using
the offsets from that table instead.
- 921600 baud works on most boards; drop to 460800 or 115200 for flaky CH340
clones.
- If esptool reports `Failed to connect`, hold **BOOT** while it tries, and
make sure the serial monitor is closed.
- `firmware.bin` alone is what an OTA update expects; the bootloader and
partition table are only written over USB.
---
## MQTT topics
| Topic | Description |
|---|---|
| `solar/mppt/data` | JSON payload with all sensor values |
| `solar/mppt/status` | `online` / `offline` (retained LWT) |
| `solar/mppt/diag` | Diagnostics: uptime, WiFi RSSI/status, heap, reconnect counts, BLE packet age |
```json
{
"battery_voltage": "12.98",
"battery_current": "-0.20",
"pv_power": "8.0",
"yield_today_wh": "350",
"charge_state": "bulk",
"charge_state_raw": 3,
"error_code": 0,
"external_load_a": "0.35",
"external_load_w": "4.5",
"data_age_s": 2
}
```
`data_age_s` is the age of the last decoded BLE packet. If it keeps growing while
`solar/mppt/diag` still updates, MQTT and WiFi are fine and the problem is BLE reception.
```json
{
"uptime_s": 86400,
"wifi_status": 3,
"wifi_rssi": -61,
"free_heap": 182344,
"min_free_heap": 140112,
"wifi_disconnects": 0,
"mqtt_connects": 1,
"publish_failures": 0,
"ble_valid": 14321,
"ble_ignored": 0,
"ble_rssi": -72,
"ble_age_s": 2
}
```
<details>
<summary><b>Charge state values</b></summary>
| Value | State |
|---|---|
| 0 | Off |
| 2 | Fault |
| 3 | Bulk |
| 4 | Absorption |
| 5 | Float |
| 7 | Equalize |
| 245 | Starting |
| 252 | External control |
</details>
## Troubleshooting
Start by subscribing to the `solar/mppt/diag` topic:
| Observation | Likely cause |
|---|---|
| `diag` updates, `ble_age_s`/`data_age_s` growing | MQTT and WiFi are fine; the MPPT advertisement is not being received (range, MPPT asleep, VictronConnect connected to it, BLE scan failed) |
| `diag` stops, `status` still `online` | WiFi or broker path; watch the serial log for `[WiFi] Disconnected, reason=` and `[MQTT] Connection lost` |
| `publish_failures` climbing | Socket/broker rejects; check broker logs for duplicate `MQTT_CLIENT_ID` |
| `free_heap` low or `min_free_heap` dropping each boot | Memory pressure; shorten the diag interval or reduce scanning |
Note that retained `online` on `solar/mppt/status` only means the broker has not yet
noticed the TCP session is gone; it does not prove the ESP32 is still publishing.
The firmware includes two safety nets (both defined in `src/main.cpp`):
- **5 minutes** without a successful publish → drop and rebuild the MQTT connection.
- **60 minutes** without any successful publish → `ESP.restart()`.
---
## Home Assistant
Once the ESP32 is running, go to **Settings → Devices & Services → MQTT**. A **Victron
SmartSolar MPPT** device appears with Battery Voltage (V), Battery Current (A), PV Power (W),
Yield Today (Wh), Charge State, Error Code, External Load (A) and External Load Power (W). No
manual YAML.
**Dashboard card** — [`dashboard.yaml`](dashboard.yaml) is ready to paste into
**Dashboard → Edit → Add Card → Manual**. It needs two HACS frontend plugins:
[Mushroom](https://github.com/piitaya/lovelace-mushroom) and
[Mini Graph Card](https://github.com/kalkih/mini-graph-card).
**Template sensors** — [`configuration_template.yaml`](configuration_template.yaml) adds two
optional calculated sensors: **Garage Net Battery Power** (current × voltage in watts,
positive = charging) and **Garage Battery Status** (Charging / Discharging / Idle). Add the
`template:` block to your `configuration.yaml` and reload YAML.
## Power consumption
| Device | Idle draw |
|---|---|
| Raspberry Pi 4 | ~3–5 W |
| ESP32 (this firmware) | ~0.15 W |
~4W saved overnight is ~40Wh per 10-hour night — roughly 3–4Ah at 12V, which is a real
fraction of a small leisure battery.
## Layout
```
├── include/
│ ├── example.config.h template — copy to config.h
│ └── victron_ble.h BLE parser header
├── src/
│ ├── main.cpp WiFi, MQTT, BLE scanning, FreeRTOS tasks
│ └── victron_ble.cpp AES decryption and MPPT data parser
├── dashboard.yaml ready-made Home Assistant dashboard card
├── configuration_template.yaml optional template sensors
└── platformio.ini build config
```
## Dependencies
PlatformIO handles all of these:
- [NimBLE-Arduino](https://github.com/h2zero/NimBLE-Arduino) — lightweight BLE stack
- [PubSubClient](https://github.com/knolleary/pubsubclient) — MQTT client
- [ArduinoJson](https://arduinojson.org) — JSON serialisation
- mbedtls — AES, bundled with the ESP32 Arduino core
## Related projects
- [victron-ble](https://github.com/keshavdv/victron-ble) — the Python library this follows; still the right answer on a Pi or Linux box
- [esphome-victron_ble](https://github.com/Fabian-Schmidt/esphome-victron_ble) — ESPHome component, if you prefer YAML
- [Victron_BLE_Advertising_example](https://github.com/hoberman/Victron_BLE_Advertising_example) — the original Arduino BLE example
## Licence
MIT.
## Support
If this got a Raspberry Pi out of your solar setup, you can buy me a coffee.
<a href="https://www.buymeacoffee.com/MrSossidge"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee" height="48"></a>