ESP32 Swarm / Local AI Nodes

Listen → parse → route → execute on $8 chips — when the house (or desk) is a mesh of bounded nodes, not one Mac Mini brain

← Cyborg wave hub · Shopping list · Principles · E‑paper · Character · M5

Status: field notes / architecture digest — not a verified full-home deploy on this desk.

Source post (long guide, heavy bookmarks): @ardchain — physical swarm architecture + ESP32 build playbook

Related code / frameworks named in the post: slvDev/esp32-ai, cactus-compute/needle (verify current README before flashing anything).

What this path is

A local / edge story that pushes back on “buy a $3k GPU box for the house.” Instead of one central brain:

Listen — cheap node + I2S mic captures voice (wake / short utterance)
Parse — small model or edge framework maps speech → structured intent
Route — deterministic code chooses the next branch
Execute — BLE macro, GPIO, Home Assistant, or a fixed action

Each microcontroller is a node with one job (wake word, kitchen intents, garage intents). Schemas (JSON tools) reduce drift between nodes. Ordinary C++ handles debounce, dedupe, and BLE — models only where language is fuzzy.

What this path is not

Not the same as our e‑paper cloud dump (Wi‑Fi → API → Arthur) — opposite default for where intelligence lives
Not a free pass to trust every flash command in a viral thread without checking the repo
Not “70B on an $8 chip” — the claim class is ~26–30M parameter / function-calling edge models, heavily constrained
Not required for day-one HEOS dumps — use cloud I/O first if you only need capture

Read with a filter (important)

The thread is useful as an architecture playbook. Replies also flag that some “complete flash + BLE + BTHome” packaging in viral posts can oversell what’s actually in public repos (e.g. demo vs full intent mesh).

Our rule: steal the patterns (bounded nodes, schemas, hardware tiering, code vs model). Treat exact bin paths, wiring pinouts, and “works offline end-to-end this weekend” claims as check against GitHub + hardware wiki before you order ten boards.

Core ideas worth keeping

Idea	Meaning	Why it matters here
Memory is the bottleneck	ESP32-S3: limited SRAM; models live in flash / mapped carefully	Don’t design “one omniscient node”
One decision per node	Wake word / intent class / BLE macro — not all three plus chat	Matches cyborg “one job per device”
Edges carry structured intent	JSON tools / fixed schemas between chips	Same as agent tool schemas — physical CLIs
Code for deterministic work	Debounce, route, BLE, timers	Don’t spend tokens on if/else
Hardware tiering	$8 chip for lights; bigger box for vision/reason	Swarm only when multi-room / offline / cost justifies it
Hyper-specialize	Kitchen model ≠ garage model	Maps to `arthur.agent` vs `janice.agent` hardware contracts

Swarm shape (fan-out → barrier → fan-in)

Phase	Job	Example
Fan-out	Independent capture	I2S mics in rooms; wake without waking the whole house
Barrier / reduce	Compress to intent	Small offline model or fixed classifier → validated JSON
Fan-in	Physical action	BLE macro, GPIO, HA automation, agent webhook

When is swarm overkill? One desk, one dump button, cloud STT already fine — stay on e‑paper / Cart A. Swarm earns its keep with multi-room, privacy, API cost, or always-on listen power budgets.

Parts (node starter)

From the guide’s “exactly two components” baseline (plus cable). Full cart: Cart D on /cyborg-shopping.

#	Part	Requirement	~Price
1	ESP32-S3 Dev Board	Prefer 16MB flash + 8MB PSRAM variants (model / mapping headroom)	~$5–15 (“$8 chip” class)
2	INMP441 I2S mic	Same class as character-path mics	~$3–8
3	USB‑C / USB data cable	Flash + serial	have / ~$5
4	Jumpers / solder	I2S wiring	~$3

I2S wiring (as stated in the guide — verify on your board)

INMP441	ESP32-S3
VDD	3.3V
GND	GND
L/R	GND (left channel)
WS	GPIO 4
SCK	GPIO 5
SD	GPIO 6

Pin maps differ by DevKit silk-screen. Confirm against your module’s pinout before soldering.

Two software lanes (from the guide)

Lane	Idea	Caution
Bare metal / flash weights	Map a tiny distilled model into flash; C++ parses intent → action	Bootloader mode, partition maps, esptool — easy to brick if you freestyle
Needle-class frameworks	~26M “simple attention” / function-calling oriented; define tool schemas; synth data; finetune; deploy checkpoint	Follow current cactus-compute/needle docs; don’t assume tweet commands match main

Stated product shape for edge agents: model returns executable JSON tools (e.g. toggle_lights), not chat prose — then C++ / HA runs the action. That matches how we already think about agent tools and CLI-shaped interfaces.

Tutorial ladder (safe order)

Flash blink / serial hello on ESP32-S3 (16MB+PSRAM board)
Wire INMP441; prove I2S samples in serial (levels, not silence)
Wake-word or push-to-talk gate (no model yet)
Offline or on-device keyword / tiny classifier if available in samples
Define 2–3 tool schemas only (e.g. log_dump, status_ping)
Execute path A: serial print JSON only (prove parse)
Execute path B: BLE or Wi‑Fi webhook to HEOS / Home Assistant
Only then: finetune / flash a specialized micro-model for one room or one agent

Skip “full offline LLM weekend” as step 1. Capture + structured intent is the cyborg win.

Cloud path vs local swarm (both cyborg)

	Cloud I/O (e‑paper / character)	Local swarm (this page)
Brain	Arthur / Janice / OpenAI on server	Tiny model + C++ on chip
Best for	Rich dumps, HEOS vault, model upgrades free	Privacy, offline, multi-room macros, low API spend
Failure mode	Wi‑Fi / API down	Wrong local intent / hard to update models
First build here	Cart A e‑paper	One S3 + INMP441 + JSON intent to serial

Hybrid is allowed: local wake + intent class, then hand high-confidence dumps to cloud Arthur. Don’t pick a religion — pick a blast radius.

Map to named agents / HEOS

Node idea	Agent / system	Intent examples
Desk dump node	`arthur.agent`	`log_dump`, `start_build` (webhook to HEOS)
Ops / “what’s next”	`janice.agent`	`next_block`, `timer_5`
Room macro node	Home Assistant / BLE	`lights_off`, `set_timer` (not every agent)

Hyper-specialize: a kitchen-only node should not know Arthur’s code tools. Same principle as separate hardware contracts on /cyborg.

First ship definition

Done when:

ESP32-S3 + INMP441 captures audio you can prove (serial / file)
One spoken or button-gated phrase becomes a fixed JSON intent
That intent triggers one action (serial log, webhook, or BLE) without a laptop UI in the loop

Stretch: two nodes with different tool schemas, no shared “do everything” model.

Related paths

Cyborg wave hub

All build paths + shared I/O architecture.

Hub →

Shopping list

Cart D: S3 + INMP441 swarm starter.

Parts →

E‑paper (cloud dump)

Complementary: brain in the cloud, not on flash.

Path →

Character device

Also uses INMP441-class mics + face states.