Skip to content
Jefferson Gonçalves
← Back to projects

VoiceBM

Voice biometrics engine for Home Assistant: identify speakers and publish identity signals via MQTT.

#VoiceBM

Local, open-source voice biometrics.

VoiceBM resolves speaker identity from live audio — entirely on-premises, with no cloud dependency — and publishes the result over MQTT. Home Assistant is the reference integration and MQTT discovery is built in, but the output is plain MQTT: anything that can read a topic can consume it. It runs alongside the Wyoming ONNX ASR pipeline and uses Sherpa-ONNX for speaker embedding.

⚠️ Running Home Assistant and the VoiceBM dashboard together? Read this first. VoiceBM has two control surfaces: Home Assistant and its own built-in dashboard. Drive settings from one of them, not both:

  • Home Assistant only → use the Home Assistant controls.
  • Dashboard only → use the dashboard controls.
  • Both running at once → from Home Assistant, everything works. From the dashboard, only ID Injection, enrollment, and the Transcript Preferred switch are guaranteed; thresholds, gallery max, and other settings may not sync — set those from Home Assistant. See Which controls to use for the full explanation.

#What it does

VoiceBM answers one question: who is speaking?

It's a set of independent primitives that publish over MQTT. Each has one job; how its output is consumed — automations, an LLM pipeline, access-control logic — is the integrator's decision. Home Assistant is the reference integration (MQTT discovery is built in), but anything that reads MQTT can consume VoiceBM.

#Components

VoiceBM is modular — run only the parts you want. Each is independent, and the two add-ons sit on the core side they extend.

VoiceBM 2.0 — the engine: identity resolution, enrollment, and the MQTT contract.

  • Active 2.0 — request-driven, per utterance. Resolves speaker identity against the enrollment gallery before STT runs; publishes identity, confidence, and decision; gates transcript output (voicebm/transcript/preferred = allowed speakers only, voicebm/transcript/debug = everything); keeps a pending buffer of recent utterances for enrollment review. Includes the STT Bridge, an OpenAI-compatible endpoint that lets OpenWebUI (or any client speaking the OpenAI transcription API) drive the identity pipeline — so VoiceBM works with platforms beyond Home Assistant.
  • Passive 2.0 — continuous, per node. Records audio from an RTSP audio source, filters non-speech with VAD, and embeds/scores against the gallery in the background to build and review your roster over time.
  • Global 2.0 — the system-wide control layer: the global "Voice Biometrics" device, identity injection control, blocklist, and settings that aren't tied to a single node or person.

Optional add-ons — each extends the side it rides on:

  • Emote 1.0 beta — speech emotion recognition on the active side. Estimates emotional tone per utterance and publishes it alongside the identity. CPU-only.
  • Ambient 1.0 beta — audio event detection on the passive side. Recognizes environmental sounds (glass, a dog, a vehicle, and so on) on a continuous background listener. CPU-only.

Everything is optional. The two editions track their own release path (pre-alpha → beta → 1.0), separate from the 2.0 core.

#Requirements

Compute baseline is CPU — no GPU or CUDA required. VoiceBM is an orchestrator: it coordinates swappable components, none of which it depends on. Identity, passive, and ambient run on ONNX Runtime (sherpa-onnx) on CPU; the active path uses the ONNX ASR container; Emote (SER) pulls torch for FunASR/SenseVoice-Small but runs CPU-only (device='cpu'). No component touches CUDA or the GPU. GPU acceleration is an optional upgrade you wire into a component yourself — it is never assumed.

  • Linux host with systemd (ARM64 or x86_64)
  • A Python environment for the dependencies — conda, a venv, or an interpreter you already manage. Conda is optional: it's a way to keep these deps off your base system if you want that, not a requirement.
  • paho-mqtt and sherpa-onnx available to that interpreter
  • Sherpa-ONNX nemo_en_titanet_small.onnx speaker model
  • Wyoming ONNX ASR running in Docker
  • MQTT broker (Mosquitto or compatible)
  • Home Assistant with MQTT integration enabled
  • FFmpeg (passive pipeline recorder)

#Installation

tar -xzf VoiceBM_v2.0_Complete.tar.gz
cd VoiceBM_v2.0_Complete
./setup_voicebm.sh

setup_voicebm.sh is the install path. It walks you through component selection, sets up the Python environment of your choice (conda, a venv, or an interpreter you manage), installs the dependencies and the speaker model, takes your node details, writes config.json for you, and then offers to run the deploy. You never hand-write configuration — the wizard builds it from your answers. It only uses sudo for the final systemd step.

#After the wizard

If you declined the deploy step at the end of the wizard, run it any time:

sudo ./scripts/deploy_global_services.sh

Add a passive node service for each node you defined (nodes are RTSP audio sources — a Pi or anything serving an RTSP audio stream, not a camera):

sudo ./scripts/replicate_node.sh living
sudo ./scripts/replicate_node.sh bedroom

Verify:

sudo systemctl status voicebm-stt.service
sudo journalctl -u voicebm-stt.service -n 50 --no-pager

Check Home Assistant for the Voice Biometrics device under Settings → Devices & Services → MQTT.

#STT Bridge & Dashboard

These two pieces let you use VoiceBM from outside Home Assistant. If you only ever drive VoiceBM through an HA voice satellite, you don't need either — but together they're what makes VoiceBM platform-agnostic.

#STT Bridge

The bridge is an OpenAI-compatible speech-to-text endpoint. It accepts the same /v1/audio/transcriptions request OpenWebUI and other OpenAI-style clients already speak, runs the audio through the full VoiceBM active pipeline (identity resolution, injection, the transcript gate), and returns the transcript. The result: any platform that can point its STT at an OpenAI endpoint gets identity-tagged transcripts from VoiceBM, with no Home Assistant in the path.

It installs as voicebm-stt-bridge.service and listens on port 8005.

To connect a client (e.g. OpenWebUI), set its STT endpoint to:

http://<voicebm-host>:8005/v1

and use model name whisper-1 (the value is ignored — the bridge always routes to your ASR container).

Transcript Preferred switch. Because bridge traffic runs the same pipeline as native HA speech, a switch (in Home Assistant and on the dashboard) controls whether bridge-driven utterances reach voicebm/transcript/preferred. Turn it off to keep an OpenWebUI session from leaking into the topic Home Assistant consumes; native satellite speech is never affected. The state lives in config.json (voicebm.transcript_preferred) and stays in sync across the dashboard and HA.

#Dashboard

The Flask dashboard (port 5000) is VoiceBM's own control surface — most useful when you're running the bridge, or on a setup without Home Assistant. It mirrors the controls HA would otherwise give you:

  • Pending Voices — review unmatched speakers from the active pipeline and enroll or reject them in one click.
  • Enrolled Identities — rename, merge, delete, and per-person blocklist.
  • Switches — ID Injection and Transcript Preferred, both writing to config.json and echoing to MQTT so HA stays in sync.

It installs as voicebm-dashboard.service. Open:

http://<voicebm-host>:5000

The dashboard reads and writes the same config.json everything else uses, so a change made here shows up in Home Assistant and vice versa. (Pending Voices and similar live panels refresh on page load; a manual refresh shows the latest.)

#Which controls to use

Threshold and similar settings can be driven from either Home Assistant or the dashboard, but not reliably from both at once:

  • Home Assistant only — use the Home Assistant controls. All of them work.
  • Dashboard only — use the dashboard controls. All of them work.
  • Both at once — from Home Assistant, all controls work normally. From the dashboard, only the ID Injection toggle, enrollment, and the Transcript Preferred switch are guaranteed. The rest (thresholds, gallery max, etc.) may or may not sync — leave those to Home Assistant.

#Upgrading from v1.0.x

cd scripts
sudo ./upgrade_v1_to_v2.sh

The upgrade script backs up all affected files before making any changes and prints rollback instructions on completion. See CHANGELOG_ACTIVE_2_0.md for the full list of breaking changes and migration steps.

#Architecture overview

             ACTIVE                                          PASSIVE
    (request-driven, per utterance)                   (continuous, per node)

    Voice satellite / assist audio                    RTSP audio node
            │                                               │
            ▼                                               ▼
    Wyoming ONNX ASR (Docker)                         rec_node.sh      RTSP -> WAV
    handler.py                                              │
        │  ▲                                                ▼
        ▼  │                                          vad_filter.py    drop non-speech
    voicebm_stt_service.py                                  │
       Sherpa embed                                         ▼
            │                                         embed_node.sh    Sherpa embed
            │                                               │
            └──────────►  ┌───────────────┐  ◄─────────────┘
              match /     │    GALLERY     │    match /
              enroll      │   enrolled     │    enroll/cluster
                          │  voiceprints   │
                          │   (Sherpa)     │
                          └───────────────┘
              -- both sides contribute to it · both match off it --
            │                                                   │
            ▼                                                   ▼
    handler.py gates, publishes:                       publish_identity_node.py
      transcript/preferred (gate-enforced)             voicebm/{node}/identity ·
      transcript/debug     (raw)                         person_id · score · accepted
      current_speaker                                         │
            │                                                 ▼
            ▼                                          Home Assistant (roster review)
    Home Assistant / LLM

The two sides are independent but not isolated: both embed with Sherpa, both write to the same gallery, and both match against it. That shared gallery is the bridge — strip either side and the other still stands.

Identity primitives published per utterance:

  • voicebm/active/identity — speaker ID, display name, confidence, decision
  • {person_id}/voice — binary sensor (ON during utterance)
  • voicebm/current_speaker — display name of current enrollment-grade sample
  • voicebm/pending_active — buffer of recent unidentified utterances for review

#Enrollment

Speakers are enrolled through the pending active buffer. When an unknown speaker is detected, the utterance is held in voicebm/pending_active. The Home Assistant Voice Biometrics device exposes controls to listen to the sample, assign a name, and enroll it into the gallery. Enrolled speakers are immediately active — no restart required.

#Troubleshooting

Services not starting:

sudo journalctl -u voicebm-stt.service -n 50 --no-pager
sudo journalctl -u voicebm-enrollment-watcher.service -n 50 --no-pager

Identity always resolving to user:
Check that the enrollment gallery exists and that the active threshold slider in Home Assistant is set appropriately for your environment.

Docker container not updating:
handler.py is deployed via docker cp — the container (your-asr-container) is never rebuilt. If the deploy script completed without error, check the container logs: docker logs your-asr-container.

#License

MIT License — see LICENSE for details.

Author: David M. Dryver Sr.
Repository: https://github.com/cybericebyte/VoiceBM
Version: 2.1

New version available.