Text-to-speech that runs on a microcontroller

vtts turns text into speech entirely on an ESP32: text normalization, pronunciation, an int8 neural voice and the vocoder all run on the chip. No cloud, no network, no Linux. Every clip on this page was generated by the chip and sent back over its serial port.

vtts> Pin còn 15%, nhiệt độ phòng là 28°C. SAID model=M audio=4.24s synth=1.08s RTF=0.25 first chunk 199 ms, playback started 274 ms, underruns 0
0.15–0.24real-time factor on ESP32-P4
1.1–2.6 MBmodel size in flash (int8)
0.26–0.49real-time factor on ESP32-S3
0.012character error rate, Vietnamese

Listen

Each column is a model with its size and real-time factor (RTF) per chip; below 1 is faster than real time. Both models run on both chips. Every clip is the chip's own output, only compressed to MP3.

How it works

Everything below runs on the microcontroller, from the raw text to the samples sent to the speaker.

  1. UTF-8 text
  2. Language tag per word
  3. Normalizer (numbers, dates, units)
  4. G2P
  5. Duration + acoustic model (int8)
  6. Vocos decoder + iSTFT
  7. PCM 22.05 kHz → I2S speaker

Runtime

C99, no malloc: the caller passes one arena whose size is known up front. Int8 weights with per-channel scales, int8 activations with one scale per frame. Streaming keeps per-layer state, so every frame is computed once.

On the chip

Int8 SIMD kernels for the ESP32-P4 (PIE) and the ESP32-S3 (128-bit SIMD), work split over both cores with FreeRTOS. Chip audio matches the reference C runtime on a PC.

Models

A VITS teacher (~24M params) is distilled into 0.8–1.7M-param students: duration predictor, acoustic model with pitch and energy, Vocos-style ConvNeXt decoder with an iSTFT head.

Front-ends

One per language, in Python for training and in C for the device, tested to give identical token ids. Vietnamese: spelling-based G2P with 6 tones. English: CMUdict plus letter-to-sound rules learned from it.

Data

The Vietnamese prototype uses a public single-speaker corpus. Bilingual voices use about 98 h each of speech synthesized with VoxCPM2 (Apache-2.0) from designed voices.

Tooling

PyTorch for training, ESP-IDF v6.1 for the firmware, PhoWhisper and Whisper for intelligibility, UTMOS and DNSMOS for naturalness and cleanliness.

Get started

gh release download v0.3.2 -R tuanpmt/vtts -p '*.vtts' -D models
cd firmware/esp32
BOARD=esp32-p4-nano ./build.sh ../../models/vtts-vi-s-v0.1.q8.vtts \
  ../../models/vtts-vi-m-v0.2.q8.vtts -p PORT flash monitor
BOARD=BoardSpeaker pathDefault volume
esp32-p4-86-panel-eth-2roESP32-P4-86-Panel-ETH-2ROES8311vol 70, gain 0
esp32-p4-nanoESP32-P4-NANOES8311vol 90, gain 12
esp32-s3-touch-lcd-2.8ESP32-S3-Touch-LCD-2.8PCM5101vol 80, gain 6
echoearEchoEar 2.0 (ESP32-S3)ES8311vol 75, gain 6

Type a sentence on the serial console and the board speaks it. Full instructions are in the README.