vall-e - An unofficial PyTorch implementation of the audio LM VALL-E, WIP

Created at: 2023-01-11 19:32:21
Language: Python
License: MIT


An unofficial PyTorch implementation of VALL-E, based on the EnCodec tokenizer.

"Buy Me A Coffee"

Get Started


Since the trainer is based on DeepSpeed, you will need to have a GPU that DeepSpeed has developed and tested against, as well as a CUDA or ROCm compiler pre-installed to install this package.


pip install git+

Or you may clone by:

git clone --recurse-submodules

Note that the code is only tested under Python 3.10.7.


  1. Put your data into a folder, e.g. data/your_data. Audio files should be named with the suffix .wav and text files with .normalized.txt.

  2. Quantize the data:

python -m vall_e.emb.qnt data/your_data
  1. Generate phonemes based on the text:
python -m vall_e.emb.g2p data/your_data
  1. Customize your configuration by creating config/your_data/ar.yml and config/your_data/nar.yml. Refer to the example configs in config/test and vall_e/ for details. You may choose different model presets, check vall_e/vall_e/

  2. Train the AR or NAR model using the following scripts:

python -m vall_e.train yaml=config/your_data/ar_or_nar.yml

You may quit your training any time by just typing quit in your CLI. The latest checkpoint will be automatically saved.

  1. Export trained models:

Both trained models need to be exported to a certain path. To export either of them, run:

python -m vall_e.export zoo/ yaml=config/your_data/ar_or_nar.yml

This will export the latest checkpoint.


python -m vall_e <text> <ref_path> <out_path> --ar-ckpt zoo/ --nar-ckpt zoo/


  • [x] AR model for the first quantizer
  • [x] Audio decoding from tokens
  • [x] NAR model for the rest quantizers
  • [x] Trainers for both models
  • [x] Implement AdaLN for NAR model.
  • [x] Sample-wise quantization level sampling for NAR training.
  • [ ] Pre-trained checkpoint and demos on LibriTTS
  • [x] Synthesis CLI


  • EnCodec is licensed under CC-BY-NC 4.0. If you use the code to generate audio quantization or perform decoding, it is important to adhere to the terms of their license.


  title={Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers},
  author={Wang, Chengyi and Chen, Sanyuan and Wu, Yu and Zhang, Ziqiang and Zhou, Long and Liu, Shujie and Chen, Zhuo and Liu, Yanqing and Wang, Huaming and Li, Jinyu and others},
  journal={arXiv preprint arXiv:2301.02111},
  title={High Fidelity Neural Audio Compression},
  author={Défossez, Alexandre and Copet, Jade and Synnaeve, Gabriel and Adi, Yossi},
  journal={arXiv preprint arXiv:2210.13438},