🤯AI Breakthrough: Audex - Speech & Sound 🗣️

CHAPTER 1: THE INTRODUCTION OF AUDEX – A NOVEL AUDIO-TEXT LLM
The NVIDIA research team has unveiled Audex (Nemotron-Labs-Audex-30B-A3B), a groundbreaking unified large language model capable of processing and generating both audio and text. This innovative model, built upon a Mixture-of-Experts (MoE) Transformer decoder architecture, represents a significant advancement in multimodal AI. Key features include a total of 30 billion parameters, with 3 billion activated per token, and a streamlined design intended to avoid the common regression issues observed in multimodal models. The model's core is the Nemotron-Cascade-2-30B-A3B backbone, a hybrid Mamba-Transformer with 52 layers, utilizing 128 routable experts and 6 activated experts. This deliberate simplicity is central to Audex's functionality.

CHAPTER 2: CORE ARCHITECTURE AND TRAINING STRATEGIES
Audex’s architecture prioritizes efficiency and performance. Audio inputs are encoded and projected into the text embedding space, while both text and quantized audio tokens are treated uniformly during the generation process. Notably, the design avoids the traditional “thinker-talker” split and stacked cascade of models, allowing Audex to run on standard LLM stacks such as Megatron-LM for training and vLLM for inference. The model supports both an instruct mode and a thinking mode, boasting a context length of 1 million tokens. The training methodology employs a multi-stage Supervised Fine-Tuning (SFT) curriculum, beginning with text SFT, followed by audio warmup, audio generation, and finally, audio understanding.

CHAPTER 3: AUDIO CODECS AND TOKENIZATION
To effectively handle audio data, Audex utilizes two distinct codecs. Speech is processed via X-Codec2 at 50 tokens per second, employing single-layer finite scalar quantization (FSQ) with a 65,536 codebook. Non-speech sound is handled by X-Codec at 200 tokens per second, utilizing four flattened residual vector quantization (RVQ) layers. This layered approach allows for variable token budgets based on the complexity of the audio, ensuring optimal processing efficiency. The interactive demo allows users to calculate token counts for various audio durations.

CHAPTER 4: PERFORMANCE AND BENCHMARKING – TEXTUAL CAPABILITIES
Audex demonstrates impressive performance on a range of textual benchmarks. It achieves a score of 86.4 on MMLU-Redux, closely matching the performance of its 86.3-scoring Nemotron-Cascade-2-30B-A3B backbone. Furthermore, Audex surpasses the baseline on IMO AnswerBench (81.1 vs. 79.3) and GPQA-Diamond. While slight drops are observed on MMLU-Pro and GPQA-Diamond, Audex consistently outperforms comparable models like Qwen3.5-35B-A3B across multiple reasoning, alignment, and instruction-following benchmarks.

CHAPTER 5: PERFORMANCE AND BENCHMARKING – AUDIO CAPABILITIES AND FUTURE DIRECTIONS
Audex excels in speech recognition, achieving an average word error rate of 6.82 on the OpenASR leaderboard, surpassing Step-Audio-R1.1-33B and Qwen3-Omni-30B-A3B-Thinking. However, performance on audio understanding tasks, such as MMAR and MMSU, lags behind the strongest audio LLMs. Despite this, Audex distinguishes itself by its ability to generate general audio, a capability absent in other leading open models. The model adheres to the ChatML template, utilizing vLLM 0.20.0, and incorporates a vLLM audio-QA script for interactive audio question answering. The research team emphasizes the importance of a multi-stage training approach to avoid context length issues, utilizing a large dataset combining 157.4B audio tokens and 320.5B text tokens across tasks like ASR, AST, TTS, text-to-audio, and audio understanding.