> For the complete documentation index, see [llms.txt](https://anxiang1836.gitbook.io/magic-tts/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://anxiang1836.gitbook.io/magic-tts/mo-xing-zu-jian/cosyvoice2.md).

# CosyVoice2

## CosyVoice2论文图

<figure><img src="/files/pZlGSza8QusP9bhqy0HX" alt=""><figcaption></figcaption></figure>

### 1. Text Tokenizer

使用的whiper的文本转化器，多语言，将文本转化成BPE

### 2. Speech Tokenizer

CosyVoice 1.0采用VQ来提取Supervised semantic codec，码本大小为4096，但是有效码本只有963。

CosyVoice 2.0采用了FSQ替换VQ，训练了6561的码本，并且码本100%激活。FSQ-Speech Tokenizer的使用使得CosyVoice 2.0在发音准确性上有明显提升。

### 3. LLM

CosyVoice 2.0采用预训练好的文本基座大模型（Qwen2.5-0.5B）替换了原来的Text Encoder + random Transformer的结构。

采用LLM进行初始化能够更好的进行文本的语义建模，使得在可控生成，音频和文本的情感匹配，多音字发音上会有明显的收益。

**精品音色**

{% hint style="info" %}
**instruct示例：**

请问你能模仿四川话的口音么？

请用YOYO的音色自然地说。
{% endhint %}

<figure><img src="/files/1So3ytWgBvQFcd3Jogrm" alt="" width="535"><figcaption></figcaption></figure>

**声音克隆**

{% hint style="info" %}
**注意：**

prompt\_text和prompt\_speech需要严格对齐；

例如，prompt\_text中识别错误，会导致合成音频中，被错误的当成待合成文本
{% endhint %}

<figure><img src="/files/JHNdzceOW0FIk1do7mJ5" alt="" width="535"><figcaption></figcaption></figure>

### 4. FlowMatching

音频采用的是帧率50Hz、采样率为24kHz的梅尔频谱作为声学特征

分词器工作频率为25Hz，即每秒生成25个语音token

因speech token 与 梅尔特征的帧率不一致，因此要对speech token进行2倍上采样( look-ahead convolution)，然后送入transformer结构（chunk-aware causal Transformer blocks），实现speech token的特征与声学特征对齐

根据speaker的embedding和参考语音的mel谱，将输出的speech token进一步解码为mel谱

<figure><img src="/files/PwSQq1MyDHhYkVQ9Ays0" alt="" width="325"><figcaption></figcaption></figure>

{% hint style="info" %}
**NOTE**

精品音色，是仅通过speaker\_embedding引入说话人特征，不指定prompt wav
{% endhint %}

### 5. Vocoder

CosyVoice 2.0中，使用HiFiGAN作为声码器，将mel谱转换为音频波形

HiFiGAN经过U-Net结构3次上采样(8倍/5倍/3倍)，总共120倍上采样生成6000长度的梅尔特征

最终经过短时傅里叶逆变换变成24000个语音信号点(帧移4)，因此最终生成的音频信号的频率是24khz

<figure><img src="/files/EEEmuOlKZJbR0TeeOs5p" alt="" width="208"><figcaption></figcaption></figure>

## CosyVoice2推理链路

<figure><img src="/files/QY3507axeU5c4V6TCd3x" alt=""><figcaption></figcaption></figure>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://anxiang1836.gitbook.io/magic-tts/mo-xing-zu-jian/cosyvoice2.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.