> For the complete documentation index, see [llms.txt](https://anxiang1836.gitbook.io/magic-tts/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://anxiang1836.gitbook.io/magic-tts/hou-xun-lian/page-1.md).
# Page 1
## CosyVoice2论文图
### 基础模型组件
#### 1. Text Tokenizer
使用的whiper的文本转化器，多语言，将文本转化成BPE
#### 2. Speech Tokenizer
CosyVoice 1.0采用VQ来提取Supervised semantic codec，码本大小为4096，但是有效码本只有963。
CosyVoice 2.0采用了FSQ替换VQ，训练了6561的码本，并且码本100%激活。FSQ-Speech Tokenizer的使用使得CosyVoice 2.0在发音准确性上有明显提升。
#### 3. LLM
CosyVoice 2.0采用预训练好的文本基座大模型（Qwen2.5-0.5B）替换了原来的Text Encoder + random Transformer的结构。
采用LLM进行初始化能够更好的进行文本的语义建模，使得在可控生成，音频和文本的情感匹配，多音字发音上会有明显的收益。
#### 4. FlowMatching
将semantic token转化成mel谱
#### 5. Vocoder
CosyVoice 2.0中，使用HiFiGAN作为声码器，将mel谱转换为音频波形
## CosyVoice2推理链路
## 后训练方法
### 1. 基模型 continue pretraining
**稳定性增强：**
构建增强训练数据，标点增强（标点与音频停顿、语义匹配）、英文增强（未见词、大写字母、大小写字母A）、中文增强（生僻字、多音字）、中英混说、支持拼音、支持音标、音色增强、口语化事件
**功能增强：**
对基模型进行属性（风格、情感）的扩充增强训练，使得模型支持属性的自定义控制
### 2. SFT
**中性拟人化：**
使用YOYO对话数据集对模型进行SFT，SFT模型生成符合YOYO特性的音频，模型能力做到：支持口语化事件、音色稳定、音质稳定、支持中英文
**YOYO情感化：**
通过SFT使得模型能够支持情感化（标签、文本自适应）
**IP音色：**
使用少量IP音色数据进行SFT，能做到风格的迁移，实现IP音色定制
#### 3. RL (DPO / RLHF等)
**提升发音稳定性：**
通过RL算法增强合成的韵律，提升合成的音质
**降低字错率：**
通过RL算法降低多音字、生僻字的错误率
**增强情感表现力：**
通过RL算法在SFT基础上提升合成音频的表现力，不同情感的差异性,可以达到情感自适应
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://anxiang1836.gitbook.io/magic-tts/hou-xun-lian/page-1.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.