请教：如何训练react类型多轮人机对话

[Sakurakdx]

RT，请教各位大佬，如何训练react类型多轮人机对话

我想训练一个能够连续对话的Agent，SFT的方案是构造下面这种多轮数据
{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
{"role": "assistant", "content": "The capital of France is Paris."},
{"role": "user", "content": "And what about Germany?"},
{"role": "assistant", "content": "The capital of Germany is Berlin."},
]
}

想请教RL该如何做？使用另一个LLM模拟user的情况下，如何做rollout？如何计算奖励

[luoyetingqiu]

1

[PanAndy]

可以使用AgenticPipeline来完成多轮会话训练，
https://alibaba.github.io/ROLL/zh-Hans/docs/User%20Guides/Pipeline/agentic_pipeline_start

可以本地调用用Sokoban/FrozenLake env观测下轨迹的形式：
https://alibaba.github.io/ROLL/zh-Hans/docs/Getting%20Started/Debugging%20Guide/debug_guide#3-%E6%9C%AC%E5%9C%B0%E8%B0%83%E8%AF%95-agentic-%E5%A4%9A%E8%BD%AE%E4%BA%A4%E4%BA%92%E8%BF%87%E7%A8%8B

trajwise的训练样本就是这个形式的，format_rollouts中只对assistant部分的token设置resonse_mask=1，其余部分是0.

{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
{"role": "assistant", "content": "The capital of France is Paris."},
{"role": "user", "content": "And what about Germany?"},
{"role": "assistant", "content": "The capital of Germany is Berlin."},
]
}

reward 设置是基于你具体的业务场景的，比如完成了某个目标，或者解对了数学题。
"另一个LLM模拟user" 以及reward计算都应该自定义在env里，rollout的逻辑在env_manager.run_rollout_loop(

ROLL/roll/pipeline/agentic/env_manager/traj_env_manager.py

Line 92 in 6ca3d10

def run_rollout_loop(self, data: DataProto):

)中，可以调试看看，就是多轮rl交互