ChatBench Simulators are a set of fine-tuned user simulators designed to enable automated, realistic evaluation of large language models (LLMs) through simulated user–AI conversations. Instead of recruiting human participants for every evaluation, you can use this simulator (chatbench-distilgpt2) to act as a proxy user. By providing a multiple-choice question (full stem plus answer options) via a simple CLI or Python API, the simulator generates natural user turns—asking clarifications, signaling understanding, or indicating errors—until the simulated user “accepts” an answer. The resulting conversations can be used to compute task success rate, coherence, user satisfaction, error recovery, and latency metrics, allowing researchers and practitioners to move beyond static benchmarks and evaluate models under interactive, user-in-the-loop conditions. — Model type: Causal LM (user simulator) — Base model: distilgpt2 (82M parameters) — Fine-tuned on: microsoft/ChatBench (≈12K multi-turn QA dialogs) — Languages: English — License: Inherits from DistilGPT2 (OpenAI license) — Developed by: Microsoft — Contacts: kevingao@microsoft.com, serinac@berkeley.edu, jmh@microsoft.com The model was…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: microsoft
Теги: gpt2, Microsoft, ChatBench, Interactive Benchmark, User Simulator, Benchmarking, en, text-generation-inference
Лайков: 4 | Загрузок: 135
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
