Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content
Train adapters

The exam

Every adapter version sits an exam before it can answer calls. This page says what the exam asks, what it compares, the target, and what each result means. It is for anyone who decides whether to trust a version.

What the exam asks

When Mubit builds a training set, it keeps at least 12 of the task's kept examples back, and never trains on them. It keeps whole conversations back, so a version isn't tested on a conversation it trained on. Only a task whose calls are nearly all one long conversation is split within it. The calls kept back are the exam.

After training, Mubit asks the new version each exam conversation, and asks the base model too, with the task's instructions. It compares each reply with the reply your provider model gave in the real call. Some replies are checked by rule. The others are graded by an automatic grader, which runs on Mubit's own account.

A new version of a task sits the same exam as the version before it, with the conversations its own training kept back added. So two versions of a task compare on the same questions.

The target

A version passes when its score is close enough to your provider model's on the same exam: within 5 points by default, or within 15 points when the exam is too small to tell the two apart. Some test cases are required: a version must pass all of them.

The adapter's line in the console says how it did, for example:

  • Matched {a}% of your provider model's answers on {n} test cases · needs {need}%;
  • Passed {a}% of {n} required test cases · needs 100%.

Test results

Each version gets one result. Select the adapter, then open History.

ResultWhat it means
PassedThe version answers close enough to your provider model.
Below targetThe version answers less like your provider model than the target asks. Its line says by how much.
Malformed tool callsSome of the version's tool calls were malformed: {k} of {n} tool calls were malformed · needs 0.
Prompt onlyThe base model, given the task's instructions, did as well as the adapter. Mubit uses the base model with the instructions in place of the adapter's weights. Requests shows its replies as Prompt only.
Not testedThe version has no result. When Mubit gives a reason, it shows when you point at the result.

While Mubit grades a version, the adapter's line reads Waiting for grading on {k} test cases.

A result measures the version on its exam. It doesn't promise every future reply: check representative calls of your own in Requests and in Try models.

How the exam acts on app.mubit.ai today

The exam advises. It does not hold a version back:

  • A version whose result is Below target, Malformed tool calls or Not tested still becomes a candidate, and can go into a silent test, a partial rollout and then live, as a passing version does. See The rollout.
  • Its result stays on the version. In Requests, each reply of such a version shows below its test beside Served by, such as Your adapter (v2) · below its test.
  • One case is held back: a version that gives the same reply to every exam question never answers calls.
  • The checks during the rollout act the same way: they record a low score on the version, and they don't hold the version back or take it offline.
ℹ️Note

If a version answers calls you don't want it to answer, take it offline, roll back, or switch the adapter off. An admin does this on the adapter's page in the console. See Take a version offline or roll back.

After a version that didn't pass

Under a version that didn't pass, the adapter shows what comes next:

  • Next: a new version after {need} new examples: send more calls of the task, or rate more of its calls good.
  • Next: retraining paused: {reason}: the reason says why the task doesn't retrain now.

Ratings make the most difference: a call rated bad is never trained on, so rate the replies you know are wrong. See Rate a call yourself.

Next