Retries and Idempotency

What the SDK retries on its own, how to tune it, how to make writes safe to repeat, and when to add an outer retry.

What the SDK retries automatically

Every Client call (the namespaces and client.raw.invoke) and every event post on the loop lane is retried up to max_retries times with exponential backoff and jitter when the failure is one of these:

• RateLimitError (429) and UnavailableError (503).
• InternalServerError: 500 and any other 5xx (UnavailableError is one, so 503 is covered twice).
• TransportError whose code is UNAVAILABLE, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED, ABORTED, INTERNAL, CANCELLED, CONNECTION_ERROR or TIMEOUT: a request or connect timeout, a dropped connection, a gRPC transient code.

It never retries:

• A refused connection or an unresolvable host (APIConnectionError.refused is True): nothing is listening, so the call fails within connect_timeout (2 s) on the first attempt.
• 400, 401, 403, 404, 409, 410 (BadRequestError, AuthenticationError, PermissionDeniedError, NotFoundError, AlreadyExistsError, GoneError), UnsupportedFeatureError and LessonRejected: caller errors; retrying repeats them.

The last error is re-raised when the retries are exhausted, so a handler sees the real class (see Errors), never a wrapper.

Tuning

The delay before attempt n (the first retry is attempt 2) is min(base × 2^(n−2), cap), then scaled by a random factor in [1 − jitter, 1 + jitter]: with the defaults, about 200 ms, 400 ms, 800 ms, ... capped at 5 s. MUBIT_RETRY_ATTEMPTS (total attempts including the first) is the deprecated 0.13 name: it is read only when MUBIT_MAX_RETRIES is unset, equals MUBIT_MAX_RETRIES + 1, prints one deprecation notice, and is removed in 1.0.

MUBIT_MAX_RETRIES=4
MUBIT_RETRY_BASE_MS=500
MUBIT_CONNECT_TIMEOUT_MS=5000

client = mubit.Client(max_retries=0)                 # fail fast in a request handler
worker = client.with_options(max_retries=5, timeout=60.0)

Idempotency keys

client.memory.remember() (and the underlying control.ingest) carry an idempotency key so a repeated write returns the existing entry instead of creating a duplicate. If you don't pass one, the key defaults to the item id (item_id, else an auto-generated remember-<timestamp>). Pin it explicitly to dedupe across retries from a queue worker:

client.memory.remember(
    "…",
    kind="fact",
    idempotency_key=f"ticket-{ticket_id}-fact-1",
    run_id=run_id,
    agent_id="support-agent",
)

client.outcomes.record(reference_id=..., idempotency_key=...) applies the outcome at most once per key, so a retried outcome write reinforces once rather than double-counting. Loop events (run.start, model.call, outcome, run.end) carry a content-derived event_id; a re-posted batch is reported under duplicates and applied once. Other writes are naturally idempotent by their own ids (client.memory.archive keys on the block id, client.agents.register on the agent id).

When to retry yourself

The built-in retries cover a transient blip. Add an outer budget only in queue workers and batch jobs that can wait longer than a few seconds, and only around the retryable classes:

• Retry RateLimitError (honour retry_after when present), InternalServerError / UnavailableError, and APIConnectionError unless refused is True.
• Don't retry BadRequestError (which includes NotFoundError and AlreadyExistsError), AuthenticationError (which includes PermissionDeniedError), GoneError, UnsupportedFeatureError, LessonRejected: fix the call instead.

import random
import time
 
import mubit
 
 
def with_retry(fn, max_attempts=4, base_ms=300):
    for attempt in range(max_attempts):
        try:
            return fn()
        except mubit.RateLimitError as e:
            delay = e.retry_after or base_ms * (2 ** attempt) / 1000
        except (mubit.InternalServerError, mubit.UnavailableError):
            delay = base_ms * (2 ** attempt) / 1000
        except mubit.APIConnectionError as e:
            if e.refused:
                raise
            delay = base_ms * (2 ** attempt) / 1000
        time.sleep(delay * (0.8 + random.random() * 0.4))
    raise RuntimeError("retries exhausted")

The 0.13 handlers (except ServerError, except TransportError) keep working: ServerError is InternalServerError and TransportError is an APIConnectionError. They do not see RateLimitError (429), which has no 0.13 parent; add it when you move to the outer-retry pattern.

See also

• Errors — status codes and the SDK exception hierarchy
• SDK configuration — timeouts, transport selection and the failure policy
• Rate limits — input caps and overload behavior