Streaming, timeouts and limits
This page lists the limits that apply to a model call: the time limits, retries, sizes and rates. It is for developers who run long calls, large requests or heavy traffic through Mubit.
Streaming
Stream as you do with your provider: set stream to true on OpenAI, OpenAI-compatible services and Anthropic, or call streamGenerateContent?alt=sse on Gemini. Mubit starts the answer to your app when the provider's status and headers arrive, then passes each part as it comes. It doesn't change the stream, and it adds nothing to it.
Mubit records the whole reply of a streamed call. Each provider page shows a streaming example: OpenAI, Anthropic, Gemini.
If your app stops reading, or closes the connection, Mubit cancels the call to the provider.
Time limits
| Limit | Value | What happens |
|---|---|---|
| Connect to the provider | 10 s | Mubit tries once more on a new connection. If that fails too, the call gets 502 upstream_unreachable, and nothing was sent. |
| First byte | 610 s | From the start of your request to the provider's status and headers. Past it, the call gets 504 upstream_timeout. |
| Silence | 15 minutes | The longest time no part of the answer goes to your app, because the provider is quiet or your app stopped reading. Past it, Mubit closes the connection. |
| Total | 60 minutes | The longest a call can run, from the start of the request to the last byte of the answer. Past it, Mubit closes the connection. |
When Mubit closes a stream at a limit, the connection breaks. Mubit adds nothing to the stream.
The OpenAI and Anthropic SDKs wait 600 s by default, so their own timeout usually ends a wait for the first byte before Mubit's does. For a long answer, stream it: the first byte then arrives early.
Retries
Mubit retries a call in two cases:
- No connection. If Mubit can't connect to the provider, it tries once more on a new connection. Nothing was sent, so a second try is safe.
- An overloaded provider. If the provider answers
503, or Anthropic answers529, and no part of the answer has reached your app, Mubit sends the call again, up to two more times, after a short pause: about 200 ms, then about 400 ms. It does this only for a request body of 4 MiB or less. No other status is retried, and a4xxnever is.
Mubit never retries a call that has started to go to the provider and then broke. The provider may already have it, and a second send could be billed twice. The call gets 502 upstream_failed.
Your SDK owns every other retry. Mubit's errors tell it what to do: x-should-retry: false on errors a retry won't fix, and retry-after on 429 and 503.
Sizes
- Request body. Mubit sets no size limit of its own on a request. It streams the body to the provider as it arrives. Your provider's limits apply.
- What Mubit records. Up to 32 MiB of a request, and the first 4 MiB and the last 256 KiB of a reply. A larger call still goes through whole. Only Mubit's record of it is cut.
Rates
Mubit applies no per-key rate limit of its own to model calls. Your provider's limits apply, and its 429 answers come back to your app as the provider sent them, with its retry-after and rate-limit headers.
Mubit limits the connections it holds open to providers. Past that limit, a call gets 503 gateway_overloaded with retry-after, and nothing was sent. An instance that is shutting down answers 503 gateway_draining. Try again in both cases.
The runtime calibration routes have their own rate limits: see Runtime calibration.