# Flash Sidecar — live operating guide (for agents)

You are talking to **Flash Sidecar**, an OpenAI-compatible reverse proxy that fronts one or more
inference backends and presents them as a single gateway. **No human runs this binary** — agents
configure and operate it. This guide is served by the running instance itself, so you can understand it
cold, over HTTP, with no access to the source repo.

This page is served at `GET /` and `GET /help` as Markdown. It is the default guide compiled into the
binary; a deployment may override it with a file (`SIDECAR_AGENT_GUIDE`) that reloads on `SIGHUP`.

## Orient yourself in three calls

```bash
curl -s $BASE/status  | jq .    # live: upstreams + health, feeds + limits, traffic pulse
curl -s $BASE/config  | jq .    # effective configuration (secrets redacted)
curl -s $BASE/v1/models | jq '.data[] | {id, model_type, display_name}'   # the catalog
```

`GET /health` is a liveness probe (passes through to the primary upstream). `GET /status` is the richest
view: which named upstreams exist and whether each is reachable, every feed with its resolved envelope
(timeouts / concurrency / body cap), the extra routes, and a live per-feed traffic pulse (requests,
errors, in-flight, streaming vs buffered, last status) that works **even when the log database is down**.

## Endpoints

| Endpoint | Purpose |
|----------|---------|
| `GET /`, `GET /help` | This guide (Markdown). |
| `GET /health`, `/v1/health` | Readiness + primary-upstream liveness. |
| `GET /status`, `/v1/status` | Live upstream + downstream state + traffic. |
| `GET /config`, `/v1/config` | Effective config, secrets redacted. |
| `GET /v1/models`, `/models` | Model catalog with per-feed metadata. |
| `POST /v1/chat/completions` | Chat (buffered or SSE streaming). |
| `POST /v1/embeddings` | Embeddings. |
| `POST /v1/tokenize`, `/v1/detokenize` | Token helpers. |
| `GET /metrics`, `GET /slots` | Upstream metrics / slot state passthrough. |
| *(config)* `routes_extra` | Any extra passthrough paths (rerank, audio, images, video jobs, …), including path parameters like `/v1/video/generations/{id}`. |

## Calling each model type

Route is chosen by the `"model"` field in the request body; an unknown model falls open to `primary`.
Discover valid model ids at `GET /v1/models`.

**Chat (buffered):**
```bash
curl -s $BASE/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"<id>","messages":[{"role":"user","content":"hi"}]}'
```

**Chat (streaming SSE)** — add `"stream": true` and read incrementally (`curl -N`). Streaming responses
carry `x-accel-buffering: no`; the final event is `data: [DONE]`.

**Embeddings:**
```bash
curl -s $BASE/v1/embeddings -H 'content-type: application/json' \
  -d '{"model":"<embedding-id>","input":"some text"}'
```

**Vision / OCR** — these are chat models that accept images in the message content array (per the feed's
`input_modalities`); call `/v1/chat/completions` as usual.

**Rerank / audio / images** — served via `routes_extra` (see `GET /status` → `routes_extra`); call the
declared path with the upstream's expected body.

**Async generation (video / some image models):** the sidecar is a **stateless passthrough** of the
upstream's own job API. Typical flow (paths come from `routes_extra`):
```bash
JOB=$(curl -s $BASE/v1/video/generations -H 'content-type: application/json' \
  -d '{"model":"<video-id>","prompt":"..."}' | jq -r .id)   # submit -> job id
curl -s $BASE/v1/video/generations/$JOB | jq .               # poll -> status/url
curl -s $BASE/v1/video/generations/$JOB/content -o out.mp4   # download (binary, streamed)
```
Binary/media responses (non-JSON/text content types) are streamed straight through, not buffered.

## Per-feed limits (the envelope)

Each feed may declare a `limits` block (visible at `GET /status` and `GET /config`):

- `read_timeout_secs` — **inactivity** timeout: a response that keeps producing bytes never trips; a
  stalled upstream trips after this many seconds of silence. This is what lets long generations stream.
- `total_timeout_secs` — optional hard total deadline for buffered responses.
- `max_concurrent` — max in-flight requests for this feed (e.g. `1` for a single-GPU video model);
  gates buffered submits and streams alike.
- `max_body_bytes` — per-feed request-body cap (large for image/video uploads).

## Authentication

If `GET /config` shows `auth.require_inbound_auth: true`, send a key on every inference/mutation call:
`Authorization: Bearer <key>` (or `api-key: <key>`). Read-only orientation (`/`, `/help`, `/health`,
`/status`, `/config`, `/v1/models`) is exempt so you can always orient; the upstream passthroughs
`/metrics` and `/slots` are gated. Keys are never emitted anywhere.

## Troubleshooting

| Symptom | Likely cause / fix |
|---------|--------------------|
| `401` | Inbound auth is on; send `Authorization: Bearer <key>`. |
| `413` | Body exceeds the feed's `max_body_bytes` (or the global cap). Check `GET /status` → the feed's `limits`. |
| `502` | Upstream unreachable/error. Check `GET /status` → `upstreams[].healthy` and the `upstream` URL in `GET /config`. |
| `504` | Upstream inactivity timeout tripped (no bytes for `read_timeout_secs`). Raise the feed's `limits.read_timeout_secs` for a genuinely slow model. |
| Stream ends early without `[DONE]` | The upstream stalled and the inactivity timeout cut it; see `/status` last_status and the request log. |
| A model lands on the wrong backend | Its id isn't in `feeds` (typo) → it fell open to `primary`. Check `GET /config` → `feeds`. |
| No rows in the log DB | Logging is best-effort; the proxy is unaffected. Check `GET /config` → `logging`, and `GET /status` → `log_dropped`. |

## Updating this guide and the configuration

- **Feeds / upstreams / routes / limits** live in the JSON config file (`SIDECAR_CONFIG`), validated by
  `config/config.schema.json`. Invalid config **fails closed at boot** with a legible message. Apply
  changes by editing that file and restarting the service; verify with `GET /config`.
- **This guide** is either the copy compiled into the binary or an external file at `SIDECAR_AGENT_GUIDE`.
  When an external file is configured, edit it and send the process **`SIGHUP`** to hot-reload it without
  a restart (a broken file keeps the last-good copy). Confirm the change by re-fetching `GET /help`.
- Persist any change through the repo's review flow — see **AGENTS.md** in the source repository for the
  development contract (research → questions → plan → approval → build) and for how to add a feed, add an
  upstream, or tune throughput. This guide covers *operating* a live instance; AGENTS.md covers *changing*
  the code.