# outpost API

> Run HTTP and TCP probes from a fleet of workers with fixed egress IPs, selected by labels, with separate DNS/connect/TLS/TTFB timings per worker. Base URL: https://outpost.neubauer-digital.at. Machine-readable spec: /openapi.json (OpenAPI 3.1).

## Authentication

Send `Authorization: Bearer <token>` on every /v1 request. Tokens look like `opk_…` and are created by an admin at /admin/.
Roles, each including the ones before it:

- `readonly`: GET endpoints only
- `operator`: also create checks and schedules
- `admin`: also agent registration and token management

`GET /v1/whoami` tells you your role. 401 = token missing, invalid, expired or revoked. 403 = role too low. Never put tokens in URLs or logs.

## Core workflow: measure a URL

1. `POST /v1/checks?wait=10s` with a CheckRequest.
2. 200 means finished (`status` is done, partial or failed). 202 means still running: poll `GET /v1/checks/{id}` every 1–2 s until `status` is terminal.
3. Read `results[]`: one entry per worker, each with `status` (job) and `result` (measurement).

```http
POST /v1/checks?wait=10s
Authorization: Bearer opk_...
Content-Type: application/json

{
  "type": "http_get",
  "http": {"url": "https://example.com/"},
  "agents": {"workload": "monitoring", "count": 2},
  "ip_version": "auto",
  "timeout_ms": 10000
}
```

TCP connect with TLS handshake, IPv4 only:

```json
{"type": "tcp_connect", "tcp": {"host": "mail.example.com", "port": 443, "tls": true},
 "agents": {"workload": "monitoring", "count": 2}, "ip_version": "ipv4"}
```

### If your proxy blocks POST

`GET /v1/run` does the same thing with query parameters and returns the same response:

```http
GET /v1/run?type=http_get&url=https%3A%2F%2Fexample.com%2F&workload=scraping&return_body=true&wait=20s&include_body=true
Authorization: Bearer opk_...
```

Parameters: `type`, `workload`, `url` (URL-encoded), `host`, `port`, `tls`, `server_name`, `follow_redirects`, `max_redirects`, `insecure_tls`, `return_body`, `count`, `ip_version`, `timeout_ms`, `max_bytes`, `label.<key>=`, `header.<Name>=` (e.g. `header.Accept-Language=de-AT`), `wait`, `include_body`. Unknown parameters get 400. Keep the token in the Authorization header, never in the URL.

## Fetching page content (HTML)

Set `"return_body": true` in `http` and use a `scraping` agent:

```http
POST /v1/checks?wait=20s&include_body=true
{"type": "http_get",
 "http": {"url": "https://example.com/page", "return_body": true, "follow_redirects": true},
 "agents": {"workload": "scraping", "count": 1},
 "max_bytes": 4194304, "timeout_ms": 15000}
```

- With `include_body=true`, each result has `body_text` (UTF-8 bodies only, such as HTML, JSON or text) plus `body: {status, sha256, bytes, content_type, url}`.
- Raw bytes (also binary): `GET /v1/checks/{id}/body` (add `?agent_id=` if several agents delivered). The response carries the original Content-Type and `X-Outpost-Sha256`. A 202 with `Retry-After` means the transfer is still being assembled.
- `?wait=` also waits for the body. If `body.status` is `pending`, poll `GET /v1/checks/{id}?include_body=true` again after about 1 s.
- Bodies are cut at `max_bytes` (max 4 MiB); check `result.http.truncated`. They are kept 7 days, then 404/410. Identical content is stored once, keyed by sha256.
- Use `monitoring` only for availability and latency. Fetch content through `scraping` agents; they reuse connections and have their own egress IPs.

## Request headers and User-Agent

`http.headers` is sent as given, including `User-Agent`, `Accept` and `Accept-Language`. Without one, the User-Agent is `outpost/1 (+https://outpost.neubauer-digital.at/docs)`, which identifies the service to site operators. Some shops block recognisable bots; outpost does not rotate agents, IPs or fingerprints and does not solve challenges. If a site blocks you (403/429), respect it rather than working around it.

## CheckRequest fields

- `type`: `http_get` or `tcp_connect`. whois, smtp, imap, pop3 and dns are planned and rejected today (400).
- `http`: `{url, headers?, follow_redirects? (false), max_redirects? (5), insecure_tls? (false), return_body? (false)}`. Absolute http(s) URL, no credentials in the URL.
- `tcp`: `{host, port, tls? (false), server_name?}`
- `agents.workload` (required): `monitoring` or `scraping`, strictly enforced (see below).
- `agents.count`: 1..50, default 1.
- `agents.labels`: optional exact-match filter. Keys: `ring`, `egress_ip`, `port25`, `browser`, `created`. Hostnames and agent ids are NOT selectors.
- `ip_version`: `auto` (default; IPv6 first when available), `ipv4`, `ipv6`. With ipv4/ipv6 there is no fallback: if the target has no address of that family, the result is `status=error`.
- `timeout_ms`: 100..60000, default 10000. `max_bytes`: body limit, default 1 MiB, max 4 MiB (bigger bodies: `http.truncated=true`).

Unknown fields are rejected with 400.

## Reading results

Check `status`:

- `queued`: waiting for a rate-limit token
- `running`: dispatched
- `done`: every selected worker delivered
- `partial`: some delivered, or a worker failed or was unreachable. A worker outage never produces `failed`.
- `failed`: every worker rejected the job

Per-worker `results[].status`: `done`, `rejected` (see `error`), `lost` (no answer within timeout + 15 s), `queued`, `dispatched` or `running`.

Measurement `results[].result`:

- `status`:
  - `ok`: got a response or connection. HTTP 4xx/5xx is still `ok`; look at `http.status_code`.
  - `error` or `timeout`: a valid measurement of a failing target
  - `blocked`: target resolved to a private, loopback, link-local, cloud-metadata or infrastructure address. Not retryable.
- `timings` (ms, separate phases, not cumulative): `dns_ms`, `connect_ms`, `tls_ms`, `ttfb_ms` (request written → first byte), `total_ms`.
- `resolved_ip`, `ip_version`, `http.{status_code, bytes, truncated, sha256, final_url}`, `tls.{version, cert_expiry, cert_subject}`
- `reused=true` (scraping only) means a keep-alive connection was reused, so DNS, connect and TLS are 0.

Compare workers by `results[].hostname` (a label) or `agent_id`. Egress IPs are in the `egress_ip` label from `GET /v1/agents`.

## Workloads (important)

- `monitoring` workers open a fresh connection per measurement. Use them for availability and latency.
- `scraping` workers keep connections alive. Use them for fetching content.

Each worker has exactly one workload. If fewer workers match than `count`, the API returns 422 with `{error, workload, eligible, requested}` and creates nothing. Jobs are never rerouted to the other workload. Lower `count` or relax `labels`; do not switch workload just to make a request succeed. Check `GET /v1/agents` (`online`, `labels`) to see what is available.

## Rate limits

Dispatch is rate limited centrally per registrable domain (for example `a.example.com` and `b.example.com` share one bucket) across all workers: default 1 request/s, burst 5. Excess jobs wait in `queued` for up to 10 min. HTTP 429/503 from a target backs off the whole domain exponentially. Checks are accepted anyway, only dispatch waits. So batch sensibly and expect `queued` under load.

## Other endpoints

- `GET /v1/checks/{id}[?include_body=true]`: check plus per-worker results
- `GET /v1/checks/{id}/body[?agent_id=]`: raw stored body
- `GET /v1/agents`: inventory: id, hostname (label), labels, online (heartbeat < 30 s), capabilities, busy
- `GET /v1/results?check_id=&agent_id=&target=&type=&since=1h&until=&limit=100`: history, newest first. `since`/`until` take RFC 3339 or a relative duration. Default window 7 days, limit max 1000. Rows are flat, so names differ from check results: `http_status` = `result.http.status_code`. Bodies are not included; use `GET /v1/checks/{id}/body`.
- `GET /v1/stats/dashboard?range=1h|24h|7d`, `GET /v1/stats/agents?range=`: aggregated statistics (readonly)
- `GET /v1/access`, `GET /v1/access/stream`: access log, history and live (admin)
- `POST /v1/schedules` `{"interval_s": 60, "check": <CheckRequest>}` (operator); `GET /v1/schedules`; `DELETE /v1/schedules/{id}`. Missed runs are skipped. A run without enough agents is recorded as a failed check.
- `GET /v1/tokens`, `POST /v1/tokens` `{name, role, expires_in_days}`, `DELETE /v1/tokens/{id}` (admin). The plaintext token appears only in the POST response.
- `GET /healthz`: no auth

## Errors

JSON `{"error": "..."}` with 400 (validation; fix the request, do not retry unchanged), 401, 403, 404, 409, 422 (not enough agents), 500 (retry with backoff).

## Links

- [OpenAPI spec](/openapi.json): complete schemas
- [Human-readable docs](/docs)
- [Admin UI](/admin/): token management (admin role)