# Operating and recovering services for agents

Observe a deployed Service, read failure evidence, and recover it when something breaks. This page is written for an AI agent. It extends the deployment protocol at `https://app.tokay.io/llms.txt`. Everything here is an escape hatch. The happy path needs none of it, so act on an explicit user request or a named readiness action, not on your own initiative.

## The operating model

- A Service reports two independent signals. `deployReadiness` is the deploy state machine and names your next action. `actualStatus` is observed runtime health (`RUNNING`, `EXITED`, `DEGRADED`, `STARTING`). Never infer one from the other.
- Failure evidence is layered. `rootFailureDiagnosis` is the single entry point and points at the failing stage with a safe message and a next action. Runtime incidents carry `runtimeLogs` captured when the incident opened, which outlive the crashed container. `logSnapshot` is a live read of the current container.
- A crash followed by an immediate restart may not open an incident. If the user reports a blip and `serviceIncidents` is empty, read `logSnapshot`. The crash output is usually still in the tail.
- Nothing is rewritten. A rollback is a new deployment that points at an old release, and a delete moves the entity into a 30-day trash. The only destructive verb is purge, and it takes a confirmation token.
- Pause is a change in deploy state, not a runtime toggle. A stopped service shows `desiredStatus: STOPPED` with `actualStatus: EXITED`, its public URL returns 404, and the resume verb is a fresh deploy.

## What to call

| To | Use |
|---|---|
| Check health and next action | `service { actualStatus deployReadiness { state action message } }` |
| Read live container logs | `service.logSnapshot(tail:)` |
| Read the root failure | `service.rootFailureDiagnosis` |
| Read scheduled run history | `service.executions` |
| Trigger a scheduled run now | `runNow`, then poll `execution` |
| Test a FUNCTION | `sendTestEvent`, then read `service.requestLogs` |
| See what changed and when | `service.serviceEvents` |
| Pause | `stopService` |
| Resume | `deployService` |
| Restore a previous version | `rollbackService` |
| Delete, list trash, restore | `removeService`, `deletedServicesForCurrentPerson`, `restoreService` |

## Read logs

```
query Logs($id: ID!, $tail: Int = 200) {
  service(id: $id) { logSnapshot(tail: $tail) { text lineCount isTruncated } }
}
```

`tail` can be any number of lines up to 1000. Start at 200. `logSnapshot` performs a live container read for one service, so never put it in a list query. `isTruncated: true` means the output hit the safety cap. A live stream is available at `GET {API_ORIGIN}/sse/logs/:serviceId` with your JWT.

## Read the root failure

```
query Failure($id: ID!) {
  service(id: $id) {
    actualStatus
    deployReadiness { state reason action actionModality message supportRef }
    rootFailureDiagnosis {
      failureDiagnosisStatus
      failureNextAction
      message
      supportRef
      serviceIncident {
        incidentType
        startedAt
        runtimeLogs { text lineCount isTruncated capturedAt }
      }
    }
  }
}
```

Prefer `runtimeLogs` for runtime incidents, since they were captured at the moment the incident opened. `rootFailureDiagnosis.message` and `deployReadiness.message` are always safe to show the user. When `rootFailureDiagnosis` is null and the service looks healthy, there is no active failure to explain.

## Scheduled runs

```
mutation Run($service: ID!) {
  runNow(input: { service: $service }) { runNowResult { channelId executionId } }
}
```

Poll the returned `executionId` directly. The row can be null until the worker starts, so treat null as pending.

```
query Result($id: ID!) {
  execution(id: $id) {
    executionTriggerType startTime endTime durationMs exitCode stdoutPreview logContent
  }
}
```

The run is finished when `endTime` is set. `executionTriggerType` distinguishes `MANUAL` runs from cron runs. Live stdout streams at `GET {API_ORIGIN}/sse/run/:serviceId/:channelId`. Past runs are in `service { executions(first: 20, orderBy: START_TIME_DESC) { nodes { ... } } }`.

## Test a FUNCTION

`sendTestEvent(input: { service: $service })` posts a synthetic event at the live handler. It requires `actualStatus: RUNNING` and fails with "Service must be running to send a test event" otherwise. Read the outcome in the request log. Test events Tokay sends have `source: "tokay"`; there is no separate test flag.

```
query Requests($id: ID!) {
  service(id: $id) {
    requestLogs(first: 5, orderBy: REQUEST_TIME_DESC) {
      nodes { requestTime method path statusCode durationMs source bodyPreview }
    }
  }
}
```

`requestBody` and `responseBody` are base64 fields on the same type when you need full payloads.

## Pause and resume

`stopService(input: { service: $service })` stops the container. Poll until `actualStatus` is `EXITED`. The public URL returns 404 while stopped, and `deployReadiness` moves to `READY_TO_DEPLOY` with `action: DEPLOY`. Resume is `deployService(input: { service: $service })`, then poll until the service is `RUNNING` and `UP_TO_DATE` again. Nothing durable is lost across a pause.

## Rollback

```
mutation Rollback($deployment: ID!) {
  rollbackService(input: { deployment: $deployment }) { rollout { id } }
}
```

Pass an older deployment id from `service.deployments(orderBy: CREATED_AT_DESC)` with `deployState: DEPLOYED`. Rollback creates a new deployment for the old release, running with current secrets. Do not wait for `deployReadiness.state: UP_TO_DATE` afterward. It will not arrive, because the service is intentionally on an older target and readiness reports `READY_TO_DEPLOY` so a roll forward stays available. Success is the new deployment reaching `deployState: DEPLOYED` with `actualStatus: RUNNING`. Under `releaseSafetyMode: MANUAL` the rollback parks at `CONFIRM_RELEASE` first, even though it runs no migrations. A code rollback does not rewind database state. See the releases reference for snapshot restore.

## Trash and restore

`removeService(input: { service: $service })` moves the service to a 30-day trash and releases its runtime immediately. List the trash with `deletedServicesForCurrentPerson(first: 20) { nodes { id name deletedAt } }` (sibling queries exist for projects, repos, and resources). `restoreService(input: { service: $service })` brings it back, but does not redeploy. Readiness returns `action: DEPLOY` and you must call `deployService` to bring it live again.

Purge is permanent and runs only when the user asks. It takes two steps. `requestPurgeConfirmation(input: { entityId: ..., operation: ... })` mints a token that expires quickly, and `purgeService(input: { service: ..., purgeToken: ... })` consumes it. Confirm with the user before calling either.

## Error shapes

| You see | It means | Do this |
|---|---|---|
| "Service must be running to send a test event" | The FUNCTION is stopped or crashed | Check `actualStatus`, resume with `deployService` first |
| `logSnapshot` GraphQL error about a missing container | No live container to read | Use `rootFailureDiagnosis.serviceIncident.runtimeLogs` for crash evidence |
| `execution(id:)` returns null after `runNow` | The worker has not started the run yet | Keep polling for a few seconds |
| `deployReadiness.action: CONTACT_SUPPORT` | Tokay cannot recover this state on its own | Show `message` and `supportRef` to the user and stop |
| "Not authorized for ACTION on TYPE id" | Your credential lacks that permission | Report the missing grant to the user instead of retrying |