Operating and recovering services for agents

Observe a deployed Service, read failure evidence, and recover it when something breaks. This page is written for an AI agent. It extends the deployment protocol at https://app.tokay.io/llms.txt. Everything here is an escape hatch. The happy path needs none of it, so act on an explicit user request or a named readiness action, not on your own initiative.

Last updated July 14, 2026

The operating model

• A Service reports two independent signals. deployReadiness is the deploy state machine and names your next action. actualStatus is observed runtime health (RUNNING, EXITED, DEGRADED, STARTING). Never infer one from the other.
• Failure evidence is layered. rootFailureDiagnosis is the single entry point and points at the failing stage with a safe message and a next action. Runtime incidents carry runtimeLogs captured when the incident opened, which outlive the crashed container. logSnapshot is a live read of the current container.
• A crash followed by an immediate restart may not open an incident. If the user reports a blip and serviceIncidents is empty, read logSnapshot. The crash output is usually still in the tail.
• Nothing is rewritten. A rollback is a new deployment that points at an old release, and a delete moves the entity into a 30-day trash. The only destructive verb is purge, and it takes a confirmation token.
• Pause is a change in deploy state, not a runtime toggle. A stopped service shows desiredStatus: STOPPED with actualStatus: EXITED, its public URL returns 404, and the resume verb is a fresh deploy.

What to call

Read logs

query Logs($id: ID!, $tail: Int = 200) {
  service(id: $id) { logSnapshot(tail: $tail) { text lineCount isTruncated } }
}

tail can be any number of lines up to 1000. Start at 200. logSnapshot performs a live container read for one service, so never put it in a list query. isTruncated: true means the output hit the safety cap. A live stream is available at GET {API_ORIGIN}/sse/logs/:serviceId with your JWT.

Read the root failure

query Failure($id: ID!) {
  service(id: $id) {
    actualStatus
    deployReadiness { state reason action actionModality message supportRef }
    rootFailureDiagnosis {
      failureDiagnosisStatus
      failureNextAction
      message
      supportRef
      serviceIncident {
        incidentType
        startedAt
        runtimeLogs { text lineCount isTruncated capturedAt }
      }
    }
  }
}

Prefer runtimeLogs for runtime incidents, since they were captured at the moment the incident opened. rootFailureDiagnosis.message and deployReadiness.message are always safe to show the user. When rootFailureDiagnosis is null and the service looks healthy, there is no active failure to explain.

Scheduled runs

mutation Run($service: ID!) {
  runNow(input: { service: $service }) { runNowResult { channelId executionId } }
}

Poll the returned executionId directly. The row can be null until the worker starts, so treat null as pending.

query Result($id: ID!) {
  execution(id: $id) {
    executionTriggerType startTime endTime durationMs exitCode stdoutPreview logContent
  }
}

The run is finished when endTime is set. executionTriggerType distinguishes MANUAL runs from cron runs. Live stdout streams at GET {API_ORIGIN}/sse/run/:serviceId/:channelId. Past runs are in service { executions(first: 20, orderBy: START_TIME_DESC) { nodes { ... } } }.

Test a FUNCTION

sendTestEvent(input: { service: $service }) posts a synthetic event at the live handler. It requires actualStatus: RUNNING and fails with "Service must be running to send a test event" otherwise. Read the outcome in the request log. Test events Tokay sends have source: "tokay"; there is no separate test flag.

query Requests($id: ID!) {
  service(id: $id) {
    requestLogs(first: 5, orderBy: REQUEST_TIME_DESC) {
      nodes { requestTime method path statusCode durationMs source bodyPreview }
    }
  }
}

requestBody and responseBody are base64 fields on the same type when you need full payloads.

Pause and resume

stopService(input: { service: $service }) stops the container. Poll until actualStatus is EXITED. The public URL returns 404 while stopped, and deployReadiness moves to READY_TO_DEPLOY with action: DEPLOY. Resume is deployService(input: { service: $service }), then poll until the service is RUNNING and UP_TO_DATE again. Nothing durable is lost across a pause.

Rollback

mutation Rollback($deployment: ID!) {
  rollbackService(input: { deployment: $deployment }) { rollout { id } }
}

Pass an older deployment id from service.deployments(orderBy: CREATED_AT_DESC) with deployState: DEPLOYED. Rollback creates a new deployment for the old release, running with current secrets. Do not wait for deployReadiness.state: UP_TO_DATE afterward. It will not arrive, because the service is intentionally on an older target and readiness reports READY_TO_DEPLOY so a roll forward stays available. Success is the new deployment reaching deployState: DEPLOYED with actualStatus: RUNNING. Under releaseSafetyMode: MANUAL the rollback parks at CONFIRM_RELEASE first, even though it runs no migrations. A code rollback does not rewind database state. See the releases reference for snapshot restore.

Trash and restore

removeService(input: { service: $service }) moves the service to a 30-day trash and releases its runtime immediately. List the trash with deletedServicesForCurrentPerson(first: 20) { nodes { id name deletedAt } } (sibling queries exist for projects, repos, and resources). restoreService(input: { service: $service }) brings it back, but does not redeploy. Readiness returns action: DEPLOY and you must call deployService to bring it live again.

Purge is permanent and runs only when the user asks. It takes two steps. requestPurgeConfirmation(input: { entityId: ..., operation: ... }) mints a token that expires quickly, and purgeService(input: { service: ..., purgeToken: ... }) consumes it. Confirm with the user before calling either.

Error shapes