Operating and recovering services for agents

This reference extends https://app.tokay.io/llms.txt with Service operation and recovery actions for AI agents. Use these actions only when the user explicitly asks or a named readiness action requires them.

Last updated October 14, 2026

AI agent? Start with llms.txt.

The operating model

• A Service reports runtime health through actualStatus and deployment assessment through deployReadiness. Inside readiness, status is deploy progression, reason preserves the cause, action names the exact next step, and attentionStatus classifies operational triage as ALARM, TASK, or NONE. Never infer one from another. A failed Build stays an alarm while Tokay prepares a fix, while an up-to-date Private service with no audience members is a task.
• Failure evidence is layered. rootFailureDiagnosis is the single entry point and points at the failing stage with a safe message and a next action. What the app printed is saved when it fails: serviceIncident.runtimeLogs when an incident opens, deployment.failureLogs when a deploy fails to start. Both outlive the crashed container, hold only the app's own output, and need PROJECT_READ_LOGS (null without it, or when nothing was captured). logSnapshot is a live read of the current container, which after a failed deploy is the previous version.
• A crash followed by an immediate restart may not open an incident. If the user reports a blip and serviceIncidents is empty, read logSnapshot. The crash output is usually still in the tail.
• Nothing is rewritten. A rollback is a new deployment that points at an old release, and a delete moves the entity to Recently Deleted for 30 days. Purge is the irreversible entity-lifecycle verb, and it takes a confirmation token.
• Pause is a change in deploy state, not a runtime toggle. A stopped service shows desiredStatus: STOPPED with actualStatus: EXITED, its public URL returns 404, and the resume verb is deployService, which redeploys the build it ran.

What to call

Usage and capacity

get_usage reports estimated service/database consumption and the workspace VM's measured usage and headroom. Images and saved data share one disk allowance. Runtime layers shared by several consumers are allocated among them, so a service's displayed usage is not a prediction of bytes reclaimed by deletion. Tokay includes platform images, retained image cache, and system overhead.

Use the returned capacity assessments rather than recomputing thresholds. Warning starts at 75% and critical at 90%; missing or stale measurements are UNKNOWN. Recommend a size increase only when the returned recommendation is UPGRADE_WORKSPACE_SIZE.

Read logs

query Logs($id: ID!, $tail: Int = 200) {
  service(id: $id) { logSnapshot(tail: $tail) { text lineCount isTruncated } }
}

tail can be any number of lines up to 1000. Start at 200. logSnapshot performs a live container read for one service, so never put it in a list query. isTruncated: true means the output hit the safety cap. A service running several copies returns up to tail lines from each, grouped by copy and prefixed [copy N]; the stream prefixes each line the same way. A live stream is available at GET {API_ORIGIN}/sse/logs/:serviceId with your API token.

Read the root failure

query Failure($id: ID!) {
  service(id: $id) {
    actualStatus
    deployReadiness {
      status reason action actionModality attentionStatus message supportRef
      sourceChangeProposal { id state hasChanges explanation }
    }
    rootFailureDiagnosis {
      state
      action
      message
      supportRef
      serviceIncident {
        type
        startedAt
        runtimeLogs { text lineCount isTruncated capturedAt ranOutOfMemory }
      }
      deployment {
        failureLogs { text lineCount isTruncated capturedAt ranOutOfMemory }
      }
    }
  }
}

Prefer runtimeLogs and failureLogs over logSnapshot, since they were captured at the moment of the failure. rootFailureDiagnosis.message and deployReadiness.message are always safe to show the user. When rootFailureDiagnosis is null and the service looks healthy, there is no active failure to explain.

When deployReadiness.action is REVIEW_SOURCE_CHANGE_PROPOSAL, the related sourceChangeProposal is guaranteed to be READY and contain changes. The readiness reason still names the original failure. Use the action to choose the operation and the reason to explain why it is needed.

Scheduled runs

mutation Run($service: ID!) {
  runServiceNow(input: { service: $service }) { runServiceNowResult { serviceExecutionId } }
}

Poll the returned serviceExecutionId directly. The row can be null until the worker starts, so treat null as pending.

query Result($id: ID!) {
  serviceExecution(id: $id) {
    triggerType startedAt endedAt durationMs exitCode stdoutPreview logContent
  }
}

The run is finished when endedAt is set. logContent, the run's last 1 MiB, is read from the workspace disk. It is empty while the run is in progress and null when the workspace is not running or the log was pruned (each service keeps its latest 500 run logs), so fall back to stdoutPreview. triggerType distinguishes MANUAL runs from cron runs. Live stdout streams at GET {API_ORIGIN}/sse/run/:serviceId/:serviceExecutionId and ends with done, even when the stream opens after the run finished. Past runs are in service { executions(first: 20, orderBy: STARTED_AT_DESC) { nodes { ... } } }.

Test a FUNCTION

Write the handler as a named export taking one Fetch Request, with the body read via await request.json(). Declare it as exports.handler = async (request) => .... Return a Response, a { statusCode, headers, body } object, or a plain JSON value.

sendServiceTestEvent(input: { service: $service }) posts a synthetic event at the live handler. It requires actualStatus: RUNNING and fails with "Service must be running to send a test event" otherwise. Read the outcome in the request log. Test events Tokay sends have source: "tokay". There is no separate test flag.

query Requests($id: ID!) {
  service(id: $id) {
    requestLogs(first: 5, orderBy: REQUESTED_AT_DESC) {
      nodes { requestedAt method path statusCode durationMs source bodyPreview }
    }
  }
}

requestBody and responseBody are base64 fields on the same type when you need full payloads. They are read from the workspace, where captures are kept for 30 days, and are null while the workspace is stopped. A read that fails on a running workspace returns an UNAVAILABLE error instead of null, so retry it rather than treating the body as missing.

Copies

setServiceReplicaCount(input: { service: $service, replicaCount: 3 }) runs that many copies, at least 1 with no upper limit, of a WEB, FUNCTION, or BACKGROUND app side by side. WEB and FUNCTION copies share requests and BACKGROUND copies each run the whole worker. Copies share the workspace's memory, so more copies than it holds get killed for running out of memory. The change applies within seconds without a deploy, and copies share the service's persistent directories, databases, and environment. Read replicaSafety first: SINGLETON means the code keeps state in one process, such as in-memory sessions or an in-app scheduler, so relay replicaSafetyReason and raise the count only if the user still wants it. Copies need an app process: canRunReplicas is false for a static site, and setServiceReplicaCount rejects it there. replicaSafety may be null when Tokay made no judgment.

Pause and resume

stopService(input: { service: $service }) stops the container. Poll until actualStatus is EXITED. The public URL returns 404 while stopped, and deployReadiness moves to READY_TO_DEPLOY with action: DEPLOY and reason: SERVICE_STOPPED. Auto-deploy never resumes a paused service, so deploy only when the user asks and never poll for it.

Resume is deployService(input: { service: $service }), then poll until the service is RUNNING and UP_TO_DATE again. It redeploys the build the service ran before the pause, so a rollback stays in place. When the project auto-deploys and newer code arrived during the pause, or a maintenance hold stands, it deploys the newest code instead. Readiness afterwards says whether newer code is waiting. deployProject leaves paused services alone unless the whole project is paused, as after restoreProject. Then it starts them on their newest code, except one paused on a rollback. Nothing durable is lost across a pause.

Rollback

mutation Rollback($deployment: ID!) {
  rollbackService(input: { deployment: $deployment }) { rollout { id } }
}

Pick the target from the service's deployments:

query Deployments($id: ID!) {
  service(id: $id) {
    currentDeployment { id commitHash }
    deployments(first: 20, orderBy: CREATED_AT_DESC) {
      nodes { id state commitHash isCurrent createdAt }
    }
  }
}

Every deployment that went live reads DEPLOYED, and isCurrent marks the one the service runs now. Choose the newest DEPLOYED deployment with isCurrent: false whose commitHash differs from currentDeployment.commitHash, unless the user named a commit. A settings change redeploys the same commit, so several deployments can share one. Rollback creates a new deployment for the old release, running with current Project Config, Secrets, and managed resource values. It runs none of that release's migrations, which already ran when it was live, and it keeps the current data.

The old code runs against the database as it is now. Changes that only add something (a table, a nullable column, an index) cannot break it, so those rollbacks start at once. If a release since that version made any other change, the first call fails with CONFIRMATION_REQUIRED, and extensions.confirmation.impact.schemaChanges lists each such release with its commitHash, appliedAt, and changes. The user chooses one of two ways forward:

• Keep the current data: repeat the call with confirmationToken. The old code may fail on the listed changes.
• Restore the data, when impact.canRestoreData is true: call again with restoreData: true. Each changed database goes back to the restore point taken just before the first of those changes, and every other service using it goes back to the version it ran then. This call answers with its own CONFIRMATION_REQUIRED, whose impact gives restoresTo and alsoRollsBack; anything written since is lost, and it needs PROJECT_DESTROY. While the data is restored, those services show their maintenance page. If the restore point is gone, the call fails with ROLLBACK_RESTORE_POINT_UNAVAILABLE.

A restore point is kept 7 days after a release that dropped or rewrote data (its expiresAt), otherwise only while it is among the newest three for that database.

Once the rollback is live, readiness reads status: UP_TO_DATE with reason: ROLLED_BACK. Auto-deploy, Config changes and workspace restarts leave the service on the version you chose until a newer push arrives. deployService on that service rolls forward on purpose while it runs. On a paused service it resumes the rollback instead. A project-wide deploy leaves it alone.

Recently Deleted and restore

deleteService(input: { service: $service }) moves the service to Recently Deleted for 30 days and releases its runtime immediately. Its data keeps its disk. List them with deletedServices(first: 20) { nodes { id name deletedAt } }, and deletedProjects, deletedRepos, and deletedProjectResources for the rest. Deleting a project moves its services and databases there with it, and deleting a repo moves its services.

A caller who can see a deleted item but lacks that action gets FORBIDDEN naming it, such as reason: PROJECT_WRITE_REQUIRED, plus requiredAction when the credential lacks the grant. A service or database whose project is still deleted fails with RESTORE_PARENT_PROJECT_FIRST, and a service whose repo is still deleted with RESTORE_PARENT_REPO_FIRST. Items deleted on their own before their parent stay in Recently Deleted when the parent comes back.

Restoring never redeploys. Every restored service is paused, with readiness action: DEPLOY and reason: SERVICE_STOPPED. Call deployService only when the user wants it running again. A restored database runs again at once.

Purge is permanent and runs only when the user asks. purgeService(input: { service: ... }) first fails with CONFIRMATION_REQUIRED and returns the impact and a short-lived token in extensions.confirmation; show the impact to the user, then repeat the call with confirmationToken added to the input.

Error shapes