Skip to main content

Troubleshoot the Hosted Agent

This runbook is organized by symptom. Find the behavior you're seeing, then work through its checks in order.

Agent never reaches ACTIVE after install​

The agent reports ACTIVE to the platform as soon as the container starts and registers. If it stays PROVISIONED in the console (the Manage Hosted Agents page) after you run the install command, check in order:

  1. Outbound HTTPS to the platform. The agent needs outbound TCP 443 to the platform host embedded in your install command (api.apisecapps.com, or api.<stage>.apisecapps.com for staged environments). Test from the Docker host:

    nc -z -w30 api.apisecapps.com 443

    The install script runs this same check before starting the container, so a script that completed cleanly usually means the host could reach the platform at install time — but confirm nothing in the path (firewall, proxy policy) has changed since.

  2. Docker is running and the container started.

    docker ps --filter "name=apisechostedagent"
    docker logs <container-name>

  3. Token problems in the logs. Authentication failures against the platform appear in the container logs. See Agent requests return 401.

  4. Certificates enabled? If you set ENABLE_CERTIFICATES=true, a certificate configuration problem aborts startup and the container restart-loops. See Container restart-loops at startup.

See Networking and egress for the full connection model.

Agent shows Unresponsive​

How the status works:

  • A running agent sends a health heartbeat every 5 minutes (HEARTBEAT_REPORT_HEALTH_FIXED, default 5m).
  • The platform marks an ACTIVE agent UNRESPONSIVE once its last heartbeat is at least 1 hour old. The check runs when the agent list is evaluated, not continuously.
note

Because the staleness threshold is 1 hour, an agent that has died can continue to display ACTIVE for up to an hour. Treat a recent ACTIVE status as "heartbeated within the last hour," not "alive right now."

To recover, check in order:

  1. Is the container running? docker ps on the host. If it exited, docker logs <container-name> shows why. The install command's --restart=unless-stopped policy restarts crashed containers automatically, so a container that is gone entirely was stopped deliberately or Docker itself is down.
  2. Can the host still reach the platform? Re-run the nc -z -w30 <platform-host> 443 check. Network or proxy changes are a common cause of an agent going quiet.
  3. Is the token still valid? A 401 in the logs means the agent is running but the platform rejects it — see the next section.

Once a healthy agent heartbeats again, the platform sets it back to ACTIVE.

Agent requests return 401​

The platform rejects the agent's token in three cases:

CauseHow to recognize itFix
Token expiredTokens are valid for 180 days. Diagnostics warn when expiry is 7 days or less away.Re-download the install command from Manage Hosted Agents. When the old token is expired, the download mints a fresh one. Redeploy with the new command.
Token rotatedSomeone rotated the agent's token. Rotation revokes all prior tokens immediately and sets the agent back to PROVISIONED.Update the running container to the new token: redeploy with the newly downloaded install command (or recreate the container with the new BROKER_TOKEN).
Agent deletedThe agent record was deleted in the console.A deleted agent cannot be revived. Create a new agent and deploy with its install command.
tip

Re-downloading the install command while the token is still valid reuses the same token identity and carries a fresh 180-day expiry, so it does not invalidate a running agent. The running container keeps the expiry the token it started with was issued with, so a redeploy is still what moves the agent onto the renewed one — and because the old token stays valid meanwhile, replicas can be rolled one at a time. See Upgrading and versions for the full token lifecycle.

Container restart-loops at startup with certificates enabled​

With ENABLE_CERTIFICATES=true, the agent validates its entire certificate configuration at startup. Any problem aborts startup, and because the container runs with --restart=unless-stopped, Docker restarts it immediately — producing a restart loop. Read the exact error from the logs:

docker logs <container-name>
Startup errorFix
Certificates directory not found at '<path>'Mount your certificate directory at the container path set by CERTIFICATES_BASE_PATH (default /app/certificates/).
Certificate configuration file not found at '<path>config.json'Add a config.json to the certificate directory.
Failed to parse certificate configuration file '<path>'Fix the JSON syntax in config.json.
Could not load certificate [<filename>]. Error: <message>The keystore is unreadable — most often a wrong passphrase. Verify the passphrase and that the file is a valid PKCS12 keystore.
PEM is not supported in the client-certificate slot [<filename>]. Use .pfx or .p12 for client certificates; .pem is only supported for CA bundles (caFilename).Convert the client certificate to PKCS12 (.pfx/.p12). Keep .pem files for the caFilename slot only.
Only .pfx, .p12 and .pem certificates are supportedRemove or convert files with any other extension.
Could not load PEM certificate [<name>]. Error: <message>The CA .pem file is unreadable — verify its contents and encoding.
No certificate specified for host: <host>Each config.json entry needs a filename, a caFilename, or both.
No host specified for certificateAdd a host value to the entry.
Hostname should not start with 'http://' or 'https://'Use a bare hostname in host — no scheme.
Certificate [<filename>] not foundPlace the referenced file in the certificate directory, or correct the filename in config.json.

See Certificates and mTLS for full setup instructions.

Scans run without mTLS after a certificate failure​

Startup validation is strict, but a certificate that fails to load at request time — for example, a file deleted or modified after the container started — does not fail the scan call. Instead the agent:

  • Logs the error with code CERT_LOAD_001, including a root cause and resolution hint.
  • Continues the request without mTLS and without the custom CA (default TLS trust).
  • Reports the load failure in its certificate status, which it sends to the platform at startup and hourly.

No separate error report is sent for this case, so watch the certificate status and the container logs rather than expecting a scan failure. To fix it, restore the certificate files and passphrases at CERTIFICATES_BASE_PATH.

Instruction failures (BROKER_INSTR_001)​

Scan work reaches the agent as instructions held in a platform-side queue, one queue per agent. When the agent fails to fetch or execute an instruction, it logs the failure with code BROKER_INSTR_001 and sends an error report to the platform. The report includes a stack trace only when SEND_BACK_ERROR_REPORT=true (the install command sets it to true).

Two behaviors worth knowing:

  • Malformed instructions are redelivered. When the agent polls, the platform reads the next message from the agent's queue and parses it before handing it over. If parsing fails, the instruction is not removed and is delivered again after a delay. A permanently malformed instruction therefore fails on every redelivery, and the agent logs BROKER_INSTR_001 on each attempt.
  • Instruction size limits are platform-side. The size limits of the instruction queue are a property of the platform, not of the agent — there is no agent-side setting that raises or changes them.

Run agent diagnostics​

Health checks are initiated from the platform: when the agent receives a health-check action, it runs its diagnostics suite and returns the results. See Manage Hosted Agents for console operations. The checks are:

CheckWhat it verifiesThresholds
Manager reachabilityThe platform health endpoint responds (3-second timeout)Fail: no response
Auth token validityThe agent's broker tokenWarn: expires in 7 days or less. Fail: expired
Certificate expiryEach configured certificateWarn: expires in 30 days or less. Fail: expired or cannot be loaded
JVM healthThe agent's Java runtime—
Disk spaceFree space on the certificates directoryWarn: less than 50 MB free. Fail: the path cannot be read

Overall status is HEALTHY only when every check passes; any warning or failure makes the overall result DEGRADED.

note

The disk-space check measures the certificates directory, which only exists when certificates are configured. On an agent left at the default (ENABLE_CERTIFICATES=false, no certificates volume mounted) the path is absent, so the check reports a failure and the overall result is DEGRADED even though nothing is wrong with the agent. Read the individual check rows rather than the overall status on an agent that does not use certificates.