Skip to main content

Playbooks

Each one starts with the fastest thing that stops the bleeding, then the diagnosis, then the cleanup. If you're reading this during an incident, do step 1 and come back.

The two moves that cover most of it

A connection's kill switch stops one integration everywhere, instantly. Stopping the gateway stops everything. Neither needs you to unpick permissions first, and neither destroys anything you'll want back.

An agent is doing something it shouldn't​

Stop it

Remove the agent — Agents → remove. Its sessions are revoked immediately and any grant that named only that agent goes with it. If you'd rather keep the agent and stop the damage, hit the kill switch on the connection it's abusing.

Work out what happened

Open the audit log and filter to that agent. You're answering one question: was it allowed, or did it find a way?

  • Every call was allowed — then policy did what you configured, and the configuration was wrong. Look at which grant permitted it. This is the common case, and it's a good outcome in the sense that nothing was bypassed.
  • Calls were denied and it kept trying — the wall held. Worth understanding why it's asking, but nothing escaped.
  • Something ran that no grant covers — that's a different class of problem. Capture the audit entry id and get in touch.

Clean up

Narrow the policy that allowed it, rather than deleting the grant, so the same mistake isn't available to the next agent. Then reconnect the agent if you still want it — it'll come back as a fresh identity under whatever you've now allowed.

A credential has leaked​

The one that matters: a leak from Permaura's side is not how this happens. Sealed credentials are write-only — nothing in the console, the CLI or the API returns a value. If a key is out, it came from the vendor's side, someone's clipboard, or a copy that existed before you sealed it.

Stop it

  1. Revoke it at the vendor. This is the only step that actually invalidates the key. Everything else limits what Permaura will do with it.
  2. Hit the connection's kill switch. Every call through it stops instantly, so nothing keeps using the compromised key while you sort out a replacement.

Replace it

Issue a new credential at the vendor, then rotate the connection's credential to the new value. The connection, its capabilities, its policies and its grants all stay as they are — you're replacing the sealed value underneath them, not rebuilding anything.

Re-enable the connection when you're ready.

Then check

Filter the audit log to that connection for the exposure window. Every call made through it is recorded with what it did, which is the basis for whatever you have to tell somebody else.

Spend has spiked​

Stop it

Pause the grant, or kill-switch the connection. Pausing the grant is usually right — it keeps the definition so you can put it back once you understand the cause.

Find the cause

Check the grant's budget consumption, then the audit log for that agent. Almost always one of three things:

  • A retry loop. The same call repeating. Something upstream is failing in a way the agent treats as retryable.
  • A widened grant. Someone allowed a capability that costs money without a budget on it.
  • A capability that arrived enabled. Anything that spends money is meant to arrive switched off, but check what's actually on for that connection.

Prevent the repeat

Put a budget on the grant — a cap on calls per rolling day is enough to turn a runaway into a nuisance. For anything genuinely expensive, move it to the needs approval rung so a human sees each one. See Approvals.

The audit chain reads broken​

What it means

The hash chain no longer verifies: the log has been altered or truncated on disk. This is exactly what the chain exists to detect, and it's the one signal here you should treat as serious by default.

Confirm it directly rather than trusting the pill alone:

curl -s http://127.0.0.1:7376/health | jq .audit_chain_intact

What to do

  1. Don't restart the gateway and don't delete anything. The broken state is the evidence.
  2. Read out what's still there — permaura audit, or GET /v1/audit-events — and keep a copy off that machine.
  3. Consider the machine untrusted until you know why. Something with write access to ~/.permaura/gateway.db did this. The mundane explanations are a disk problem, a backup tool, or a file restored from a snapshot; the alarming one is someone covering a track.
  4. Get in touch with what /health returned and roughly when you last saw it verified.

What it isn't

It isn't a sign that a secret leaked or that an agent escaped policy. The chain covers the record, not the enforcement.

Somebody has to be locked out right now​

Not a resignation — a laptop in a taxi, or worse.

  1. Remove them from the organisation — Settings → Members. Access ends across every workspace on their next request.
  2. Remove the agents they connected. These are separate identities and survive the person. Anything running on their machine keeps working until you remove the agent.
  3. Revoke their approval device — Workspace → Devices — but pair a replacement first if it's the only one. Revoking your last device does not lower the approval bar; it just leaves nobody able to clear the queue. On a hosted gateway that replacement is held for their device to confirm, so revoke first and pair after, and expect to sign in again at each step if theirs is the last usable device; see the last-device playbook.
  4. Rotate credentials they set up. They never saw the sealed values, but they may hold the originals.

If you want everything frozen while you work through it, stop the gateway or pause the broker. Nothing is lost by doing that first.

Full version, for a planned departure: offboarding.

Agents are being refused and you don't know why​

Deny-by-default is doing its job; the question is which rule.

Filter the audit log to denials. The entry names the capability and connection, and that's usually the whole answer. Then check, in this order:

  1. Is the capability enabled on the connection? A disabled capability is refused for everyone, whatever your policies say.
  2. Is the connection's kill switch off?
  3. Is the grant enabled rather than paused, within its budget, and before its expiry?
  4. Does a policy deny it? Deny beats needs-approval beats allow, across every policy that applies — one deny anywhere wins.
  5. Is the broker paused? That refuses everything, and it's easy to forget you did it.

Troubleshooting covers the same ground from the end user's side.

The approval queue is stuck behind a device nobody has​

The sticky-floor case. Once a device was paired, the gateway stopped accepting unsigned approvals, and unpairing didn't lift that.

Pair a replacement device — that's the clean fix, and it restores the queue.

If you can't, the bar can be lowered, and where from depends on where the gateway runs:

  • Local or tunnelled: only from the machine running the gateway, never over the network and never from the console. That restriction is the whole point: it means an attacker holding your account session can't clear it.
  • Hosted: from the console, under Workspace → Devices, after proving it's you again with your password or a two-step code. While you still have one working device, that device also has to sign the change; the sign-in alone is enough only once nothing usable is left, which is the next playbook.

Actions explicitly marked Device-signed still require a signature regardless.

See Approvals for the exact behaviour.

Hosted: the last approval device is gone​

The one shape of this that has no host to walk to. On a hosted gateway there is no machine of yours to run a local command on, so the recovery runs through your account and is deliberately narrow.

If the device is lost but you can still get a new one enrolled, do that first. It is the clean fix, and it costs two sign-ins:

  1. Install the Permaura app on the replacement device.
  2. Revoke the lost device under Workspace → Devices, and prove it's you again when asked, with your password or a two-step code. This matters before, not after: while the old key is still trusted, the gateway will hold the new enrolment waiting for the lost device to confirm it, which nothing can satisfy.
  3. Pair the replacement in the app, and sign in again when it asks. With no usable device left, it enrols on that fresh sign-in alone, exactly like the first one did.

Two re-authentications, in that order. They are separate on purpose: what the first one issues can only revoke, what the second one issues can only enrol, so neither step can be spent as the other. Nothing holding your account session alone can do either.

Step 2 asks for a sign-in only because it is your last usable device. Emptying the keyring returns a hosted worker to trusting the next device on first use, and that is the one revocation that loosens rather than tightens. Revoking a device while another still works asks for nothing, on any gateway.

If you want console approvals back rather than a new device, revoke every device first, then lower the approval floor from the console, under Workspace → Devices. That is the same two sign-ins: one to take out the last device, one to lower the floor. With nothing usable left there is no device to co-sign the lowering, which is the whole reason revoking comes first.

Revoke, then pair, in that order

Pairing a replacement while the lost device is still trusted produces a request that waits for a signature from the phone you no longer have. It will sit there indefinitely, because held requests do not expire. Revoking first turns the same enrolment into an immediate one.

Everything here is written to the audit log under permaura.approver.pair, and the account owner is emailed. If either shows an enrolment you did not perform, treat it as an account compromise: change the password, turn on two-step verification if it is off, and revoke the unknown device.

Everything is offline and you don't know if it's you or us​

Ask the gateway directly:

curl -s http://127.0.0.1:7376/health
  • It answers — your gateway is fine, and the problem is between the console and it. Check the port, the browser URL under Workspace → Gateway → Advanced, and whether "unverified" is just the expected state for a local gateway.
  • It doesn't answer — the gateway isn't running. Start it.
  • It answers but permaura.com is down — your local gateway keeps enforcing policy and serving agents. Registration was a one-time handshake, not a runtime dependency. You lose the console and remote sign-in until we're back; your agents don't stop.

Troubleshooting has the longer list.

What to send us​

For anything you escalate:

  • what you were doing, and what happened instead
  • the output of curl -s http://127.0.0.1:7376/health — it contains no secrets
  • whether the gateway is local, tunnelled, or hosted
  • the audit entry id, if there is one

Never send a credential, a gateway token, or a pairing code. We never need one, and anything pasted into a support thread should be treated as burned.

support@permaura.com