Skip to main content

Playbooks

Each one starts with the fastest thing that stops the bleeding, then the diagnosis, then the cleanup. If you're reading this during an incident, do step 1 and come back.

The two moves that cover most of it

A connection's kill switch stops one integration everywhere, instantly. Stopping the gateway stops everything. Neither needs you to unpick permissions first, and neither destroys anything you'll want back.

An agent is doing something it shouldn't​

Stop it

Remove the agent: Agents → remove. Its sessions are revoked immediately and any grant that named only that agent goes with it. If you'd rather keep the agent and stop the damage, hit the kill switch on the connection it's abusing.

Work out what happened

Open the audit log and filter to that agent. You're answering one question: was it allowed, or did it find a way?

  • Every call was allowed: then policy did what you configured, and the configuration was wrong. Look at which grant permitted it. This is the common case, and it's a good outcome in the sense that nothing was bypassed.
  • Calls were denied and it kept trying: the wall held. Worth understanding why it's asking, but nothing escaped.
  • Something ran that no grant covers: that's a different class of problem. Capture the audit entry id and get in touch.

Clean up

Narrow the policy that allowed it, rather than deleting the grant, so the same mistake isn't available to the next agent. Then reconnect the agent if you still want it; it'll come back as a fresh identity under whatever you've now allowed.

A credential has leaked​

The one that matters: Permaura never shows you one sealed value, but an export hands over all of them. Nothing in the console, the CLI or the API shows you one credential on its own. A vault export is the exception that matters: it writes every credential in the vault into one bundle, sealed under a passphrase the person exporting chose. If a key is out, it came from the vendor's side, someone's clipboard, a copy that existed before you sealed it, or an export, and the last of those is the one you can rule in or out from here (see Then check).

Stop it

  1. Revoke it at the vendor. This is the only step that actually invalidates the key. Everything else limits what Permaura will do with it.
  2. Hit the connection's kill switch. Every call through it stops instantly, so nothing keeps using the compromised key while you sort out a replacement.

Replace it

Issue a new credential at the vendor, then rotate the connection's credential to the new value. The connection, its capabilities, its policies and its grants all stay as they are; you're replacing the sealed value underneath them, not rebuilding anything.

Re-enable the connection when you're ready.

Then check

Filter the audit log to that connection for the exposure window. Every call made through it is recorded with what it did, which is the basis for whatever you have to tell somebody else.

Then check whether anyone exported the vault. Whoever holds Export the vault (owners, and admins or members only if an owner switched it on; see Roles & permissions) can write every credential in the vault into one bundle under their own passphrase. Look in the audit log for permaura.vault.export events since the leaked key was sealed. If one is unexplained, or you cannot tell whose it is, treat every credential that export could have read as out, not just this one: revoke and rotate all of them the same way.

Spend has spiked​

Stop it

Pause the grant, or kill-switch the connection. Pausing the grant is usually right: it keeps the definition so you can put it back once you understand the cause.

Find the cause

Check the grant's budget consumption, then the audit log for that agent. Almost always one of three things:

  • A retry loop. The same call repeating. Something upstream is failing in a way the agent treats as retryable.
  • A widened grant. Someone allowed a capability that costs money without a budget on it.
  • A capability that arrived enabled. Anything that spends money is meant to arrive switched off, but check what's actually on for that connection.

Prevent the repeat

Put a budget on the grant: a cap on calls per rolling day is enough to turn a runaway into a nuisance. For anything expensive, move it to the needs approval rung so a human sees each one. See Approvals.

The audit chain reads broken​

What it means

The hash chain no longer verifies: the log has been altered or truncated on disk. This is exactly what the chain exists to detect, and it's the one signal here you should treat as serious by default.

Confirm it directly rather than trusting the pill alone:

curl -s http://127.0.0.1:7376/health | jq .audit_chain_intact

What to do

  1. Don't restart the gateway and don't delete anything. The broken state is the evidence.
  2. Read out what's still there (permaura audit, or GET /v1/audit-events) and keep a copy off that machine.
  3. Consider the machine untrusted until you know why. Something with write access to ~/.permaura/gateway.db did this. The mundane explanations are a disk problem, a backup tool, or a file restored from a snapshot; the alarming one is someone covering a track.
  4. Get in touch with what /health returned and roughly when you last saw it verified.

What it isn't

It isn't a sign that a secret leaked or that an agent escaped policy. The chain covers the record, not the enforcement.

Somebody has to be locked out right now​

Not a resignation: a laptop in a taxi, or worse.

  1. Remove them from the organization (Settings → Members). Access ends across every workspace on their next request.
  2. Remove the agents they connected. These are separate identities and survive the person. Anything running on their machine keeps working until you remove the agent.
  3. Revoke their approval device (Workspace → Devices) as an owner or admin, since a member can revoke only their own; it asks you to confirm it's you. From gateway 1.3.46 it takes effect at once when the gateway's owner does it, or an owner or admin holding Remove any approval device alone (organization owners hold it; admins only if an owner switched it on). Anyone else's removal waits until a device paired to someone other than them confirms it, or the gateway's owner removes it, and lapses after 24 hours. When it cannot wait, have one of the people who remove alone do it, or have someone else with a paired device confirm it straight away. Pair a replacement first if it's the only one. Revoking your last device does not lower the approval bar; it just leaves nobody able to clear the queue. On a hosted gateway that replacement is held for their device to confirm, so revoke first and pair after, and expect to confirm it's you again when you pair; see the last-device playbook.
  4. Rotate credentials. Whoever holds Export the vault (owners, and admins or members only if an owner switched it on) can write every credential in the vault into one bundle sealed under their own passphrase. With a device paired and the approval floor not lowered, a paired device has to confirm each export, and from gateway 1.3.46 the exporter's own phone cannot confirm one asked for through permaura.com while a device paired to someone else is trusted, unless the exporter is the gateway's owner. Check the audit log for permaura.vault.export events while they had access. If one is theirs, or you cannot tell whose it is, they may hold every credential in that vault: rotate all of them. Otherwise rotate the ones they set up: they never saw those sealed values through Permaura, but they may hold the originals.

If you want everything frozen while you work through it, stop the gateway or pause the broker. Nothing is lost by doing that first.

Full version, for a planned departure: offboarding.

Agents are being refused and you don't know why​

Deny-by-default is doing its job; the question is which rule.

Filter the audit log to denials. The entry names the capability and connection, and that's usually the whole answer. Then check, in this order:

  1. Is the capability enabled on the connection? A disabled capability is refused for everyone, whatever your policies say.
  2. Is the connection's kill switch off?
  3. Is the grant enabled rather than paused, within its budget, and before its expiry?
  4. Does a policy deny it? Deny beats needs-approval beats allow, across every policy that applies: one deny anywhere wins.
  5. Is the broker paused? That refuses everything, and it's easy to forget you did it.
  6. On a hosted gateway, has its plan lapsed, or is the gateway being removed? From gateway 1.3.51 either one pauses every agent on it; see What happens if a plan lapses.

Troubleshooting covers the same ground from the end user's side.

The approval queue is stuck behind a device nobody has​

The sticky-floor case. The gateway's approval floor requires a device for every approval (you raised it, or the gateway had a device paired before 1.3.17 and kept that rule), the device is gone, and unpairing didn't lift the requirement.

Pair a replacement device. That's the clean fix, and it restores the queue.

If you can't, the bar can be lowered, and where from depends on where the gateway runs:

  • Local or tunneled: only from the machine running the gateway, never over the network and never from the console. That restriction is the whole point: it means an attacker holding your account session can't clear it.
  • Hosted: from the console, under Workspace → Devices, after confirming it's you. While you still have one working device, that device also has to sign the change; the confirmation alone is enough only once nothing usable is left, which is the next playbook.

Actions explicitly marked Device-signed still require a signature regardless.

See Approvals for the exact behavior.

Hosted: the last approval device is gone​

The one shape of this that has no host to walk to. On a hosted gateway there is no machine of yours to run a local command on, so the recovery runs through your account and is deliberately narrow.

If the device is lost but you can still get a new one enrolled, do that first. It is the clean fix, and it costs two confirmations:

  1. Install the Permaura app on the replacement device. The iPhone app is on the App Store.
  2. Revoke the lost device under Workspace → Devices, and confirm it's you when asked. This matters before, not after: while the old key is still trusted, the gateway will hold the new enrollment waiting for the lost device to confirm it, which nothing can satisfy. Because it is the last trusted device, from gateway 1.3.46 the gateway's owner has to revoke it, or an owner or admin of the organization who holds Remove any approval device alone or whose device it was; on an older gateway any owner or admin of the organization can. Anyone else's removal of it is refused, or from gateway 1.3.46 held for a confirmation that only the lost device could give, so it would lapse unanswered after 24 hours.
  3. Pair the replacement in the app, and confirm it's you again when it asks. With no usable device left, it enrolls on that confirmation alone, exactly like the first one did, and like the first one it has to be paired by the gateway's owner or an owner or admin of its organization.

Two confirmations, in that order. They are separate on purpose: what the first one issues can only revoke that one device, what the second one issues can only enroll, so neither step can be spent as the other. Nothing holding your account session alone can do either.

If the lost phone was also your account's approver, confirm both steps with your passkey or your password instead. A password changed in the last seven days takes a second step as well, and that has to be something else you hold: your authenticator app, an emailed code (held back if you reset the password from your inbox) or a backup code. Afterwards, add a replacement sign-in factor and remove the lost phone from your account too; Two-step verification covers that side, including what changes if you had to reset your password.

If you want console approvals back rather than a new device, revoke every device first, then lower the approval floor from the console, under Workspace → Devices. That is the same two confirmations: one to take out the last device, which again takes the gateway's owner, or from gateway 1.3.46 an owner or admin who holds Remove any approval device alone or is removing their own device (any owner or admin on an older gateway), and one to lower the floor. With nothing usable left there is no device to co-sign the lowering, which is the whole reason revoking comes first.

Revoke, then pair, in that order

Pairing a replacement while the lost device is still trusted produces a request that waits for a signature from the phone you no longer have. Nothing can confirm it: from gateway 1.3.46 it lapses after 24 hours, and on an older gateway it waits indefinitely. Revoking first turns the same enrollment into an immediate one.

Everything here is written to the audit log under permaura.approver.pair, and the account owner is emailed. If either shows an enrollment you did not perform, treat it as an account compromise: change the password, remove any sign-in factor you don't recognize under Password & security (see Two-step verification), remove any agent you don't recognize under Agents (a new password does not cut off an agent that is already connected), and revoke the unknown device.

Everything is offline and you don't know if it's you or us​

Ask the gateway directly:

curl -s http://127.0.0.1:7376/health
  • It answers: your gateway is fine, and the problem is between the console and it. Check the port, the browser URL under Workspace → Gateway → Advanced, and whether "unverified" is just the expected state for a local gateway.
  • It doesn't answer: the gateway isn't running. Start it.
  • It answers but permaura.com is down: your local gateway keeps enforcing policy and serving agents. Registration was a one-time handshake, not a runtime dependency. You lose the console and remote sign-in until we're back; your agents don't stop.

Troubleshooting has the longer list.

What to send us​

For anything you escalate:

  • what you were doing, and what happened instead
  • the output of curl -s http://127.0.0.1:7376/health (it contains no secrets)
  • whether the gateway is local, tunneled, or hosted
  • the audit entry id, if there is one

Never send a credential, a gateway token, or a pairing code. We never need one, and anything pasted into a support thread should be treated as burned.

support@permaura.com

Last updated on