When the Pager Goes Off: Engineering Emergency Access That Zero Trust Can Still Vouch For
Photo: AMISOM Public Information, CC0, via Wikimedia Commons
It is 02:14 on a Sunday and a duty engineer cannot reach the payments cluster. The identity provider is up, the policy engine is up, and the engineer's just-in-time role has expired because the approval workflow depends on a colleague who is asleep. Somewhere in a physical safe there is a laminated card with a break-glass password that has not been rotated since the last audit. Everyone knows it exists. Nobody wants to be the person who uses it.
This is the failure mode zero-trust programmes are least willing to discuss. Architecture that assumes continuous verification struggles with the night when verification itself is the obstacle. The temptation is to carve out an exception: a standing account, exempt from conditional access, held by a small group. That exception is usually the single most valuable credential in the estate.
The shape of the trap
Break-glass designs fail in predictable ways. The account exists permanently and is used more often than anyone admits, usually for convenience rather than crisis. The password is shared, so attribution collapses the moment two people know it. There is no expiry, so the elevation outlives the incident by months. Logging exists but sits in the same platform that was unavailable, which is precisely why the account was needed. And the review, scheduled for the following week, is quietly deferred.
Regulators in the UK are not asking for the absence of emergency access. They are asking whether risk is managed and evidenced. The FCA's operational resilience expectations, the NCSC's Cyber Assessment Framework and NHS data security standards all converge on the same question: can you demonstrate that a privileged exception was authorised, bounded and reviewed? A break-glass mechanism that cannot answer that is an unmanaged control, however well intentioned.
Four properties, no exceptions
Defensible emergency access has four properties, and each one must hold during a real incident, not just in the design document.
Named. Every use is attributable to a specific person, using their own identity with elevation, never a shared account. Where a shared account is unavoidable in legacy systems, the credential is split so that no single individual can reconstruct it.
Time-bound. Elevation expires by default, measured in minutes or hours. Expiry must be enforced by the platform, not by a calendar reminder.
Approved. Either by a duty approver with a hard response window, or by the requestor themselves with mandatory justification and immediate notification to a quorum of peers who can revoke in flight.
Observed. Session activity, commands and justification are captured to a log sink the requestor cannot modify, and which does not depend on the systems being rescued.
A fifth property, rotation, follows from use: any credential touched during an incident is replaced before the incident is closed.
A three-tier blueprint
Tiering keeps the common case fast and reserves ceremony for the rare one.
Tier one — contained emergency. Self-authorised elevation to a scoped role, 60-minute expiry, hardware security key issued to the duty engineer and stored in a tamper-evident sealed bag with a serialised number. Breaking the seal is itself the notification trigger. Suitable for restarting a service, restoring a configuration or reaching a management plane.
Tier two — dual control. Two-person approval with a 30-minute response target from a separate duty approver rota, four-hour maximum elevation and full session recording. Used for domain-level administration or access to regulated data stores.
Tier three — catastrophic recovery. A sealed credential set held in at least two geographically separate locations, reconstructed only by quorum, with the chief information security officer and the on-call director notified at the moment of assembly. Eight-hour ceiling, mandatory post-use review within 24 hours, and a written account of why the lower tiers were insufficient.
Approval chains that survive three in the morning
Rota design matters more than policy wording. The duty approver must sit outside the requesting team, be reachable by at least two channels, and have a documented escalation path when they do not respond within the window. The fallback should not be silence: after the window elapses, tier-one style self-authorisation may proceed, with automatic notification and a mandatory review the next working day.
Public-sector teams should test this against outsourced service desks and shared-service arrangements, where the person holding approval authority may be in a different organisation under a different contract. Name them in the runbook. A supplier's on-call number that routes to a generic mailbox is not an approval chain.
Logging that holds up under review
Capture, at minimum: the identity of the requestor and approver, the justification text as typed, the systems reached, the commands or API calls executed, session start and end, and a hash of any recorded artefact. Send it to an independent, append-only store with adequate clock synchronisation, retention aligned to your regulatory obligations and a documented legal hold process.
Two practical tests tell you whether this is real. First, can the log sink be reached during the exact failure you are preparing for? Second, can an investigator reconstruct a session six months later without help from the engineer who ran it? If either answer is no, the logging is decorative.
Prove it in a tabletop, then prove it again
Run the exercise quarterly, with one unannounced scenario inside an agreed window each year. Measure time to access, time to approval, time to revocation and the proportion of touched credentials rotated within 24 hours. Then inject complications: the approver is unreachable, the identity provider is degraded, the log pipeline is down, the incident coincides with a change freeze.
Record the findings as you would a penetration test, with owners and dates. The evidence pack that satisfies an auditor is not the runbook. It is the runbook plus three exercises where the design bent and did not break.
Emergency access is not a hole in zero trust. It is a controlled exception with a receipt, and the receipt is what makes the architecture credible on the morning after.
