Skip to content
SkyKeephelp

Troubleshooting

Symptoms quote what the vault actually emits, so searching this page for the error text you see should land on its entry. Several 'errors' below are the vault's fail-closed design working as intended — those entries say so explicitly.

The browser refuses to open the vault: NET::ERR_CERT_AUTHORITY_INVALID, ERR_CERT_COMMON_NAME_INVALID, or "Warning: Potential Security Risk Ahead"

No certificate material is mounted, so the front door generated a self-signed one at start. A self-signed certificate is not signed by an authority your browser trusts, and it never will be — this warning is correct and expected on a development or demonstration deployment. ERR_CERT_COMMON_NAME_INVALID specifically means the certificate does not carry the name you typed; the generated pair covers the loopback name and loopback address in both families, plus the deployment-internal service name, so seeing that error means you reached the vault by some other name.

  1. For a deployment only you use: click through the warning. Nothing is wrong. The traffic is still encrypted; what is missing is a third party vouching for who is on the other end.
  2. If you reach the vault by another name — a hostname, a LAN address — put it in SKYKEEP_TLS_SAN using openssl subjectAltName syntax, a comma-separated list where each entry is prefixed DNS: for a name or IP: for an address. Then delete the generated pair and restart so it is made again. Names you add are ADDED to the defaults, never substituted for them, so loopback keeps working.
  3. For a real installation, do not use the generated pair at all: bind-mount your own tls.crt and tls.key at /etc/skykeep/tls. The front door leaves mounted material untouched, and this warning goes away because a certificate your organisation's authority signed is one your browsers already trust.

a document sits on the Processing page and never moves

A document is worked in stages and only an ADMITTED document is searchable (ADR-0038). An item stops moving for one of three ordinary reasons, and the page says which: it is waiting for a person, because the upload asked to approve each stage and the approval buys exactly one stage; it is waiting for the vault, because a stage was approved and the worker has not reached it yet; or it STOPPED, because a scan gate caught it, a stage failed, or somebody abandoned it. None of these is a fault by itself.

  1. Read the stage column: 'converted, waiting for a summary' means it is yours to approve, and 'approved — the vault is working on it' means it is not.
  2. An item whose approval was recorded but which never moves means no worker is running against this deployment; the stages are `skykeep.workflow.stages` driven by `skykeep.workflow.runner`.
  3. A stopped document keeps everything: it is stored, encrypted, listed to those who can reach it, and downloadable byte for byte. Stopping is an outcome, not a deletion.
  4. A worker that died holds nothing: its lease expires and the item becomes claimable again on its own. There is no queue to drain by hand and no item to unstick.

invitations refuse with "no mail relay is configured" (HTTP 503)

Inviting requires the vault to be able to send: the bulk invite pre-flights the relay BEFORE creating any account, so an unconfigured relay refuses the whole submission rather than minting accounts nobody could be told about (ADR-0016). This is the fail-closed design working, not a fault.

  1. Configure the relay at /admin (mail relay, SMTP host, port, From address, transport security) — it applies to the next send with no restart.
  2. Or set the SKYKEEP_MAIL_RELAY / SKYKEEP_SMTP_* variables in the environment as the baseline, then restart the stack.
  3. Send a test message first with `skykeep.sh emtest`, which validates the credentials end to end before you invite anyone.

an invited person never received their invitation, and the emails-sent page shows the message as bounced

The relay refused that recipient while the vault was still on the wire — usually a mistyped address. The account was created and left in place deliberately (ADR-0016 § 4): that account plus the bounced record is exactly what the correct-and-resend workflow acts on.

  1. On the emails-sent page, correct the address in place on the bounced row and resend. That updates the person's address on file and issues a fresh temporary password; the re-invitation is audited against that account, and the address itself is deliberately not recorded, because `create-user` already omits addresses from the trail and the resend keeps that pattern (`dal/adminconsole.py`, `reissue_invite`). It does not create a second account for them.
  2. A message shown as `sent` means only that the relay ACCEPTED it; the vault cannot confirm delivery beyond that point, so a person who received nothing despite a `sent` row is a question for the mail provider, not the vault.
  3. A `sent` invitation gets no checkbox and cannot be resent from the page. That is deliberate: a resend issues a fresh temporary password, so resending delivered invitations in bulk would lock out everyone still holding the first one. If somebody genuinely needs re-inviting at an address the relay accepted, that is a deliberate per-message act, and the audit trail records it as one.

compose fails: required variable "SKYKEEP_…" is missing

Every configuration value arrives via the environment and the stack fails closed on any unset required variable.

  1. Compare your .env against env.example — every uncommented key in the template is required unless marked optional.
  2. After adding keys, run `docker compose up` again; no rebuild is needed for environment changes.

signed in, but everything answers "denied": "password change required before anything else"

The account carries the forced-rotation flag (ADR-0012) — every fresh vault's admin001 starts this way, and the vault refuses every operation until a new password is set. This is the bootstrap credential ending its useful life, not a fault.

  1. Use the portal's password-change form (it appears at sign-in when the flag is set), or POST /api/account/rotate with the current and new password in the request body.
  2. The new password must be at least 8 characters and differ from the current one; refusals of your own proposed value say why.
  3. After the change, sign-in and every interface work normally; the old password can never be used again.

a person has forgotten their password and cannot sign in

The vault ships PASSWORD CHANGE, not password reset: the only rotation path is the signed-in form (and its /api/account/rotate body), and there is deliberately no self-service reset link, because a reset mailed to an address would be a second way into the account. The vault stores only password hashes, so a lost password cannot be recovered — only replaced.

  1. An administrator re-invites the person from the emails-sent page. The re-invite issues a FRESH temporary password to the address on file, under forced first-login rotation, exactly as the original invitation was born; the re-invitation is audited against that account.
  2. If the original invitation BOUNCED, the row is selectable on the page and the address can be corrected in place before the resend.
  3. If the original invitation was DELIVERED, the resend is refused unless the administrator passes the explicit per-item override — the guard lives in the act's own transaction (`dal/adminconsole.py`, `reissue_invite`), not only in the page, so a scripted caller meets it too. That is deliberate: the resend retires the temporary password the person was already given, so re-inviting a delivered address has to be somebody's explicit choice, and both the delivery status and the override land in the audit detail.
  4. The person replaces the temporary password at their next sign-in; the old one can never be used again.

sign-in is denied for an account whose password is correct (and the audit trail says no-email-on-file)

The account has no email address on file, and one-time codes are delivered to the account's email (ADR-0013) — never to the identifier, and never to a guessed address. On the wire the refusal is indistinguishable from a wrong password — since ADR-0079 both are the same plain 'bad username or password', where they used to be the same one-time-code challenge that no code could complete (anti-enumeration); the audit trail carries the real reason. Typical on a vault upgraded from a build that predates per-account email, where an admin rotated before any email could be collected.

  1. An administrator sets the account's email: for an upgraded vault's own admin account, one operator SQL statement (UPDATE auth_local_credentials SET email = '<address>' WHERE identifier = '<identifier>') — new invites always carry an email, and a forced first-login rotation collects one.
  2. Accounts still awaiting their forced first password change are unaffected: that login is password-only by design and collects the email.

every sign-in takes about three seconds, successful ones included, and the vault otherwise seems responsive

That is the sign-in delay, and it is working (ADR-0079). Every attempt at POST /api/login/begin and POST /api/login/complete is held to a floor of SKYKEEP_LOGIN_DELAY_SECONDS — default 3 — whether it succeeded or failed. It is a brute-force brake and a timing-oracle cover: a refusal that came back instantly would tell an attacker the work was skipped, and would let one source try thousands of passwords a minute. It is deliberately applied to the SUCCESS path too, because a delay only on failure is itself the oracle. It costs the vault nothing while it waits: the delay holds no database connection and no worker thread, so a sign-in flood cannot starve anything else.

  1. Nothing, if three seconds is acceptable — this is the shipped posture and the figure the vault's owner chose.
  2. To retune it, set SKYKEEP_LOGIN_DELAY_SECONDS to a number of seconds above 0 and at most 60. Anything else — blank, unparseable, zero, negative, or above the ceiling — falls back to 3 rather than disabling the delay: a control never fails open, so there is no value that switches it off.
  3. If sign-in takes much LONGER than the configured delay, the delay is not the cause and is not hiding one: it pads a fast attempt up to the floor and never slows a slow one down. Look at the database, the mail relay, or (under the directory provider) the directory server.

the vault is "refusing to start" and names SKYKEEP_SCANNER

This installation is a PRODUCTION installation (it carries no demonstration marker) and SKYKEEP_SCANNER is set to 'pattern'. That engine is a byte-signature TEST DOUBLE: it detects exactly the strings in SKYKEEP_SCANNER_SIGNATURES and passes everything else, so the vault would accept every upload unscanned while reporting a configured scanner. Refused twice, on purpose — once by `scripts/skykeep.sh start` before any container is created, and again by the webserver itself, so a bare `docker compose up` is covered too.

  1. On a real engagement, install a scanner and repoint the engine — this is the whole change: SKYKEEP_SCANNER=command with SKYKEEP_SCANNER_COMMAND=/usr/bin/clamscan --no-summary.
  2. If this deployment is the demonstration vault, set the demonstration marker (SKYKEEP_DEMO_INSTALLATION). A demonstration is an already-declared weakening, so it needs no second acknowledgment for the engine.
  3. If this is a disposable verification stack, set SKYKEEP_SCANNER_PATTERN_ACK to its exact phrase. Only that phrase is an acknowledgment: 'true', '1' and a typo are all still refused.
  4. Clearing SKYKEEP_SCANNER is NOT the fix and is not refused here: an unconfigured scanner quarantines every upload. The ingest scan gates cannot be turned off by any setting; only the engine changes.

every upload answers "quarantined"

No malware scanner is configured, and an unconfigured scanner quarantines everything — the gate fails closed rather than waving content through unscanned. This is the vault working, not breaking.

  1. Set SKYKEEP_SCANNER in this deployment's env file and restart. It is NOT in the admin console and cannot be: a console that could change the scanner could downgrade the malware gate past the start-up refusal, which reads the environment and not the database. The template ships it EMPTY, which is why a fresh installation quarantines everything until somebody chooses. Two engines exist and neither can turn the gate OFF: 'pattern' is a byte-signature double for development, and 'command' runs an external scanner in ClamAV's convention (SKYKEEP_SCANNER_COMMAND).
  2. A scanner with an empty signature list, or the 'command' engine with no command, still counts as unconfigured — it could never detect anything.
  3. Under the 'command' engine, check the scanner really runs: a command that is missing, not executable, times out, or exits with anything but 0 (clean) or 1 (infected) quarantines, because a scanner that could not answer has not answered 'clean'. The quarantine reason names which of those happened.
  4. Already-quarantined items stay in the review queue: release them (ambiguous and auditor holds only) or reject them in the admin console.

every upload PASSES the malware gate, and you are not sure anything is actually being scanned

SkyKeep ships a STUB scanner (containers/scanner/clamscan-stub.sh, installed in the webserver image at /app/scanner/clamscan-stub.sh) and the shipped SKYKEEP_SCANNER_COMMAND points at it. IT ANSWERS CLEAN WITHOUT SCANNING ANYTHING. It exists so the gate is wired, exercised and identical in shape to a real deployment before ClamAV is installed — not so that a deployment can ship without malware detection. On a real engagement this means the vault is detecting nothing.

  1. Read the webserver's log. The stub announces itself on stderr on EVERY invocation; a deployment running a real scanner prints no such line. That is the fastest way to tell which one you have.
  2. Install ClamAV and repoint the setting. That is the whole change — no code, no image rebuild: SKYKEEP_SCANNER_COMMAND=/usr/bin/clamscan --no-summary (or /usr/bin/clamdscan --fdpass for the daemon).
  3. Verify with the standard antivirus test file: a real scanner quarantines it and names the signature in the reason, and the stub passes it. That check takes one upload.
  4. Do not try to switch the stub off by clearing the setting and expecting uploads to flow. An unconfigured scanner quarantines EVERY upload; the gate cannot be turned off, only its engine changed.

one upload answers "unidentified content"

Since ADR-0030 the vault decides a file's type from its CONTENT, not from its name, and it refuses bytes it cannot positively identify rather than guessing at them. Guessing plain text is how a mis-detected binary reaches a text parser, so an unidentified file is refused and recorded instead. Note the refusal is about the BYTES: an unfamiliar extension, or no extension at all, is not itself a problem and such a file ingests normally when its content is recognisable.

  1. Check the file opens in the application that produced it — a truncated download or a partial copy is the usual cause, and a zip container missing its own parts is refused rather than reported optimistically as the format it resembles.
  2. Compare the file against /help/formats, which is generated from the same table the pipeline dispatches against, so it cannot advertise something this deployment does not do.
  3. A format listed there as original_only is not this symptom either, but it is also not accepted: such a file is refused today, with a message naming the format, because no converter has been adopted for it. Storing an unconvertible file for its own sake is designed and not yet built. Nothing about such a file is stored, so there is nothing to search and nothing to download back.

the webserver exits at start: "refusing to start"

Local-first enforcement: the configured model endpoint points outside the deployment, and the vault refuses to serve rather than quietly send content off-box.

  1. Point model_base_url (or SKYKEEP_OLLAMA_BASE_URL) at an endpoint inside the deployment — the compose service name is the normal value.
  2. If off-box egress is genuinely intended, set the explicit operator override variable to the documented acknowledgment token (see env.example) — every model call under the active override is written to the audit trail.

every portal call answers 503 "portal not configured"

A portal dependency is missing: identity material, the engagement key, or the model configuration. The portal refuses service outright rather than running degraded.

  1. Check SKYKEEP_ENGKEY is set (64 hex characters).
  2. Check SKYKEEP_IDENTITY_DIR: the container self-enrolls dev material into an empty directory; provisioned deployments mount real certificates there instead.
  3. Check model_provider and embed_model resolve to values (the admin console's settings page shows each key's effective value and where it came from).

sign-in is refused for every account: "auth provider not configured"

Sign-in fails closed, and four different states produce this same refusal on purpose — the vault will not guess which provider you meant. auth_provider may be UNSET. It may be set to a name this vault does not implement: only 'local' and 'directory' are accepted, and anything else — a typo, or a provider that has not been built — refuses every sign-in rather than falling back to local. It may be set to 'directory' while no directory server is configured, which refuses identically and deliberately, so that a half-configured directory can never quietly become local sign-in. Or the provider is fine and the mail relay is unconfigured, so one-time codes cannot be delivered. A single disabled account refuses too, but only for that person — if others can sign in, look there first.

  1. Read auth_provider's effective value, and where it came from, on the admin console's settings page. An override set in the console outranks the SKYKEEP_AUTH_PROVIDER environment baseline, so a restart will not undo one.
  2. For password sign-in against this vault's own accounts, set auth_provider to 'local' (environment baseline SKYKEEP_AUTH_PROVIDER, or the admin console). A fresh vault needs this: the bootstrap administrator admin001 is a local credential and signs in under no other value.
  3. For directory sign-in, set auth_provider to 'directory' AND configure SKYKEEP_DIRECTORY_HOST together with the rest of the SKYKEEP_DIRECTORY_* values. Setting the provider on its own refuses every sign-in with this same message; so does a directory configuration that is half filled in.
  4. Under 'directory', a person who exists only in this vault cannot sign in at all — admin001 included. Directory sign-in authenticates people who ALREADY hold an account here; it creates nobody. If switching the provider has locked everyone out of the console, the recovery is to delete the auth_provider row from the vault_settings table in the database: an override outranks the environment, so editing the environment baseline will not rescue you.
  5. Configure the SMTP relay values — an unconfigured relay refuses code delivery fail-closed.
  6. Check the account is enabled in the admin console's Users panel.

changing a password or a code-delivery address is refused: "the configured auth provider does not manage local credentials"

This vault is configured for directory-backed sign-in (auth_provider = 'directory'). A directory-backed person's password and mail attribute live in the directory server, and SkyKeep only ever READS that server — the connector ships bind, search and unbind, and no write operation at all (ADR-0060). So the vault refuses instead of editing a local credential row that nobody's sign-in consults, which would report success while changing nothing that authenticates anyone. This is the fail-closed design working, not a fault.

  1. Change the password in the directory itself, using whatever tool the directory's administrator normally uses. The vault reads the new secret at the next sign-in; nothing here needs updating afterwards.
  2. Nothing on the vault side is broken and nothing needs repairing. If you did intend this vault to manage passwords itself, auth_provider must be 'local' — but changing it also changes who can sign in at all, so read the entry above before you turn that dial.

search returns fewer results than requested

Under-return honesty: fewer matching documents exist within your compartment reach than you asked for, and the vault says so instead of padding the list. Not an error.

  1. Nothing to fix on the vault side; the flag is the feature.
  2. If you expected more results, check which compartments your grants admit — an administrator can review grants in the console.

a request answers a bare "denied" with no reason

Denials are deliberately detail-free — explaining what was denied can leak what exists. The reason is in the audit trail, not the response.

  1. An administrator can read the denial's audited reason and adjust grants if the denial was not intended.

an agent commit answers "stale-base"

The document gained a newer version after the agent's checkout. Nothing was lost: versions are never overwritten.

  1. Checkout again, re-apply the edit onto the fresh base, and commit with the new base_version_id.

the admin console refuses every act for a signed-in user

Console authority requires an admin-kind principal; a plain member gets the uniform denial on every console surface (and each attempt is audited).

  1. Sign in with an admin account, or have an existing admin provision one in the Users panel.

the initial administrator password is lost, or a from-scratch reinstall is wanted

The bootstrap credential's useful life ends at first login (ADR-0012), and the vault stores only password hashes — a lost password cannot be recovered, only replaced.

  1. To start over COMPLETELY, destroying every document, user, and both audit trails: run `scripts/skykeep.sh reinstall_complete_wipes_all_data` and type the acknowledgment phrase it asks for. The fresh vault reseeds admin001 with the documented initial password under forced first-login rotation. There is no undo.
  2. Without the exact typed acknowledgment the command refuses and touches nothing.

the service log (skykeep.sh dumplog) repeats "audit relay: drain failed"

The webserver's relay task (ADR-0014) could not reach the audit instance, so queued audit events stay in the primary outbox. Nothing is lost — the outbox holds every event until delivery succeeds — but the separate audit trail is not receiving entries while this repeats.

  1. Check the auditstore service: `docker compose ps auditstore` and its log. Restart it if it is down.
  2. Delivery resumes by itself on the next drain tick after the audit instance is reachable again — no operator action, no webserver restart. `skykeep.sh dumplog` shows the queued events moving from the outbox to delivered entries.

the demonstration installation prints "SKYKEEP_DEMO_INSTALLATION does not carry the demonstration acknowledgment phrase"

The demo mode stands up a DELIBERATELY WEAK vault (one shared inbox, one shared weak password, no forced first-login rotation), so it refuses unless the deployment is explicitly marked for demonstration use. The marker is an exact typed phrase, never a boolean: true, 1 and yes all refuse. Nothing was created, changed, or deleted.

  1. Only on a deployment that holds no real data, set SKYKEEP_DEMO_INSTALLATION to the acknowledgment phrase printed in env.example, then re-run `scripts/skykeep.sh demo_setup`.
  2. If this appeared on a deployment that DOES hold real data, the fence worked as designed — leave the marker unset.

the demonstration installation prints "the vault already holds data demo_setup did not create"

The mode refuses on any vault holding principals, compartments, or documents it did not itself create — which is how pointing it at the wrong deployment costs nothing. A freshly migrated vault holds exactly two principals (the system attribution row and the bootstrap administrator) and no compartments at all. The message names the counts it found. Nothing was created, changed, or deleted.

  1. Check which deployment the command reached. If it is the intended one and it is genuinely disposable, wipe it (`scripts/skykeep.sh reinstall_complete_wipes_all_data`) and re-run — the wipe destroys all data and asks for typed confirmation.
  2. Re-running against an already-built DEMO vault is fine and does not trigger this: the mode's own compartments, accounts, and documents are not foreign to it.

the demonstration installation refuses with a list of unset SKYKEEP_DEMO_* settings, or the corpus source cannot be fetched

The demo's own configuration has no built-in defaults (ADR-0002): the shared address and password, the bootstrap administrator's identifier and secret, and the corpus expansion directory must all be configured. The corpus itself is a committed archive shipped inside the product and expanded into SKYKEEP_DEMO_CORPUS_DIR — nothing is downloaded unless SKYKEEP_DEMO_CORPUS_URL is set to an alternative corpus zip, so a fetch failure can only come from that override.

  1. Fill the SKYKEEP_DEMO_* block in .env from env.example; the refusal names exactly which values are missing.
  2. If the fetch fails, unset SKYKEEP_DEMO_CORPUS_URL to use the packaged corpus, or point it at a local zip file path — the override accepts either. A complete expansion already in SKYKEEP_DEMO_CORPUS_DIR short-circuits every source entirely.

the admin console Kind pulldown offers no clientadmin

A console built before the pulldown was widened. It offered only human and admin for as long as the clientadmin kind existed; the route behind it always accepted clientadmin, so this was a short pulldown rather than a missing power. A console still showing only two kinds is serving stale static files.

  1. Reload the page; if it persists, the webserver image predates the change and needs rebuilding (docker compose up --build).
  2. Use the portal's Invite page and tick 'Invite as clientadmin' before submitting — that is the shipped clickable path, and it mails the person a temporary password the administrator never sees.
  3. Or POST /api/admin/users as an administrator with kind 'clientadmin'. That path creates the account but sends no mail, so telling the person is your job.
  4. See the clientadmin provisioning page in this runbook for the whole path and what the account can then do.

sign-in answers that code delivery is unavailable on a demonstration vault

Almost certainly you are signing in as admin001, because the six MEMBER accounts do not need a relay at all. Since ADR-0025 the one-time code is a policy rather than a fixed leg of sign-in, and demo_setup creates all six accounts already enrolled with the code turned off — a state any account can choose from its own profile page. The bootstrap administrator was never put in that state, so the ordinary policy asks it for a code like anybody else, and with no relay configured there is no way to deliver one. The vault refuses rather than letting anyone in.

  1. Configure the mail relay in the admin console (or the SMTP environment settings) and sign in again — no restart is needed, the effective configuration is read per request.
  2. All six demo accounts share one address, so one working inbox is enough to demonstrate every viewpoint.

You need a backup of the vault, or you are about to take a host-level snapshot of the machine instead

There is one supported backup, and it is the verb scripts/skykeep.sh backup. A whole-instance snapshot is the obvious alternative and it is the wrong one: it re-captures the key wraps beside the ciphertext, so destroying one document version no longer destroys it everywhere, and there is no point in a physical restore at which the record of what was purged can be replayed. The verb writes FOUR artifacts under one run id — the corpus dump, a separate escrow artifact holding the key wraps under a DIFFERENT key for a different custodian, a full audit dump, and a manifest tying the three together. It forces the audit relay to drain first and refuses outright if any event is still queued, because the record of what was purged is the audit log itself and a backup whose ledger might be short is worse than no backup. What a backup is NOT: it carries NO predicate-signing key, NO workload-identity CA key, and NO engagement key in any form. Nor is the bulk dump ciphertext-only — summaries, filenames, keywords and indexed words are plain columns, which is exactly why every artifact is encrypted in its own right.

  1. Set SKYKEEP_BACKUP_DIR, SKYKEEP_BACKUP_ESCROW_DIR, SKYKEEP_BACKUP_KEY and SKYKEEP_BACKUP_ESCROW_KEY in the environment file. All four refuse when unset, the two directories must differ, and the two keys must differ and must not equal the engagement key.
  2. Run scripts/skykeep.sh backup. It reads the data volumes and never writes or removes one, and it never deletes, lists or truncates anything in the backup location — rotation is your object-lifecycle policy, your share policy, or a cron, never the vault.
  3. Point the backup location at a mount this vault cannot delete from: an object store with object-lock and a write-only credential, an append-only share, or a directory whose files this uid may create but not unlink.
  4. After any restore, install a predicate key slot with scripts/skykeep.sh predicate_key. The backup deliberately does not carry one, and the vault refuses to serve until you supply it — one predicate-TTL of denial, no re-encryption, no data loss.
  5. Escrowing the ENGAGEMENT key is your own act, out of band. The vault ships no verb that exports it, because an artifact holding both the ciphertext and the key that decrypts it is not a backup.

Somebody has asked you to erase a document, and you need to say what the purge did and did not destroy

A purge destroys the version inside this vault and the per-version key that seals it; it does not reach a backup artifact already written, and the derived plaintext inside one — the summary, filename, keywords and indexed words — goes on existing until that artifact is retired. That is the whole of it, and it is not a defect being worked around: the summary, the filename, the keywords and the indexed words were never sealed under the per-version key, so destroying that key cannot reach a copy of them that is already written. Inside the vault they ARE destroyed, by the same transaction and with the engine refusing to commit a purge that left any of them behind. The vault also cannot delete from the backup location by design — that is what makes the location trustworthy — so retiring an artifact is your act on your mount, on your retention policy. Restoring an artifact does not resurrect the document: the purge ledger is replayed over whatever was loaded, so the version is destroyed again on the way in. The residue is what sits in the artifact at rest, between the purge and the day that artifact is retired.

  1. Bridge-mode engagements take no backups at all, so there is no artifact of this kind to retire and the purge is the whole story. This entry applies to permanent-store engagements.
  2. Read your backup retention window as the answer to how long a purged document goes on existing in derived form, and say that number when you are asked — it is a destructibility bound, not only a storage bound.
  3. Retire the artifacts on your own mount when the window expires. The vault issues no delete against the backup location, and giving it one would undo the property that makes the location worth backing up to.
  4. A version under a legal hold is not purged at all, and a restore will not purge it either: the replay refuses that ledger entry, records the refusal and carries on. Release the hold first if the erasure must proceed.

The console or the API answers "Purge certificates are disabled on this deployment. An administrator can enable them in the admin console under Settings (purge_certificates). A certificate already issued is unaffected and still verifies offline."

Purge certificates are a CONFIGURABLE capability, not a mandatory feature on every deployment (requirements § 5.2, ADR-0087 § 9), and this deployment has the capability switched off. It is one setting, purge_certificates in the admin console under Settings, with SKYKEEP_PURGE_CERTIFICATES as its environment baseline in the env file the deployment was started with. Unset means ON, which is what a deployment that has never touched the dial does. Every refused request is in the audit trail, so the answer to "was this vault issuing certificates in March?" is a query and not a recollection.

  1. Turn it on in the admin console under Settings: set purge_certificates to true. A console value overrides the env file and takes effect on the next request, with no restart.
  2. Or set SKYKEEP_PURGE_CERTIFICATES=true in the env file and restart, which is the right half to use when the decision belongs to whoever deploys rather than to whoever administers.
  3. Turning the capability OFF invalidates nothing. A certificate already issued goes on verifying, offline, with scripts/skykeep.sh verify_purge_certificate, for as long as its engagement key slot is installed — the authenticator covers the certificate's own bytes and asks no vault anything. The switch governs issuing, not checking.
  4. If the container refused to START naming this setting, the value is neither true nor false. That is refused rather than guessed at in either direction, because both guesses look correct to whoever typed it. Fix the value or unset it.

You need to restore this vault from a backup run, or a restore refused and you want to know whether anything was destroyed

scripts/skykeep.sh restore RUN_ID REPLACES the corpus database from the artifacts of one run, KEEPS the audit database (it is the purge ledger, ADR-0076 § 3), and then REPLAYS that ledger over what it loaded. The replay is the point of the whole mechanism: a backup taken before a purge still contains the purged version, so restoring it without a replay would undo the one act this product promises has no undo. The ledger is the audit log itself — every purge event with outcome success — read in order, and every version it names that is still whole in the restored corpus is destroyed again through the same purge_version() function a live purge uses, never a second path. Refusals come in two kinds and it matters which you got. BEFORE the load — an artifact whose digest does not match the manifest, artifacts from two different runs, an engagement-key fingerprint that is not this vault's, a schema head this checkout does not have, or a deployment that still holds documents and no acknowledgment phrase -- nothing was touched. AFTER the load — an audit chain that does not walk clean to its anchor, a constraint that will not validate once the escrow artifact is merged, or a ledger entry the replay could not apply — the databases have been replaced and the vault is left REFUSING TO SERVE on purpose, because it may hold versions the trail says were destroyed. THE ORDER OF THE VERB IS PART OF THE MECHANISM (ADR-0076 § 4.11): it PAUSES the webserver, restores, RESUMES the webserver and then RESTARTS it. Nothing may write into the vault while it is being replaced — a live webserver's disk sampler alone inserts into disk_space_samples through the reset identity sequence between the dump's COPY and its setval, which is a duplicate id, a primary key that cannot be built, and a restore that half-happened. The restart is not optional either: a merely-thawed webserver still holds the predicate key slots and the engagement keyring it read at startup, which the restore has just replaced.

  1. Run scripts/skykeep.sh restore RUN_ID with the same four backup settings exported that the backup used. The run id is the one scripts/skykeep.sh backup printed; there is no latest and no default, because the vault cannot list the mount it writes to.
  2. If the restore refuses saying the webserver is still running, it refused BEFORE touching the vault and nothing was destroyed. Let scripts/skykeep.sh restore do the pausing; it resumes the container afterwards whether the restore succeeded or failed. If you must quiesce by hand, use docker pause and NOT stop or down: a stopped webserver un-declares the deployment's posture (ADR-0042 reads the marker out of the running container) and its ownership by a test lane (ADR-0048), so stopping it trades this refusal for two others.
  3. Restoring into a deployment that still holds documents asks you to type an acknowledgment phrase on stdin. The prompt names it. There is no flag and no environment variable that supplies it: a mistype, an empty line and a script piping nothing all refuse, untouched.
  4. If the vault refuses to start afterwards saying a restore did not finish, the restore got past the load and stopped. Read its message, fix what it names, and run the same verb again. Do NOT clear the marker by hand — clearing it is the act of asserting the purge ledger was replayed.
  5. A restore carries no predicate-signing key, no workload-identity CA key and no engagement key. Install a key slot with scripts/skykeep.sh predicate_key before starting; the vault refuses to serve until you do, and re-enrol any workload identities.
  6. The restore drill — back up, purge a version, restore, confirm the purged version reads as nothing — is what gates permanent-store mode (ADR-0076 § 6.3). Permanent store must not be offered on a deployment whose drill has not passed.

a question takes tens of seconds to come back, and nothing else about the vault looks unhealthy

The meaning leg of a search keeps walking the vector index until it has enough passages the asker is actually allowed to read. Every candidate it steps over is one the caller's compartment predicate has to judge — the verified-identity check included — and that judgement happens once per candidate, inside the query. On a vault where a reader reaches a small share of a large corpus, most of the passages nearest a question belong to somebody else, so walking past them is where the time goes. Measured on a test lane (Postgres 17, pgvector 0.8.6): one pass that walked past roughly 2,000 unreachable candidates took about 50 seconds to come back. SKYKEEP_HNSW_MAX_SCAN_TUPLES bounds that walk, and it is the dial between how complete an answer is and how long it takes — not a safety ceiling that is free to raise.

  1. Confirm it is this before changing anything. Time the slow question, then set SKYKEEP_HNSW_ITERATIVE_SCAN=off in this deployment's environment file, recreate the webserver, and time the same question again. A question that comes back promptly with the scanning off and slowly with it on is this entry. One that is slow either way is not — go to the datastore sizing page, which is about the memory the index needs rather than the work one question does.
  2. Decide which failure this engagement would rather have, because both directions fail. Lowering SKYKEEP_HNSW_MAX_SCAN_TUPLES buys back the seconds and gives up results that exist inside the asker's own reach; the vault reports that as an under-return rather than hiding it, but a reported short answer is still a short answer. Raising it buys the results and spends the seconds. There is no value that is safe in both directions.
  3. Leaving SKYKEEP_HNSW_ITERATIVE_SCAN=off is the fastest setting and the worst answer: it is the shipped pgvector behaviour, one pass, and it is the value that makes a compartmented vault hand back fewer results than the asker could have seen. Use it to diagnose, not to finish.
  4. Do not reach for SKYKEEP_SEARCH_OVERFETCH_CAP here. It cannot be raised past 1000 whatever you write, because 1000 is pgvector's own ceiling on hnsw.ef_search and not this vault's, and it neither lengthens nor shortens this walk.
  5. Both settings are read from the environment, so editing the environment file changes nothing until the webserver container is recreated. A bad value is not guessed at: the vault refuses to start and names the setting.