Chapter 8: production readiness

DocDuck works. It also stores every policy in a JavaScript object that vanishes on restart, runs one process, and reports nothing. This last chapter takes it to production: a database-backed adapter, production mode, cache invalidation across instances, metrics, and a health probe.

What you should already have

src/roles.ts, src/policies.ts, src/documents.ts, src/access.ts (adapter, hooks, engine), src/server.ts from chapter 6, and a client from chapter 7. Everything below edits src/access.ts and adds two routes to src/server.ts.

Learning goals

• Swap the memory adapter for a persistent one without touching a policy.
• Understand what production mode changes and what it costs you.
• Keep caches correct across multiple instances.
• Emit latency and allow/deny metrics, and expose a health probe.
• Run through the pre-deploy checklist.

What a production deployment looks like

Each instance keeps its own caches and reads through to the same database on a miss. When one instance writes a policy through the admin API, it broadcasts an invalidation over Redis so the others drop their stale copies instead of serving them until their TTL expires.

Step 1: a persistent adapter

The engine talks to an IamAdapter.IAdapter - policy store, role store, subject store. Every adapter implements the same interface, so swapping one is a single-line change and no policy or role definition moves.

import { PrismaClient } from '@prisma/client'
import { IamPrismaAdapter } from '@gentleduck/iam/adapters/prisma'

const prisma = new PrismaClient()
const adapter = new IamPrismaAdapter(prisma)

The Prisma adapter expects four models - accessPolicy, accessRole, accessAssignment, accessSubjectAttr. The adapters page lists the schema; IamDrizzleAdapter, IamRedisAdapter, IamFileAdapter and IamHttpAdapter cover the other shapes, and IamAdapter.IAdapter is the contract if you write your own.

Two things follow from persistence:

• IamMemory.IInit.assignments is gone. The seed block from chapter 5 becomes a migration or a seed script that writes rows once, not a promise your app awaits at boot.
• Every read is now I/O. adapterTimeoutMs (default 5000) bounds it; an over-budget read aborts and the check fails closed rather than hanging the request.

Step 2: production mode

export const engine = new IamEngine({
  adapter,
  mode: process.env.NODE_ENV === 'production' ? 'production' : 'development',
  scopeMode: 'hierarchical',
  cacheTTL: 60,
  maxCacheSize: 5_000,
  adapterTimeoutMs: 3_000,
  maxConcurrentSubjectLoads: 200,
  hooks, // beforeEvaluate, onDeny, onError, onMetrics - unchanged since chapter 4
})

What production mode changes:

• check() and permissions() return plain booleans instead of decision objects. can() returns boolean in both modes, so guards and middleware are unaffected.
• explain() throws - explain() is not available in production mode. Keep a staging deployment in development mode for debugging.
• Evaluation runs against a compiled lookup table plus a role bitmask, rebuilt lazily and guarded by a generation counter so a concurrent invalidation cannot install a stale table.

Two configurations the constructor refuses outright, both at construction rather than at request time:

• mode: 'production' with policyCombine: 'first-applicable' - the fast path cannot represent it.
• defaultEffect: 'allow' without allowFailOpen: true - fail-open is a footgun. Even with the opt-in the engine logs a loud startup warning, on purpose.

Step 3: invalidation across instances

With one process, engine.cache.invalidatePolicies() after an admin write is enough. With several, instance B keeps serving its cached copy until the TTL lapses. Wire a broadcaster:

import { createIamRedisInvalidator } from '@gentleduck/iam/invalidators/redis'

const invalidator = createIamRedisInvalidator({
  client: { publish: (ch, msg) => pub.publish(ch, msg), subscribe: (ch, h) => sub.subscribe(ch, h) },
  secret: process.env.IAM_INVALIDATE_SECRET,
  tenantId: process.env.TENANT_SLUG,
  onPublishError: (err, channel) => alerting.warn('iam invalidate publish failed', { err, channel }),
})

export const engine = new IamEngine({ adapter, invalidator /* ...the rest */ })

tenantId is the multi-tenant shortcut: it appends :tenant:<id> to the channel so tenant A's revoke cannot wipe tenant B's caches. It is validated against /^[A-Za-z0-9_-]{1,64}$/, so an attacker-supplied tenant slug cannot inject a channel name.

onPublishError matters more than it looks. A publish failure is non-fatal locally - the writing instance already dropped its own caches - but the broadcast is lost, so the other instances stay stale until TTL. A quiet Redis outage is a silent correctness drift; alert on it.

The invalidation facets, local or broadcast:

Each takes { broadcast?: boolean }. Call engine.dispose() on shutdown to unsubscribe.

Step 4: metrics and health

import { iamCreateMetricsAggregator } from '@gentleduck/iam/observability/metrics'

export const metrics = iamCreateMetricsAggregator({ sampleSize: 2_000 })

export const engine = new IamEngine({
  adapter,
  mode: process.env.NODE_ENV === 'production' ? 'production' : 'development',
  scopeMode: 'hierarchical',
  // ...the rest of the step 2 config
  hooks: {
    ...hooks, // beforeEvaluate stays: drop it and every ownership rule reads an empty attribute bag
    onMetrics: metrics.record,
    onDeny: (request, decision) => auditLog.write({ subject: request.subject.id, reason: decision.reason }),
    onError: (err, request) => logger.error({ err, subject: request.subject.id }, 'iam check failed'),
    onMutation: (event) => auditLog.write(event),
  },
})

metrics.record closes over its own state and never reads this, so passing it unbound is safe. onMutation is the other half of the audit trail: onDeny records refused requests, onMutation records the writes that changed who could ask.

import { engine, metrics } from './access'

app.get('/metrics', (_req, res) => res.json({ ...metrics.snapshot(), caches: engine.stats.get() }))

app.get('/healthz', async (_req, res) => {
  const health = await engine.healthCheck()
  res.status(health.ok ? 200 : 503).json(health)
})

await engine.preload({ validator: true })

metrics.snapshot() returns total, allow, deny, failOpen, p50, p95, p99, max, and samples over a rolling ring buffer (default 1000 samples, evicting oldest). reset() zeroes it.

engine.stats.get() returns { hits, misses, size } for each of the five caches - policies, roles, rbacPolicy, mergedPolicies, subjects. Counters accumulate from construction; engine.stats.reset() zeroes them. A collapsing subjects hit rate usually means maxCacheSize is too small for your active user count.

engine.healthCheck() does one timed adapter round-trip (listPolicies) and returns { ok, adapter: 'ok' | 'fail', cacheHitRate, adapterLatencyMs, lastError? }. It is cheap enough for a readiness probe, and ok: false tells your orchestrator to pull the instance.

engine.preload() warms the merged-policy cache and builds the compiled table concurrently, so the first request after boot does not pay the full load-and-index cost. { validator: true } also loads the lazy 12 KB validator chunk up front - worth it on any instance that serves admin writes, skippable on read-only instances.

Step 5: validate before you ship

The validator is a separate entry point on purpose, so apps that never write policies do not pay for it:

import { validatePolicy, validateRoles } from '@gentleduck/iam/core/validate'

const result = validateRoles(roles)
if (!result.valid) {
  for (const issue of result.issues) console.error(`${issue.type} ${issue.code}: ${issue.message}`)
  process.exit(1)
}

IamValidate.IResult is { valid, issues }, where each issue carries a type of 'error' or 'warning' plus a code. Warnings do not set valid: false - decide in your own script whether to fail on them. The admin router runs the same validator before every write and throws on an error-level issue, so a bad policy never reaches the database through that path.

What just happened

None of your policies, roles, or checks changed in this chapter. Only what surrounds them did - where the data lives, how stale it can get, and how you find out when it breaks. That is the point of the adapter interface.

The failure direction is consistent throughout. An adapter timeout, a shed subject load, a policy-load error, an over-cap read: all of them deny, fire onError, and answer 403. The only way to get an allow is for a rule to actually fire.

Pre-deploy checklist

Try it

1. Point DocDuck at Postgres through IamPrismaAdapter, restart the process, and confirm Bob's scoped roles survive the restart.
2. Set mode: 'production' and call engine.explain(...). Confirm it throws, then confirm engine.can(...) still returns the same booleans as in development.
3. Construct an engine with defaultEffect: 'allow' and no allowFailOpen. Read the error - that is the guard rail working.
4. Run two instances against one database, change a policy through the admin router on instance A with an invalidator wired, and confirm instance B sees the new decision immediately. Remove the invalidator and watch it lag by cacheTTL.
5. Hit /metrics after some traffic and check that failOpen is 0 and the subjects cache hit rate is high. Drop maxCacheSize to 2 and watch it collapse.
6. Stop the database and hit /healthz. Confirm ok: false, adapter: 'fail', and a populated lastError, and that checks deny rather than hang.

See also

• Production guide - the operational reference this chapter condenses
• Engine modes - what production mode compiles and what it gives up
• Caching - the five caches, TTLs, and invalidation semantics
• Redis invalidator - envelope format, signing, replay window
• Metrics - snapshot fields and exporter wiring
• Adapters and Prisma adapter - schemas and trade-offs
• Validation - every issue code
• Troubleshooting - symptoms to causes