Making Deploys Invisible

One morning this month, a user at one of the shelters running Shelter Sense sent 25 bug reports to our support inbox in about four minutes. They were all the same report. One of them, in capitals, said "PLEASE TEST YOUR WORK".

My first reaction was the unproductive one. My second reaction, about ten seconds later, was that they were completely right — the app was broken for them, badly, and it did not matter at all that the twenty-five emails were themselves a bug. Somebody trying to sign a dog back in from a shelter floor should not be the person discovering this.

What they were wrong about was the cause. They thought we had shipped an untested feature. We had not. We had shipped a perfectly ordinary deploy, and the deploy itself — the sixty-second window where a Fly machine is replaced and a Cloudflare Worker is swapped out — was the entire problem. Everything they saw was an artifact of that window, plus a connectivity system that turned a two-second blip into a ten-minute outage.

There is one more sentence in that pile of emails, and it is the most useful thing anyone sent me. Buried in one of the duplicates, almost as an aside: "Reloading doesn't always fix it."

That sentence is worth more than the other twenty-four emails combined, and I will come back to why. Working through all of it took days rather than hours, because it was not one bug. It was about a dozen, and several of them were only visible once I went looking for them specifically.

"Reloading doesn't always fix it" is not a complaint about a feature. It is a statement about a cache. Users describe mechanisms without knowing it.

01

Why 25 emails

Start with the noise, because the noise turned out to be a real bug with a real lesson in it.

Shelter Sense feedback goes: user submits a form, a server action pushes a message onto a Cloudflare Queue, the queue consumer in the worker calls the Nest API, and Nest sends the email. Perfectly reasonable. Here is the consumer config:

// apps/clients/apps/shelter-sense/wrangler.jsonc
"consumers": [
  {
    "queue": "shelter-sense-jobs",
    "max_batch_size": 10,
    "max_batch_timeout": 5,
    "max_retries": 3,
    "dead_letter_queue": "shelter-sense-jobs-dlq"
  }
]

max_retries: 3 means four delivery attempts — a number the consumer has to know about too, and does:

/** First delivery + `max_retries` from the wrangler.jsonc consumer config. */
const MAX_JOB_DELIVERY_ATTEMPTS = 4;

And the email is sent inside the consumer's HTTP request to Nest. So if Nest sends the email and then the response is lost — which is exactly what happens while Fly is rolling machines — the worker concludes delivery failed and redelivers. Nest sends the email again. Four times.

Then the message dead-letters, and we send ourselves a report about it. Then the user, staring at a form that had helpfully kept their text, clicks Send again. Five submissions, each worth up to four deliveries plus a dead-letter report, and you land somewhere around twenty-five.

The thing that made this possible was not the retry count. Retries are correct; the queue is doing its job. The problem was that nothing anywhere held an idempotency key, and — this is the embarrassing part — feedback was never persisted at all. There was no row. There was nothing to deduplicate against. The email was both the delivery mechanism and the only record that a delivery had happened.

The fix is a Feedback model with a unique key, and the interesting design decision is what that key identifies:

model Feedback {
  id                 String    @id @default(cuid())
  /// Browser-generated, stable across retries of the same submission.
  clientSubmissionId String    @unique @map("client_submission_id")
  ...
  /// Non-null once some writer has claimed and sent the notification email.
  emailSentAt        DateTime? @map("email_sent_at") @db.Timestamptz(6)
}

clientSubmissionId is minted once per form open, in the browser, and reused across every retry. That is the whole trick. It identifies the user's intent — "this person wants to tell us this one thing" — rather than a delivery attempt. Every layer below it (server action, queue message, retry, HTTP call) inherits the same id, so the entire fan-out collapses onto one row. It is retired only on a confirmed success:

// feedback-dialog.component.tsx
// Only a confirmed success retires the submission id. Resetting on failure
// would defeat deduplication precisely when it is needed — the retry.
useEffect(() => {
  if (showSuccess) {
    resetIdentity();
  }
}, [showSuccess, resetIdentity]);

Then the send itself becomes a claim, not a send:

const claimResult = await this.prismaService.client.feedback.updateMany({
  where: { id: feedbackRecord.id, emailSentAt: null },
  data: { emailSentAt: new Date() },
});
if (claimResult.count === 0) {
  logger.info({ message: 'FeedbackEmailClaimedByAnotherDelivery', ... });
  return { emailSent: false };
}

A conditional update on emailSentAt: null, and only the winner sends. The ordering matters more than the mechanism: claim before sending, not after. It is tempting to send first and record afterwards, because that way you never mark something sent that wasn't. But the reverse loses the exact race it exists to prevent — two concurrent deliveries would both find the row unclaimed, and both would send. A claim you might have to reconcile beats a duplicate you can never un-send.

And one last detail I enjoy, because it is the kind of thing that only shows up in an incident. Our dead-letter email said:

A user's feedback email was never sent.

That was the opposite of the truth. The one incident it ever fired on was an incident where the same feedback had already been delivered four times. The email is sent inside the consumer's request, so the most common reason this job dead-letters is a response lost after the send succeeded. The wording now says what is actually knowable:

case 'SEND_FEEDBACK':
  return `A user's feedback job exhausted its retries. It may still have been
    delivered to ${SUPPORT_EMAIL_ADDRESS} — check the feedback record's
    emailSentAt before assuming it was lost.`;

An alert that confidently asserts the wrong thing is worse than no alert. It sends you looking for a lost message that is sitting in your inbox four times.

02

The app lied about being offline

Now the actual complaint. Users were seeing this banner while both the Worker and the Fly API were completely healthy:

Showing cached data captured just now.
Sign-outs and sign-ins made now will sync when you're back online.

This is not a cosmetic problem. server-down is not an advisory state in this app — it is a routing decision:

export function shouldRouteMutationThroughOfflineOutbox(
  onlineStatus: OnlineStatus,
): boolean {
  if (isSessionExpired(onlineStatus)) {
    return false;
  }
  return isConnectivityInterrupted(onlineStatus) || isNavigatorOffline();
}

When Shelter Sense believes it is offline it swaps the live walk list for an IndexedDB snapshot and writes real sign-ins into an IndexedDB outbox instead of sending them. Staff on the floor are then working against a frozen list and queueing writes, for no reason, while the server sits there answering requests perfectly happily.

There were three separate causes, all on the connectivity health probe, and they compounded.

No hysteresis. useHealthProbeLoop committed the status of every probe. The whole loop body ended in one line:

setStatus(mapProbeResult(probeResult));

One failed sample took the entire dashboard offline. Not two in a row, not a rolling window — one. On a 10-second poll, a single cold TCP handshake was enough to flip the whole app into offline mode. It is a gate now, and the asymmetry is deliberate:

const applyProbeResult = (probeResult: HealthProbeResult) => {
  if (probeResult !== 'server-down') {
    consecutiveServerDownCount = 0;
    commitStatus(mapProbeResult(probeResult));
    return;
  }

  consecutiveServerDownCount += 1;
  // Only a working connection is protected by the run requirement. Once
  // already disconnected, a `server-down` reading is a re-classification
  // (e.g. from `session-expired`) and applies at once.
  if (
    committedStatusRef.current === onlineStatus &&
    consecutiveServerDownCount < consecutiveFailuresBeforeDisconnect
  ) {
    return;
  }
  commitStatus(mapProbeResult(probeResult));
};

Demotion off online takes two consecutive failures; promotion back still takes one. Being wrong about "you're online" costs a retry. Being wrong about "you're offline" costs a shelter a morning of divergent data. navigator.onLine === false keeps its immediate flip, because that is a local fact rather than a network sample.

A 4-second probe timeout. The endpoint itself is trivial, but the request crosses browser → Worker → Fly proxy → Nest. I measured it against production rather than guessing: p50 around 190ms once warm, and 10 to 22 seconds on first contact over a cold connection. A 4s timeout does not measure whether the server is up. It measures whether the connection is already warm.

- export const DEFAULT_HEALTH_PROBE_TIMEOUT_MS = 4_000;
+ export const DEFAULT_HEALTH_PROBE_TIMEOUT_MS = 10_000;
+ export const ONLINE_STATUS_CONSECUTIVE_FAILURES_BEFORE_DISCONNECT = 2;

Those two constants have to move together. A longer timeout on a one-strike system only makes the false positive slower; a two-strike rule with a 4s timeout just needs two cold probes instead of one. Neither is a fix on its own, which is a good sign you are looking at one bug rather than two.

There was a third contributor in the same loop that I only noticed while writing tests for the first two: nothing read visibilityState. A backgrounded tab kept probing under browser timer throttling, generating failures that said nothing whatsoever about the network. Polling pauses while hidden now, discards in-flight probes, and takes a fresh reading on becoming visible rather than trusting a sample from before the laptop lid closed.

The probe endpoint was auth-gated, and the auth cache was silently switched off. This one is my favourite, because nothing was broken in the sense of throwing an error.

The health endpoint is authenticated on purpose, so anonymous internet noise cannot poll us:

@Authentication({ type: 'private' })
@Controller('offline-sync')
export class OfflineSyncController {
  @Get('/health')
  @Header('Cache-Control', 'no-store')
  health(): OfflineHealthResponse {
    return this.offlineSyncService.health();
  }
}

Fine. Except UniversalCacheService in our Nest libraries is injected with @Optional(), and it is only provided when an app enables the cache module — which hard-requires an explicit Redis connection. Shelter Sense has no Redis; its read cache is in-process, so the root module says exactly what you would want it to say:

export const RootModule = createStandardRedwoodRootModule({
  ...
  enableCache: false,

Which means the service was simply absent, and every call site in the auth guard looked like this:

await this.cacheService?.get<IsTokenValidResponseDto>(cacheKey);

Optional chaining on an undefined service. Every get was undefined. Every set was a no-op. No error, no warning, no log line — just a cache that had never once returned a hit in production. Which meant every request carrying a cookie made a live round trip to an external auth server, including every 10-second health probe, and including every single bridge call behind a server-component render. One dashboard page load fanned out into one internet hop per bridge call, each holding one of a machine's 50 concurrency slots for the duration.

That is how a healthy server ends up reporting itself unreachable: it was busy answering its own health check.

The fix is a process-local fallback store used only when no shared cache is configured, and it is deliberately not a general-purpose cache. TTLs are clamped to 60 seconds, far below the 5 minutes and 2 hours the call sites ask for, because Redis is shared and a logout's del reaches every instance while a per-process map cannot. A token invalidated on one machine would otherwise keep working on another for the full TTL. Sixty seconds bounds that while still collapsing the burst of validations a single page render produces, which is where essentially all of the benefit was anyway.

(Two existing service specs started failing on that change, because they reuse token strings across cases and were, for the first time in their lives, actually hitting a cache.)

The tell that all of this was a false positive was sitting in the banner text the whole time. "Captured just now." The snapshot hydrator only runs while status is online. If the snapshot was captured seconds ago, the server was demonstrably reachable seconds ago. The app was contradicting itself in a single sentence of UI copy, and it took days before anyone — me included — actually read it.

There was one more thing hiding underneath, found while writing tests for the others. The probe treated 401 as "session expired" and everything else as "server down". But our auth guard answers 401 only for missing credentials. A cookie that is present but rejected — expired, rotated, signed out on another device — comes back 403. That is the common case.

const SESSION_EXPIRED_HEALTH_STATUSES: readonly number[] = [401, 403];

Before that one-line list, an expired session was reported to the user as a server outage: they saw "Server unreachable" instead of the sign-in gate, and their saves queued into an outbox against a session that could only ever reject them. We had already audited for and fixed that exact failure once. It walked back in through a different status code.

03

The toast storm, and a test that encoded the bug it was named after

One user described the experience as "easily 1 every 10 seconds", which is a very precise bug report if you know that the health poll interval is 10 seconds.

We had a 45-second flap cooldown on the connectivity toast specifically to stop this. It did not work, and the reason is a single clause:

if (
  nowMs - lastDisconnectToastAtMs < CONNECTIVITY_TOAST_DISCONNECT_FLAP_COOLDOWN_MS &&
  fromStatus !== 'online' &&
  !isDisconnectReclassification(fromStatus, toStatus)
) {
  return; // suppress
}

Read the middle clause carefully. The cooldown only applies when the previous status was not online. But every online -> server-down transition has fromStatus === 'online' by definition. That is what the transition is. So during a flap — the exact scenario the cooldown was written for — the condition could never be satisfied, and the cooldown never applied once.

I can even reconstruct the reasoning that produced that clause, because it sounds so sensible: the first drop after a healthy stretch always deserves a toast. Which is true! It is just that in a flap, every single cycle looks like the first drop after a healthy stretch.

It is a guard that is structurally incapable of firing in the situation it was built for. And that pattern turned out to be the theme of the whole investigation.

Worse, the -> online branch had no cooldown at all, so each flap cycle also produced a cheerful "Back online" toast — announcing recovery from an outage the user was never told about, at ten-second intervals, on top of whatever they were trying to do. (A separate report complained that the feedback prompt "popped and then disappeared before I even had a chance to type anything in the box, and I didn't click away from it." It was being dismissed by an ambient outside-click event. Guess what was appearing behind it every ten seconds.)

The fix is boring — cool down on elapsed time alone, track whether a disconnect was actually announced, and only announce recovery if it was, or if there is queued work the user should know about. The interesting part is what I found in the test suite. There was already a test called:

it('allows disconnect toast after cooldown when coming from online', () => {
  showConnectivityTransitionToast({
    ...baseArgs,
    fromStatus: 'online',
    toStatus: 'offline',
    lastDisconnectToastAtMs: 80_000,
    nowMs: 100_000,
  });

  expect(toastMock.warning).toHaveBeenCalled();
});

Its two timestamps are 20 seconds apart. The cooldown is 45 seconds. That test was asserting that a toast appears inside the cooldown window, and it passed — it passed only via the escape hatch it was named after. The test did not miss the bug. The test encoded the bug, gave it a reassuring name, and then guarded it against being fixed.

That exact 80_000 / 100_000 pair now lives in a test asserting the opposite (suppresses a repeat disconnect toast inside the cooldown from online), and the original test got the honest gap it always claimed to have.

I do not think there is a tool that catches that. The only thing that catches it is reading the numbers in your own test fixtures and asking whether they mean what the test title claims.

04

Why reloading didn't fix it

Back to the best sentence anyone sent me: "Reloading doesn't always fix it."

If a page is broken because of a transient server problem, reloading fixes it. Always. That is what transient means. So a user reporting that reloading does not reliably fix it is not describing a flaky backend at all — they are describing something between them and the backend that is answering the reload itself. They are describing a cache. They just do not have the vocabulary for it, and they should not need to.

Shelter Sense is a PWA with a service worker, because shelter WiFi is dreadful and the walk list has to keep working in a concrete kennel block. Authenticated dashboard HTML is served NetworkFirst:

const dashboardHtmlStrategy = new NetworkFirst({
  cacheName: getHtmlCacheName(),
  networkTimeoutSeconds: DASHBOARD_HTML_NETWORK_TIMEOUT_SECONDS,
  ...
});

That constant was 3. And here is the thing I had genuinely not internalised about networkTimeoutSeconds, which I have now written into the source so the next person does not have to learn it from an incident:

/**
 * The timeout is the single most user-visible number in this file. When the
 * network does not answer within it, serwist resolves the request from cache
 * instead — so this is not "how long before we give up", it is "how long before
 * we start serving a page that may be a day old".
 */
export const DASHBOARD_HTML_NETWORK_TIMEOUT_SECONDS = 8;

networkTimeoutSeconds is not a giving-up threshold. It is a switchover threshold. Below it, you get the network. Above it, you get a document out of the cache with a 24-hour max age — silently, with no indication that anything unusual happened. Combine that with the cold-connection latency I measured earlier (10 to 22 seconds), and a routine reload during a deploy was being answered from a cache entry that could be a day old.

Which is precisely why a hard reload worked and an ordinary one did not. A hard reload bypasses the service worker entirely. The user had, without knowing it, discovered the exact boundary of the bug and reported it accurately in six words.

Eight seconds is comfortably above a healthy response and below the point where a person concludes the page is broken. It is deliberately not larger, because while genuinely offline this timeout is the only thing standing between the user and the cached page they actually need.

05

The framework was reloading the page out from under people

This is the one that made me wince the most, because the user-visible symptom is "I typed a long note about a dog and pressed Save and everything vanished."

RedwoodSDK's client transport ships a default handleResponse. Here it is, from rwsdk/dist/runtime/client/navigation.js:

if (!response.ok) {
  // Redirect to the current page (window.location) to show the error
  // This means the page that produced the error is called twice.
  abortPendingNavigation();
  window.location.href = window.location.href;
  return false;
}

A full page reload, on any non-ok response — and critically, that includes server actions, not just navigations. So during the handful of seconds when a Fly machine is being replaced, a user who clicked Save got a hard page reload: dialog gone, form state gone, everything they had typed gone, and no message of any kind explaining what happened. The comment is honest about the intent ("to show the error"), and for a navigation that is a reasonable call. For a mutation it is data loss.

It was also completely invisible to our own error handling, which is the part that kept it alive for so long. The transport returns nothing useful on that path, so our useFormAction hook saw neither a thrown error nor a result. The pending flag cleared and nothing rendered — the exact silent "Save does nothing" failure that hook exists to prevent, arriving through a door underneath it.

The override is short:

function handleResponse(response: Response): boolean {
  if (response.status >= 300 && response.status < 400) {
    return rwsdkHandleResponse(response);
  }

  if (!response.ok) {
    reportClientErrorToSentry(
      new Error(
        `Transport response ${response.status} for ${response.url || window.location.pathname}`,
      ),
      { errorBoundary: 'rwsdk-transport-response', ... },
    );
    return false;
  }

  return rwsdkHandleResponse(response);
}

Returning false hands control back to the caller's own error path, so the user keeps their work and sees a real message. Redirects still delegate to rwsdk, because those it handles correctly. And the failure now reaches Sentry naming a status and a URL, which — see the next section — is a recurring theme.

While in there I also turned on rwsdk's opt-in recovery for tabs left holding the previous deploy's bundle:

initClient({ onModuleNotFound: 'reloadWhenReady' });

That polls the current route until it serves a real 200 document before reloading, which is a genuinely nice design — a new build id can be live while the specific route the user needs still is not. It is sanctioned and, as far as I can tell, entirely undocumented; I found it by reading PR #1222.

It does not cover the common case, though, and the reason is worth writing down. reloadWhenReady's gate only accepts a dynamic-import TypeError. But the shape that actually happens on nearly every deploy is different: a browser sitting on deploy N receives an RSC payload from deploy N+1 that names a 'use client' component added by that deploy. The loaded bundle's lookup table has no entry for it, so loadModule throws a plain Error reading No module found for '…' in module lookup — raised before the try block that feeds the recovery gate (upstream issue #1279). Recovery never fires. React surfaces a blank page. Every deploy that adds a client component can do this.

So there is a small util that closes it from the React side, with the one guard that this kind of thing absolutely requires:

/**
 * A second reload cannot fix what the first one did not, so without a guard a
 * genuinely-missing module becomes an endless reload loop — considerably worse
 * than the blank page. One attempt per window.
 */
const STALE_BUILD_RELOAD_COOLDOWN_MS = 60_000;

Auto-reload recovery without a sentinel is how you turn a blank page into a browser that will not stop refreshing. Ask me how confident I am about that.

06

The misleading error that hid all of it

Here is what Sentry had been showing us this whole time:

SyntaxError: Unexpected end of JSON input
  at index.mjs:442

For months I had filed that mentally as "something flaky in the bridge, probably a truncated payload, look into it when it gets worse". It was not a JSON bug. It was every deploy-window outage we have ever had, wearing a disguise.

callAppService — the single function every server component and server action in the app uses to reach Nest — called response.json() without ever checking response.ok. That is it. That is the whole bug.

When Fly returns a 502 with an HTML error page, response.json() throws SyntaxError: Unexpected token '<'. When a response is cut off mid-stream because the machine went away, it throws Unexpected end of JSON input. Neither error mentions the status code. Neither mentions which handler was being called. Both name a line inside a bundled file. So a routine machine roll arrived in Sentry looking like a parsing bug in our own code, and got triaged accordingly — for months.

It now reads the body as text first, and there is no bare response.json() left anywhere in the file:

const rawResponseBody = await response.text();

if (!response.ok) {
  const bridgeUnavailableError = new AppServiceUnavailableError(
    methodName,
    response.status,
    rawResponseBody,
  );
  reportCallAppServiceError(bridgeUnavailableError, methodName);
  throw bridgeUnavailableError;
}

AppServiceUnavailableError carries the status, the method name, and a 200-character body preview, so the Sentry title is now App service bridge call "getAnimalManagementPageData" failed with status 502: <!DOCTYPE html>… — which is a sentence you can act on. It also validates the response shape, so a 2xx of the wrong form cannot throw a TypeError while reaching for bridgeResponse.error.message and bury the real response underneath a second, bogus error.

The lesson is not "check response.ok", which everybody already knows. The lesson is that an error message that does not name the transport failure will be misfiled as an application bug indefinitely, and you will not notice, because it is a real error in a real file and it looks completely plausible.

07

And nobody could see any of it

There is a reason this all took days and not hours. Every Worker frame reaching Sentry was unmapped. index.mjs:754:143254 was as good as it got — a column number in the hundred-thousands, which is what a minified megabyte-wide line looks like from the outside.

The cause is genuinely fun, in a "I cannot believe I found this" way. rwsdk builds the worker environment twice: a worker pass emits dist/worker/index.mjs, then a linker pass re-bundles that output together with the SSR bridge and overwrites it. Only the linker output ships. Sentry's bundler plugin decides whether to inject a debug ID by calling hasExistingDebugID(), which inspects only the first 6000 bytes of a bundle (and the last 500). In the final worker bundle, the inlined SSR bridge's debug-ID snippet lands at around byte 1651.

So the plugin looked at the deployed bundle, found an existing debug ID within its 6000-byte window, and concluded its work was already done. Except that ID belonged to an intermediate build. Sentry then dutifully resolved it to a 335-line artifact, went looking for line 754, did not find it, and gave up. Silently. Forever.

Fixing it took two changes. The first is scoping the plugin's hooks to bundles that actually ship, which leaves the intermediates untagged so the real bundle gets a real ID:

const shipsOutput = (environmentName: string | undefined): boolean => {
  if (environmentName === 'client') {
    return true;
  }
  if (environmentName === 'worker') {
    return process.env.RWSDK_BUILD_PASS === 'linker';
  }
  return false;
};

The second is that the intermediates' own source maps were never fed to the linker at all, so the final map listed exactly two sources — ssr_bridge.js and index.js — and every symbolicated frame landed on a megabyte-wide generated line instead of the .tsx it came from. Rolldown will not read a //# sourceMappingURL comment off disk by itself; returning map from a load hook is the only way to make it compose. Chaining them takes the source list from 2 entries to 3802, of which 1603 are real app and lib files, with sourcesContent.

I want to flag the general shape here, because it is the second time this exact structure showed up in one investigation: a tool that inspects a prefix of a file and makes a global decision from it. hasExistingDebugID() reading 6000 bytes is not wrong, exactly — it is a reasonable heuristic that is silently invalidated by a build pipeline that concatenates two bundles. There is no error. There is just a stack trace that never resolves, and an engineer who assumes the frames are simply unmappable.

08

PID 1, and a health check that always said ok

At this point I had fixed a lot of symptoms of the deploy window. It was time to ask why the deploy window was violent in the first place.

Nothing in the entire workspace handled a shutdown signal. Not one process.on('SIGTERM'). And the runtime image uses exec-form CMD:

CMD [ "node", "server.js" ]

Exec form means node is PID 1. And PID 1 does not get the kernel's default signal dispositions — it ignores any signal it has installed no handler for. So Fly's kill_signal did precisely nothing. The machine sat out the full 30-second kill_timeout, doing nothing, and was then SIGKILLed with every in-flight request severed mid-response.

That is a big chunk of why our deploys were visible at all. Users lost whatever they were doing at the exact moment their machine was replaced, every single time.

registerGracefulShutdown now lives in our shared Nest utilities, so every Fly-deployed app gets it. Three details in it are load-bearing.

It handles both signals:

const SHUTDOWN_SIGNALS = ['SIGTERM', 'SIGINT'] as const;

Our fly.toml sends SIGINT while the platform default elsewhere is SIGTERM. Handling only the one we configure today would regress silently the day that config changes, and the failure would look exactly like the bug we just fixed.

It flips readiness first, waits for the proxy to observe the failing health check and stop routing, and then closes:

isDraining = true;
// Health checks now fail; give the proxy time to notice before we stop
// accepting. Skipped when the delay is zero (dev/tests).
if (drainDelayMs > 0) {
  await waitFor(drainDelayMs);
}
await Promise.race([app.close(), waitFor(shutdownTimeoutMs).then(...)]);

The order is the entire point. Closing first strands the requests the proxy is still sending you, which is the thing we set out to stop. And it calls enableShutdownHooks(), without which Nest never runs the hooks Prisma disconnects in.

The numbers are picked to nest inside each other, which I recommend writing down somewhere: the health check runs every 10s, the drain delay is 15s (about 1.5 cycles, so the proxy definitely sees a failure), the close timeout is 10s, total 25s — inside Fly's kill_timeout = '30s'.

Which brings us to the health check itself. /api/healthcheck returned this:

return { status: 'ok' };

Unconditionally. No database check, no draining flag, nothing. That made it useless at both ends of a rollout. A machine passed the moment Nest bound the port, so Fly routed live traffic to it before it could reach the database. And nothing ever reported unhealthy, so the proxy happily kept feeding requests to a machine that was thirty seconds from being killed.

It is a real readiness probe now:

async isReady(): Promise<boolean> {
  if (isApplicationDraining()) {
    return false;
  }
  if (this.hasVerifiedDatabaseAtBoot) {
    return true;
  }
  try {
    await this.prismaService.client.$queryRaw`SELECT 1`;
    this.hasVerifiedDatabaseAtBoot = true;
    return true;
  } catch {
    return false;
  }
}

503 while draining, and 503 until the database has been reached at least once. Note that the result is cached from boot rather than re-run per request: the check fires every 10 seconds per machine, and running a query each time adds load exactly when a cold machine can least absorb it. The lazy retry is there so a machine that booted before the database was reachable can still recover instead of being stuck unhealthy forever.

Only then was it safe to change this:

[deploy]
  release_command = 'npx prisma migrate deploy'
  strategy = 'bluegreen'   # was 'canary'
  wait_timeout = '5m'

Canary destroys and replaces machines one at a time, so mid-rollout the app runs on half its capacity against a hard limit of 50 concurrent requests per machine — precisely when a deploy is also generating a wave of reconnects and reloads. Bluegreen boots a full second set, waits for it to pass health checks, then cuts over, so the old machines keep serving until the new ones are actually ready.

But bluegreen is only as good as your definition of "ready", and before the readiness probe, "passed health checks" meant nothing more than "bound the port". Flipping that strategy first would have made things worse, with more confidence. The one-line config change was the easy part; earning the right to make it was the work.

And one last deploy-window bug, which I only went looking for because I was already elbow-deep in the build output for a different reason. rwsdk's double build (yes, the same double build as the source maps) means dist/worker/index.mjs contains two independent registrations of rwsdk's request-info AsyncLocalStorage — you can verify it, grep -c requestInfoStore returns 2, against two different registry objects. Vite's resolve.dedupe cannot merge them, because they come from separate rollup passes. rwsdk only ever populates one of them, so on the other code path our cookie lookup always fell through to a "last-resort" fallback... which was a single mutable string on globalThis.

A single mutable string, serving every concurrently in-flight request in the isolate, overwritten on each inbound fetch. Two overlapping requests could send one user's cookie on the other's bridge call, or send none at all and render an authenticated user as signed out. And bridge calls are slowest exactly when overlap is most likely — during a deploy. It is an AsyncLocalStorage now, with the store itself keyed on globalThis so both duplicated copies of the module resolve the same instance.

The comment above it used to say the global was a fallback that "must never be preferred when ALS is available". Which was true, and also completely beside the point: in the deployed bundle it was not a fallback at all. It was the only path on one of the two copies.

09

The one we found by accident

While auditing the build pipeline for deploy-related problems, I grepped the built client bundle for something unrelated and found a 64-character service key sitting in it. Verbatim. In dist/client/assets/v2/client-*.js. A file served to every browser that has ever loaded the app.

The mechanism is short enough to state in one sentence: Vite's define has no per-environment scoping. Our vite.config.mts carried a define entry for process.env.WILK_SOFT_SERVICE_KEY, and define substitutes into the client build as readily as the worker build. Rolldown also replaces the bracket form (process.env['WILK_SOFT_SERVICE_KEY']) just as happily as the dot form, contrary to a note elsewhere in that same config file.

Here is the part I want to dwell on, because it is the single most instructive thing I learned in the whole investigation. The read site was guarded:

function isBrowserRuntime(): boolean {
  return typeof window !== 'undefined' && typeof document !== 'undefined';
}

That guard is correct. It works. It does exactly what it says. And it is completely irrelevant, because it is a runtime check, and the problem is a literal already sitting in the file. Nobody needs to execute your code to read a string out of it. I had looked at that guard more than once and felt reassured by it.

What that key granted anyone who opened devtools was not small: it makes /api/internal/app-service skip session auth entirely on an internet-facing origin, reaching every registered bridge handler for any tenant; it gates the internal SyncedState publish endpoint; and it is the HMAC root for the per-org socket connect tokens.

The key is now resolved from the Worker's secret binding at runtime, so there is nothing left for a build step to inline:

async function resolveBridgeServiceKey(): Promise<string | undefined> {
  if (isBrowserRuntime()) {
    return undefined;
  }
  const workerEnv = await getWorkerEnv();
  const fromWorkerBinding = workerEnv?.['WILK_SOFT_SERVICE_KEY'];
  ...
}

In the browser the binding cannot resolve, so the bridge falls back to cookie session auth — which is what a browser caller should have been doing all along.

And then the guard, which is the actual fix. We already had a build-time guard for leaked env. It scanned dist/client for the names — process.env.WILK_SOFT_SERVICE_KEY and friends. That guard structurally could not catch this, and it is worth being precise about why: a defined secret leaves behind a bare string literal with no process.env text to match. The name-based scan was looking for the fingerprint of a leak that had not happened, while the leak that had happened left no fingerprint at all.

The guard now scans dist/client/** for the values of WILK_SOFT_SERVICE_KEY, SENTRY_AUTH_TOKEN, DATABASE_URL and STRIPE_SECRET_KEY. It also had to be taught to load .env, without which every secret would have been undefined and the check would have silently passed on precisely the build it exists to reject. I probed it in both directions: it fails on the old build, and passes on the rebuilt one.

That both-directions habit is not optional any more. It is the only reason I trust any of the guards I wrote during this.

10

What actually connects these

Reading back over the list, three things run through nearly all of it.

The best bug report was a sentence about a mechanism. "Reloading doesn't always fix it" is not feedback about a feature. It is a claim that something is intercepting the reload, and the only thing that intercepts a reload is a cache. The twenty-four emails that said the app was broken told me nothing I could act on. The one that described when it was broken pointed straight at networkTimeoutSeconds. Users describe mechanisms constantly without knowing that is what they are doing, and the useful skill is not "listen to your users" in the greeting-card sense — it is noticing which of their sentences is a mechanism.

Half of these were guards that could not fire. A flap cooldown with a clause that is false by definition during a flap. A test asserting the bug it was named after. A health check that returned ok unconditionally. A secret scanner that only checked names. A runtime browser guard around a value inlined at build time. Every one of these was written by someone (me) who then felt protected. None of them had ever been probed in the failing direction. A guard you have only ever seen pass is not a guard; it is a comment with a test runner attached.

The one-line fixes and the architectural ones mattered about equally. networkTimeoutSeconds: 3 -> 8 is a one-character diff and it is possibly the single most user-visible change in the whole batch. registerGracefulShutdown plus a real readiness probe plus bluegreen is a proper piece of infrastructure work, and it matters just as much. There is a temptation to rank these — to treat the config constants as trivia and the architecture as the real work. They were the same size to the person on the shelter floor.

11

And they were right, mostly

I want to close on the "PLEASE TEST YOUR WORK" email, because I think the honest read is more interesting than either of the easy ones.

They were wrong about the cause. This was not shipped-untested code; the features worked. Almost everything they hit was a deploy-window or connectivity artifact, and a couple of them (the double-registered ALS, the 6000-byte debug-ID window) are the sort of thing you find by reading a bundler's output at 11pm, not by writing another test.

But "we tested it, your diagnosis is wrong" is a garbage response, and it would also have been false in the way that matters. Our tests did not catch any of this, because our tests never ran during a deploy. The one test that touched the flap cooldown was actively protecting the bug. And the user's actual claim — the app is unreliable, and it wastes my time on a shelter floor — was completely accurate. Twenty-five emails is not an overreaction to that. It is what happens when the Send button appears to do nothing five times in a row.

Both things are true. The diagnosis was wrong and the experience was genuinely bad, and only one of those is the user's job to get right.

The deploy that shipped all of this went out on a Tuesday afternoon. Nobody noticed. That was the entire goal.

Key Takeaways

• "Deploy day breaks everything" was not one bug. It was about a dozen — an idempotency-free queue, a one-strike health probe, a 3s service-worker cache switchover, a framework that hard-reloaded on any 5xx, a bridge that never checked response.ok, unmapped source maps, and no signal handler anywhere.

• Idempotency keys should identify intent, not delivery. A clientSubmissionId minted once per form open collapses every retry, re-click and redelivery onto one row. And claim before you send — the reverse loses the race it exists to prevent.

• networkTimeoutSeconds is a switchover threshold, not a give-up threshold. Below it you serve the network; above it you silently serve a page that may be 24 hours old. That is why hard reload worked and ordinary reload did not.

• Probe every guard in both directions. A cooldown that cannot apply during a flap, a health check that always returns ok, a test named after the bug it encodes, and a secret scanner that only checks names all pass forever without ever protecting anything.

• A build-time leak cannot be fixed by a runtime guard. Vite's define has no per-environment scoping, so a secret becomes a bare literal in the client bundle — with no process.env text left for a name-based scanner to find.

• Node as PID 1 ignores signals it has no handler for. Exec-form CMD plus no SIGTERM handler means your machine sits out its whole kill_timeout and gets SIGKILLed with requests in flight. Fix that and the readiness probe before you touch the deploy strategy.

Shelter Sense, when it is doing its job, is boring

A live walk list, enrichment tracking, and reporting your whole team can rely on — including during a deploy.

Explore Shelter SenseSee pricing

The architecture all of this runs on is covered in Leaving Next.js for RedwoodSDK, and the reason any of it exists is in The Origin of Shelter Sense.