Josep Piguillem
a32c937765
The Redis consumer used to leave a failed event pending and go on to the next ones, retrying it only at the next start. With Mongo down for a while, a later change of the same ticket was then projected first, and the version check dropped the earlier one for good (a status change could vanish from the read model), besides the read side lagging until a restart. Now a failed event is retried in place, waiting longer each time (up to 30 s), and the next one waits for it. Failures of the infrastructure (Mongo, Redis, the network: isTransientError) wait as long as it takes, so the read side catches up by itself; an event that keeps failing on its own goes, after 5 attempts, to <stream>:dead with its error, so it does not hold up the rest. GET /api/health reports the way from write to read (events waiting in the outbox, pending to be projected, set aside) and says `degraded`, still with 200, when something is stuck. Tried for real: with Mongo stopped, two status changes went in; the first failed to project with ECONNREFUSED, and once Mongo was back the ticket's history had both steps, in order, without restarting the API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
37 lines
1.2 KiB
TypeScript
37 lines
1.2 KiB
TypeScript
/** Errors of the infrastructure itself: waiting is the cure, not skipping. */
|
|
const TRANSIENT_NAMES =
|
|
/^(Mongo(Network|NetworkTimeout|ServerSelection|NotConnected|TopologyClosed|PoolCleared)Error|MaxRetriesPerRequestError)$/;
|
|
const TRANSIENT_CODES = new Set([
|
|
'ECONNREFUSED',
|
|
'ECONNRESET',
|
|
'ETIMEDOUT',
|
|
'EPIPE',
|
|
'EHOSTUNREACH',
|
|
'ENETUNREACH',
|
|
'ENOTFOUND',
|
|
'EAI_AGAIN',
|
|
]);
|
|
const TRANSIENT_MESSAGES =
|
|
/connection (is )?closed|connection refused|timed out|getaddrinfo|socket hang up/i;
|
|
|
|
/**
|
|
* Whether an error comes from a database or the network being down (and
|
|
* will go away by itself) rather than from the event or the code that
|
|
* handles it. Follows `cause`, as drivers wrap the original error.
|
|
*/
|
|
export function isTransientError(error: unknown): boolean {
|
|
for (let current = error, depth = 0; current && depth < 5; depth++) {
|
|
const e = current as {
|
|
name?: string;
|
|
code?: string;
|
|
message?: string;
|
|
cause?: unknown;
|
|
};
|
|
if (e.name && TRANSIENT_NAMES.test(e.name)) return true;
|
|
if (e.code && TRANSIENT_CODES.has(e.code)) return true;
|
|
if (e.message && TRANSIENT_MESSAGES.test(e.message)) return true;
|
|
current = e.cause;
|
|
}
|
|
return false;
|
|
}
|