Designing Resilient Systems Through Failure

A reliable system is not one that never fails. It is one that can detect failures, limit their impact, and recover in a predictable way.

This distinction matters whenever an application depends on a database, external API, message queue, or another service.

Assume dependencies can fail

Imagine an application that requests data from an external API. The API might become slow, return errors, or stop responding altogether.

Without safeguards, a single dependency can consume application resources and affect unrelated requests.

Use timeouts

Every network request should have a reasonable timeout. Otherwise, an operation may wait longer than the application can tolerate.

async function fetchProfile(url) {
  const response = await fetch(url, {
    signal: AbortSignal.timeout(5000),
  });

  if (!response.ok) {
    throw new Error(`Request failed: ${response.status}`);
  }

  return response.json();
}

This example uses a five-second timeout. Choose an appropriate value based on the service’s expected latency and the application’s requirements.

Retry carefully

Retries can help with temporary failures, but indiscriminate retries can make an overloaded service even less healthy.

A retry policy should consider:

• Whether the error is transient.
• Whether the operation is safe to repeat.
• How many attempts are allowed.
• Whether exponential backoff and jitter are needed.
• Whether the request has an overall deadline.

Choose a failure strategy

Technique	Purpose
Timeout	Limits how long an operation waits
Retry	Recovers from some temporary failures
Circuit breaker	Temporarily stops calls to an unhealthy dependency
Bulkhead	Limits how much one failure can affect other work
Graceful degradation	Preserves essential functionality when optional features fail

These patterns solve different problems. They should be selected according to the system’s failure modes rather than applied indiscriminately.

Understand the system boundary

A timeout at one service does not automatically cancel work across the entire system. Similarly, a successful HTTP response does not necessarily mean a downstream operation completed correctly.

For that reason, reliability requires observability as well as defensive code.

Track latency, error rates, saturation, and relevant business outcomes. Correlate logs across service boundaries where possible, and make sure alerts represent actionable failures.

Review the recovery path

A useful reliability review asks:

1. What happens when a dependency becomes slow?
2. What happens when it returns an error?
3. Can retries duplicate a write or payment?
4. Does an optional dependency block a critical feature?
5. How will an operator recognize and investigate the failure?

Answering these questions before an incident makes the system easier to operate under pressure.

Conclusion

Resilience comes from understanding how failures propagate and deliberately limiting their effects. Timeouts, retries, isolation, observability, and graceful degradation work together to make a system more predictable.