API Integration Retry Policies and Strategies

Every API integration fails sometimes. Networks drop packets, DNS resolution stalls, TLS handshakes timeout, load balancers rotate targets, databases get busy, queues fill, SaaS providers throttle traffic, and deployments temporarily remove capacity. A retry policy is the set of rules that decides whether the client should try again, when it should try again, how many times it should try, and how it prevents duplicated side effects.

For business systems, retry behaviour is not just a technical detail. It affects payment capture, booking confirmation, CRM lead creation, ERP order sync, ecommerce fulfilment, healthcare workflows, reporting pipelines, and customer notifications. This guide explains the retry strategies VaniTech usually considers when designing API integrations, with sample code and practical policy defaults.

What Is an API Retry Policy?

An API retry policy is a structured rule set for repeating failed outbound calls. It normally defines retryable failures, maximum attempts, per-attempt timeout, delay algorithm, maximum delay, total elapsed time, jitter, idempotency requirements, logging, metrics, and fallback behaviour.

Microsoft's Azure transient fault guidance says applications should use a retry strategy that fits the requirement, including how many times to retry, the delay between attempts, and the action after failure. It also recommends using built-in retry mechanisms where they fit and warns against aggressive strategies that can worsen overload. AWS Well-Architected guidance similarly recommends exponential backoff with jitter and a maximum retry value, while warning about retries at multiple layers, non-idempotent calls, and untested custom mechanisms.

The most important distinction is this: retrying is not error handling by itself. It is one tool inside a resilience design that also needs timeouts, circuit breakers, rate limits, idempotency, queues, dead-letter handling, observability, and clear ownership.

Retryable vs Non-Retryable Failures

Retry only when a repeated request has a reasonable chance of succeeding. A retry policy should normally treat these as candidates:

• Network faults: connection reset, DNS blip, socket timeout, TCP disconnect.
• Timeouts: request timed out before a response was received, provided the operation is safe to repeat.
• HTTP 408: request timeout.
• HTTP 429: too many requests or rate limiting, especially when the response includes Retry-After.
• HTTP 500, 502, 503, 504: selected server-side or gateway failures.

Do not normally retry validation failures, authentication failures, authorization failures, missing resources, unsupported media types, schema errors, malformed requests, or business-rule rejections. Repeating a bad request usually creates noise, cost, and load without improving reliability.

Retry Strategies Defined

The Retry Policy Template

A practical retry policy should be written down, not scattered through helper methods. Use this template for each dependency:

• Dependency: payment API, CRM API, ERP API, search API, internal service, queue consumer, database client.
• Operation: read customer, create order, update stock, submit payment, sync invoice, poll job status.
• Safety: idempotent, conditionally idempotent, non-idempotent, or protected by an idempotency key.
• Retryable signals: exception types, HTTP status codes, provider error codes, Retry-After, timeout type.
• Attempts: maximum retry attempts in addition to the first request.
• Per-attempt timeout: how long each request can run before being abandoned.
• Total budget: maximum elapsed time across all attempts.
• Delay strategy: immediate, fixed, incremental, exponential, exponential with jitter, or server-directed.
• Maximum delay: cap the wait between attempts.
• Fallback: fail request, enqueue, return cached data, show degraded response, open circuit, alert support.
• Observability: logs, metrics, traces, retry count, final failure, dependency name, idempotency key, correlation ID.

Suggested Starting Policies

TypeScript Sample: Fetch With Retry-After, Backoff and Jitter

This sample is deliberately small enough to audit. It retries only selected status codes, honours Retry-After, caps delay, adds jitter, and requires the caller to decide whether a mutating request is safe.

type RetryPolicy = {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  perAttemptTimeoutMs: number;
  retryableStatuses: number[];
};

const defaultPolicy: RetryPolicy = {
  maxAttempts: 4,
  baseDelayMs: 250,
  maxDelayMs: 5000,
  perAttemptTimeoutMs: 8000,
  retryableStatuses: [408, 429, 500, 502, 503, 504]
};

const sleep = (ms: number, signal?: AbortSignal) =>
  new Promise<void>((resolve, reject) => {
    const timer = setTimeout(resolve, ms);
    signal?.addEventListener('abort', () => {
      clearTimeout(timer);
      reject(signal.reason ?? new Error('Aborted'));
    }, { once: true });
  });

function parseRetryAfter(value: string | null): number | null {
  if (!value) return null;
  const seconds = Number(value);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
  const dateMs = Date.parse(value);
  if (Number.isFinite(dateMs)) return Math.max(0, dateMs - Date.now());
  return null;
}

function jitteredBackoff(attempt: number, policy: RetryPolicy): number {
  const exponential = Math.min(policy.maxDelayMs, policy.baseDelayMs * 2 ** attempt);
  return Math.floor(Math.random() * exponential); // full jitter
}

export async function fetchWithRetry(input: RequestInfo, init: RequestInit = {}, policy = defaultPolicy) {
  let lastError: unknown;

  for (let attempt = 0; attempt < policy.maxAttempts; attempt++) {
    const controller = new AbortController();
    const timeout = setTimeout(() => controller.abort(new Error('Attempt timed out')), policy.perAttemptTimeoutMs);

    try {
      const response = await fetch(input, { ...init, signal: controller.signal });
      clearTimeout(timeout);

      if (!policy.retryableStatuses.includes(response.status) || attempt === policy.maxAttempts - 1) {
        return response;
      }

      const serverDelay = parseRetryAfter(response.headers.get('retry-after'));
      const delay = Math.min(policy.maxDelayMs, serverDelay ?? jitteredBackoff(attempt, policy));
      await sleep(delay);
      continue;
    } catch (error) {
      clearTimeout(timeout);
      lastError = error;
      if (attempt === policy.maxAttempts - 1) throw error;
      await sleep(jitteredBackoff(attempt, policy));
    }
  }

  throw lastError;
}

// Mutating calls should include an idempotency key when the API supports one.
await fetchWithRetry('https://api.example.com/orders', {
  method: 'POST',
  headers: {
    'content-type': 'application/json',
    'idempotency-key': crypto.randomUUID()
  },
  body: JSON.stringify({ customerId: '123', sku: 'ABC', quantity: 1 })
});

This wrapper is not a complete production library. In production, add structured logging, metrics, tracing, retry budgets, provider-specific error codes, cancellation from the caller, and tests for timeout, retry-after, 429, 503, and duplicate write scenarios.

C# Sample: Polly Retry Strategy for HTTP APIs

Polly's retry strategy supports retry predicates, backoff type, jitter, maximum attempts, delay caps, and retry callbacks. The sample below retries transient HTTP outcomes and uses jittered exponential backoff.

using Polly;
using Polly.Retry;
using System.Net;

static ResiliencePipeline<HttpResponseMessage> BuildApiRetryPipeline(ILogger logger)
{
    return new ResiliencePipelineBuilder<HttpResponseMessage>()
        .AddRetry(new RetryStrategyOptions<HttpResponseMessage>
        {
            MaxRetryAttempts = 3,
            Delay = TimeSpan.FromMilliseconds(300),
            MaxDelay = TimeSpan.FromSeconds(5),
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true,
            ShouldHandle = args =>
            {
                if (args.Outcome.Exception is HttpRequestException or TaskCanceledException)
                {
                    return PredicateResult.True();
                }

                var status = args.Outcome.Result?.StatusCode;
                return status is HttpStatusCode.RequestTimeout
                    or (HttpStatusCode)429
                    or HttpStatusCode.InternalServerError
                    or HttpStatusCode.BadGateway
                    or HttpStatusCode.ServiceUnavailable
                    or HttpStatusCode.GatewayTimeout
                    ? PredicateResult.True()
                    : PredicateResult.False();
            },
            OnRetry = args =>
            {
                logger.LogWarning(
                    "Retrying API call. Attempt={Attempt} Delay={Delay} Outcome={Outcome}",
                    args.AttemptNumber + 1,
                    args.RetryDelay,
                    args.Outcome.Exception?.Message ?? args.Outcome.Result?.StatusCode.ToString());

                return ValueTask.CompletedTask;
            }
        })
        .Build();
}

public static async Task<HttpResponseMessage> SendWithRetry(
    HttpClient client,
    HttpRequestMessage request,
    ResiliencePipeline<HttpResponseMessage> pipeline,
    CancellationToken cancellationToken)
{
    return await pipeline.ExecuteAsync(
        async token => await client.SendAsync(request, token),
        cancellationToken);
}

For production .NET systems, also consider IHttpClientFactory, typed clients, dependency-specific policies, provider SDK defaults, and a separate timeout strategy. Polly's own documentation recommends dedicated strategies for different failure domains rather than one policy that handles unrelated work such as HTTP transport and JSON deserialization together.

Idempotency: The Rule That Makes Retries Safe

A retry can create two outcomes that look the same from the client but differ inside the provider. The server might receive a payment request, process it successfully, then fail before the response reaches the caller. If the caller retries without idempotency protection, it might charge the customer twice.

Google Cloud's retry guidance says request safety depends on both the response and the idempotency of the request. It classifies some operations as always idempotent, some as conditionally idempotent when preconditions are supplied, and some as never idempotent. Stripe's API documentation describes idempotency keys for safely retrying create or update requests; repeated requests with the same key return the first result rather than performing the action again.

Use these design patterns for mutating API integrations:

• Client-generated idempotency key: send a unique key with every business command, such as order creation or payment capture.
• Server-side dedupe store: store key, request hash, response status, response body, expiry time, and processing state.
• Preconditions: use ETags, generation checks, version numbers, or If-Match headers where the provider supports them.
• Natural business key: use a unique order number, invoice number, booking reference, or external transaction ID.
• Outbox pattern: commit business state and outbound integration events together, then dispatch reliably from the outbox.
• Inbox pattern: dedupe inbound webhook and message IDs before processing.

Retry-After and Rate-Limited APIs

HTTP 429 Too Many Requests indicates rate limiting. RFC 6585 says a 429 response may include Retry-After indicating how long to wait before making another request. RFC 9110 defines Retry-After as a response header that tells the user agent how long to wait before a follow-up request, including with 503 Service Unavailable responses.

A well-behaved API client should parse both valid forms: seconds and HTTP-date. If the provider says wait 60 seconds, do not retry after your local 500ms backoff just because the client-side formula says so. Treat server-provided retry timing as the stronger signal, then still cap total waiting time to protect your own workflow.

Testing Retry Policies

Retry code is easy to write and hard to trust unless it is tested against real failure modes. Test these cases before relying on the policy:

• first attempt succeeds with no retry;
• one timeout then success;
• 429 with seconds-based Retry-After;
• 429 with HTTP-date Retry-After;
• 503 without Retry-After uses jittered backoff;
• 400, 401, 403 and 404 are not retried;
• maximum attempts are respected;
• total elapsed time is capped;
• caller cancellation stops retries;
• mutating requests use idempotency keys;
• dead-letter or fallback behaviour happens after final failure;
• metrics record attempts, delays, and final outcomes.

Observability Checklist

Retries should be visible but not noisy. Log early retry attempts as informational or warning events and final failure as an error. Track:

• dependency name and operation name;
• attempt number and maximum attempts;
• delay selected and whether it came from Retry-After;
• HTTP status, exception type, or provider error code;
• idempotency key or dedupe key where safe to log;
• correlation ID and trace ID;
• total elapsed time;
• final outcome: success after retry, failed, queued, dead-lettered, circuit opened, or fallback served.

Alert on retry rate, final failure rate, circuit breaker state, queue age, dead-letter count, and dependency latency. Occasional retries are expected. A rising retry rate is often an early warning that a dependency is degraded or that your integration is exceeding a rate limit.

Final Recommendation

For most API integrations, start with provider SDK retry defaults where available. For custom HTTP clients, use finite exponential backoff with jitter, respect Retry-After, retry only known transient failures, and protect every mutating operation with idempotency or dedupe. Keep user-facing retry windows short, move long-running work to queues, and add circuit breakers or retry budgets when dependency failures can cascade.

The best retry policy is boring in production: it quietly absorbs rare transient faults, backs off when a dependency is struggling, avoids duplicate business actions, and gives operators enough telemetry to see when the integration is becoming unhealthy.

Sources Checked

• Microsoft Learn: Transient fault handling
• Microsoft Learn: Retry pattern
• AWS Well-Architected: Control and limit retry calls
• AWS Architecture Blog: Exponential Backoff and Jitter
• Google Cloud: Retry strategy
• RFC 9110: HTTP Semantics
• RFC 6585: Additional HTTP Status Codes
• Stripe API Reference: Idempotent requests
• Polly: Retry resilience strategy
• Polly: Resilience strategies