resilience-engineering_skill

Overview

This skill teaches practical resilience engineering for Shopify apps to handle API rate limits (429), retries, throttling, and circuit breakers. It focuses on concrete patterns: honoring Retry-After headers, queuing and throttling bulk work, using circuit breakers for unreliable services, and ensuring webhook idempotency. The guidance is geared for high-traffic apps and large sync jobs.

How this skill works

It inspects common failure modes from Shopify and related services and prescribes code-level strategies. Key behaviors include reading the Retry-After header on 429 responses, applying exponential backoff and retry policies, scheduling requests through a limiter or background queue, and wrapping unstable dependencies with a circuit breaker. It also enforces webhook idempotency by tracking unique webhook delivery IDs.

When to use it

• When your app receives 429 Too Many Requests from Shopify or third-party APIs.
• During large data syncs or bulk operations (thousands of products, orders, etc.).
• When integrating with unreliable external services where failures can cascade.
• When processing Shopify webhooks to avoid duplicate handling.
• When you need predictable throughput and to avoid service throttling.

Best practices

• Always respect the Retry-After header and implement a capped backoff before retrying failed requests.
• Move heavy or batched work to background workers and use a rate limiter (e.g., Bottleneck) instead of tight loops.
• Combine retry policies with circuit breakers to stop hammering failing services and allow recovery periods.
• Persist webhook IDs with a TTL (Redis or DB) to achieve idempotency for at-least-once deliveries.
• Monitor error rates, throttling responses, and circuit breaker state to tune timeouts, thresholds, and concurrency.

Example use cases

• Syncing 10,000 products: schedule requests through a Bottleneck limiter and run jobs in a worker queue.
• Handling intermittent carrier API failures: wrap calls with a circuit breaker and retry with exponential backoff.
• Responding to 429 responses: parse Retry-After and requeue the request after the specified delay.
• Processing orders/create webhooks: store X-Shopify-Webhook-Id in Redis for 24 hours to ignore duplicates.
• Scaling billing or inventory updates: limit concurrent requests and stagger bursts to stay within Shopify’s leaky-bucket model.