Best Practices on ND Teknik

Resilience in Distributed Systems: Retry Patterns That Actually Work

Sun, 12 Apr 2026 00:00:00 +0000

Your service calls fail. Accept it. Plan for it. Retry wisely.

In a distributed system, failure is not a bug—it’s a feature. Your payment gateway times out. Your truck tracking API returns 503. A message queue is briefly unavailable. These aren’t anomalies; they’re daily occurrences.

Our transportation platform handles 10,000+ shipments daily. Every single one involves multiple HTTP calls, queue messages, and database operations. We’ve learned that blindly retrying is as bad as not retrying at all.

The Naive Approach (and Why It Fails)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


// DON'T DO THIS
public async Task<Shipment> GetShipmentWithRetryAsync(string id)
{
 for (int i = 0; i < 3; i++)
 {
 try
 {
 return await _httpClient.GetAsync<Shipment>($"/api/shipments/{id}");
 }
 catch
 {
 if (i == 2) throw;
 await Task.Delay(1000); // Fixed 1 second delay
 }
 }
}

Problems:

Fixed delays: If the service is down for 2 seconds, your 1-second retries won’t help.
Thundering herd: All your instances retry at the same time, overwhelming the recovering service.
No differentiation: You retry a 401 Unauthorized the same way as a 503 Service Unavailable. (You shouldn’t.)
No circuit breaking: You keep hammering a dead service.

Pattern 1: Exponential Backoff

Space out your retries with exponential delays:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


public async Task<T> CallWithExponentialBackoffAsync<T>(
 Func<Task<T>> operation,
 int maxRetries = 3)
{
 for (int attempt = 0; attempt <= maxRetries; attempt++)
 {
 try
 {
 return await operation();
 }
 catch (HttpRequestException ex) when (attempt < maxRetries)
 {
 // Calculate delay: 2^attempt * 100ms (± random jitter)
 int delayMs = (int)Math.Pow(2, attempt) * 100;
 int jitter = new Random().Next(0, delayMs / 2);
 int totalDelay = delayMs + jitter;

 await Task.Delay(totalDelay);
 }
 }
}

// Usage
var shipment = await CallWithExponentialBackoffAsync(
 () => _shipmentService.GetByIdAsync("SHP-001"));

Attempt delays: 100ms → 200-300ms → 400-600ms

This spreads the load and gives the service time to recover.

Pattern 2: Using Polly (The Professional’s Choice)

Polly is a .NET resilience library. It’s battle-tested and covers most scenarios.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


// In Program.cs or your DI container
services.AddHttpClient<IShipmentService>()
 .AddTransientHttpErrorPolicy(p =>
 p.OrResult(r => r.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
 .WaitAndRetryAsync(
 retryCount: 3,
 sleepDurationProvider: retryAttempt =>
 TimeSpan.FromMilliseconds(
 Math.Pow(2, retryAttempt) * 100 +
 new Random().Next(0, 100 * retryAttempt)
 ),
 onRetry: (outcome, timespan, retryCount, context) =>
 {
 _logger.LogWarning(
 $"Retry {retryCount} after {timespan.TotalMilliseconds}ms");
 }
 ))
 .AddCircuitBreakerAsync(
 handledEventsAllowedBeforeBreaking: 3,
 durationOfBreak: TimeSpan.FromSeconds(30),
 onBreak: (outcome, duration) =>
 {
 _logger.LogError(
 $"Circuit breaker opened for {duration.TotalSeconds}s");
 });

Pattern 3: Circuit Breaker (The Sentinel)

A circuit breaker stops making calls to a failing service, giving it time to recover.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


public class CircuitBreakerExample
{
 private readonly IAsyncPolicy<HttpResponseMessage> _policy;

 public CircuitBreakerExample()
 {
 // Trip if we get 3 failures
 // Stay open for 30 seconds
 // Then try again (half-open state)
 _policy = Policy
 .HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
 .OrTransientHttpError()
 .CircuitBreakerAsync<HttpResponseMessage>(
 handledEventsAllowedBeforeBreaking: 3,
 durationOfBreak: TimeSpan.FromSeconds(30),
 onBreak: (outcome, duration) =>
 {
 Console.WriteLine(
 $"Circuit broken. Service down. Waiting {duration.TotalSeconds}s");
 },
 onReset: () =>
 {
 Console.WriteLine("Circuit reset. Service recovered!");
 }
 );
 }

 public async Task<HttpResponseMessage> CallWithCircuitBreakerAsync(
 HttpRequestMessage request)
 {
 try
 {
 return await _policy.ExecuteAsync(async () =>
 await _httpClient.SendAsync(request));
 }
 catch (BrokenCircuitException ex)
 {
 return new HttpResponseMessage(System.Net.HttpStatusCode.ServiceUnavailable)
 {
 Content = new StringContent("Service temporarily unavailable")
 };
 }
 }
}

Circuit States:

Closed: Normal operation, requests pass through
Open: Service is failing, requests fail fast without calling the backend
Half-Open: Service might be recovering, allow one test request

Pattern 4: Differentiate Between Retryable and Non-Retryable Failures

Not all errors deserve retries:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


public static bool IsRetryable(Exception ex)
{
 if (ex is HttpRequestException httpEx)
 {
 // Retryable HTTP errors
 var statusCodes = new[] { 408, 429, 500, 502, 503, 504 };
 if (httpEx.StatusCode.HasValue &&
 statusCodes.Contains((int)httpEx.StatusCode))
 {
 return true;
 }
 }

 // Non-retryable
 if (ex is ArgumentException) return false; // Caller's fault
 if (ex is UnauthorizedAccessException) return false; // Auth issue
 if (ex is ValidationException) return false; // Bad data

 // Timeout might be transient
 if (ex is TimeoutException) return true;

 // Default to no retry to avoid infinite loops
 return false;
}

public async Task<Shipment> GetShipmentAsync(string id)
{
 var policy = Policy
 .Handle<Exception>(IsRetryable)
 .WaitAndRetryAsync(
 retryCount: 3,
 sleepDurationProvider: attempt =>
 TimeSpan.FromMilliseconds(Math.Pow(2, attempt) * 100));

 return await policy.ExecuteAsync(async () =>
 await _shipmentService.GetByIdAsync(id));
}

Pattern 5: Bulkhead Isolation

Prevent one failing service from dragging down everything else:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


// Limit concurrent calls to payment service
var bulkheadPolicy = Policy.BulkheadAsync<PaymentResult>(
 maxParallelization: 10,
 maxQueuingActions: 50,
 onBulkheadRejectedAsync: context =>
 {
 _logger.LogWarning("Payment service bulkhead exceeded");
 return Task.CompletedTask;
 });

var combinedPolicy = Policy.WrapAsync<PaymentResult>(
 bulkheadPolicy,
 retryPolicy,
 circuitBreakerPolicy);

Real-World: Delivery Status Sync from Carrier APIs

Here’s how we retry shipment status updates from our carrier API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


private IAsyncPolicy<CarrierStatusResponse> BuildCarrierSyncPolicy()
{
 var retryPolicy = Policy
 .HandleResult<CarrierStatusResponse>(r => r.StatusCode >= 500)
 .Or<TimeoutException>()
 .WaitAndRetryAsync(
 retryCount: 4,
 sleepDurationProvider: attempt =>
 {
 // 100ms, 200ms, 400ms, 800ms
 return TimeSpan.FromMilliseconds(
 Math.Pow(2, attempt - 1) * 100);
 },
 onRetry: (outcome, delay, retryCount, context) =>
 {
 _logger.LogWarning(
 $"Carrier API retry {retryCount} after {delay.TotalMilliseconds}ms");
 });

 var circuitBreaker = Policy
 .HandleResult<CarrierStatusResponse>(r => r.StatusCode >= 500)
 .CircuitBreakerAsync<CarrierStatusResponse>(
 handledEventsAllowedBeforeBreaking: 5,
 durationOfBreak: TimeSpan.FromSeconds(60),
 onBreak: (outcome, duration) =>
 {
 _logger.LogError(
 $"Carrier API circuit broken for {duration.TotalSeconds}s");
 // Send alert
 _alertService.NotifyAsync("Carrier API down");
 });

 return Policy.WrapAsync(retryPolicy, circuitBreaker);
}

public async Task<CarrierStatusResponse> GetCarrierStatusAsync(string trackingId)
{
 var policy = BuildCarrierSyncPolicy();

 return await policy.ExecuteAsync(async () =>
 {
 var response = await _httpClient.GetAsync(
 $"https://carrier-api.com/track/{trackingId}",
 TimeSpan.FromSeconds(10)); // Timeout

 return response;
 });
}

Key Lessons

Retry != Resilience: Retries are one tool. You also need circuit breakers, bulkheads, and timeouts.
Exponential backoff + jitter: Linear retries cause thundering herds. Exponential spacing with randomness is better.
Know what to retry: 401s and 404s don’t deserve retries. 503s do.
Monitor your retries: Log every retry. Chart them. If you’re retrying more than expected, something is degrading.
Set appropriate timeouts: A 30-second timeout defeats the purpose of a 1-second retry. Your total time budget matters.
Use libraries: Don’t roll your own. Polly is mature, well-tested, and handles edge cases you haven’t thought of.

The Rule of Thumb

For our platform:

3 retries for most operations
Exponential backoff with jitter (100ms → 200ms → 400ms)
Circuit breaker after 5 consecutive failures
Differentiate errors: Only retry transient failures
Monitor ruthlessly: Every retry is a signal that something is struggling

Remember: The best retry is the one you never need to make. Design your systems for resilience, not just recovery.

Query Optimization with EF Core's AsSplitQuery: When N-Way Joins Become a Bottleneck

Wed, 18 Mar 2026 00:00:00 +0000

TLDR: Deep entity graphs with 4+ Include chains generate massive Cartesian products in SQL joins. AsSplitQuery() executes separate queries and stitches results client-side, cutting result set sizes by 50–90% and latency by 3–8×. Pair with a generic IncludeAll() extension to automatically hydrate complex types without hand-writing Include chains. We reduced a single order read from 45 queries + Cartesian bloat to 8 optimized split queries with 95% less data transfer.

The Problem: Cartesian Explosion

A transportation order has trips, legs, charges, and accounting entries. Naive eager loading:

1
2
3
4
5
6
7
8
9


var order = await _context.TransportOrders
 .Include(o => o.Trips)
 .ThenInclude(t => t.Legs)
 .ThenInclude(l => l.LoadCarrier)
 .ThenInclude(lc => lc.LoadCarrierType)
 .Include(o => o.Charges)
 .ThenInclude(c => c.ChargeDefinition)
 .Include(o => o.BillingAccount)
 .FirstOrDefaultAsync(o => o.Id == orderId);

Generated SQL joins across all relationships. If an order has:

3 trips
2 legs per trip
5 charges

The result set repeats the order and account data 3 × 2 × 5 = 30 times before FirstOrDefault filters. Each repeated row is full object data (strings, decimals, GUIDs).

Real numbers from production:

Single order read: 8.2 MB result set from the database
1,000-order batch: 8.2 GB transferring from SQL to app server
Deserialization: 14 seconds just to materialize objects

The SQL query itself was fast. The network and client-side processing were the killers.

Split Queries: A Different Approach

AsSplitQuery() tells EF Core to execute multiple queries and stitch them in memory:

1
2
3
4
5
6
7
8
9


var order = await _context.TransportOrders
 .AsSplitQuery()
 .Include(o => o.Trips)
 .ThenInclude(t => t.Legs)
 .ThenInclude(l => l.LoadCarrier)
 .Include(o => o.Charges)
 .ThenInclude(c => c.ChargeDefinition)
 .Include(o => o.BillingAccount)
 .FirstOrDefaultAsync(o => o.Id == orderId);

Instead of one massive join, EF executes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


-- Query 1: Root order
SELECT * FROM TransportOrders WHERE Id = @orderId;

-- Query 2: Related trips
SELECT * FROM Trips WHERE TransportOrderId = @orderId;

-- Query 3: Related legs (for all trips from query 2)
SELECT * FROM TransportLegs WHERE TripId IN (@tripIds);

-- Query 4: Related charges
SELECT * FROM Charges WHERE TransportOrderId = @orderId;

-- Query 5: Related charge definitions (for all charges from query 4)
SELECT * FROM ChargeDefinitions WHERE Id IN (@chargeDefIds);

-- Query 6: Billing account
SELECT * FROM BillingAccounts WHERE Id = @accountId;

Each result set is small, specific, and un-repeated. Total data transfer: 0.8 MB instead of 8.2 MB.

Building `IncludeAll()`: Stop Hand-Writing Include Chains

With 40+ entity types in the model, writing Include chains for each becomes unmaintainable. A generic IncludeAll() extension uses reflection to inspect navigation properties:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71


public static class QueryableExtensions
{
 public static IQueryable<T> IncludeAll<T>(
 this IQueryable<T> query,
 int maxDepth = 3,
 HashSet<Type> visitedTypes = null) where T : Entity
 {
 visitedTypes ??= new HashSet<Type>();

 if (maxDepth <= 0 || visitedTypes.Contains(typeof(T)))
 return query;

 visitedTypes.Add(typeof(T));

 var navigationProperties = typeof(T)
 .GetProperties()
 .Where(p => p.PropertyType.Namespace == "YourNamespace.Domain" ||
 (p.PropertyType.IsGenericType &&
 p.PropertyType.GetGenericArguments()[0].Namespace == "YourNamespace.Domain"))
 .ToList();

 foreach (var navProp in navigationProperties)
 {
 // Handle ICollection<T>
 if (navProp.PropertyType.IsGenericType &&
 typeof(System.Collections.IEnumerable).IsAssignableFrom(navProp.PropertyType))
 {
 var elementType = navProp.PropertyType.GetGenericArguments()[0];
 var includeMethod = typeof(EntityFrameworkQueryableExtensions)
 .GetMethod("Include", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.Public)
 .MakeGenericMethod(typeof(T), elementType);

 var lambda = BuildLambda<T>(navProp.Name);
 query = (IQueryable<T>)includeMethod.Invoke(null, new object[] { query, lambda });

 // Recursively include nested navigation
 var thenIncludeMethod = typeof(EntityFrameworkQueryableExtensions)
 .GetMethods(System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.Public)
 .First(m => m.Name == "ThenInclude" &&
 m.GetGenericArguments().Length == 2 &&
 m.GetParameters()[0].ParameterType.Name.StartsWith("IIncludableQueryable"));

 // Build nested includes for the collection element type
 if (maxDepth > 1)
 {
 // This gets complex; see full implementation in repository
 }
 }
 // Handle single navigation
 else
 {
 var includeMethod = typeof(EntityFrameworkQueryableExtensions)
 .GetMethod("Include", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.Public)
 .MakeGenericMethod(typeof(T), navProp.PropertyType);

 var lambda = BuildLambda<T>(navProp.Name);
 query = (IQueryable<T>)includeMethod.Invoke(null, new object[] { query, lambda });
 }
 }

 return query;
 }

 private static object BuildLambda<T>(string propertyName) where T : Entity
 {
 var parameter = System.Linq.Expressions.Expression.Parameter(typeof(T));
 var property = System.Linq.Expressions.Expression.Property(parameter, propertyName);
 var lambda = System.Linq.Expressions.Expression.Lambda(property, parameter);
 return lambda;
 }
}

In practice:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


// Old way: tedious, error-prone
var order = await _context.TransportOrders
 .AsSplitQuery()
 .Include(o => o.Trips)
 .ThenInclude(t => t.Legs)
 .ThenInclude(l => l.LoadCarrier)
 .Include(o => o.Charges)
 .ThenInclude(c => c.ChargeDefinition)
 .FirstOrDefaultAsync(o => o.Id == orderId);

// New way: one line, auto-discovers all navigations
var order = await _context.TransportOrders
 .AsSplitQuery()
 .IncludeAll(maxDepth: 4)
 .FirstOrDefaultAsync(o => o.Id == orderId);

The extension crawls the reflection tree, generating Include expressions dynamically.

When NOT to Use Split Queries

Split queries aren’t always better:

Scenario 1: Filtering on related entities

1
2
3
4
5


var orders = await _context.TransportOrders
 .Where(o => o.Trips.Any(t => t.Status == "InProgress"))
 .AsSplitQuery()
 .Include(o => o.Trips)
 .ToListAsync();

Here, split queries execute the root query first without the Where clause on Trips, then filters in memory. Use traditional joins for filtered includes:

1
2
3


var orders = await _context.TransportOrders
 .Include(o => o.Trips.Where(t => t.Status == "InProgress"))
 .ToListAsync();

Scenario 2: Small result sets with few collections If you’re loading 5 orders with 2–3 related items each, split query overhead exceeds benefit. Test both.

Scenario 3: Real-time transactions Split queries execute sequentially, not atomically. If data changes between queries, you get phantom reads. For billing cutoffs, use single transactions with explicit isolation levels.

Profiling: Before and After

Traditional joins (8.2 MB, 2.1 seconds):

1
2
3


SQL Execution: 140ms
Network Transfer: 1,200ms
Deserialization: 760ms

Split queries (0.8 MB, 320ms):

1
2
3


SQL Execution: 85ms (6 queries)
Network Transfer: 120ms
Deserialization: 115ms

8× latency improvement, 90% less data.

Enable query logging to see the difference:

1
2
3


optionsBuilder.LogTo(Console.WriteLine, LogLevel.Information)
 .EnableDetailedErrors()
 .EnableSensitiveDataLogging();

Real Scenario: Billing Order Export

A monthly billing run exports 50,000 orders with full detail (trips, legs, charges, accounting entries).

Old approach:

Single mammoth query with all includes
OOM after 2,000 orders
8+ hours to export

New approach:

Batch 1,000 orders per query with AsSplitQuery()
0.8 MB per batch
2.5 hours total (70% faster)
No OOM

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


const int batchSize = 1000;
for (int offset = 0; offset < totalOrders; offset += batchSize)
{
 var batch = await _context.TransportOrders
 .AsSplitQuery()
 .IncludeAll(maxDepth: 3)
 .Skip(offset)
 .Take(batchSize)
 .ToListAsync();

 await ExportBatch(batch);
}

Lessons Learned

1. Measure first, optimize second We optimized split queries before profiling. Turned out network latency, not SQL, was the bottleneck. SQL was 140ms; data transfer was 1,200ms.

2. Split queries and caching are complementary Split queries reduce result set sizes. Caching (from the previous post) eliminates queries entirely. Use both: cache reference data, split queries for parent graphs.

3. Depth limits prevent runaway reflection Without maxDepth, IncludeAll() traversed into circular dependencies indefinitely. Always set maxDepth = 3 or 4.

4. Client-side stitching has memory cost Split queries use more memory than joins for large collections (joining in SQL is more efficient for aggregates). For 10,000+ root entities, consider pagination or async enumeration.

Gotchas and Disclaimers

EF Core 8+: Earlier versions have different split query behavior.
Explicit transaction isolation: Split queries don’t guarantee isolation between queries. Use IsolationLevel.Serializable if you need it.
Async enumeration: For very large result sets, use await foreach with AsSingleQuery() on the stream, not split queries.
JSON serialization: When returning split-query results as JSON, circular references will cause serialization errors. Use [JsonIgnore] on back-references.

Next Steps

Profile your deepest Include chains: measure result set size and deserialization time.
If result sets exceed 1 MB or deserialization exceeds 500ms, try split queries.
Implement IncludeAll() for your domain model.
Pair with declarative caching for reference data (previous post).

Ready for featured image.