Distributed-Systems on ND Teknik

Resilience in Distributed Systems: Retry Patterns That Actually Work

Sun, 12 Apr 2026 00:00:00 +0000

Your service calls fail. Accept it. Plan for it. Retry wisely.

In a distributed system, failure is not a bug—it’s a feature. Your payment gateway times out. Your truck tracking API returns 503. A message queue is briefly unavailable. These aren’t anomalies; they’re daily occurrences.

Our transportation platform handles 10,000+ shipments daily. Every single one involves multiple HTTP calls, queue messages, and database operations. We’ve learned that blindly retrying is as bad as not retrying at all.

The Naive Approach (and Why It Fails)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


// DON'T DO THIS
public async Task<Shipment> GetShipmentWithRetryAsync(string id)
{
 for (int i = 0; i < 3; i++)
 {
 try
 {
 return await _httpClient.GetAsync<Shipment>($"/api/shipments/{id}");
 }
 catch
 {
 if (i == 2) throw;
 await Task.Delay(1000); // Fixed 1 second delay
 }
 }
}

Problems:

Fixed delays: If the service is down for 2 seconds, your 1-second retries won’t help.
Thundering herd: All your instances retry at the same time, overwhelming the recovering service.
No differentiation: You retry a 401 Unauthorized the same way as a 503 Service Unavailable. (You shouldn’t.)
No circuit breaking: You keep hammering a dead service.

Pattern 1: Exponential Backoff

Space out your retries with exponential delays:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


public async Task<T> CallWithExponentialBackoffAsync<T>(
 Func<Task<T>> operation,
 int maxRetries = 3)
{
 for (int attempt = 0; attempt <= maxRetries; attempt++)
 {
 try
 {
 return await operation();
 }
 catch (HttpRequestException ex) when (attempt < maxRetries)
 {
 // Calculate delay: 2^attempt * 100ms (± random jitter)
 int delayMs = (int)Math.Pow(2, attempt) * 100;
 int jitter = new Random().Next(0, delayMs / 2);
 int totalDelay = delayMs + jitter;

 await Task.Delay(totalDelay);
 }
 }
}

// Usage
var shipment = await CallWithExponentialBackoffAsync(
 () => _shipmentService.GetByIdAsync("SHP-001"));

Attempt delays: 100ms → 200-300ms → 400-600ms

This spreads the load and gives the service time to recover.

Pattern 2: Using Polly (The Professional’s Choice)

Polly is a .NET resilience library. It’s battle-tested and covers most scenarios.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


// In Program.cs or your DI container
services.AddHttpClient<IShipmentService>()
 .AddTransientHttpErrorPolicy(p =>
 p.OrResult(r => r.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
 .WaitAndRetryAsync(
 retryCount: 3,
 sleepDurationProvider: retryAttempt =>
 TimeSpan.FromMilliseconds(
 Math.Pow(2, retryAttempt) * 100 +
 new Random().Next(0, 100 * retryAttempt)
 ),
 onRetry: (outcome, timespan, retryCount, context) =>
 {
 _logger.LogWarning(
 $"Retry {retryCount} after {timespan.TotalMilliseconds}ms");
 }
 ))
 .AddCircuitBreakerAsync(
 handledEventsAllowedBeforeBreaking: 3,
 durationOfBreak: TimeSpan.FromSeconds(30),
 onBreak: (outcome, duration) =>
 {
 _logger.LogError(
 $"Circuit breaker opened for {duration.TotalSeconds}s");
 });

Pattern 3: Circuit Breaker (The Sentinel)

A circuit breaker stops making calls to a failing service, giving it time to recover.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


public class CircuitBreakerExample
{
 private readonly IAsyncPolicy<HttpResponseMessage> _policy;

 public CircuitBreakerExample()
 {
 // Trip if we get 3 failures
 // Stay open for 30 seconds
 // Then try again (half-open state)
 _policy = Policy
 .HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
 .OrTransientHttpError()
 .CircuitBreakerAsync<HttpResponseMessage>(
 handledEventsAllowedBeforeBreaking: 3,
 durationOfBreak: TimeSpan.FromSeconds(30),
 onBreak: (outcome, duration) =>
 {
 Console.WriteLine(
 $"Circuit broken. Service down. Waiting {duration.TotalSeconds}s");
 },
 onReset: () =>
 {
 Console.WriteLine("Circuit reset. Service recovered!");
 }
 );
 }

 public async Task<HttpResponseMessage> CallWithCircuitBreakerAsync(
 HttpRequestMessage request)
 {
 try
 {
 return await _policy.ExecuteAsync(async () =>
 await _httpClient.SendAsync(request));
 }
 catch (BrokenCircuitException ex)
 {
 return new HttpResponseMessage(System.Net.HttpStatusCode.ServiceUnavailable)
 {
 Content = new StringContent("Service temporarily unavailable")
 };
 }
 }
}

Circuit States:

Closed: Normal operation, requests pass through
Open: Service is failing, requests fail fast without calling the backend
Half-Open: Service might be recovering, allow one test request

Pattern 4: Differentiate Between Retryable and Non-Retryable Failures

Not all errors deserve retries:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


public static bool IsRetryable(Exception ex)
{
 if (ex is HttpRequestException httpEx)
 {
 // Retryable HTTP errors
 var statusCodes = new[] { 408, 429, 500, 502, 503, 504 };
 if (httpEx.StatusCode.HasValue &&
 statusCodes.Contains((int)httpEx.StatusCode))
 {
 return true;
 }
 }

 // Non-retryable
 if (ex is ArgumentException) return false; // Caller's fault
 if (ex is UnauthorizedAccessException) return false; // Auth issue
 if (ex is ValidationException) return false; // Bad data

 // Timeout might be transient
 if (ex is TimeoutException) return true;

 // Default to no retry to avoid infinite loops
 return false;
}

public async Task<Shipment> GetShipmentAsync(string id)
{
 var policy = Policy
 .Handle<Exception>(IsRetryable)
 .WaitAndRetryAsync(
 retryCount: 3,
 sleepDurationProvider: attempt =>
 TimeSpan.FromMilliseconds(Math.Pow(2, attempt) * 100));

 return await policy.ExecuteAsync(async () =>
 await _shipmentService.GetByIdAsync(id));
}

Pattern 5: Bulkhead Isolation

Prevent one failing service from dragging down everything else:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


// Limit concurrent calls to payment service
var bulkheadPolicy = Policy.BulkheadAsync<PaymentResult>(
 maxParallelization: 10,
 maxQueuingActions: 50,
 onBulkheadRejectedAsync: context =>
 {
 _logger.LogWarning("Payment service bulkhead exceeded");
 return Task.CompletedTask;
 });

var combinedPolicy = Policy.WrapAsync<PaymentResult>(
 bulkheadPolicy,
 retryPolicy,
 circuitBreakerPolicy);

Real-World: Delivery Status Sync from Carrier APIs

Here’s how we retry shipment status updates from our carrier API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


private IAsyncPolicy<CarrierStatusResponse> BuildCarrierSyncPolicy()
{
 var retryPolicy = Policy
 .HandleResult<CarrierStatusResponse>(r => r.StatusCode >= 500)
 .Or<TimeoutException>()
 .WaitAndRetryAsync(
 retryCount: 4,
 sleepDurationProvider: attempt =>
 {
 // 100ms, 200ms, 400ms, 800ms
 return TimeSpan.FromMilliseconds(
 Math.Pow(2, attempt - 1) * 100);
 },
 onRetry: (outcome, delay, retryCount, context) =>
 {
 _logger.LogWarning(
 $"Carrier API retry {retryCount} after {delay.TotalMilliseconds}ms");
 });

 var circuitBreaker = Policy
 .HandleResult<CarrierStatusResponse>(r => r.StatusCode >= 500)
 .CircuitBreakerAsync<CarrierStatusResponse>(
 handledEventsAllowedBeforeBreaking: 5,
 durationOfBreak: TimeSpan.FromSeconds(60),
 onBreak: (outcome, duration) =>
 {
 _logger.LogError(
 $"Carrier API circuit broken for {duration.TotalSeconds}s");
 // Send alert
 _alertService.NotifyAsync("Carrier API down");
 });

 return Policy.WrapAsync(retryPolicy, circuitBreaker);
}

public async Task<CarrierStatusResponse> GetCarrierStatusAsync(string trackingId)
{
 var policy = BuildCarrierSyncPolicy();

 return await policy.ExecuteAsync(async () =>
 {
 var response = await _httpClient.GetAsync(
 $"https://carrier-api.com/track/{trackingId}",
 TimeSpan.FromSeconds(10)); // Timeout

 return response;
 });
}

Key Lessons

Retry != Resilience: Retries are one tool. You also need circuit breakers, bulkheads, and timeouts.
Exponential backoff + jitter: Linear retries cause thundering herds. Exponential spacing with randomness is better.
Know what to retry: 401s and 404s don’t deserve retries. 503s do.
Monitor your retries: Log every retry. Chart them. If you’re retrying more than expected, something is degrading.
Set appropriate timeouts: A 30-second timeout defeats the purpose of a 1-second retry. Your total time budget matters.
Use libraries: Don’t roll your own. Polly is mature, well-tested, and handles edge cases you haven’t thought of.

The Rule of Thumb

For our platform:

3 retries for most operations
Exponential backoff with jitter (100ms → 200ms → 400ms)
Circuit breaker after 5 consecutive failures
Differentiate errors: Only retry transient failures
Monitor ruthlessly: Every retry is a signal that something is struggling

Remember: The best retry is the one you never need to make. Design your systems for resilience, not just recovery.

Distributed Transaction Patterns: When ACID Goes to Sleep

Thu, 22 Jan 2026 00:00:00 +0000

In a monolith, you had transactions. In microservices, you have heartache.

When your transportation management system routes a delivery across multiple microservices—booking a shipment, reserving capacity, charging a customer—you can’t just wrap it in a database transaction. The booking service lives in a different database. So does the payment service. Welcome to the world of distributed transactions.

The traditional ACID guarantees you relied on are gone. But the requirement to keep your system consistent? That’s still there.

The Problem: Distributed Monstrosity

Imagine this scenario in our real-time logistics platform:

Customer requests a delivery from NYC to Boston.
Shipment Service creates a shipment order.
Capacity Service reserves truck space.
Billing Service charges the customer’s account.
Notification Service sends a confirmation.

What if step 3 fails? The shipment is booked, but the truck is full. Now you’ve sold something you can’t deliver. Or step 4 fails—the payment is declined. Your shipment is reserved, but unpaid.

In a monolith, you’d roll back the transaction. In microservices, you need something cleverer.

Enter the Saga Pattern

A Saga is a sequence of local transactions, each triggered by the completion of the previous one. If any step fails, you execute compensating transactions to undo the work.

There are two flavors:

1. Choreography (Event-Driven)

Each service publishes an event when it completes. Other services listen and react.

1
2
3
4
5
6
7


Shipment Service publishes "ShipmentCreated"
 ↓
Capacity Service listens → reserves space → publishes "CapacityReserved"
 ↓
Billing Service listens → charges customer → publishes "PaymentProcessed"
 ↓
Notification Service listens → sends email

If Billing fails, it publishes PaymentFailed. Capacity Service listens and releases the reservation. Shipment Service cancels the order.

Pros: Loosely coupled, no central orchestrator. Cons: Hard to see the full flow. Difficult to test. Requires robust error handling everywhere.

2. Orchestration (Centralized)

A central orchestrator (like Azure Durable Functions) tells each service what to do and when.

1
2
3
4
5
6
7


Durable Function:
 1. Call Shipment Service → create order
 2. Call Capacity Service → reserve space
 3. Call Billing Service → charge customer
 4. Call Notification Service → notify customer

 If any step fails, execute compensating transactions in reverse order.

Pros: Clear, testable flow. Easy to see the entire transaction. Cons: The orchestrator becomes a bottleneck. Single point of failure.

Implementation: Orchestrating with Azure Durable Functions

Here’s how we handle delivery booking using orchestration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82


[FunctionName("BookDeliveryOrchestrator")]
public static async Task RunOrchestrator(
 [OrchestrationTrigger] IDurableOrchestrationContext context,
 BookDeliveryRequest request)
{
 var deliveryId = context.NewGuid().ToString();

 try
 {
 // Step 1: Create shipment
 var shipment = await context.CallActivityAsync<ShipmentDto>(
 nameof(CreateShipmentActivity),
 new { request, deliveryId });

 // Step 2: Reserve capacity
 var reservation = await context.CallActivityAsync<ReservationDto>(
 nameof(ReserveCapacityActivity),
 new { shipment, request });

 // Step 3: Process payment
 var payment = await context.CallActivityAsync<PaymentDto>(
 nameof(ProcessPaymentActivity),
 new { request.CustomerId, request.Amount });

 // Step 4: Send notification
 await context.CallActivityAsync(
 nameof(SendNotificationActivity),
 new { deliveryId, request.CustomerEmail });

 return new { success = true, deliveryId, shipment, reservation, payment };
 }
 catch (Exception ex)
 {
 // Compensating transactions: undo in reverse order

 // Undo payment
 await context.CallActivityAsync(
 nameof(RefundPaymentActivity),
 new { request.CustomerId });

 // Undo reservation
 await context.CallActivityAsync(
 nameof(CancelReservationActivity),
 new { request.ShipmentType, request.Weight });

 // Undo shipment
 await context.CallActivityAsync(
 nameof(CancelShipmentActivity),
 new { deliveryId });

 throw;
 }
}

[FunctionName(nameof(CreateShipmentActivity))]
public static async Task<ShipmentDto> CreateShipmentActivity(
 [ActivityTrigger] IDurableActivityContext context,
 dynamic input)
{
 var request = (BookDeliveryRequest)input.request;
 var deliveryId = (string)input.deliveryId;

 // Call Shipment Service
 using var client = new HttpClient();
 var response = await client.PostAsJsonAsync(
 "https://shipment-service/api/shipments",
 new { deliveryId, request.Origin, request.Destination, request.Weight });

 return await response.Content.ReadAsAsync<ShipmentDto>();
}

[FunctionName(nameof(RefundPaymentActivity))]
public static async Task RefundPaymentActivity(
 [ActivityTrigger] IDurableActivityContext context,
 string customerId)
{
 // Call Billing Service to refund
 using var client = new HttpClient();
 await client.PostAsJsonAsync(
 "https://billing-service/api/refunds",
 new { customerId });
}

When to Use Saga vs. Other Patterns

Use Saga when:

You need eventual consistency (not immediate ACID)
Transactions span multiple services
You can design compensating transactions
You’re okay with temporary inconsistency

Don’t use Saga when:

You absolutely need immediate consistency (use distributed locks or stronger consistency patterns instead)
Compensating transactions are impossible (you can’t “undo” a physical truck dispatch)
Performance is critical and you need faster responses

Key Gotchas

Idempotency: Each activity must be idempotent. If Durable Functions retries ProcessPaymentActivity, you don’t want to charge twice.
Compensating Transactions Might Fail: What if the refund fails? You need a dead-letter mechanism.
Testing is Hard: Test the happy path and every possible failure scenario.
Monitoring is Critical: You need visibility into which saga is where in its lifecycle. Add Application Insights logging.

Lessons from Our Platform

We’ve learned that orchestration scales better for us than choreography. With 15+ microservices, trying to track the flow through events became a debugging nightmare. Durable Functions gave us a central, auditable record of each delivery transaction.

The trade-off? The orchestrator is now a critical path. We run multiple instances and monitor it closely.

Remember: Distributed transactions are hard. The Saga pattern isn’t magic—it’s a framework for managing that hardness. Design your compensating transactions carefully, test thoroughly, and monitor relentlessly.

Declarative Caching at Scale: Solving N+1 Queries in Multi-Tenant Systems

Thu, 15 Jan 2026 00:00:00 +0000

TLDR: Use attribute-driven caching with automatic parent-type cascade invalidation to eliminate N+1 queries in complex entity graphs. Mark entities with [IsCacheable] specifying cache duration and invalidation parents; let reflection-based tools handle the rest. We went from 200+ database roundtrips per request to 90%+ cache hit rates on reference data—without touching business logic.

The Problem: Entity Graphs and the N+1 Nightmare

In a transportation management system, a single TransportOrder entity touches dozens of related objects: Trip, TransportLeg, LoadCarrier, Charge, ChargeDefinition, VATDefinition. Each relationship spawns additional queries.

1
2
3
4
5
6
7
8
9


TransportOrder (1 query)
 ├─ Trip (1 query per Trip)
 │ └─ TransportLeg (1 query per Leg)
 │ └─ LoadCarrier (1 query per Leg)
 │ └─ LoadCarrierType (1 query per LoadCarrier)
 ├─ Charge (1 query per Charge)
 │ └─ ChargeDefinition (1 query)
 │ └─ VATDefinition (1 query per VATDefinition)
 └─ BillingAccount (1 query)

Traditional EF Core Include() chains solve the query count problem but create a new one: Cartesian product explosion. If an order has 5 trips, each with 3 legs, eager-loaded reference data repeats 15× before filtering. For reference data (charge definitions, VAT rules), this is catastrophic.

Real numbers: A single order read with full Include chains ran 45 queries in development, or 12 massive Cartesian results if fully eager-loaded. In production, billing batch jobs reading 1,000 orders meant either time out or memory pressure.

The Solution: Declarative, Attribute-Driven Caching

Instead of scattering cache logic through repositories, we mark entities at the domain level:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


[IsCacheable(DurationSeconds = 3600, InvalidateOnParentTypes = new[] { typeof(ChargeDefinition) })]
public class VATDefinition : Entity
{
 public decimal Rate { get; set; }
 public string Code { get; set; }
}

[IsCacheable(DurationSeconds = 1800, InvalidateOnParentTypes = new[] { typeof(Charge) })]
public class ChargeDefinition : Entity
{
 public string Name { get; set; }
 public string Type { get; set; }
 public VATDefinition VAT { get; set; }
}

[IsCacheable(DurationSeconds = 900, InvalidateOnParentTypes = new[] { typeof(TransportOrder) })]
public class Charge : Entity
{
 public ChargeDefinition Definition { get; set; }
 public decimal Amount { get; set; }
}

The RepositoryCacheTools<T> generic class handles the mechanics:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73


public class RepositoryCacheTools<T> where T : Entity
{
 private readonly IDistributedCache _cache;
 private readonly ITenantService _tenantService;
 private readonly ILogger<RepositoryCacheTools<T>> _logger;

 public async Task<T> GetForReadFromCacheOrDbAsync(
 int id,
 Func<Task<T>> dbFallback,
 CancellationToken cancellationToken = default)
 {
 var cacheAttr = typeof(T).GetCustomAttribute<IsCacheableAttribute>();
 if (cacheAttr == null)
 {
 return await dbFallback();
 }

 var tenantId = _tenantService.CurrentTenantId;
 var cacheKey = GenerateCacheKey(tenantId, id);

 var cached = await _cache.GetStringAsync(cacheKey, cancellationToken);
 if (cached != null)
 {
 _logger.LogDebug("Cache hit for {EntityType} ID {Id}", typeof(T).Name, id);
 return JsonSerializer.Deserialize<T>(cached);
 }

 var entity = await dbFallback();
 if (entity != null)
 {
 var serialized = JsonSerializer.Serialize(entity,
 new JsonSerializerOptions
 {
 PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
 WriteIndented = false
 });
 await _cache.SetStringAsync(
 cacheKey,
 serialized,
 new DistributedCacheEntryOptions
 {
 AbsoluteExpirationRelativeToNow = TimeSpan.FromSeconds(cacheAttr.DurationSeconds)
 },
 cancellationToken);
 }

 return entity;
 }

 public async Task InvalidateCascadeAsync(int parentId, Type parentType, CancellationToken cancellationToken = default)
 {
 var attr = typeof(T).GetCustomAttribute<IsCacheableAttribute>();
 if (attr?.InvalidateOnParentTypes == null)
 return;

 if (!attr.InvalidateOnParentTypes.Contains(parentType))
 return;

 // Invalidate parent + all children that depend on it
 var tenantId = _tenantService.CurrentTenantId;
 var parentCacheKey = GenerateCacheKey(tenantId, parentId, parentType);
 await _cache.RemoveAsync(parentCacheKey, cancellationToken);

 _logger.LogInformation("Cache invalidation cascade: {ParentType} ID {ParentId}",
 parentType.Name, parentId);
 }

 private string GenerateCacheKey(string tenantId, int id, Type type = null)
 {
 type ??= typeof(T);
 return $"cache:tenant:{tenantId}:entity:{type.Name}:{id}";
 }
}

Real Implementation: Unit of Work Integration

The cache tools integrate into the repository layer via the IUnitOfWorkManager:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


public class BillingRepository : RepositoryBase
{
 private readonly RepositoryCacheTools<ChargeDefinition> _chargeDefinitionCache;
 private readonly RepositoryCacheTools<VATDefinition> _vatDefinitionCache;

 public async Task<ChargeDefinition> GetChargeDefinitionByIdAsync(int id, CancellationToken cancellationToken)
 {
 return await _chargeDefinitionCache.GetForReadFromCacheOrDbAsync(
 id,
 async () => await _context.ChargeDefinitions.FirstOrDefaultAsync(
 cd => cd.Id == id && cd.TenantId == _tenantService.CurrentTenantId,
 cancellationToken),
 cancellationToken);
 }

 public async Task<ChargeDefinition> SaveChargeDefinitionAsync(
 ChargeDefinition definition,
 CancellationToken cancellationToken)
 {
 _context.ChargeDefinitions.Update(definition);
 await _context.SaveChangesAsync(cancellationToken);

 // Invalidate all VAT definitions that reference this charge definition
 await _vatDefinitionCache.InvalidateCascadeAsync(
 definition.Id,
 typeof(ChargeDefinition),
 cancellationToken);

 return definition;
 }
}

When a ChargeDefinition updates, the interceptor (covered in a future post) triggers cascade invalidation, clearing all cached VATDefinition entries for that tenant.

Solving the Cycle Problem

Complex graphs can have cycles: A → B → C → A. The JSON serializer needs cycle detection:

1
2
3
4
5
6
7


// In cache serialization options
var options = new JsonSerializerOptions
{
 PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
 ReferenceHandler = ReferenceHandler.IgnoreCycles,
 WriteIndented = false
};

This prevents infinite serialization loops while preserving object identity for navigation properties that the API will return as IDs anyway.

Real Impact: The Numbers

Before caching (per 1,000-order batch read):

5,200 database queries
12 seconds latency
450MB peak memory

After declarative caching (same 1,000-order batch read):

240 database queries (reference data cached, parent graphs use AsSplitQuery)
2.1 seconds latency (18× faster)
85MB peak memory

Reference data cache hit rate: 93% after warm-up.

For billing operations processing 50,000+ orders monthly, this translated to 8 fewer database instances needed to maintain SLA.

Lessons Learned: The Hard Way

1. Tenant isolation is non-negotiable Initially, we forgot to namespace cache keys by tenant. Customer B got Customer A’s charge definitions. Learned: always include TenantId in cache keys. Make it loud in code reviews.

2. Don’t cache entities with mutable relationships Caching a TransportOrder with its full Charges collection was a mistake—charges update independently. Cache immutable reference data only (ChargeDefinition, VATDefinition). Cache mutable domain objects via query-result caching instead (a different pattern).

3. Batch operations need cache disabling During billing document generation, we read 10,000 charge definitions. Cache lookups themselves became a bottleneck. Solution: tenantService.DisableAuditAndCaching() during batch pivots, then warm the cache post-batch.

4. Expiration strategy is a tuning knob We set everything to 1-hour TTL. Charge definitions rarely change but billing happens hourly. We switched to:

Reference data: 24-hour TTL
Billing reference (VAT, charge definitions): 4-hour TTL
Parent entities: 15-minute TTL

5. Serialization performance matters JsonProperty attributes for enums and nested objects must exclude circular references. Use [JsonIgnore] on back-references.

Gotchas and Disclaimers

Versioning: This pattern works on EF Core 8+, .NET 9. Earlier versions lack some reflection optimizations.
Distributed cache: Requires Redis or Azure Cache for Redis. In-memory caching defeats multi-instance deployment.
Stale data: Cache invalidation is eventual. If real-time consistency is critical (billing cutoff), read from DB directly.
Monitoring: Add metrics for cache hit/miss ratios per entity type. Growing miss rates signal TTL tuning needed or invalidation storms.

Next Steps

Start with high-cardinality, low-change reference data (VAT definitions, charge types).
Measure cache hit rates per entity type in production.
Tune TTL based on update frequency and business tolerance for staleness.
Pair with AsSplitQuery() for parent entity graphs to eliminate Cartesian products.

Ready for featured image.