Event-Driven Architecture Pitfalls: Lessons Learned the Hard Way

Everything works until it doesn’t.

And when event-driven systems break, they break silently.

We’ve built our transportation SaaS platform on Event-Driven Architecture from the ground up. We publish events when shipments are created, tracked, delivered, returned. We have 15+ services listening to those events. It’s beautiful—until you’re at 3 AM debugging why deliveries are stuck in a Pending state, and nobody can tell you why.

Here are the pitfalls we hit. Hopefully you can avoid them.

Pitfall 1: Message Ordering Chaos

You publish ShipmentCreated, then immediately ShipmentDispatched. But what if they arrive out of order?

The Scenario:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
// Service A
await _serviceBus.SendAsync(new ShipmentCreatedEvent { 
    ShipmentId = "SHP-001", 
    Status = "Created" 
});

await _serviceBus.SendAsync(new ShipmentDispatchedEvent { 
    ShipmentId = "SHP-001", 
    Status = "Dispatched",
    Timestamp = DateTime.UtcNow.AddSeconds(2)
});

Service B receives:

1
2
Event 2: ShipmentDispatched (Processing started...)
Event 1: ShipmentCreated (Hmm, no shipment found for SHP-001)

Now your shipment is orphaned. It was dispatched before it was created. Your database might enforce a foreign key constraint. Exception. Dead-letter. Silent failure if not monitored.

The Solution:

For events with ordering guarantees, use Azure Service Bus with Sessions:

1
2
3
4
5
6
7
var message = new ServiceBusMessage(eventJson)
{
    SessionId = shipment.ShipmentId, // Same shipment = same session
    CorrelationId = shipment.ShipmentId,
};

await _sender.SendMessageAsync(message);

Or use Azure Event Hubs with Partition Key:

1
2
var sendOptions = new SendEventOptions { PartitionKey = shipment.ShipmentId };
await _producer.SendAsync(new[] { new EventData(eventBytes) }, sendOptions);

This guarantees messages for the same shipment are processed in order.

But: This has a cost—throughput drops by 50% because you can only process one message per partition/session at a time.

Pitfall 2: Duplicate Message Processing

You process a PaymentProcessedEvent, charge the customer, publish ShipmentConfirmedEvent. Then the message broker retries the event (it didn’t get an ACK fast enough). You process it again. Double charge.

The Catch:

Message brokers deliver “at-least-once,” not “exactly-once.” It’s a fundamental limitation.

The Solution: Idempotency Key

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
public class PaymentProcessedEvent
{
    public string IdempotencyKey { get; set; } // Required
    public string CustomerId { get; set; }
    public decimal Amount { get; set; }
}

public async Task HandleAsync(PaymentProcessedEvent evt)
{
    // Check if we've seen this before
    var existingCharge = await _chargeRepository.GetByIdempotencyKeyAsync(evt.IdempotencyKey);
    if (existingCharge != null)
    {
        _logger.LogInformation($"Duplicate payment already processed: {evt.IdempotencyKey}");
        return; // Silently ignore
    }
    
    // First time: process and store idempotency key
    var charge = await _billingService.ChargeAsync(evt.CustomerId, evt.Amount);
    await _chargeRepository.SaveWithIdempotencyKeyAsync(charge, evt.IdempotencyKey);
}

The idempotency key should be deterministic and unique per event:

1
var idempotencyKey = $"{evt.CustomerId}:payment:{evt.TransactionId}:{evt.Amount}";

Pitfall 3: Message Ordering + Retries = Deadlock

You have Session-based ordering (for guaranteed order). A message fails and gets retried. But the retry is blocking the entire session. Now messages 3, 4, 5 are stuck behind message 2’s retry loop.

1
2
3
4
5
Message 1: Processed ✓
Message 2: Failed (retrying for 5 minutes...)
Message 3: Blocked (waiting for message 2)
Message 4: Blocked (waiting for message 2)
Message 5: Blocked (waiting for message 2)

After 5 minutes, Message 2 gets dead-lettered. Finally, 3-5 can process. Your “ordered delivery” now delivered everything with a 5-minute delay.

The Solution: Retry Budget

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
var options = new ServiceBusProcessorOptions
{
    MaxConcurrentCalls = 1, // Process one per session
    ReceiveMode = ServiceBusReceiveMode.PeekLock,
};

var processor = _client.CreateProcessor(queueName, options);

processor.ProcessMessageAsync += async args =>
{
    var message = args.Message;
    int retryCount = GetRetryCount(message);
    
    try
    {
        await HandleEventAsync(message);
        await args.CompleteMessageAsync(message);
    }
    catch (Exception ex) when (retryCount < 3)
    {
        // Reschedule with backoff
        int delayMs = (int)Math.Pow(2, retryCount) * 100;
        var rescheduleTime = DateTime.UtcNow.AddMilliseconds(delayMs);
        
        // Defer the message (Service Bus holds it until rescheduleTime)
        await args.DeferMessageAsync(message);
        _logger.LogWarning($"Deferred message {retryCount} times");
    }
    catch (Exception ex)
    {
        // Max retries exceeded: dead-letter
        _logger.LogError($"Sending to dead-letter: {ex.Message}");
        await args.DeadLetterMessageAsync(message, "MaxRetriesExceeded");
    }
};

Pitfall 4: Silent Dead-Letter Queue Failures

You set up dead-lettering. Messages that can’t be processed go to a dead-letter queue. Everything is fine. Until someone asks, “How many messages are in our dead-letter queue?”

Silence.

Nobody’s monitoring it. Messages sit there for months. Unprocessed orders. Unrefunded payments. Stranded shipments.

The Solution: Dead-Letter Queue Monitoring

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
// In your health check
public class DeadLetterQueueHealthCheck : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        var queueProperties = await _managementClient.GetQueueRuntimePropertiesAsync(
            queueName: "payments-dlq", 
            cancellationToken: cancellationToken);
        
        long messageCount = queueProperties.Value.MessageCountDetails.DeadLetterMessageCount;
        
        if (messageCount > 0)
        {
            return HealthCheckResult.Degraded(
                $"Dead-letter queue has {messageCount} messages");
        }
        
        return HealthCheckResult.Healthy();
    }
}

// Add to health check
services.AddHealthChecks()
    .AddCheck<DeadLetterQueueHealthCheck>("dlq-check");

// Alert if degraded
app.MapHealthChecks("/health", new HealthCheckOptions
{
    ResultStatusCodes =
    {
        { HealthStatus.Degraded, StatusCodes.Status503ServiceUnavailable }
    }
});

Also: Alert on message age. If messages have been in the queue for > 1 hour, page someone.

Pitfall 5: Debugging Event-Driven Systems is Horrible

Traditional request-response debugging: I hit an endpoint, I see the response, I trace the stack. Done.

Event-driven debugging: Message published at 2:15 PM, processed at 2:16 PM, published another event at 2:17 PM, which was dead-lettered at 2:18 PM, but I only checked logs at 3 PM, and the retention window was 2 hours, so the evidence is gone.

The Solution: Distributed Tracing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
// In the event publisher
var traceContext = Activity.Current ?? new Activity("PublishEvent").Start();

var message = new ServiceBusMessage(eventJson)
{
    CorrelationId = traceContext.Id,
    ApplicationProperties =
    {
        { "TraceParent", traceContext.Id },
        { "TraceState", traceContext.TraceStateString },
        { "PublishedAt", DateTime.UtcNow.ToIso8601String() }
    }
};

await _sender.SendMessageAsync(message);

In the event handler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
var activity = new Activity("ProcessPaymentEvent").Start();

if (message.ApplicationProperties.TryGetValue("TraceParent", out var traceParent))
{
    activity.SetParentId((string)traceParent);
}

activity.SetTag("event.shipment_id", evt.ShipmentId);
activity.SetTag("event.amount", evt.Amount);

try
{
    await _billingService.ChargeAsync(evt.CustomerId, evt.Amount);
    activity.SetTag("charge.success", true);
}
catch (Exception ex)
{
    activity.SetTag("charge.error", ex.Message);
    activity.SetStatus(ActivityStatusCode.Error, ex.Message);
    throw;
}
finally
{
    activity.Stop();
}

Now when someone asks “Why didn’t delivery SHP-001 get confirmed?”, you can:

1
2
3
4
5
6
7
8
9
// Query Application Insights
var trace = await _appInsights.GetTraceChain(
    correlationId: "SHP-001",
    timeRange: TimeSpan.FromHours(1));

foreach (var activity in trace.Activities)
{
    Console.WriteLine($"{activity.Timestamp}: {activity.OperationName} - {activity.Result}");
}

Pitfall 6: Cascading Failures

One service is slow. It backs up its event queue. Now the broker is retrying messages. Those retries compete with new messages. The queue grows. Processing gets slower. Timeouts. Backoff. More retries. Exponential degradation.

The Solution: Bulkheads + Circuit Breakers

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
var bulkheadPolicy = Policy.BulkheadAsync(
    maxParallelization: 5,
    maxQueuingActions: 10,
    onBulkheadRejectedAsync: context =>
    {
        _logger.LogWarning("Shipment service bulkhead rejected");
        // Could emit metric, fire alert
        return Task.CompletedTask;
    });

var circuitBreakerPolicy = Policy
    .Handle<ServiceBusException>()
    .CircuitBreakerAsync(
        handledEventsAllowedBeforeBreaking: 3,
        durationOfBreak: TimeSpan.FromSeconds(30));

var combinedPolicy = Policy.WrapAsync(bulkheadPolicy, circuitBreakerPolicy);

processor.ProcessMessageAsync += async args =>
{
    await combinedPolicy.ExecuteAsync(async () =>
    {
        await HandleEventAsync(args.Message);
    });
};

The Observability Checklist

For every event-driven system, ensure you have:

Distributed tracing (Application Insights or Open Telemetry)
Dead-letter queue monitoring (alerts if count > 0)
Message age tracking (how long messages wait)
Retry metrics (how many retries per service)
Processing duration (P50, P95, P99 latencies)
Circuit breaker state (open/closed/half-open counts)
Idempotency key collisions (duplicate handling rate)
Message ordering violations (out-of-order event counts)
End-to-end flow dashboards (from event publish to final state)

The Reality

Event-driven architecture is powerful. But it’s not magic. It’s a tradeoff:

You gain: Decoupling, scalability, async processing You lose: Observability, testability, simplicity

Use it when you need the gains. But go in with eyes open—you’ll need robust monitoring, comprehensive testing, and a very strong understanding of your message flow.

The golden rule: If you can’t observe it, you can’t fix it. And if you can’t fix it quickly, you don’t have event-driven architecture—you have a distributed disaster.