Every hop in the relay had sessions enabled. Messages for one order still arrived out of order. Twice, for two different reasons.

I have recommended Service Bus sessions for years as the answer to ordering. They are still the answer, but they answer a narrower question than I used to think. This post is about the two ways my relay broke in 2026 with sessions switched on everywhere, and the two fixes.

The relay

The transport management system I work on relays events between tenants through an integration hub. A driver reports from the app, the report goes through an external API onto Service Bus, the TMS of the executing tenant applies it, the TMS sends a status event to the hub, the hub maps it and delivers it on Service Bus to the ordering tenant. Four or five hops, every one of them a sessions enabled subscription with a sessions enabled trigger.

1
2

[ServiceBusTrigger("sbt-driver-app-order-report-event", "driver-app-order-report-event-func",
    Connection = "SbConnString", IsSessionsEnabled = true)]

A session guarantees that messages with the same session id are handled one at a time, in enqueue order, by one receiver. That is a strong guarantee. It is also the only guarantee, and both of my failures lived outside it.

Failure one, the session key meant something different at every hop

A testing session in September produced a list of nine problems, and the ugliest was that status, milestone and document events for one order sometimes applied in the wrong order on the receiving side. I audited every sender. Sessions were on at every hop. The session id was not the same thing at every hop.

The driver app hop used the leg uid. The hop into the hub used the uid of the flow that created the order. Inside the hub, each event type had its own flow, so each event type got its own session. By the time the events reached the receive function of the ordering tenant, one order had fanned out into three concurrent lanes. Each lane was perfectly ordered. The lanes were not.

The fix is one rule, written once, used by every sender.

1
2
3
4
5

public static class ServiceBusLanes
{
    public static string ForOrder(string? correlationId, Guid integrationFlowUid) =>
        string.IsNullOrEmpty(correlationId) ? integrationFlowUid.ToString() : correlationId;
}

The correlation id is the key of the ordering tenant for that leg. It is the session id on every hop inside the hub and on both senders in the TMS, and the hub also sets ReplyToSessionId to it so replies travel the same lane back. The flow uid is only a fallback for ingress that has no correlation id yet. The driver app side moved from leg uid to transport order uid, through a cached lookup, so a trip with several legs stays in one lane too.

Both TMS senders now refuse a queue item that has no correlation id. Before, they sent it without a session id to a subscription that requires one, and the message was never delivered to anyone. It did not fail. It just sat there.

Failure one and a half, the retry that let later messages overtake

The audit found a second thing in the delivery function, and I had written it myself, proudly. On a transient failure the function completed the message and scheduled a clone thirty seconds in the future, with the same session id. Clean, no blocking, no poison.

Except a scheduled clone is a new enqueue. Every message of the same order that arrived in those thirty seconds went first. The retry did keep the session; it lost its place in it.

1
2
3
4
5

// Pace the retry while still holding the lock (lock duration PT1M, auto renewed by
// the host), then abandon: the message stays first in its session, so order is kept.
await Task.Delay(retryDelay);
await messageActions.AbandonMessageAsync(message);
return;

The wait blocks the session slot, which is why it is seconds now and not thirty. The subscription MaxDeliveryCount is above the five attempt ladder, so the ladder decides when a failure is terminal and only terminal failures reach the dead letter queue.

Failure two, the order was lost before the first enqueue

October 15, a few minutes before nine in the evening, in production. A driver reported arrival at a stop and, five seconds later, departure. The report stamped 20:56:50 was applied after the one stamped 20:56:55. The later report set the arrival time back to null, the departure then landed with the arrival in one payload, no milestone was created, and the wait time report of the driver stayed unlinked to anything. One support ticket, three symptoms, one cause.

The driver app writes reports to a Firebase out queue. A relay forwards every push to the external API in its own invocation, and two pushes a second apart can reach the API in either order. The session on the Service Bus side then replayed that order faithfully. Sessions preserve enqueue order. They cannot know what the enqueue order should have been.

I could not fix the relay; it is not mine. What I could change is what a message means. The out queue entry stays in Firebase until the TMS has committed it, so the queue is the exact set of reports not yet applied. A message is now a wake up for its leg. The consumer reads every pending entry for that uid, sorts them by the report stamp of the app, and applies each one in its own scope and unit of work, deleting each after its commit.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

pending = await CollectPendingAsync(driverAppProvider, queueMessage, tenantId);

if (pending.Count == 0)
{
    // Drained by an earlier message of the same leg; nothing to apply.
    await messageActions.CompleteMessageAsync(message);
    return;
}

// Oldest first. A failure stops the drain at that report; the earlier ones are
// committed and deleted, the message is retried, and the retry resumes from there.
foreach (var report in pending)
    await ApplyReportAsync(report, tenantId, tenantName, communicationProviderUid);

await messageActions.CompleteMessageAsync(message);

Every status is still written and every automation still fires, in the order the driver acted. A message whose entry was already drained by an earlier wake up completes as a no op. Before this change the same situation dead lettered after ten retries, because the consumer looked for one entry and found nothing.

The query needs an index on uid in the Firebase rules. Until the live rules have it the provider reads the whole node and logs a warning. I would rather read too much for a day than ship a consumer that fails on a missing index.

The bug I shipped with the fix

The drain moved the provider factory resolution out of the using block around the scoped DbContext. The repositories of the factory were built on a disposed context, and every report on QA failed with ObjectDisposedException. One line in the commit message says it all. The scope disposes the context; nothing else should. I now let the scope own the context and build everything that needs it inside the same block.

What sessions actually promise

Sessions promise that messages with one key are handled one at a time in enqueue order. Three things sit outside that promise, and each of them bit me.

The key has to mean the same thing at every hop. If one hop sessions by leg and the next by flow, you have ordered lanes that do not line up.

Retries must not re enqueue. Abandon keeps the place in the lane. A scheduled clone does not.

Enqueue order is not event order when something upstream can reorder. When that upstream keeps a durable record of what is pending, let the message be a wake up and let the record be the truth.

None of this is a reason to stop using sessions. It is a reason to write down, per hop, what the session key is and who guarantees the order before the first enqueue. I had not, and it cost me a month of odd tickets.