Azure-Service-Bus on ND Teknik

Unit of Work Batching for Long-Running Operations: From Request to Background Job to Queue

Wed, 22 Apr 2026 00:00:00 +0000

TLDR: Long-running batch operations (invoice generation, document export) that process thousands of items need explicit transaction scope management, not implicit per-item commits. Use IUnitOfWorkManager to batch 50–100 items per SaveChanges, Hangfire for orchestration, Service Bus for async distribution, and Durable Functions for multi-phase workflows. Report progress every 5% to avoid database thrashing. We cut invoice generation from 45 minutes to 8 minutes for 50,000 orders—and made progress visible without killing the database.

The Problem: Implicit Transactions Don’t Scale

A naive batch operation to generate billing documents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


public async Task GenerateBillingDocumentsAsync(int[] orderIds, CancellationToken cancellationToken)
{
 foreach (var orderId in orderIds)
 {
 var order = await _context.Orders.FindAsync(orderId);
 var document = new BillingDocument { OrderId = orderId, Amount = order.Total };
 _context.BillingDocuments.Add(document);
 await _context.SaveChangesAsync(cancellationToken); // One SaveChanges per order!
 }
}

For 50,000 orders:

50,000 SaveChanges calls
50,000 database roundtrips
50,000 audit log entries written per-item
Each SaveChanges acquires a transaction, validates constraints, writes audit trail, releases lock
Total time: 45 minutes

The API request times out. Background job thrashes the database. Progress is invisible.

The Solution: Explicit Batching with IUnitOfWorkManager

Define a unit of work manager that groups operations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


public interface IUnitOfWorkManager
{
 Task<T> ExecuteInUnitOfWorkAsync<T>(Func<Task<T>> work);
 Task ExecuteInUnitOfWorkAsync(Func<Task> work);
 void DisableAudit();
 void EnableAudit();
}

public class UnitOfWorkManager : IUnitOfWorkManager
{
 private readonly BillingContext _context;
 private bool _auditEnabled = true;

 public async Task ExecuteInUnitOfWorkAsync(Func<Task> work)
 {
 try
 {
 _context.ChangeTracker.QueryTrackingBehavior = QueryTrackingBehavior.TrackAll;
 await work();
 await _context.SaveChangesAsync();
 }
 catch
 {
 _context.ChangeTracker.Clear();
 throw;
 }
 }

 public void DisableAudit() => _auditEnabled = false;
 public void EnableAudit() => _auditEnabled = true;
 public bool IsAuditEnabled => _auditEnabled;
}

Now refactor the batch operation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


public async Task GenerateBillingDocumentsAsync(
 int[] orderIds,
 IProgress<BatchProgress> progress,
 CancellationToken cancellationToken)
{
 const int batchSize = 100;
 var documents = new List<BillingDocument>();
 var totalBatches = (orderIds.Length + batchSize - 1) / batchSize;
 var lastProgressReport = 0;

 for (int i = 0; i < orderIds.Length; i += batchSize)
 {
 var batch = orderIds.Skip(i).Take(batchSize).ToArray();

 await _unitOfWorkManager.ExecuteInUnitOfWorkAsync(async () =>
 {
 foreach (var orderId in batch)
 {
 var order = await _context.Orders
 .AsNoTracking()
 .FirstOrDefaultAsync(o => o.Id == orderId, cancellationToken);

 if (order == null) continue;

 var document = new BillingDocument
 {
 OrderId = orderId,
 Amount = order.Total,
 CreatedAt = DateTime.UtcNow
 };
 _context.BillingDocuments.Add(document);
 }
 });

 // Report progress at 5% intervals only, not per-item
 var currentProgress = (int)((i + batchSize) / (double)orderIds.Length * 100);
 if (currentProgress - lastProgressReport >= 5)
 {
 progress?.Report(new BatchProgress
 {
 CompletedItems = i + batchSize,
 TotalItems = orderIds.Length,
 PercentComplete = currentProgress
 });
 lastProgressReport = currentProgress;
 }
 }
}

Result:

500 SaveChanges calls (instead of 50,000)
500 database roundtrips (100× fewer)
8 minutes (5.6× faster)
Progress updated every 5%, not per-item

Architecture: Three Phases

For truly large batches (100,000+ items), separate into phases:

Phase 1: Hangfire (API → Background Job)

The API request initiates the job, then returns immediately:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


[ApiController]
[Route("api/billing")]
public class BillingController : ControllerBase
{
 private readonly IBackgroundJobClient _jobClient;

 [HttpPost("generate-documents")]
 public ActionResult QueueBillingGeneration([FromBody] GenerateBillingRequest req)
 {
 var jobId = _jobClient.Enqueue<BillingService>(
 x => x.GenerateBillingDocumentsAsync(req.OrderIds, null, default));

 return Accepted(new { jobId });
 }
}

The Hangfire job batches 100 orders per UoW, writes documents to the database:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


[BackgroundJob]
public async Task GenerateBillingDocumentsAsync(
 int[] orderIds,
 IJobCancellationToken cancellationToken)
{
 var jobId = GetCurrentJobId(); // Hangfire provides this
 var progressStore = new HangfireProgressStore(_jobStore);

 var progress = new Progress<BatchProgress>(p =>
 {
 // Report to Hangfire job store every 5%
 progressStore.UpdateProgress(jobId, p.PercentComplete);
 });

 try
 {
 await GenerateBillingDocumentsAsync(orderIds, progress, cancellationToken.ShutdownToken);
 }
 catch (Exception ex)
 {
 progressStore.MarkFailed(jobId, ex.Message);
 throw;
 }
}

Phase 2: Service Bus (Batch → Distribution)

Once documents are created, enqueue them for distribution to external carriers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


public async Task EnqueueDocumentsForDistributionAsync(
 List<int> documentIds,
 CancellationToken cancellationToken)
{
 const int batchSize = 50;
 var sender = _serviceBusClient.CreateSender("carrier-distribution");

 for (int i = 0; i < documentIds.Count; i += batchSize)
 {
 var batch = documentIds.Skip(i).Take(batchSize).ToArray();

 var messages = batch.Select(docId => new ServiceBusMessage
 {
 Body = new BinaryData(new { DocumentId = docId }),
 CorrelationId = Guid.NewGuid().ToString(),
 TimeToLive = TimeSpan.FromHours(24),
 ScheduledEnqueueTime = DateTime.UtcNow
 }).ToArray();

 await sender.SendMessagesAsync(messages, cancellationToken);
 }
}

Service Bus handles fanout, retry, and dead-lettering. If a carrier is temporarily down, the message stays in the queue.

Phase 3: Durable Functions (Orchestration + Retry)

For multi-step workflows (create docs → validate → push to carriers → post accounting entries):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


[FunctionName("DocumentDistributionOrchestrator")]
public static async Task RunOrchestrator(
 [OrchestrationTrigger] IDurableOrchestrationContext context)
{
 var documentIds = context.GetInput<int[]>();

 var activityOptions = new RetryOptions(
 firstRetryInterval: TimeSpan.FromSeconds(5),
 maxNumberOfAttempts: 3)
 {
 Handle = ex => ex is HttpRequestException
 };

 try
 {
 // Phase 1: Push to external carriers (with 3 retries)
 await context.CallActivityWithRetryAsync(
 "PushToCarriersAsync",
 activityOptions,
 documentIds);

 // Phase 2: Post accounting entries (with 3 retries)
 await context.CallActivityWithRetryAsync(
 "PostAccountingEntriesAsync",
 activityOptions,
 documentIds);

 // Phase 3: Update document status (no retries; fail fast if DB issues)
 await context.CallActivityAsync("MarkDocumentsCompleteAsync", documentIds);
 }
 catch (Exception ex)
 {
 await context.CallActivityAsync("MarkDocumentsFailedAsync",
 new { DocumentIds = documentIds, Error = ex.Message });
 throw;
 }
}

[FunctionName("PushToCarriersAsync")]
public async Task PushToCarriers([ActivityTrigger] int[] documentIds)
{
 foreach (var docId in documentIds)
 {
 var document = await _context.BillingDocuments.FindAsync(docId);
 await _carrierClient.PushAsync(document);
 }
}

Disabling Audit During Batch Operations

Audit logging on every item multiplies database load. Disable during batching:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


public async Task GenerateBillingDocumentsAsync(int[] orderIds, CancellationToken cancellationToken)
{
 _unitOfWorkManager.DisableAudit(); // Skip audit writes per-item

 try
 {
 // ... batching loop ...
 }
 finally
 {
 _unitOfWorkManager.EnableAudit();
 }

 // Post-batch: create single audit entry for the entire operation
 _auditService.LogBulkOperation(new BulkAuditEntry
 {
 Operation = "BillingDocumentGeneration",
 ItemCount = orderIds.Length,
 CompletedAt = DateTime.UtcNow
 });
}

This keeps audit trail (compliance requirement) while eliminating per-item overhead.

Real Scenario: 50,000-Order Monthly Invoice Run

Timeline (end-to-end):

T+0m: API call queues Hangfire job, returns 202 Accepted
T+2m: Hangfire reads 50,000 orders from analytics DB, batches by 100
T+8m: Creates 50,000 billing documents in primary DB (8 units of work)
T+8.5m: Enqueues all documents to Service Bus (fire-and-forget)
T+8.5m–T+12m: Service Bus distributes to 500 carriers in parallel
T+12m–T+18m: External carriers process, return EDI confirmations
T+18m–T+22m: Durable Functions post accounting entries per carrier batch
T+22m: Mark documents complete, update billing account status

Total: 22 minutes (vs 45 minutes without batching + async phases)

Lessons Learned: The Hard Way

1. Batch size is not universal We used batch size = 50 initially. For orders with many charges, creating 50 documents per UoW overwhelmed the change tracker. Switched to:

Simple entities (no navigation): 200 per batch
Complex entities (many relationships): 50 per batch
Measure with context.ChangeTracker.Entries().Count() logging

2. Progress reporting at the wrong granularity kills performance Reporting progress per-item meant 50,000 database writes to the progress table. Switched to 5% intervals: 20 writes for the entire operation. Progress UI polls the job status endpoint instead of hitting the table per-item.

3. Async operations need explicit timeouts Service Bus enqueue has no timeout; if the queue backs up, the entire job hangs. Added explicit CancellationToken with timeout after 5 minutes.

4. Rollback at scale is expensive A constraint violation in item #47,382 doesn’t fail the entire batch in EF Core—just that item. But rolling back and retrying means regenerating work for items 1–47,381. Better: validate before batching, or skip invalid items with logging.

5. Load distribution matters Running all 50,000 generates in a single Hangfire worker overloaded the DB. Split into 10 parallel jobs (5,000 items each) with staggered start times.

Gotchas and Disclaimers

EF Core 8+: Earlier versions handle change tracker differently; batching thresholds vary.
Distributed transactions: Service Bus + Durable Functions + Database are not ACID. Expect eventual consistency. Implement idempotent operations.
Memory leaks: context.ChangeTracker in loops holds references. Call context.ChangeTracker.Clear() after each batch.
.NET 9+: Recommended for TimeProvider improvements and better async cancellation.

Next Steps

Measure current batch operation duration and database impact.
Introduce explicit UoW batching with size = 100 (tune from there).
Add progress reporting at 5% intervals, not per-item.
Offload async distribution to Service Bus + Durable Functions.

Ready for featured image.

Distributed Transaction Patterns: When ACID Goes to Sleep

Thu, 22 Jan 2026 00:00:00 +0000

In a monolith, you had transactions. In microservices, you have heartache.

When your transportation management system routes a delivery across multiple microservices—booking a shipment, reserving capacity, charging a customer—you can’t just wrap it in a database transaction. The booking service lives in a different database. So does the payment service. Welcome to the world of distributed transactions.

The traditional ACID guarantees you relied on are gone. But the requirement to keep your system consistent? That’s still there.

The Problem: Distributed Monstrosity

Imagine this scenario in our real-time logistics platform:

Customer requests a delivery from NYC to Boston.
Shipment Service creates a shipment order.
Capacity Service reserves truck space.
Billing Service charges the customer’s account.
Notification Service sends a confirmation.

What if step 3 fails? The shipment is booked, but the truck is full. Now you’ve sold something you can’t deliver. Or step 4 fails—the payment is declined. Your shipment is reserved, but unpaid.

In a monolith, you’d roll back the transaction. In microservices, you need something cleverer.

Enter the Saga Pattern

A Saga is a sequence of local transactions, each triggered by the completion of the previous one. If any step fails, you execute compensating transactions to undo the work.

There are two flavors:

1. Choreography (Event-Driven)

Each service publishes an event when it completes. Other services listen and react.

1
2
3
4
5
6
7


Shipment Service publishes "ShipmentCreated"
 ↓
Capacity Service listens → reserves space → publishes "CapacityReserved"
 ↓
Billing Service listens → charges customer → publishes "PaymentProcessed"
 ↓
Notification Service listens → sends email

If Billing fails, it publishes PaymentFailed. Capacity Service listens and releases the reservation. Shipment Service cancels the order.

Pros: Loosely coupled, no central orchestrator. Cons: Hard to see the full flow. Difficult to test. Requires robust error handling everywhere.

2. Orchestration (Centralized)

A central orchestrator (like Azure Durable Functions) tells each service what to do and when.

1
2
3
4
5
6
7


Durable Function:
 1. Call Shipment Service → create order
 2. Call Capacity Service → reserve space
 3. Call Billing Service → charge customer
 4. Call Notification Service → notify customer

 If any step fails, execute compensating transactions in reverse order.

Pros: Clear, testable flow. Easy to see the entire transaction. Cons: The orchestrator becomes a bottleneck. Single point of failure.

Implementation: Orchestrating with Azure Durable Functions

Here’s how we handle delivery booking using orchestration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82


[FunctionName("BookDeliveryOrchestrator")]
public static async Task RunOrchestrator(
 [OrchestrationTrigger] IDurableOrchestrationContext context,
 BookDeliveryRequest request)
{
 var deliveryId = context.NewGuid().ToString();

 try
 {
 // Step 1: Create shipment
 var shipment = await context.CallActivityAsync<ShipmentDto>(
 nameof(CreateShipmentActivity),
 new { request, deliveryId });

 // Step 2: Reserve capacity
 var reservation = await context.CallActivityAsync<ReservationDto>(
 nameof(ReserveCapacityActivity),
 new { shipment, request });

 // Step 3: Process payment
 var payment = await context.CallActivityAsync<PaymentDto>(
 nameof(ProcessPaymentActivity),
 new { request.CustomerId, request.Amount });

 // Step 4: Send notification
 await context.CallActivityAsync(
 nameof(SendNotificationActivity),
 new { deliveryId, request.CustomerEmail });

 return new { success = true, deliveryId, shipment, reservation, payment };
 }
 catch (Exception ex)
 {
 // Compensating transactions: undo in reverse order

 // Undo payment
 await context.CallActivityAsync(
 nameof(RefundPaymentActivity),
 new { request.CustomerId });

 // Undo reservation
 await context.CallActivityAsync(
 nameof(CancelReservationActivity),
 new { request.ShipmentType, request.Weight });

 // Undo shipment
 await context.CallActivityAsync(
 nameof(CancelShipmentActivity),
 new { deliveryId });

 throw;
 }
}

[FunctionName(nameof(CreateShipmentActivity))]
public static async Task<ShipmentDto> CreateShipmentActivity(
 [ActivityTrigger] IDurableActivityContext context,
 dynamic input)
{
 var request = (BookDeliveryRequest)input.request;
 var deliveryId = (string)input.deliveryId;

 // Call Shipment Service
 using var client = new HttpClient();
 var response = await client.PostAsJsonAsync(
 "https://shipment-service/api/shipments",
 new { deliveryId, request.Origin, request.Destination, request.Weight });

 return await response.Content.ReadAsAsync<ShipmentDto>();
}

[FunctionName(nameof(RefundPaymentActivity))]
public static async Task RefundPaymentActivity(
 [ActivityTrigger] IDurableActivityContext context,
 string customerId)
{
 // Call Billing Service to refund
 using var client = new HttpClient();
 await client.PostAsJsonAsync(
 "https://billing-service/api/refunds",
 new { customerId });
}

When to Use Saga vs. Other Patterns

Use Saga when:

You need eventual consistency (not immediate ACID)
Transactions span multiple services
You can design compensating transactions
You’re okay with temporary inconsistency

Don’t use Saga when:

You absolutely need immediate consistency (use distributed locks or stronger consistency patterns instead)
Compensating transactions are impossible (you can’t “undo” a physical truck dispatch)
Performance is critical and you need faster responses

Key Gotchas

Idempotency: Each activity must be idempotent. If Durable Functions retries ProcessPaymentActivity, you don’t want to charge twice.
Compensating Transactions Might Fail: What if the refund fails? You need a dead-letter mechanism.
Testing is Hard: Test the happy path and every possible failure scenario.
Monitoring is Critical: You need visibility into which saga is where in its lifecycle. Add Application Insights logging.

Lessons from Our Platform

We’ve learned that orchestration scales better for us than choreography. With 15+ microservices, trying to track the flow through events became a debugging nightmare. Durable Functions gave us a central, auditable record of each delivery transaction.

The trade-off? The orchestrator is now a critical path. We run multiple instances and monitor it closely.

Remember: Distributed transactions are hard. The Saga pattern isn’t magic—it’s a framework for managing that hardness. Design your compensating transactions carefully, test thoroughly, and monitor relentlessly.