Serverless was the slow part; moving two Function Apps off Flex Consumption

Wed, 16 Sep 2026 00:00:00 +0000

An order took 70 to 110 seconds to travel from one tenant to another. Warm, the same trip takes about two seconds.

I have written a lot of kind words about serverless on this blog, and I stand by them. This is the post where I describe the workload serverless was wrong for, and what it cost me to find out.

The chain

The transport management system I work on lets one tenant order a leg and another tenant execute it. In between sits an integration hub, a separate product with its own Function App. A dispatch goes through roughly this.

1

TMS tenant A ──Service Bus──► Hub receive ──Service Bus──► Hub mapping ──Service Bus──► Hub delivery ──Service Bus──► TMS tenant B

Every arrow is a Service Bus hop and every box is a function. Status reports from the driver app walk a similar chain in the other direction. In QA, both Function Apps ran on Flex Consumption. It was the modern choice, it was cheap, and it scaled to zero between messages. All three facts turned out to matter.

Where the time went

Flex scales each app to zero when nothing runs. The next message pays a host cold start, and for a .NET isolated app with forty odd functions and a few hundred megabytes of dependencies that start was 10 to 30 seconds in my measurements. One cold start is tolerable. The chain above pays it at every hop, in two different apps, and the hops are serial. A dispatch measured 20 to 34 seconds. A full order with its status replies measured 70 to 110 seconds.

Nobody noticed in the first weeks because the testers sent one message at a time and went for coffee. Real users send a dispatch and then watch the screen.

The first fix was a half fix, and I want to be honest about it. The hub app also runs Durable Functions, and a timer polled an external system every few minutes. On Flex that timer alone caused 15 host restarts in two hours, with the starter function at 1.2 seconds per run instead of a few milliseconds. I pinned one always ready instance to the durable scaling group. Durable got warm. The Service Bus and timer functions kept scaling from zero, because always ready instances on Flex are assigned per function group, not per app. The chain stayed slow.

The arithmetic

Flex lets you buy always ready instances per group. To keep the whole chain warm I counted eight groups across the two apps. At the QA prices of the time that was about 168 USD per environment per month, for QA and production each. The point of serverless was supposed to be paying for nothing while idle; paying to never be idle felt like the wrong tool defending itself.

So I looked at what the apps actually used. Seven days of telemetry showed about two busy instances on average and a host peak of 1.8 GB. That is a small dedicated plan. A B2 for the TMS app and a B1 for the hub, both always on, came to 39 USD per environment. The B2 alone keeps all 42 TMS functions warm for 26 USD. No function pays a cold start outside a deploy or platform maintenance, and the scale up path is a SKU change on the plan, B3 and then Premium v3 for autoscale, none of which recreate the app.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


resource plan 'Microsoft.Web/serverfarms@2023-12-01' = {
 name: planName
 location: resourceGroup().location
 kind: 'linux'
 sku: planSku // { name: 'B2', tier: 'Basic' }
 properties: {
 reserved: true
 zoneRedundant: false
 }
}

// in the site config
siteConfig: {
 alwaysOn: true
 linuxFxVersion: 'DOTNET-ISOLATED|10.0'
 appSettings: [
 { name: 'FUNCTIONS_EXTENSION_VERSION', value: '~4' }
 { name: 'FUNCTIONS_WORKER_RUNTIME', value: 'dotnet-isolated' }
 { name: 'WEBSITE_RUN_FROM_PACKAGE', value: '1' }
 ...
 ]
}

Two settings differences bit me on the way. A dedicated plan takes the runtime from FUNCTIONS_WORKER_RUNTIME and linuxFxVersion; Flex took it from functionAppConfig and rejected those settings. And a dedicated plan keeps the deployed package on its own disk, so WEBSITE_RUN_FROM_PACKAGE with no content share is enough.

ARM will not convert the plan for you

This is the part that turned an afternoon into a week. A site cannot change plan kind in place. ARM refuses the serverFarmId change, and it refuses to turn an FC1 plan into a Basic one. The only path is to delete the app and the plan and let the deployment recreate both.

The app holds no state, so deletion sounds harmless. It is not, for three reasons.

Deleting the app deletes its managed identity. The new app gets a new principal. My role assignments are named after the site, not the principal, so the tombstoned assignments of the dead identity collide with the new ones and the deployment fails with RoleAssignmentUpdateNotPermitted. The deploy script now lists assignments by principal id and removes them before the template runs. Not by assignee; once the identity is gone from Entra, the CLI cannot resolve it and exits with an error. The Cosmos SQL role assignments and the Managed Redis access policy assignment bind the principal too, so they go as well.

The SQL and PostgreSQL users were created FROM EXTERNAL PROVIDER, which binds the object id. The template cannot fix those. The script prints the steps at the end and a human runs them.

In production the app integrates into a subnet, and the delegation of that subnet changes from Microsoft.App/environments to Microsoft.Web/serverFarms. The subnet accepts the new delegation only once the service association link of the Flex app is gone, and deleting a site without disconnecting it first can leave that link orphaned. The script disconnects first, deletes, then polls the subnet for up to five minutes. If the link is still there it prints the purge call and stops.

1
2
3


az rest --method POST \
 --uri "/subscriptions/$SUB/providers/Microsoft.Web/locations/swedencentral/purgeUnusedVirtualNetworkIntegration?api-version=2024-04-01" \
 --body "{'subnetResourceId': '$OLD_FUNC_SUBNET'}"

The guard in the deploy script checks the live plan tier, runs once, and never fires again. What if mode warns that the real run would delete first, because the what if diff shows an in place update that ARM would refuse.

Was Flex wrong, or was I?

Both, a little. Flex Consumption is a good product. For bursty HTTP, for a queue worker that does one job, for anything where a cold start is paid once and then amortized over a burst, I would pick it again tomorrow. I still run other apps on it.

A chain is different. Latency compounds along it, idle time between hops is the normal state, and the thing a user waits for is the sum of every cold start on the path. For that shape, two small always on boxes are faster and cheaper than the serverless plan, and they are boring to operate.