Day 39: Externalized Configuration

Externalized Configuration means the compiled artifact (a container image or a published build) contains no environment-specific settings and no secrets.

Intro

Externalized Configuration means the compiled artifact (a container image or a published build) contains no environment-specific settings and no secrets. Connection strings, feature toggles, URLs, and credentials are supplied at runtime from outside, from environment variables, a central configuration service, and a secret store such as Azure Key Vault. The same image then runs unchanged in Dev, Test, and Prod, and a password can be rotated without a rebuild or redeploy.

Why we need this

• Same artifact everywhere. The 12-factor rule “store config in the environment” lets you build once and promote the same image through environments. If config is baked in, you rebuild per environment and no longer test what you ship.
• Secrets must not live in source or images. Anything in a Git repo or image layer is readable by everyone with pull access and stays in history and registries. Leaked connection strings in appsettings.json are among the most common causes of cloud breaches.
• Operations without releases. Changing a timeout, a claims-approval threshold, or rotating a password should be an operational action with an audit trail, not a code change plus a pipeline run.
• Many services multiply the problem. With 30 microservices, every one has database, broker, and identity settings. Without a central place, drift and copy-paste errors are guaranteed.
• Compliance. Standards such as PCI DSS, ISO 27001 and SOC 2 expect secret rotation, least-privilege access, and access logs, which a secret store gives you and a config file cannot.

What problem it solves

Problem: configuration baked into images breaks 12-factor and leaks secrets.

In our insurance claims system, the ClaimsService needs a SQL Server connection string, a Service Bus namespace, a fraud-scoring API key, and a “auto-approve limit” business setting. Without externalization:

• appsettings.Production.json with the SQL password is committed to Git; a contractor’s laptop or a CI log exposes it.
• To change the auto-approve limit from 500 to 750, a developer edits the file, and the team rebuilds, re-tests, and redeploys the image.
• Prod and Test images differ, so “it worked in Test” proves little.
• Rotating the SQL password after an employee leaves means touching every service repo.
• Nobody can answer “who changed the fraud-API key and when?”

When it is needed (and when it is NOT)

Needed

• Any service that runs in more than one environment (practically all).
• Any service holding secrets: DB credentials, API keys, signing keys, certificates.
• Containerized or scaled-out services where instances must pick up changes without rebuilds.
• Regulated domains such as insurance, where access to secrets must be audited.

Overkill or wrong

• A throwaway prototype or a local-only tool: dotnet user-secrets and a JSON file are enough.
• Do not put everything in a central store. Static, non-environment-specific defaults (for example a JSON serializer option) belong in code or the bundled appsettings.json.
• Do not use a config service as a database. Values are small key/value settings, not business data.
• Do not make startup depend on a central service with no fallback. If the store is down, new instances must still start (cached or last-known-good values), otherwise config becomes a single point of failure.

How to identify the problem (key signals)

1. Secrets, passwords, or Server=...;Password=... strings appear in Git history, PR diffs, or a secret scanner report (GitHub secret scanning, gitleaks).
2. There are separate builds or Dockerfile ARGs per environment (“build-for-prod”).
3. A one-line setting change requires a full release cycle.
4. Environments drift: Test works, Prod fails, and the difference turns out to be a config value nobody tracked.
5. Secret rotation is a multi-day, multi-team event, or nobody dares to rotate.
6. docker history or an image scan shows credentials in layers.
7. Incident reviews ask “who changed this setting?” and nobody can say.

Flow Diagram

One image everywhere; settings and secrets are supplied at runtime with a managed identity.

flowchart LR
    CI["Build once"] --> IMG["Container image - no config"]
    IMG --> DEV["Dev"]
    IMG --> TEST["Test"]
    IMG --> PROD["Prod"]
    PROD -- "managed identity" --> AC["App Configuration - settings, flags"]
    AC -- "Key Vault reference" --> KV["Key Vault - secrets"]
    PROD -- "Entra auth, no password" --> SQL[("Azure SQL")]
    OPS["Operator bumps sentinel key"] --> AC

Level 1: Beginner

Analogy: A hotel room key. The lock (your code) is the same in every room. The key card (configuration) is issued at check-in, can be cancelled at any time, and is never welded into the door.

Minimal example. ASP.NET Core already layers configuration sources; later sources override earlier ones. WebApplication.CreateBuilder loads, in order: appsettings.json, appsettings.{Environment}.json, user secrets (Development only), environment variables, then command-line args.

// Program.cs (.NET 10, minimal API)
var builder = WebApplication.CreateBuilder(args);

// Bind a section to a typed class.
builder.Services.Configure<ClaimRules>(
    builder.Configuration.GetSection("ClaimRules"));

var app = builder.Build();

app.MapGet("/claims/limit", (Microsoft.Extensions.Options.IOptions<ClaimRules> opts)
    => opts.Value.AutoApproveLimit);

app.Run();

public class ClaimRules
{
    public decimal AutoApproveLimit { get; set; } = 500m;
}

// appsettings.json  (safe defaults only, no secrets)
{ "ClaimRules": { "AutoApproveLimit": 500 } }

Override at runtime without touching the image. The double underscore maps to the : section separator:

docker run -e ClaimRules__AutoApproveLimit=750 claims-service:1.4.2

For local development, keep secrets out of the repo:

dotnet user-secrets init
dotnet user-secrets set "ConnectionStrings:ClaimsDb" "Server=localhost;Database=Claims;User Id=dev;Password=dev-only;TrustServerCertificate=true"

Level 2: Intermediate

.NET: Options pattern, Key Vault, App Configuration

Pick the right Options interface:

Validate at startup so a bad value fails the deploy, not the first customer:

builder.Services
    .AddOptions<ClaimRules>()
    .Bind(builder.Configuration.GetSection("ClaimRules"))
    .Validate(r => r.AutoApproveLimit is > 0 and <= 10_000,
              "AutoApproveLimit must be between 1 and 10000")
    .ValidateOnStart();

Load secrets from Azure Key Vault and settings from Azure App Configuration, authenticating with a managed identity (no credentials in config). Packages: Azure.Identity, Azure.Extensions.AspNetCore.Configuration.Secrets, Microsoft.Azure.AppConfiguration.AspNetCore.

using Azure.Identity;
using Microsoft.Extensions.Configuration.AzureAppConfiguration;

var builder = WebApplication.CreateBuilder(args);

var credential = new DefaultAzureCredential();

if (!builder.Environment.IsDevelopment())
{
    // Non-secret settings + Key Vault references, with dynamic refresh.
    builder.Configuration.AddAzureAppConfiguration(o =>
    {
        o.Connect(new Uri(builder.Configuration["AppConfig:Endpoint"]!), credential)
         .Select("ClaimsService:*", labelFilter: builder.Environment.EnvironmentName)
         .ConfigureKeyVault(kv => kv.SetCredential(credential))
         .ConfigureRefresh(r => r
             .Register("ClaimsService:Sentinel", refreshAll: true)
             .SetRefreshInterval(TimeSpan.FromSeconds(30)));
    });
    builder.Services.AddAzureAppConfiguration();
}

var app = builder.Build();

if (!app.Environment.IsDevelopment())
    app.UseAzureAppConfiguration(); // triggers refresh checks per request

Only the App Configuration endpoint (a non-secret URL) is in local config. Secrets arrive as Key Vault references, which App Configuration resolves through the app’s identity.

Angular: config is different, the browser is public

Anything shipped to the browser is visible to the user. Angular has no secrets; it only receives non-secret runtime settings such as the API base URL. Do not use environment.prod.ts for per-environment URLs if you want one build for all environments, since it is compiled into the bundle. Load a JSON file at startup instead (Angular 19+ style with provideAppInitializer):

import { ApplicationConfig, inject, provideAppInitializer } from '@angular/core';
import { provideHttpClient } from '@angular/common/http';
import { AppConfigService } from './app-config.service';

export const appConfig: ApplicationConfig = {
  providers: [
    provideHttpClient(),
    provideAppInitializer(() => inject(AppConfigService).load()),
  ],
};

import { Injectable, inject } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { firstValueFrom } from 'rxjs';

export interface AppConfig { apiBaseUrl: string; }

@Injectable({ providedIn: 'root' })
export class AppConfigService {
  private http = inject(HttpClient);
  private cfg!: AppConfig;

  async load(): Promise<void> {
    // /assets/config.json is replaced per environment at deploy time.
    this.cfg = await firstValueFrom(this.http.get<AppConfig>('/assets/config.json'));
  }
  get apiBaseUrl() { return this.cfg.apiBaseUrl; }
}

The container entrypoint or pipeline writes config.json per environment; the built bundle stays identical.

Database side

The connection string lives in Key Vault, or better, is not a password at all: use Microsoft Entra authentication (managed identity) to Azure SQL and PostgreSQL Flexible Server so there is no secret to store.

// Azure SQL with managed identity, no password in config
// Connection string: "Server=tcp:claims-sql.database.windows.net;Database=Claims;Authentication=Active Directory Default;Encrypt=True"
builder.Services.AddDbContext<ClaimsDbContext>(o =>
    o.UseSqlServer(builder.Configuration.GetConnectionString("ClaimsDb")));

Level 3: Advanced

Precedence and layering. Order sources deliberately. A typical stack, lowest to highest priority: bundled defaults, App Configuration (shared across environments via labels), Key Vault references, environment variables set by the platform, and emergency command-line overrides. Document it; surprise overrides are the top cause of “why is this value wrong”.

Refresh strategy. Polling every request is wasteful. Use a sentinel key: services watch one key and reload everything when it changes, so an operator updates many keys and bumps the sentinel last. Alternatively, push refresh via Event Grid events from App Configuration to avoid polling. Remember IOptions<T> never refreshes.

Failure modes

• Central store outage at startup. Cache the last-known-good config (or mark optional sources optional: true for non-critical keys) and use retries. Fail fast only for values without which the service is unsafe.
• Throttling. Key Vault has request limits per vault and region. Hundreds of pods reading secrets at startup can hit HTTP 429. Read once at startup, cache, and stagger refresh; do not call Key Vault per request.
• Partial refresh. Updating 5 related keys one by one causes a window with inconsistent values. Sentinel keys and immutable versioned config solve this.
• Bad value pushed to all instances at once. Validation on reload (IValidateOptions<T>) plus keeping the previous value on failure limits the blast radius.

Security

• Managed identity + Azure RBAC (Key Vault Secrets User, App Configuration Data Reader), scoped to one vault per environment or per service. Avoid the legacy access-policy model for new vaults.
• Enable soft delete and purge protection on Key Vault; restrict network access with Private Endpoints and disable public access for production.
• Turn on diagnostic logs to Log Analytics; alert on unexpected SecretGet callers.
• Rotate: use versionless secret URIs so consumers pick up the latest version, and automate rotation (Key Vault rotation policies or Event Grid SecretNearExpiry events).
• Never log configuration objects wholesale. Mark sensitive properties and redact.

Common mistakes

1. Committing .env or appsettings.Production.json “just this once”.
2. Putting a “client secret” in Angular’s config.json (public by definition).
3. Using IOptions<T> and expecting live updates.
4. One shared Key Vault for all environments and all services (huge blast radius).
5. Storing a Key Vault client secret in config to authenticate to Key Vault (the “secret zero” problem; use managed identity or workload identity).
6. Baking config via Docker ENV or ARG in the Dockerfile (visible in image history).

Level 4: Expert and Architect view

Alternatives compared

Patterns it combines with

• Microservice Chassis (Day 40): the chassis wires the configuration providers, options validation, and Key Vault client once for all services.
• Service Template (Day 41): ships the pipeline step that injects config.json and the IaC for vault and RBAC.
• Health Check API (Day 36): a readiness check can verify that required configuration loaded.
• Access Token (Day 38): signing keys and issuer authority are fetched from config and Key Vault.
• Log Deployments & Changes (Day 37): config changes should emit change markers just like deployments.
• Feature flags: App Configuration feature management (Microsoft.FeatureManagement) uses the same store.

ADR: Externalized configuration with App Configuration and Key Vault

Status: Proposed

Context: ClaimsService, PolicyService, and PaymentsService run as containers in three environments. Secrets currently live in appsettings.Production.json; a rotation of the SQL credential requires rebuilding all images. The audit team requires evidence of who accessed production secrets.

Decision: Build one image per service version. Non-secret settings and feature flags go in Azure App Configuration (one store per environment, label per service). Secrets go in Azure Key Vault (one vault per environment), referenced from App Configuration. Services authenticate with managed identity and Azure RBAC. Where supported, use Entra authentication to databases so no password exists. The Angular app reads a runtime config.json containing only non-secret values.

Consequences:

• (+) Build once, promote everywhere; rotation and setting changes need no rebuild.
• (+) Audit through diagnostic logs and change history.
• (-) New runtime dependency; mitigated by cached last-known-good values and retries.
• (-) Added platform cost and operational skill (RBAC, private networking).

Alternatives rejected: Kubernetes Secrets alone (weaker rotation and audit, coupled to cluster); self-hosted HashiCorp Vault (operational overhead, no multi-cloud requirement today).

Azure implementation

Services

• Azure Key Vault for secrets, keys, and certificates.
• Azure App Configuration for settings, feature flags, and Key Vault references.
• Managed identities (system- or user-assigned) and Azure RBAC for access.
• Azure Container Apps / AKS / App Service for hosting. App Service and Container Apps can reference Key Vault secrets directly in app settings/secrets; AKS uses the Azure Key Vault provider for Secrets Store CSI Driver with workload identity.
• Event Grid for rotation and refresh events; Azure Monitor / Log Analytics for audit logs; Private Endpoints for network isolation.

Configuration steps (Azure CLI)

# 1. Vault with RBAC, soft delete on by default, purge protection on
az keyvault create -n kv-claims-prod -g rg-claims-prod -l westeurope \
  --enable-rbac-authorization true --enable-purge-protection true

# 2. App Configuration store
az appconfig create -n appcs-claims-prod -g rg-claims-prod -l westeurope --sku Standard

# 3. Give the service's managed identity least-privilege roles
az role assignment create --assignee <claims-service-principal-id> \
  --role "Key Vault Secrets User" \
  --scope $(az keyvault show -n kv-claims-prod --query id -o tsv)
az role assignment create --assignee <claims-service-principal-id> \
  --role "App Configuration Data Reader" \
  --scope $(az appconfig show -n appcs-claims-prod --query id -o tsv)

# 4. Store a secret and a setting that references it
az keyvault secret set --vault-name kv-claims-prod -n FraudApiKey --value "<value>"
az appconfig kv set -n appcs-claims-prod --key "ClaimsService:AutoApproveLimit" --value 750 --label Production -y
az appconfig kv set-keyvault -n appcs-claims-prod --key "ClaimsService:FraudApiKey" \
  --secret-identifier https://kv-claims-prod.vault.azure.net/secrets/FraudApiKey --label Production -y

Pricing and tier considerations (confirm current figures on the Azure pricing pages before budgeting, as rates and tier names change and vary by region)

• Key Vault: Standard and Premium tiers. Premium adds HSM-protected keys. Secrets and software-key operations are billed per 10,000 operations, and certificates carry per-renewal charges. Managed HSM is a separate, hourly-billed service and is much more expensive; use it only for strict key-custody requirements. Operations volume is low if you cache at startup.
• App Configuration: a Free tier (limited storage and daily requests, no SLA, one store per subscription per region) suits dev/test. Standard is the production choice: SLA, higher limits, private endpoints, and geo-replication options. Check the pricing page for the newer lower-cost developer tier and its limits before choosing.
• Cost driver to watch: request volume from refresh polling. Fewer instances polling at a longer interval, or Event Grid push refresh, keeps costs and throttling low.

Reference architecture (text)

1. A pipeline builds one image per service and pushes it to Azure Container Registry. The image contains no environment values.
2. IaC (Bicep/Terraform) provisions per-environment resource groups, each with a Key Vault, an App Configuration store, and a managed identity per service, wired by RBAC.
3. Container Apps (or AKS) start the image. The service authenticates with its managed identity, loads settings from App Configuration and secrets via Key Vault references, and connects to Azure SQL using Entra authentication.
4. Operators change a value in App Configuration and bump the sentinel key; services refresh within the polling interval.
5. Key Vault and App Configuration send diagnostic logs to Log Analytics; alerts fire on unexpected access. Private Endpoints keep traffic off the public internet.
6. The Angular static site (Static Web Apps or Blob + CDN) serves a config.json generated per environment in the release step.

Teaching guide for my team

2-minute beginner explanation “Our code is like a car, and configuration is the fuel and the destination. We build the car once. Where it drives and what fuel it uses is given when we start it. Passwords are never written inside the car; we hand them over from a locked safe (Key Vault) when the car starts. That way the same car drives in Test and Prod, and if a password leaks we change it in the safe without building a new car.”

5-minute intermediate explanation Walk the layers: defaults in appsettings.json, environment overrides, App Configuration for shared settings, Key Vault for secrets. Show IOptions vs IOptionsSnapshot vs IOptionsMonitor, then ValidateOnStart. Explain managed identity as the answer to “how does the app prove who it is without a password”. Stress that Angular can never hold secrets and reads a runtime config.json. Close with failure modes: store outage and throttling.

Hands-on exercise

1. Create a .NET minimal API with a ClaimRules class and ValidateOnStart.
2. Run it with AutoApproveLimit from appsettings.json (500), then override with docker run -e ClaimRules__AutoApproveLimit=750 without rebuilding.
3. Set a bad value (-5) and observe the app fail at startup with the validation message.
4. Stretch: create a free-tier App Configuration store, read a key, and change it with a sentinel refresh.

Expected outcome: the same image returns 500, then 750, then refuses to start on -5. No rebuild happens, and no secret appears in the Git repo or docker history.

Interview questions

1. Why not put the connection string in appsettings.Production.json? It ends up in Git and the image, cannot be rotated without a rebuild, and cannot be access-audited.
2. Difference between IOptionsSnapshot and IOptionsMonitor? Snapshot is scoped and re-read per request; Monitor is a singleton that notifies on change. IOptions never reloads.
3. How does an app authenticate to Key Vault without a secret? With a managed identity (or workload identity in AKS) assigned an RBAC role, used through DefaultAzureCredential.

Mastery checklist

• I can list the .NET configuration sources and their override order.
• I can choose IOptions, IOptionsSnapshot, or IOptionsMonitor correctly and validate on start.
• I can explain why the same image must run in every environment.
• I can configure Key Vault and App Configuration with managed identity and RBAC.
• I can explain why Angular cannot hold secrets and implement runtime config.json loading.
• I can design for store outage and throttling (caching, last-known-good, sentinel refresh).
• I can describe a secret rotation process with no downtime.
• I can write an ADR justifying the store choice against Kubernetes Secrets and HashiCorp Vault.

Key takeaway

Build once, configure at runtime: keep settings and secrets outside the artifact, fetch them with a managed identity, validate them at startup, and prefer having no secret at all over guarding one well.