Day 14: Polling Publisher

Summary.

Intro

Summary. A Polling Publisher is a small background worker that periodically reads a service’s outbox table, publishes the pending messages to a message broker, and marks them as delivered. It is the simplest way to make “save data + announce it” reliable when your database cannot stream its transaction log (no CDC) or you are not allowed to touch the platform. You trade a little latency and some polling load for very low operational complexity. In our insurance claims system, it is how ClaimsService reliably tells the rest of the world “Claim CLM-1042 was approved” without ever losing or inventing an event.

Why we need this

When a claim is approved, two things must happen: the claim row is updated in SQL Server, and a ClaimApproved event is published so that Payments, Notifications and Reporting can react. The database and the broker are two separate systems with no shared transaction. If we do both “at the same time” in application code (a dual write), one can succeed while the other fails.

The Transactional Outbox pattern (Day 12) fixes this by writing the event into an OutboxMessages table in the same local ACID transaction as the business change. But an outbox table alone delivers nothing; something must move rows from the table to the broker. Polling Publisher is that “something”, in its simplest form.

Business reasons: no lost payments, no missed customer notifications, and consistent downstream data. Technical reasons: broker outages must not fail claim processing, and the design must work on any relational database with no special privileges.

What problem it solves

Problem (from the topic list): the database lacks CDC (Change Data Capture) or platform access is restricted, so we cannot use Transaction Log Tailing (Day 13), yet we still need outbox messages delivered.

What goes wrong without it:

• Outbox rows pile up and nobody publishes them, so the outbox is a write-only graveyard.
• Teams “fix” this by publishing inside the request handler after SaveChanges(). If the process crashes between the commit and the publish, the event is lost forever. If they publish first and the commit fails, they announce something that never happened.
• Teams write ad-hoc cron scripts per service, each with different retry, ordering and locking behaviour, and duplicates or gaps appear under load.

Polling Publisher gives one well-understood, testable relay loop: read pending -> publish -> mark done, with at-least-once delivery.

When it is needed (and when it is NOT)

Fits when:

• The database has no CDC / you cannot enable it (e.g., SQL Server Standard on a locked-down server, managed DB with restricted permissions, legacy DB owned by another team).
• Throughput is moderate (up to a few hundred to low thousands of messages per second per service, depending on batch size and DB).
• A latency of 100 ms to a few seconds between commit and publish is acceptable (claim status notifications, audit feeds, reporting updates).
• You want the least moving parts: one worker, one table, no Debezium/Kafka Connect cluster.
• You are already using the Outbox pattern and need a first, simple relay you can replace later.

Not a fit when:

• You need near-real-time (tens of milliseconds) propagation. Use log tailing/CDC or, on Azure, evaluate Change Feed (Cosmos DB) instead.
• Very high volume makes polling queries a measurable DB load. Log tailing scales better.
• The service does not have a database transaction to begin with (e.g., it only calls other APIs). There is nothing to make atomic.
• Strict global ordering across many partitions is required and you cannot partition the outbox by aggregate. Polling with several workers can reorder.
• You can publish inside the same transaction resource (some brokers with XA/DTC). Rare in cloud-native systems; usually not recommended.

How to identify the problem (key signals)

You probably need a Polling Publisher (or the outbox behind it) when you see:

1. “Ghost” or missing events: Payments never got ClaimApproved for a claim that is Approved in the database. Reconciliation reports show mismatches between services.
2. Event published for rolled-back data: downstream shows a claim as approved that the database says is still Pending.
3. Code smell: await _db.SaveChangesAsync(); await _bus.PublishAsync(...) back to back in a handler, with a try/catch that only logs.
4. Incidents that correlate with broker or deploy events: inconsistencies appear right after a Service Bus throttling period or a pod restart.
5. Manual “replay” scripts in the runbook that someone runs whenever data drifts.
6. Outbox health metrics (once you have the pattern): COUNT(*) WHERE ProcessedOnUtc IS NULL growing steadily, or the age of the oldest unprocessed row rising.
7. Support tickets: “Customer never got the approval email, but the claim shows Approved.”

Flow Diagram

A worker polls pending outbox rows, publishes them, and marks them processed.

flowchart TD
    START(["Timer tick"]) --> Q["SELECT TOP 50 pending WITH UPDLOCK, READPAST"]
    Q --> ANY{"Rows found?"}
    ANY -- "no" --> WAIT["Back off and sleep"] --> START
    ANY -- "yes" --> SEND["Send to Service Bus with MessageId"]
    SEND --> OK{"Sent?"}
    OK -- "yes" --> MARK["Set ProcessedOnUtc"]
    OK -- "no" --> FAIL["Attempts++, LastError, stop batch"]
    MARK --> COMMIT["Commit"]
    FAIL --> COMMIT
    COMMIT --> START

Level 1: Beginner

Analogy. Think of an office out-tray. Whenever you finish a claim, you put a note in the out-tray as part of finishing the paperwork (same action, cannot be separated). Every 5 seconds an office assistant walks by, collects the notes, delivers them to other departments, and stamps each note “delivered”. If the assistant is sick for a day, notes wait safely in the tray; nothing is lost.

Minimal example (C#, console, in-memory to show the loop only):

// Minimal in-memory demo of the polling loop (top-level statements, .NET 10 console app).
var outbox = new List<OutboxItem>
{
    new(Guid.NewGuid(), "ClaimApproved", """{"claimId":"CLM-1042"}""")
};

while (true)
{
    var pending = outbox.Where(o => !o.Processed).Take(10).ToList();
    foreach (var item in pending)
    {
        Console.WriteLine($"Publishing {item.Type}: {item.Payload}");
        item.Processed = true;               // mark only AFTER publish succeeds
    }
    await Task.Delay(TimeSpan.FromSeconds(2));
}

class OutboxItem(Guid id, string type, string payload)
{
    public Guid Id { get; } = id;
    public string Type { get; } = type;
    public string Payload { get; } = payload;
    public bool Processed { get; set; }
}

Key beginner rules:

• The business row and the outbox row are written in one transaction (that is the outbox pattern).
• The publisher marks a row as processed only after the broker accepted it.
• If the publisher crashes after publishing but before marking, the message will be sent again. That is normal: it is at-least-once delivery, so consumers must tolerate duplicates (Day 17).

Level 2: Intermediate

A real .NET 10 + EF Core + SQL Server + Azure Service Bus implementation, with an Angular UI that only sees the effect.

6.1 Outbox table and writing in the same transaction

using Microsoft.EntityFrameworkCore;

public class Claim
{
    public Guid Id { get; set; }
    public string Number { get; set; } = "";
    public string Status { get; set; } = "Pending";
    public decimal ApprovedAmount { get; set; }
}

public class OutboxMessage
{
    public long Id { get; set; }                    // identity => stable ordering
    public Guid MessageId { get; set; }             // consumers dedupe on this
    public string Type { get; set; } = "";
    public string Payload { get; set; } = "";       // JSON
    public string? PartitionKey { get; set; }       // e.g., ClaimId, for ordering per aggregate
    public DateTime CreatedOnUtc { get; set; }
    public DateTime? ProcessedOnUtc { get; set; }
    public int Attempts { get; set; }
    public string? LastError { get; set; }
}

public class ClaimsDbContext(DbContextOptions<ClaimsDbContext> options) : DbContext(options)
{
    public DbSet<Claim> Claims => Set<Claim>();
    public DbSet<OutboxMessage> Outbox => Set<OutboxMessage>();

    protected override void OnModelCreating(ModelBuilder b)
    {
        b.Entity<OutboxMessage>(e =>
        {
            e.ToTable("OutboxMessages");
            e.HasIndex(x => x.MessageId).IsUnique();
            // Filtered index: only pending rows are indexed, so polling stays cheap
            e.HasIndex(x => x.Id).HasFilter("[ProcessedOnUtc] IS NULL")
             .HasDatabaseName("IX_Outbox_Pending");
        });
    }
}

using System.Text.Json;

public class ApproveClaimHandler(ClaimsDbContext db, TimeProvider clock)
{
    public async Task HandleAsync(Guid claimId, decimal amount, CancellationToken ct)
    {
        var claim = await db.Claims.FindAsync([claimId], ct)
                    ?? throw new InvalidOperationException("Claim not found");

        claim.Status = "Approved";
        claim.ApprovedAmount = amount;

        db.Outbox.Add(new OutboxMessage
        {
            MessageId = Guid.NewGuid(),
            Type = "ClaimApproved",
            PartitionKey = claim.Id.ToString(),
            Payload = JsonSerializer.Serialize(new { claimId = claim.Id, claim.Number, amount }),
            CreatedOnUtc = clock.GetUtcNow().UtcDateTime
        });

        await db.SaveChangesAsync(ct);   // ONE transaction: claim + outbox row
    }
}

6.2 The Polling Publisher (BackgroundService)

using Azure.Messaging.ServiceBus;
using Microsoft.EntityFrameworkCore;

public sealed class OutboxPollingPublisher(
    IServiceScopeFactory scopes,
    ServiceBusSender sender,
    ILogger<OutboxPollingPublisher> log) : BackgroundService
{
    private const int BatchSize = 50;
    private static readonly TimeSpan IdleDelay = TimeSpan.FromSeconds(1);

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            int published = 0;
            try
            {
                published = await PublishBatchAsync(stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                log.LogError(ex, "Outbox publish cycle failed");
            }

            // Drain quickly while there is work; sleep only when idle or failing
            if (published == 0)
                await Task.Delay(IdleDelay, stoppingToken);
        }
    }

    private async Task<int> PublishBatchAsync(CancellationToken ct)
    {
        using var scope = scopes.CreateScope();
        var db = scope.ServiceProvider.GetRequiredService<ClaimsDbContext>();

        await using var tx = await db.Database.BeginTransactionAsync(ct);

        // UPDLOCK + READPAST: several publisher instances can run without
        // taking the same rows (READPAST skips rows locked by another worker).
        var batch = await db.Outbox
            .FromSql($"""
                SELECT TOP ({BatchSize}) *
                FROM OutboxMessages WITH (UPDLOCK, READPAST, ROWLOCK)
                WHERE ProcessedOnUtc IS NULL
                ORDER BY Id
                """)
            .ToListAsync(ct);

        if (batch.Count == 0) return 0;

        foreach (var m in batch)
        {
            var msg = new ServiceBusMessage(m.Payload)
            {
                MessageId = m.MessageId.ToString(),   // enables duplicate detection on the topic
                Subject = m.Type,
                PartitionKey = m.PartitionKey,
                ContentType = "application/json"
            };

            try
            {
                await sender.SendMessageAsync(msg, ct);
                m.ProcessedOnUtc = DateTime.UtcNow;
            }
            catch (ServiceBusException ex)
            {
                m.Attempts++;
                m.LastError = ex.Message[..Math.Min(ex.Message.Length, 500)];
                log.LogWarning(ex, "Send failed for outbox {Id}", m.Id);
                break;   // preserve order: stop at first failure, retry next cycle
            }
        }

        await db.SaveChangesAsync(ct);
        await tx.CommitAsync(ct);
        return batch.Count(m => m.ProcessedOnUtc is not null);
    }
}

using Azure.Identity;
using Azure.Messaging.ServiceBus;
using Microsoft.EntityFrameworkCore;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddDbContext<ClaimsDbContext>(o =>
    o.UseSqlServer(builder.Configuration.GetConnectionString("Claims")));

builder.Services.AddSingleton(_ =>
    new ServiceBusClient(builder.Configuration["ServiceBus:Namespace"],   // e.g. myns.servicebus.windows.net
                         new DefaultAzureCredential()));
builder.Services.AddSingleton(sp =>
    sp.GetRequiredService<ServiceBusClient>().CreateSender("claim-events"));

builder.Services.AddSingleton(TimeProvider.System);
builder.Services.AddScoped<ApproveClaimHandler>();
builder.Services.AddHostedService<OutboxPollingPublisher>();

var app = builder.Build();
app.MapPost("/claims/{id:guid}/approve",
    async (Guid id, ApproveRequest r, ApproveClaimHandler h, CancellationToken ct) =>
    { await h.HandleAsync(id, r.Amount, ct); return Results.Accepted(); });
app.Run();

record ApproveRequest(decimal Amount);

Packages: Microsoft.EntityFrameworkCore.SqlServer, Azure.Messaging.ServiceBus, Azure.Identity.

6.3 Angular side (what the UI must understand)

The API returns 202 Accepted: the claim is saved but downstream effects are eventually consistent. The Angular UI should not promise “email sent” instantly.

// claim.service.ts (Angular 22, standalone, functional style)
import { Injectable, inject } from '@angular/core';
import { HttpClient } from '@angular/common/http';

@Injectable({ providedIn: 'root' })
export class ClaimService {
  private http = inject(HttpClient);

  approve(id: string, amount: number) {
    return this.http.post<void>(`/api/claims/${id}/approve`, { amount });
  }
}

import { Component, inject, signal } from '@angular/core';
import { ClaimService } from './claim.service';

@Component({
  selector: 'app-approve-button',
  template: `
    <button (click)="approve()" [disabled]="busy()">Approve claim</button>
    @if (done()) { <p>Approved. Notifications will follow shortly.</p> }
  `,
})
export class ApproveButtonComponent {
  private svc = inject(ClaimService);
  busy = signal(false);
  done = signal(false);

  approve() {
    this.busy.set(true);
    this.svc.approve('9b2f6c1e-0000-0000-0000-000000000000', 1500).subscribe({
      next: () => { this.done.set(true); this.busy.set(false); },
      error: () => this.busy.set(false),
    });
  }
}

6.4 SQL Server and PostgreSQL notes

• SQL Server: the UPDLOCK, READPAST, ROWLOCK hint pattern above.
• PostgreSQL: SELECT ... FROM outbox_messages WHERE processed_on_utc IS NULL ORDER BY id LIMIT 50 FOR UPDATE SKIP LOCKED gives the same multi-worker behaviour. A partial index WHERE processed_on_utc IS NULL keeps polling cheap.

Level 3: Advanced

Performance.

• Use a filtered/partial index on pending rows; without it every poll scans a growing table.
• Poll adaptively: when a batch returns rows, loop immediately; when empty, back off (e.g., 200 ms -> 1 s -> 5 s). A fixed 1 s sleep caps latency at about 1 s plus publish time.
• Batch sends with ServiceBusSender.CreateMessageBatchAsync to cut network round trips (one call for many messages). Watch the 256 KB (Standard) / 100 MB (Premium) message and batch size limits.
• Do not hold the DB transaction while doing slow broker calls if batches are large. An alternative is a lease approach: in a short transaction set LockedUntilUtc and LockedBy, commit, publish, then mark processed in a second short transaction.
• Cleanup: delete or archive processed rows on a schedule (e.g., older than 7 days) in small batches, otherwise the table and its indexes grow without bound.

Scalability.

• Multiple instances are safe only with row-claiming (READPAST / SKIP LOCKED) or leases. Without them, two pods publish the same rows.
• Ordering caveat: with several workers, message 2 for a claim can be published before message 1 if the workers claim different rows. If order per claim matters, either run a single active publisher (leader election, e.g., via an Azure Blob lease), partition workers by hash(PartitionKey) % N, or use Service Bus sessions with SessionId = ClaimId and have consumers process by sequence number.

Security.

• Use Managed Identity + DefaultAzureCredential; grant only the Azure Service Bus Data Sender role on the specific topic.
• Do not put PII (national ID, medical details) in payloads when a reference ID suffices; the outbox table is a long-lived copy of your data and needs the same encryption (TDE) and retention rules.

Failure modes and what happens.

Common mistakes.

1. Marking rows processed before publishing (turns at-least-once into at-most-once and loses events).
2. Assuming exactly-once delivery. Service Bus duplicate detection only covers a configured time window.
3. Ordering by CreatedOnUtc instead of the identity column.
4. No index on pending rows, and no cleanup job.
5. Publishing from the request path “as an optimisation” and forgetting the outbox row is still needed.
6. Ignoring poison messages so one bad row blocks the whole queue.
7. Long-running transactions holding locks during broker outages (set a send timeout).

Level 4: Expert and Architect view

8.1 Alternatives compared

8.2 Patterns it combines with

• Transactional Outbox (Day 12): Polling Publisher is one relay implementation of it; log tailing is the other.
• Idempotent Consumer (Day 17): mandatory partner because delivery is at-least-once.
• Domain Event (Day 11): aggregates raise events; the outbox stores them.
• Saga (Day 7): saga steps use the outbox to send commands and replies reliably.
• Messaging (Day 16), Circuit Breaker (Day 26) and Retry & Backoff (Day 28): wrapping broker calls; the polling loop itself is a retry mechanism.
• Health Check API (Day 36) and Metrics (Day 33): expose outbox lag.

8.3 ADR-style justification

ADR-014: Use a Polling Publisher to relay outbox messages from ClaimsService

Status: Accepted.

Context: ClaimsService (SQL Server, on a managed instance where the platform team does not permit CDC) must publish ClaimApproved and related events to Azure Service Bus without dual-write inconsistencies. Expected peak: about 30 events/second; acceptable propagation delay: under 5 seconds.

Decision: Write events to an OutboxMessages table in the same transaction as the aggregate change. A BackgroundService polls the table (adaptive interval, batch 50, UPDLOCK/READPAST), sends to a Service Bus topic with MessageId set to the outbox id, then marks rows processed. Ordering per claim is preserved via SessionId = ClaimId on the topic subscriptions.

Consequences: (+) No new infrastructure, works on any RDBMS, simple to test. (+) Broker outages do not affect the write path. (-) Adds polling load and up to ~1 s latency. (-) At-least-once delivery, so all consumers must be idempotent. (-) Requires cleanup job and lag monitoring.

Revisit when: sustained volume exceeds ~1,000 events/second, latency SLO drops below 1 second, or CDC becomes available.

Azure implementation

Services.

Configuration.

1. Create a Service Bus namespace and topic claim-events. Enable duplicate detection on the topic (window of e.g. 10 minutes; default 10 minutes when enabled) so a re-sent MessageId inside that window is dropped by the broker. Enable sessions on subscriptions that need per-claim ordering.
2. Assign the app’s managed identity the Azure Service Bus Data Sender role at topic scope; consumers get Data Receiver on their subscription.
3. In Container Apps, run the publisher inside the API container or as a separate single-replica container app (minReplicas: 1, maxReplicas: 1) if you choose “single active publisher”. With multiple replicas rely on READPAST.
4. Configure health probes (Day 36) and log the outbox lag as a custom metric; alert when lag exceeds e.g. 60 seconds.
5. Azure SQL: create the filtered index; schedule the cleanup job (SQL Agent on Managed Instance, or an Azure Function/Container Apps Job on a timer for Azure SQL Database, which has no SQL Agent).

Pricing and tier considerations (verify current numbers on the Azure pricing pages before budgeting):

• Service Bus Basic: queues only, no topics, so it does not fit topic-based fan-out.
• Service Bus Standard: topics, sessions, duplicate detection; pay-as-you-go with a monthly base charge plus per-operation charges. Message size limit 256 KB. Good default for this use case.
• Service Bus Premium: dedicated capacity in messaging units, larger messages (up to 100 MB), VNet/Private Link support, predictable latency, geo-disaster recovery. Choose it for high throughput, network isolation or compliance needs.
• Azure SQL Database: polling cost is mostly a few small indexed queries; on serverless tiers, constant polling prevents auto-pause, so budget as always-on or increase the idle interval.
• Container Apps: consumption plan billing per vCPU-second/GiB-second; a 0.25 vCPU always-on replica is cheap, but note that it cannot scale to zero if it must poll.

Reference architecture (text).

Angular SPA -> Azure Front Door / API Management -> ClaimsService (Container Apps, .NET 10) writes Claims + OutboxMessages in one transaction to Azure SQL Database (private endpoint). The OutboxPollingPublisher (same app or a dedicated single-replica container app) reads pending rows and sends to Service Bus topic claim-events using Managed Identity. Subscriptions payments, notifications, reporting feed their own services (each with idempotent consumers and dead-letter queues). Application Insights collects traces (Day 31), the outbox_lag_seconds metric and alerts; Key Vault holds any remaining secrets.

Teaching guide for my team

10.1 Explain to a beginner in 2 minutes

“When we approve a claim, we have to update our database and tell other systems. If we try to do both directly, one might fail and we end up with a lie somewhere. So instead we do one safe thing: in the same database save, we also write a note in an Outbox table. Then a little worker checks that table every second, sends each note to the message system, and ticks it off. If the worker or the network fails, the notes just wait. Worst case a note gets sent twice, so receivers must be able to ignore repeats.”

10.2 Explain to an intermediate developer in 5 minutes

1. Show the dual-write bug: SaveChanges() then Publish(); ask “what if the pod dies between?”.
2. Introduce the outbox row written in the same transaction (draw two boxes: business table + outbox table, one transaction).
3. Show the relay loop: select pending with UPDLOCK, READPAST -> send -> mark processed.
4. Explain guarantees: at-least-once, order per partition key if designed, duplicates possible.
5. Explain trade-offs versus log tailing: simpler, but polling latency and load.
6. Finish with operational needs: filtered index, cleanup job, lag metric and alert, poison message handling.

10.3 Hands-on exercise

Task: Build the ClaimsService sample locally with SQL Server (Docker) and the Azure Service Bus emulator (or a Console sender stub).

1. Create the Claims and OutboxMessages tables via EF Core migrations.
2. Implement POST /claims/{id}/approve writing both rows in one SaveChanges.
3. Implement the OutboxPollingPublisher.
4. Run two publisher instances at once, approve 100 claims, and log the MessageId of every send.
5. Kill the publisher process mid-batch (Ctrl+C while debugging with a breakpoint after SendMessageAsync), restart it.

Expected outcome: all 100 events are eventually delivered; after the kill, at least one message is delivered twice (same MessageId), proving at-least-once behaviour; no two instances ever claim the same row at the same time; then add an IdempotentConsumer (dedupe table on MessageId) and show that the duplicate no longer causes a double effect.

10.4 Interview-style questions

1. Why not just publish to the broker right after SaveChanges()? Because the two are not atomic; a crash in between loses the event, and publishing before the commit can announce a rolled-back change. The outbox makes the intent durable in the same transaction.
2. What delivery guarantee does a Polling Publisher give, and what does that imply for consumers? At-least-once. Duplicates happen when the worker crashes after sending but before marking; consumers must be idempotent (e.g., dedupe on MessageId).
3. How do you scale it to multiple instances without duplicate or out-of-order publishing? Claim rows with UPDLOCK, READPAST (SQL Server) or FOR UPDATE SKIP LOCKED (PostgreSQL), or use leases. For per-aggregate ordering, partition by key (or use Service Bus sessions) or run one active publisher via leader election.

Mastery checklist

• I can explain the dual-write problem and show a failing code example.
• I can implement the outbox write and the polling loop with EF Core, including the claiming query.
• I can state exactly why delivery is at-least-once and where duplicates originate.
• I can design ordering per aggregate (partition key, sessions, or single publisher).
• I can add a filtered index, cleanup job, poison-message handling and a lag alert.
• I can compare Polling Publisher with Transaction Log Tailing and justify the choice in an ADR.
• I can configure Service Bus (topic, duplicate detection, sessions, RBAC with Managed Identity) for this pattern.
• I can pair it with an idempotent consumer and test the duplicate scenario.

Key takeaway

Polling Publisher is the simplest reliable relay for the Transactional Outbox: commit the event with the data, then let a small worker read, publish and mark it. Accept at-least-once delivery, and design for idempotent consumers, ordering and lag monitoring.