From Accidental to Intentional Architecture

The momentum of building a product, shaping abstract ideas into something concrete, comes with compromises and limitations. We quickly connect components and take shortcuts to speed up deliveries. This sets the scene for unpredictability.

Even a prototype is composed of several interacting components with growing complexity. That complexity leads to emergent behaviour:

A system-level property that arises from the interaction of simpler components and is not defined by any single component.

Emergent behaviour can be neutral or positive. Here, I’m specifically interested in emergent failure modes – undefined or hard to enforce invariants, scalability issues, and unexpected runtime errors.

Architectural pressure is the force cracking the design in a specific place leading to failure modes. By recognising such pressure, we feel the drive to evolve the architecture.

It signals that requirements have outgrown current design and highlights the necessary next steps. We discover these cracks through incidents or by deliberately pushing on areas that seem sensitive.

This essay uses the initial design of a web crawler to demonstrate emergent failure modes as a concrete phenomenon, and how recognising architectural pressure can significantly expand capabilities.

Part 1. Emergent Behaviour

Meet the Crawler

I need a crawler to search for signals through changes in specific web pages. For example, new trends in software engineering.

The first version relied on an Orchestrator to drive the entire flow: initiates crawling through a seed URL, fetches the web page, process the page URLs, and only enqueue supported, normalised, unseen URLs.

Pages are processed concurrently. Each crawled page is processed sequentially as described in the diagram above.

The process function implements the pipeline above.

▶ Orchestrator#process (click to expand for simplified Scala code)

			
private def process(
  queue: Queue[F, Uri], inflightWork: Ref[F, Long]
)(url: Uri): F[Unit] =
  // enqueue URL and track in-flight work
  def enqueue(url: Uri) =
    inflightWork.update(_ + 1) *> queue.offer(url)
  (for
    // fetch webpage
    htmlPage <- fetcher.fetch(url)
    // extract urls from the page
    pageLinks <- linkExtractor.extract(htmlPage)
    // emit all page urls to stdout
    _ <- emitter.emit(pageLinks)
    acceptedLinks = pageLinks.links.filter(urlFilter.accept)
    // canonicalise urls
    canonicalisedLinks = acceptedLinks.map(UrlNormaliser.canonicalise)
    // deduplicate urls
    deduplicatedLinks <- canonicalisedLinks.toVector.filterA(link =>
      deduplicator.hasSeen(link).map(!_)
    )
    // add all valid newly discovered urls to the queue
    _ <- deduplicatedLinks.traverse_(
      enqueue
    )
  yield ())
    .guarantee {
      // decrease in-flight work counter
      inflightWork.update(_ - 1)
    }

		

There are 3 fundamental components powering the crawler:

A bounded URL queue preventing OOM
Fibers supporting cooperative multi-tasking
Orchestrator defining the data flow and key invariants such as termination.

Emergent Design Pressure

All components work as expected locally:

The queue blocks operations when full
Fibers enable concurrent page processing without blocking threads
Orchestrator composes the dataflow and makes sure termination is correct.

Observe the failure mode:

Orchestrator discoveries many URLs for each URL it processes
The bounded queue reaches capacity
Each Orchestrator concurrent task finishes processing the page
Since the queue is full, in-flight tasks are suspended when enqueuing newly discovered URLs
Since in-flight Orchestrator tasks are suspended, they cannot dequeue URLs
Termination depends on completing all in-flight work
Result: deadlock. The crawler is unable to make progress or terminate.

Components are locally sound. Their interaction leads to an undesirable global property: the application may stall with very high page URLs fan-out.

There is a missing invariant: the crawler must always be able to make progress regardless of the load.

The liveness issue is an accidental global property. It is an architectural pressure: a signal that the crawler design doesn’t meet requirements. It needs immediate attention.

The Source of Tension

That tension originates from the initial, deliberate coupling between control-flow and data-flow in the Orchestrator.

That coupling also makes it very difficult to implement politeness and fairness.

These limitations were accepted for a controlled prototype. The deadlock was not – I stumbled upon the liveness issue.

Part 2. Conscious Architectural Evolution (From Emergence to Intent)

Planning Next Steps Guided By Pressure

The liveness issue and the difficulty in implementing key features are direct consequences of the current design: the Orchestrator couples work admission, control and data processing in a single component.

These pressures point to the next architectural evolutionary steps:

Convert the Orchestrator into a Worker, responsible only for processing URLs
Introduce Ingress to admit URLs into the system
Admission must never stall Workers (it may block on I/O)
Control URL dispatch with a Scheduler.

▶ Scheduler (algebra)

			
trait Scheduler[F[_]]:
  def dispatch: fs2.Stream[F, Uri]

▶ Ingress (algebra)

			
trait Ingress[F[_], T]:
  def publish(events: T): F[Unit]
  def publish(events: Vector[T]): F[Unit]
  def stream: fs2.Stream[F, T]

The initial control policy is simple: the Scheduler dispatches URLs to workers at a steady rate.

The Ingress sole responsibility is transport: it’s the entry point in the system and has well-defined semantics:

Idempotently admit URLs into the system
Expose a stream where each admitted URLs is emitted exactly once.

The diagram below demonstrates the revised architecture:

▶ FifoScheduler#dispatch

			
class FifoScheduler[F[_]: Temporal] private (
  source: fs2.Stream[F, Uri], // from Ingress
  dispatchInterval: FiniteDuration // rate limiting work admission
) extends Scheduler[F]:
  def dispatch: fs2.Stream[F, Uri] =
    source.metered(dispatchInterval)

		

▶ Ingress.makeQueuedIngress (in-memory)

			
object Ingress:
  def makeQueuedIngress[F[_]: Temporal, T](
    queue: Queue[F, T] // unbounded queue
  ): Ingress[F, T] =
    new Ingress[F, T]:
      override def publish(event: T): F[Unit] =
        queue.offer(event)
      override def publish(events: Vector[T]): F[Unit] =
        events.traverse_(queue.offer)
      override def stream: fs2.Stream[F, T] =
        fs2.Stream.fromQueueUnterminated(queue)

		

▶ Worker#process

			
// Unbounded `targets` stream replaces queue
class Worker[F[_]: Async](targets: fs2.Stream[F, Uri], . . .):
  def run = targets.parEvalMap(maxConcurrency)(process)
  private def process(url: Uri): F[Unit] =
    def publish(newlyDiscovered: Vector[Uri]) =
      tracker.track(newlyDiscovered.size) *>
        ingress.publish(newlyDiscovered)
    (for
      htmlPage <- fetcher.fetch(url)
      pageLinks <- linkExtractor.extract(htmlPage)
      _ <- emitter.emit(pageLinks)
      acceptedLinks = pageLinks.links.filter(urlFilter.accept)
      canonicalisedLinks = acceptedLinks.map(UrlNormaliser.canonicalise)
      deduplicatedLinks <- canonicalisedLinks.toVector.filterA(link => deduplicator.hasSeen(link).map(!_))
      // Publish new links to Ingress instead of offering to queue directly
      _ <- publish(deduplicatedLinks)
    yield ())
      .guarantee {
        tracker.completed()
      }

		

These changes fix the liveness issue and unlock new options. They do not provide fairness or politeness yet, but introduce a central point where these new policies can be defined.

They also introduce a new pressure.

Pressure Shifts

In the original design, the Orchestrator both produced and buffered URLs using a single bounded queue.

Now, Workers immediately publish discovered URLs. The Ingress emits a stream of admitted URLs, and the Scheduler dispatches them to Workers at a controlled frequency. Because most webpages contain many links, Workers produce URLs far faster than they can process.

The in-memory Ingress implementation relies on an unbounded queue to hold pending URLs. This removes the deadlock, but shifts the pressure elsewhere: newly-discovered URLs grow without bounds, potentially leading to OOM.

The redesign does not eliminate the risks created by the in-memory queue. It isolates the risks.

If memory-growth becomes a problem, the in-memory Ingress can be swapped for a persistent implementation.

▶ PostgresIngress (durable)

			
object PostgresIngress:
  def make[F[_]](): Resource[F, Ingress[ConnectionIO, Uri]] =
    Resource.pure(new PostgresIngress())
private class PostgresIngress private () extends Ingress[ConnectionIO, Uri]:
  override def publish(url: Uri): ConnectionIO[Unit] =
    Statements.insertUrl(url)
  override def publish(urls: Vector[Uri]): ConnectionIO[Unit] =
    Statements.insertUrls(urls)
  override def stream: fs2.Stream[ConnectionIO, Uri] =
    fs2.Stream.repeatEval(Statements.dequeueUrl).unNone

		

Since the architecture is defined by contracts, the other parts of the system remain unchanged.

The in-memory Ingress still provides several benefits:

Extremely high throughput
No external dependencies
Fully non-blocking operations.

Keeping both implementations provides options for different execution environments: simple, fast, and riskier vs. safer, slightly slower and operationally heavier.

The crawler evolution thus far:

			
Prototype
   ↓
Deadlock
   ↓
Admission (In-memory ingress)
   ↓
Control / URL processing separation
   ↓
Optional durable ingress.

		

Part 3. Politeness & Fairness (A Gentle and Effective Crawler)

As it stands, the crawler can only be used under tight control, for example by restricting it to a single domain. Otherwise it risks behaving as a DoS tool.

To unlock its full potential – extracting meaningful signal from a diverse set of webpages – it must implement politeness and fairness.

Politeness requires an interval between requests to the same domain.

Fairness preserves throughput. While waiting for the cooldown period of one domain, the crawler can continue to make progress by fetching URLs from others.

With a clear separation between admission, control and processing, supporting the new control policies can be achieved by swapping the current FifoScheduler for a more involved Coordinator. The other parts of the crawler remain intact.

Enter the Coordinator

The Coordinator uses a priority queue of DomainQueues to find the next eligible URL
A DomainQueue is initialised during admission, when the first URL for a domain arrives
The priority queue keeps the DomainQueue with the earliest next-eligible URL at the head, from which the next URL is taken
After dispatching a URL, the Coordinator advances the domain’s eligibility time by the cooldown period.

Since each domain has its own queue, hot domains do not starve lower-traffic ones.

Internally, a TreeSet ordered by (nextEligibilityAt, domain) provides priority queue semantics – preferred over a binary heap since reordering requires efficient removal, which TreeSet can handle in O(log n).

The Coordinator – the new Scheduler implementation – is more involved than the previous snippets. Both the source and the spec are available on GitHub.

The pressure that initially signaled a broken design has, in the end, helped shape a better one.

With a few remaining features in place, the crawler will reach a point where I can shift the focus from building a crawler to extracting signal from the vast web.

Reflections

In practice, the most careful and competent engineer will be surprised and think “How haven’t I seen this before?” at some point. And that is the work.

Intelligence, creativity and sophistication often compound the problem, particularly at a system-level. Early adoption of technologies, elaborate interactions between components and abstractions lead to unjustified complexity when not strongly grounded on actual necessity. This often translates into fragility. Boring and battle-tested is good.

A simple heuristic is to start simple and address emergent failure modes as they appear, identifying and isolating the architectural pressure as early as possible, using it to inform deliberate architectural evolution.

Missing invariants may be inevitable. The architectural pressure will surface them. Let it guide the next evolutionary step.

Property-Based Testing as a Design Tool

Part 1 – Motivation and Foundations

Property-based testing is a powerful tool for designing correct, unambiguous programs through executable specifications.

It helps us answer an essential question:

What types of universal claims about a program are worth defining and enforcing?

A property is a proposition of the form:

$\forall \; x_1 \in D_1,\; x_2 \in D_2,\; \ldots,\; x_n \in D_n,\; P(x_1,\ldots,x_n)$

where:

Each $D_i$ represents the domain of discourse for which the operation is defined (the set of possible values for each input parameter).
$P$ is a logical predicate describing the program’s behaviour.

Property-based tests do not attempt to prove such universal claims.

Instead, they try to falsify them by sampling the input domain in search of counterexamples. A single counterexample is enough to invalidate the claimed property (Popperian philosophy).

The Domains $D_i$ and Generators

Property-based testing needs concrete representations of the domains $D_i$ over which a property is quantified. Generators encode how to sample values from the space of inputs a component operates on.

Property-based tests aim to ensure that properties hold across the entire domain $D_i$ , not merely for a curated set of examples.

The randomness in generators serves not to produce random tests, but to increase the chances that the domain is well-sampled. Since sampling can be biased, falsification is only as effective as the generators it depends on.

The quality of a generator directly affects test effectiveness: a missing counterexample in an uncovered region of the domain will remain undetected.

The Predicate $P$

Predicates describe the externally observable behaviour of a component that must remain true: its contract.

They:

Encode reference implementations, algebraic laws, relations, domain rules.
Are agnostic to implementation details.

Effective predicates lead to reduced ambiguities and clear understanding of component semantics.

This is particularly valuable when composing components, reasoning about interaction between operations, or designing stateful components, where behaviour emerges from a sequence of actions.

Preconditions

The domain of discourse can be further restricted using implications and preconditions:

$\forall x_1 \in D_1,\; \ldots,\; x_n \in D_n,\; \operatorname{Pre}(x_1,\ldots,x_n) \Rightarrow \operatorname{Post}(x_1,\ldots,x_n)$

Preconditions define the conditions in which a property must always hold, such as valid inputs or states.

They restrict the domain of discourse and may reduce the chances of finding a counterexample if they shrink the input space excessively.

Part 2 – Property-Driven Design in Practice

These ideas become concrete when applied to a real system design.

Suppose we are tasked with building a Scheduling API. Such components provide critical operations: checking availability, detecting overlaps, manipulating time spans, to name a few. Scheduling will be built on top of more fundamental components: discrete intervals.

A sensible first step is to model intervals as ranges of numbers rather than timestamps. A minimal structure consisting of start and end, preserving start < end, may appear sufficient. It rarely is.

This is where ambiguity begins – the unspoken assumptions:

Does containment include the endpoints?
When do two intervals intersect?
Fundamentally: how do interval operations relate to each other?

None of these questions are answered by the data structure alone. I’ve tried extensive documentation (still important, but complementary) as well as large types and function names, without significant improvement.

Ambiguity remained. And worse, it created a false sense of understanding, which eventually led to unpredictable behaviour, particularly when composing operations.

Domain Pressure Trims the Design

Scheduling systems care about overlapping, adjacency, ordering, containment of time spans.

Rather than trying to encode the complete theory of intervals, the Interval API is restricted to the set of operations required to express and support schedule invariants.

Domain-Driven Interval Properties

The following tables map required scheduling features to the interval semantics needed to support them, and the corresponding properties that enforce them.

Validity and Error Modes

Components are expected to reject, sanitise or fail fast when the input is invalid.

Scheduling Concern	Interval Properties (Spec)	Formal Property
Valid schedules are constructible	Smart constructors accept all valid bounds	$\forall (s, e)\in\mathcal{T}.\, \\ \operatorname{valid}(s, e) \Rightarrow \operatorname{constructor}(s, e) \downarrow$
Invalid schedules are rejected	Smart constructors reject invalid bounds	$\forall (s, e)\in\mathcal{T}.\, \\ \neg\operatorname{valid}(s, e) \Rightarrow \operatorname{constructor}(s, e) \uparrow$

Conflicts & Coexistence

Scheduling systems must detect overlapping schedules to detect conflicts, allow new disjoint ones to coexist, handle overrides.

Scheduling Concern	Interval Properties (Spec)	Formal Property
Overlapping schedules are detected	Intersecting intervals report intersection	$\forall (a,b)\in\mathcal{I}_{\text{intersecting}} \; .\, \\ \operatorname{intersects}(a,b)$
Disjoint schedules are allowed	Disjoint intervals never intersect	$\forall (a,b)\in\mathcal{I}_{\text{disjoint}} \; .\, \neg \operatorname{intersects}(a,b)$
Overlapping detection is consistent (A conflicts with B ⇔ B conflicts with A)	Intersection is symmetric	$\forall a,b.\\ \operatorname{intersects}(a,b) \iff \operatorname{intersects}(b,a)$
Overlapping imply conflicts	Containment implies intersection	$\forall a,b.\\ \operatorname{contains}(a,b) \Rightarrow \operatorname{intersects}(a,b)$

Continuity, Adjacency and Gaps

Schedule systems need to understand if schedules are back-to-back (adjacent) or if there is room for new ones between them.

Scheduling Concern	Interval Properties (Spec)	Formal Property
Back-to-back schedules do not conflict (no hidden overlaps at the boundaries)	Adjacent intervals never intersect	$\forall (a,b)\in\mathcal{I}_{\text{adjacent}} \; .\, \neg\operatorname{intersects}(a,b)$

Partial Order

Ensuring that contains forms a partial order enables safe composition, local reasoning about nested structures (avoiding examining entire hierarchies), and defines proper equality semantics.

Scheduling Concern	Interval Properties (Spec)	Formal Property
Every schedule contains itself	Containment is reflexive	$\forall a. \operatorname{contains}(a, a)$
Nested schedules compose safely (safe chaining of containment checks)	Containment is transitive	$\forall a,b,c.\\ \operatorname{contains}(a,b)\land \operatorname{contains}(b,c) \Rightarrow \\ \operatorname{contains}(a,c)$
Uniqueness is well-defined (equality checks, deduplication)	Containment is antisymmetric	$\forall a,b. \\ \operatorname{contains}(a,b)\land \operatorname{contains}(b,a) \Rightarrow a = b$

A Few Curated Examples

Four Interval Types, One Consistent API

Grouping the properties that must hold across all interval types goes beyond DRY. It is a powerful tool for designing unified APIs with consistent, well-defined semantics.

  checkIntervalProperties("closed", Intervals.makeClosed[Int])
  checkIntervalProperties("open", Intervals.makeOpen[Int])
  checkIntervalProperties("half-open right", Intervals.makeHalfOpenRight[Int])
  checkIntervalProperties("half-open left", Intervals.makeHalfOpenLeft[Int])

Crucially, not all operations are total: some are undefined for certain interval types. For example, adjacency has no meaning for open intervals, because open intervals have no boundary points.

Property-based testing must navigate this partial landscape enforcing laws only where operations are defined.

Encoding Validity Through Construction

Smart constructors (makeClosed, makeOpen, and similar factory functions) ensure that only intervals satisfying their invariants can be constructed.

The behaviour of interval smart constructors is specified using implications:

\forall (s,e) \in \mathcal{T}.\; \text{valid}(s,e) \Rightarrow \text{constructor}(s,e)\downarrow

This states that construction must succeed whenever the validity precondition holds.

In practice, instead of encoding this implication directly in the tests, we restrict the generator to produce only inputs satisfying the precondition.

This preserves the specification while increasing the chances of falsification and making the domain of discourse explicit.

    property(s"$name intervals are valid"):
      forAll(properBounds) { case (start, end) =>
        val result = factory(start, end)
        alg.validBounds(result.start, result.end) &&
        result.start == start &&
        result.end == end
      }

Conversely, when the input is invalid, the smart constructor must not construct an interval.

\forall (s, e)\in\mathcal{T}.\, \\ \neg\operatorname{valid}(s, e) \Rightarrow \operatorname{constructor}(s, e) \uparrow

In this design, invalid inputs result in immediate error rather than a recoverable value (such as Option or Either): trying to create an interval is considered a bug, not a valid use case.

    property(s"$name intervals reject invalid bounds"):
      forAll(reversedBounds) { case (start, end) =>
        throws(classOf[IllegalArgumentException]) {
          factory(start, end)
        }
      }

Together, these two properties specify the observable success and failure behaviour of interval construction over the domain of bounds.

Domain-Specific Generators

In the IntervalSpec, most interval properties are defined over specific domain subsets, rather than through implications and preconditions. For example:

\forall (a,b)\in\mathcal{I}_{\text{disjoint}} \; .\, \neg \operatorname{intersects}(a,b)

These properties require custom generators that sample directly from the relevant domain of discourse ( $\mathcal{I}_{\text{disjoint}}$ in this case), rather than relying on implications and on discarded invalid samples.

Examples of such interval generators include: properIntervals, adjacentIntervals, intersectingIntervals, and disjointIntervals.

Non-Total Properties and Vacuous Truth

Adjacency is not defined for all interval types. In particular, it is undefined for open intervals: for example, the open intervals (6, 7) and (7, 8) are not adjacent since they do not share a boundary point.

Rather than expressing adjacency laws through implications, we can define them over a restricted domain:

\forall (a, b)\in\mathcal{I}_{\text{adjacent}} \; .\, \neg\operatorname{intersects}(a, b)

This formulation also holds for open intervals, but only vacuously. When $\mathcal{I}_{\text{adjacent}} = \emptyset$ , the property still holds, since $\forall x \in \emptyset,\; P(x) \equiv \text{true}$ .

We can encode this behaviour directly in the generator. The adjacentIntervals generator produces:

Some((a,b)) when a pair of adjacent intervals exists (closed or half-open intervals)
None when adjacency is undefined (open intervals).

The property can then be written as:

    property(s"adjacent $name intervals never intersect"):
      // Vacuously true when no adjacent intervals exist
      forAll(adjacentIntervals) { adjs =>
        adjs.forall { case (a, b) => a.isAdjacent(b) &amp;&amp; !a.intersects(b) }
      }

Relationships, Implications and Vacuity

Some relationships between operations are inherently conditional.

For example, consider the relationship between containment and intersection: if an interval contains another, they intersect. Formally:

\forall a,b.\\ \operatorname{contains}(a,b) \Rightarrow \operatorname{intersects}(a,b)

In such cases, tests are often expressed as implication. Using De Morgan’s laws, implications can be rewritten into a form more suitable for programming languages:

P \Rightarrow Q \;\equiv\; \neg P \lor Q

However, such implication-based properties are prone to vacuous truth and may convey no information at all if the antecedent rarely holds.

property(s"containing intervals intersect for $name intervals"):
  forAll(properIntervals, properIntervals) { case (a, b) =>
    // a.contains(b) ⇒ a.intersects(b)
    !a.contains(b) || a.intersects(b)
  }

If !a.contains(b) always evaluates to true, the test will always succeed, even if a.intersects(b) is broken. This typically occurs when the generator rarely produces input satisfying the antecedent.

Takeaway

Through this lens, property-based testing is not primarily about testing. It is about deliberately constraining API semantics to what the domain demands.

Collapsing an unbounded space of possible behaviours into a small set of precise properties is difficult – but necessary. Ambiguous components do not compose into reliable systems.

Deadlines and pressures are real, but core quality cannot be negotiated. Some parts of a system must be correct, predictable, and composable by design.

The goal is to build systems that fulfill their objectives. Property-based testing is one of the best tools for designing the composable and predictable components that make this possible.

The Path to Idempotency: Preserving Business Outcomes

I’ve written about navigating the uncertainties of a complex integration stack. One of the biggest recurring issues we faced was duplicates: bursts of repeated updates that caused major incidents for our customers.

Our core operations were not idempotent – multiple identical instructions were performed with side-effects as many times as they were issued.

In our domain, it caused several problems:

Heavy, expensive reprocessing on both our side and our customers’ side.
Long-running computations that could take hours to complete.
In extreme cases, hundreds of duplicate instructions piling up and blocking time-sensitive work.

This was unsustainable. We needed a deterministic, systemic fix.

This post walks through:

The primary cause of non-idempotent behaviour in our pipeline
Why idempotency was essential to preserving business outcomes
How we redesigned the system to eliminate duplicates without altering semantics.

The Problem: Duplicates After Projection

Designing an idempotent distributed system often requires handling duplicates that arise beyond transport-level guarantees such as at-least-once deliveries.

Our pipeline ingested and merged data from multiple sources, then produced several different downstream projections. These projections reflected our business domain: each customer required a different subset of the combined data.

Since each projection was a view of the merged data, source updates could collapse into the same customer-facing representation – a duplicate.

To illustrate, consider the example:

			
Combined Data:
{
  "banana_id": "B-007",
  "name": "Mr. Banana",
  "ripeness_level": "Mostly Yellow"
}
Sent to Adapter/Customer 1:
{
  "banana_id": "B-007",
  "ripeness_level": "Mostly Yellow"
}
Sent to Adapter/Customer 2:
{
  "banana_id": "B-007"
}

		

Now Mr. Banana’s ripeness_level changes:

			
Combined Data (Updated):
{
  "banana_id": "B-007",
  "name": "Mr. Banana",
  "ripeness_level": "Perfectly Yellow"
}
Sent to Adapter/Customer 1:
{
  "banana_id": "B-007",
  "ripeness_level": "Perfectly Yellow"
}
Sent to Adapter/Customer 2:
{
  "banana_id": "B-007"
}

		

Adapter 1 receives two distinct updates.

Adapter 2 receives the exact same message twice.

Combined Data → Adapter 1 → OK
→ Adapter 2 → OK

Combined Data (Updated) → Adapter 1 → OK
→ Adapter 2 → DUPLICATE! 🍌

The core issue: the underlying entity changed, but Adapter 2’s projection did not, so it ended up with a duplicate. Any adapter whose projection omits the updated field is affected in exactly the same way.

This type of deduplication must happen after projections, at the edge of the pipeline, on a per-customer adapter basis.

Deduplication sat between projection generation and adapter-specific processing, acting as a dam before downstream side effects occurred.

When a duplicate was detected, adapters skipped processing and tracked the occurrence. Deduplication errors or timeouts were handled like any other pipeline failure: through replays rather than best-effort processing.

The First Solution: Last Seen Deduplication

When the first serious duplicate-related incident hit us, a colleague proposed a straightforward and effective approach:

Serialise the message to a deterministic JSON representation
Hash the payload
Retrieve the previously stored hash by ID
Compare the previously stored hash with the new one
If identical -> return duplicate
If different -> return not duplicate and store the new hash.

This gave us last-seen-message deduplication. For example:

A -> B -> A -> B -> A : no duplicates
A -> A -> A -> A -> B : only the first A and B are non-duplicates

We needed this to work immediately, and we were already late in the day (we wanted a fix before going home). A key/value store seemed like the obvious, modern choice: O(1) lookups, no schema changes, and Redis instances already running. We could implement the fix within minutes.

The dam was built. Problem solved.

How the Dam Cracked

The first failure mode was subtle. It occasionally returned not duplicate for the same message:

			
Sent to Service 1 (First):
{
  "banana_id": "B-007",
  "ripeness_level": "Perfectly Yellow"
}
Sent to Service 1 (Second):
{
  "ripeness_level": "Perfectly Yellow",
  "banana_id": "B-007"
}

		

A change in field order was enough to bypass the deduplication logic.

We needed deduplication based on structural equivalence instead of the serialised payload. Two messages should be considered equal if their JSON trees have the same fields, values and nesting, regardless of field order.

Then came the second, more serious failure, making the dam collapse: the Redis instances had no persistent storage. One day they vanished along with the entire deduplication state.

Time for a Sustainable Strategy

We considered a few alternatives before deciding how to approach deduplication across the pipeline.

Idempotency keys / versioning: These require producers to understand semantics that only emerge at adapter, downstream-level.
Kafka log compaction: It also operates before projection. More importantly, compaction does not guarantee that consumers only see the latest version of an event.

Neither approach addressed the real problem. We decided to build a dedicated deduplicator service.

The New Deduplicator

We kept the core idea – hash-based, last-seen-message deduplication – incorporated the lessons and extracted it into a dedicated service.

That would offer us:

Consistent Behaviour: Same deduplication semantics makes the system easier to reason about.
Clear Properties & Policies: Transparency around performance, failures and retries.
Consistent Tooling: Deduplication needs operational tooling for analysis, diagnosis and reprocessing.

We envisioned a growing number of services relying on this new deduplication service.

It needed to support:

Synchronous Check: Dowstream services decide if continue or skip processing.
Moderate throughput: < 100 RPS average target to process millions of requests per day with capacity headroom for bursts.
Bounded Latency: 99th percentile < 250ms target; worst-case rare slow paths < 2s tolerated without triggering clients timeouts.
Strong Consistency: Linearizable dedup decisions – no false positives/negatives, even under concurrency.
Domain Agnostic: Perform structural comparisons on deterministic representations defined by the clients. Avoid JSON canonicalision or semantic equivalence to avoid domain interpretation, restricting flexibility of usage.
Application-level Retries: Retries must restart from the beginning of the flow to prevent processing stale data.

Core Invariants:

Safe and Deterministic

Each deduplication request was identified by a composite key:

messageId: Corresponded to the underlying entityId; globally unique and stable across the business.
serviceId: Mapped to a projection of that entity, preventing cross-services interference.

This ensured isolated and deterministic comparisons.

Preserving Order Guarantees

Events within the pipeline were partitioned by entity identifier (the messageId received by the deduplicator). Kafka, combined with a sound partition strategy, preserved ordering within each partition.

Deduplication was a synchronous step in the pipeline, so requests arrived in the same order as events.

For any given (serviceId, messageId), deduplication decisions were strictly ordered.

Safe Retries

The pipeline ensured that only the latest version of an event could be replayed, preventing stale data during reprocessing.

If processing failed after deduplication, we used operational tooling to reset the dedup state for a specific (messageId, serviceId) before replaying the event.

This path was rare and manual by design.

Hash Collisions: Theoretical, Not Practical

We used cryptographic hash of the parsed JSON tree to detect changes.

While hash collisions are theoretically possible – with the last-seen deduplication approach – they would require two distinct and successive JSON structures for the same key to hash identically. Given non-adversarial inputs and strong hash functions, we treated collisions as practically impossible.

Guaranteeing absolute collision freedom would have added significant complexity for negligible benefit.

Bounded State

The service only stored keys and the associated latest hashed message.

Thus, the state was bounded by:

number of entities * number of projections

The number of entities was large, but finite and well-understood.

Correctness over Availability:

Wrong deduplication decisions were unacceptable in our domain:

False negatives => expensive repeated work.
False positives => missed deliveries.

We needed linear behaviour: requests for a given (messageId, serviceId) pair must be processed one-by-one, preserving ordering by key.

This ruled out caching: stale reads would have broken correctness.

The Database Layer

We initially implemented the service using PostgreSQL with SERIALIZABLE isolation to guarantee linearizability. The goal was to validate the approach, build new integrations on top of it and eliminate recurring operational incidents.

This version of the service ran in production for several years and sustained the required throughput.

It worked in practice because contention was limited:

There was no contention between deduplication requests triggered by different services.
Concurrent deduplication requests for the same (messageId, serviceId) pair were rare.
Clear retry policies ensured correctness when contention did occur.

Performance Improvements

The initial version traded latency by certainty and speed of development. As adoption grew, traffic patterns evolved. Bursts became more frequent, and tail latency (p99) occasionally climbed to seconds.

Two hot paths were tightened to address this without changing semantics and invariants.

Atomic Compare-and-Swap

The core duplication logic already had a compare-and-swap pattern:

Update the stored hash only if it differs from the previous value.
Compare and update atomically.
Operate on state stored in a single database row.

It was reduced to a single atomic operation.

			
UPDATE dedup_state
SET hashed_message = :new_hash
WHERE message_id = :message_id
  AND service_id = :service_id
  AND hashed_message IS DISTINCT FROM :new_hash
RETURNING 1;

		

Thus, in this case, the same correctness guarantee could be achieved using READ COMMITTED isolation level.

Replacing SERIALIZABLE improved latency and throughput by decreasing overhead, lowering contention and eliminating retries caused by transaction conflicts.

Faster Structural Comparisons

JSON comparisons were on the hot path: every request depended on it. Improving it would have system-wide impact.

The initial comparison was built with an abstraction-rich, allocation-heavy implementation.

A memory-efficient recursive tree walk reduced memory allocation, garbage collection pressure, GC-induced stalls, and, consequently, tail latency.

Infrastructure Overview

This is how the pieces fit together:

Horizontally scalable API layer behind a load balancer.
Stateless application nodes, with state stored in the database, allowing ephemeral, replaceable web servers.
Graceful degradation under load: Unbounded request queues resulted in slower responses under heavy-load instead of dropped requests.
Single-region deployment aligned with our business constraints. Global linearizability in a multi-region setup would have required a fundamentally different approach (e.g. distributed SQL database).

Observability

We used Prometheus and Grafana to track:

Latency, globally and per service
Request volumes, globally and per service
Duplicate rates
System-level metrics.

This visibility helped us to identify the biggest sources of duplicates and eliminate unnecessary upstream noise.

Single Point of Failure (SPOF)

As a consequence of our architectural decisions, the deduplicator and its PostgreSQL database became a single point of failure.

In case of an outage, the correct behaviour was to halt processing instead of allowing the system to continue with incorrect deduplication.

In such scenarios, alerting and well-defined response procedures are essential for timely recovery and for turning a potentially stressful incident into a clear operational task.

Conclusion

Over several years of operation, the deduplication layer never failed us.

We had concrete results:

Fewer major incidents
Improved customer trust
Lower operational costs
Stronger delivery guarantees.

Serendipity

Along with the new pipeline and the tooling we built around it, we discovered an additional benefit: the ability to filter unnecessary events during ingestion, before they entered the system.

Many events differed only in fields irrelevant to our domain, such as timestamps. By stripping out these before deduplicating projections, we discarded thousands of redundant updates every day.

Despite fan-out increasing beyond 20, the number of daily events flowing through the pipeline decreased significantly.

By eliminating redundant updates early, we kept the load on the deduplicator itself under control.

TL;DR

In this pipeline, idempotency wasn’t a transport-level concern but a business one.

Treating it that way shaped how we designed the system and the tradeoffs we were willing to make.

From Integration Chaos to a Deterministic Pipeline

I was once part of a team working on an integration layer responsible for collecting data from many upstream services and processing it into content made available to multiple external customers according to their requirements.

Our job was to make upstream and external system work as a single coherent whole, despite having no control over either side. This is a classic setting for emergent chaos, and I think of it as “a place between places”.

The first few integrations we released seemed fine: there was silence, which is usually a sign of success in this kind of work. At the time, we were focused on building new integrations.

But we were about to learn just how much noise errors can make.

The Chaos (as in Deterministic Non-Linear Systems)

After releasing one of our early integrations, we started seeing recurring operational issues and growing frustration. Our services were behaving unpredictably and breaching key expectations daily. We had no alerts and no sense of the severity or frequency of errors. We learned about problems only when customers reported them.

In environments like this, panic can spread quickly. People reach for simple explanations or “obvious” culprits in an attempt to regain a sense of control. There’s usually only one way out of this situation: stay calm, put a tactical solution in place to stop the bleeding, buy time to find the real cause. Then, create a plan and explain the situation to key stakeholders.

This particular integration was more complex than the earlier ones, and small data discrepancies (introduced upstream for “convenience”) had unpredictable and severe consequences. Hundreds of pieces of content were becoming unavailable to customers every day – always exactly ninety days after they were published. We also began seeing severe DoS-like incidents due to missing idempotency across the board. Until then, we had no idea our own services could generate duplicates.

Not only did we have little control over the systems around us, but our own stack had become opaque to us.

The original plan was to fix the bugs we had discovered and continue building new integrations to meet the deadline. But was that even possible?

The New Plan

At a high-level, there was a single root problem:

Our stack did not provide the right level of abstraction.

Abstract too little, and your services become another layer of indirection with extra cost and no value. Abstract too much, and it won’t be possible to meet all the diverse requirements of downstream systems.

So we landed on a “simple” solution:

Provide a unified interface that shields developers from upstream complexity and allows us to focus on external integrations.

I spent many days thinking about this interface: a beautiful, coherent API with clear semantics paired with proper update events, enabling us to build scalable, reliable and predictable integrations.

How you pursue a goal like this depends heavily on your organisation and technical landscape. In our case, we realised that the “simple plan” required re-architecting the entire integration pipeline, something no stakeholder wants to hear. We needed a platform.

The New Integration Stack

Once we had the green light to redesign our new integration stack, we split it into two main components: the Platform and the Adapters.

External services never interacted with the platform directly. Instead, each adapter subscribed to the Platform’s new-data-available notification, consumed only the data required for its particular integration, and transformed it according to the rules and constraints of a specific external system.

This setup brought several benefits:

Focus: Devs could concentrate entirely on the requirements of a single external system.
Isolation: Integration logic was fully contained within the service responsible for that system.
Scalability: Each adapter could scale up or down independently according to the throughput demands of the system it supported.
Adaptability: A consequence of independent scalability, we could limit the throughput of specific integrations to avoid overwhelming downstream services.
Operational Tooling: Each adapter became the natural place to build tools to address that system’s recurring operational issues.
Speed of Development: More developers could work in parallel on new integration without stepping on each other’s toes.

The main challenge with this approach was balancing innovation with consistency across adapters. A small group of developers maintained more than twenty services, so consistency in design and implementation was essential. We could not afford to let teams reinvent solutions in isolated ways that ignored valuable lessons learned elsewhere in the system.

Adapters required a reliable central source of all the data they depended on, delivered with reasonable latency, correct new-data-available notifications (signals, not payloads), and consistent behaviour. The Platform offered that and more.

Enter the Platform

Chaos had become routine. Missed deliveries, costly duplicates, frozen stack. All silently building up from the unlucky combination of tiny, isolated events compounded over time, amplified by total absence of observability.

The upstream services were owned by different teams and reflected different business domains. Each was part of a non-trivial pipeline and had its own upstream dependencies. They behaved unpredictably at times, produced conflicting updates, and generated thousands of duplicate events every day. An environment like this is not just noisy; it’s adversarial.

To bring order to this landscape, we needed a layer capable of collecting data from all these disparate sources and exposing a coherent view through an unified, versioned API. Key to achieve those goals was splitting the Platform into two main components: Ingestion and API Layer.

Ingestion:

We needed high-throughput ingestion while preserving event ordering, so we defined a deterministic pipeline:

consume event -> pre-process -> de-duplicate -> decode into domain events -> process -> store

This structure unlocked several capabilities:

Early anomaly detection: Discard malformed or inconsistent data before they entered the Integration stack.
Coherent Integration view: Decode data into a unified representation of the broader business domain, forming a clear contract between Upstream -> Integration -> External Systems.
Stable downstream consumption: Data was stored in queryiable form, protecting adapters from upstream instabilities.
Source-level insights: Track and analyse events by source to understand upstream behaviour.
Extended diagnoses tooling: Persist raw events and build tools around them to support analyses and troubleshooting.
Flow-event reduction: Use of heuristics to suppress low-value updates, and avoid unnecessary cascades through the platform.

We used Kafka – combined with a carefully chosen partitioning strategy – to achieve high-throughput parallel processing and preserve events ordering for each upstream source. This allowed us to deliver correct, time-sensitive content in a deterministic way.

By configuring each topic with compaction and infinite retention, we enabled log-based replication. This gave us the ability to replay the entire pipeline deterministically, a massive win in terms of reliability.

API Layer:

Downstream teams interacted with the Platform through a client library backed by a few GraphQL endpoints. They defined a query describing the data their adapter needed in order to produce content in accordance with system contracts.

The client library brought several benefits: type-safety, auto-completion and versioning.

This API Layer model was heavily challenged, and I used the lens of category theory morphisms to help me clarify our approach: building a structure-preserving projection from the full platform domain model to a minimal, integration specific view required by each external system.

The benefits:

Adapters received exactly the data they needed and nothing more.
A lossless, deterministic view of the model, enabling safe-retries.
Multiple views of the domain without fragmenting the ingestion model.

The tradeoff with this approach was a descent, but not “light speed”, latency. Beyond the network call, other factors added to the overhead: operational tasks embedded in the library, the GraphQL layer translating queries to SQL, and PostgreSQL executing them. This was acceptable in our case: throughput mattered far more than single-request latency.

Idempotency

The Platform also integrated well with our new deduplication layer. A single duplicate event (a “spark”) could trigger costly downstream work with potentially exponential consequences (“fire”).

The challenge was that duplicates were semantic, not transport level. Different external systems (via adapters) consumed different projections of the same underlying entity, meaning that different source updates could collapse into the same business update downstream.

We needed idempotency, not as a transport concern, but as a business outcome.

Low-Value Updates

The Platform also revealed another class of issues: low-value events that looked like genuine updates to the integration stack and bypassed the deduplication entirely. This caused expensive operations or even downstream disruptions.

The Success and The Day After

New adapters powered by the Platform were released one after another, from relatively simple to profoundly complex. We had built more than a stable platform; we had created a recipe to plug new systems into our stack. Aside from the occasional celebration after a successful launch, there was mostly silence: the sound of “nothing is broken”.

It was a success.

Then came a new flagship product. New services, new people, new stakeholders. I assumed this one would be more straightforward: we had proven we could navigate uncertainties and carve order out of chaos. I was wrong. The new voices carried none of the scars of the long fire we had walked through, yet they spoke confidently. And back to the frontline.

We were a team built to succeed in a bounded context. We never really tame chaos. We only address it. You build a temporary shelter, layer by layer, one challenge after the other. And when the storm passes, another will eventually come. That’s the work.

Notes on Software Design

Reflections from the field