Any enterprise of meaningful size runs on more than one system, and those systems disagree with each other constantly. A customer exists in the CRM, in the billing platform, in the support desk, and in the data warehouse, and each copy has its own idea of the customer’s current state. Reconciliation is the work of deciding, across all of those copies, what is actually true and where truth has broken. Done naively, reconciliation produces a flood of false alarms, because most disagreements between systems are temporary and expected. Done well, it separates the noise of normal propagation from the signal of a real fault.
The difficulty is that the systems were never designed to be compared. They use different identifiers, update on different schedules, and model the same business concept with different fields. Reconciliation has to bridge all three gaps before it can say anything trustworthy.
Identity resolution comes first
You cannot compare two records until you are confident they describe the same entity, and cross-system identity is rarely a shared key. The CRM may key customers by an internal GUID, billing by an account number, and the warehouse by a surrogate that changes on reload. Identity resolution is the process of establishing that these different keys point at the same real-world thing, using whatever stable evidence exists: shared external identifiers, deterministic mapping tables where they exist, and matching on business attributes where they do not.
Getting identity wrong poisons everything downstream. If you match the wrong records, every field comparison between them becomes a false discrepancy, and the reconciliation drowns in noise it created itself. It is worth being conservative here: a high-confidence match set that covers most entities is more useful than an aggressive match set that pairs entities incorrectly. Unmatched records are themselves a finding, because an entity present in one system and absent from another is exactly the kind of gap reconciliation exists to catch.
Eventual consistency is not a fault
The central mistake in cross-system reconciliation is treating every difference as an error. Modern architectures are eventually consistent by design. A change made in the system of record propagates to downstream systems through queues, batch jobs, and integration pipelines, and during that propagation the systems legitimately disagree. A customer who upgraded their plan a minute ago will show the new plan in the CRM and the old plan in billing until the update flows through. That is the system working as designed, not a defect.
To reconcile without false alarms, you have to model this expected lag. A difference is only suspicious once it has persisted longer than the normal time for a change to propagate between those two systems. Before that window elapses, the difference is presumed to be in-flight. After it, the difference becomes a candidate fault worth investigating.
Learn propagation windows from observed history
The propagation window between two systems is not a constant you can guess. It depends on the integration, the time of day, batch schedules, and load. The reliable way to know it is to learn it from observed history: measure how long changes have actually taken to reconcile in the past, and derive a window from that distribution rather than from an assumption. A high-percentile window, such as a p99 lag learned from observed history, gives you a threshold that tolerates almost all legitimate delay while still flagging the differences that have clearly overstayed.
- Measure historical convergence times per system pair and per field family, not one global number.
- Use a high percentile as the tolerance so ordinary variance does not trip alarms.
- Account for known schedules, since a nightly batch produces a very different window than a streaming feed.
- Re-learn windows as integrations change, because a stale window is a source of both false positives and missed faults.
A discrepancy is only meaningful relative to how long the systems normally take to agree. Compare against the learned window, not against zero.
Localise the boundary where truth diverges
When a difference does persist beyond its window, the next question is where the truth actually diverged. In a chain of systems, a wrong value in the last system might originate anywhere upstream. Boundary localisation is the work of walking the propagation path and finding the specific hop where the value stopped agreeing. If the CRM and the integration layer agree but billing does not, the fault lives at the billing boundary. If the CRM already disagrees with its own outbound events, the fault is further upstream than it first appeared.
Localising the boundary changes reconciliation from an alarm into a diagnosis. Instead of reporting that two endpoints disagree, you report which link in the chain failed to carry the change, which is what an owner needs to fix it. It also prevents the common waste of investigating the wrong system, because the endpoint that shows the wrong value is often not the one that caused it.
Reconciliation as a continuous discipline
Reconciliation is most valuable when it runs continuously rather than as a quarterly fire drill. Once identity resolution, learned windows, and boundary localisation are in place, the same machinery can watch the boundaries over time, distinguishing the normal churn of in-flight changes from the accumulation of genuine faults. The output is not a wall of differences but a short list of persistent, localised divergences with the evidence for each.
In controlled, synthetic validation with injected propagation delays and injected faults, learning windows from observed history let the reconciliation pass over benign lag while still isolating the injected faults to the correct boundary. The general principle holds well beyond that setup: reconcile against what the systems normally do, not against an idealised expectation that they agree instantly, and always answer the question of where, not just whether, the truth diverged.