Eight Chargers Throttled. No Model Decided One.
On the live run, a substation went past its thermal limit — 485,442 W against a declared 450,000 W — and eight EV charging sessions were throttled to bring it back.
Not one of those eight decisions was made by a model. No model was consulted, no forecast was scored, no feature was served into them. All eight came out of a deterministic fallback rule, and every one of them carries a marker saying so, from the actuator into the decision record and out the far end.
That is the system working, not the system degraded. And the reason it is worth an article is a distinction that most engineering instincts get backwards.
Fail closed is the wrong reflex on a grid. In a report factory, refusing to publish is safe: nothing happens, and nothing happening is fine. On an electricity distribution network it is not — the transformer keeps heating while nobody is deciding. So the safe state cannot be silence. It has to be a conservative deterministic action, computable with no model and no fresh features, and the requirement that costs something is the second half: a fallback has to be visible all the way to the end. A fallback that looks like a model decision is worse than an outage, because it is silent, and it teaches an operator to trust something nobody validated.
Everything below rests on one rule that never moved: a decision emitted from a window that has not closed is a build failure. Statistical models forecast load, score tampering and rank inspections. Deterministic code owns whether a window is closed, every published number, the decision to fall back, and whether a decision may be actuated at all.
What it is
Watermark is a real-time decision platform for a distribution system operator — roughly 250,000 smart meters, 2,000 public EV chargers and 400 substations. Telemetry arrives continuously, out of order, sometimes twice, sometimes three days late.
Three decisions come out of that one stream, and they could not be more different from each other. Curtailment — throttle EV charging as a substation approaches its thermal limit — in seconds, with a physical consequence. Meter anomaly — flag a meter and dispatch an inspector — in hours, with a significant effect on a person. Settlement — hourly consumption totals — over days, restated whenever late data arrives.
The interesting engineering is in none of them individually. It is that one stream feeds all three, and each is only correct under a different definition of "we have seen enough".
It runs on IoT Core, Kinesis and Managed Service for Apache Flink, over an Iceberg lakehouse on S3 queried through Athena and dbt, with SageMaker Feature Store, Pipelines, Model Registry and a real-time endpoint, Lake Formation for access, Step Functions for erasure, and Terraform for six infrastructure layers. It has been deployed to a real AWS account, driven, promoted from, served from, and destroyed again.
The name is the Flink term and the thesis at once. A watermark is the system's claim about what it has seen. Every wrong decision in a streaming system is, underneath, a decision taken on data that had not arrived yet.
rows closed before their interval ended: 0 is claim 1 written as SQL, re-runnable by anybody against the deployed table. distinct lineage ids equals rows merged, so no two rows share an identity. 285 restatements, 285 of them naming what they replaced — a correction never erases what was previously stated. Below it, Glue Data Quality against the deployed table: 6 of 6, score 1.00.The safe state is not silence
The two fallback rules in this repository sit in two contract files, four directories apart, and reading them side by side is the fastest way to see the doctrine.
contracts/decisions/curtailment.yaml declares its fallback as proportional_on_measured_load: throttle every active session in proportion to how far measured load exceeds the declared limit, using the last telemetry reading and the point-in-time limit and nothing else. The contract is blunt about the trade it makes — "more aggressive than the forecast would be: it costs customers charging speed and it does not cost the substation."
contracts/decisions/meter_anomaly.yaml declares its fallback as no_queue_entry: produce nothing. And it says why the difference is not arbitrary — "this is the one path where silence is the safe state… no meter is queued, no inspector is dispatched, and nothing in the physical world moves. Compare curtailment, where producing nothing means the substation keeps heating while nobody decides."
# contracts/decisions/curtailment.yaml effect: physical
fallback:
id: proportional_on_measured_load
uses_model: false
uses_features: false
permitted_actions: [throttle]
# contracts/decisions/meter_anomaly.yaml effect: significant_on_person
fallback:
id: no_queue_entry
uses_model: false
uses_features: false
permitted_actions: [dismiss]Same field, opposite answers, and the reason is in the effect line above each of them. uses_model: false and uses_features: false are not documentation — a fallback that needed either would not be a fallback, because the case it exists for is the case where neither is available.
That is where the eight throttles at the top of this article came from. 485,442 W against a declared 450,000 W → 8 sessions throttled, on the deployed estate, every one of them marked as a fallback in the record that leaves the system. That figure comes from the capture's run log, and there is no screenshot of it in this article, because there is no screenshot of it in the repository — quoting it as if there were would be the small dishonesty this whole project is an argument against.
Here is the part that does not flatter the project. The curtailment model does not exist. curtailment.yaml declares model: curtailment_forecast; nothing is registered under that name. So every throttle in the live capture came from the fallback — not because the model was unavailable that day, but because the model half of that decision has never been served at all.
I could have written this section without that paragraph and it would still have been true. It would also have been the exact sleight of hand this project exists to argue against: a designed safe state and an unbuilt primary path produce identical evidence, and only one of them is an achievement. What the run demonstrates is that the fallback works, fires on real load and labels itself. What it does not demonstrate is a model being overridden. The README says so in What this does not do, and this article is not allowed to claim more than the README does.
No decision comes out of a window that has not closed
The claim the project is named after. Offline it is seven labelled situations. Live it is a SQL statement.
quiet_substation, a stalled_stream, a device three hours fast, a partition already down at start-up — and a_window_that_must_close, which exists so the suite cannot pass by refusing everything. A harness that only ever withholds scores 100% and proves nothing.
held_back, 1 of those naming the substation holding it. A held-back watermark that cannot say who is holding it is useless to an operator at 03:00.That second frame is the interesting one, and it took a while to work out why. held_back means no window closed, so the evidence for it is an absence — and an absence is precisely what a job that never started also produces. The two are indistinguishable from the output alone.
What makes them distinguishable is that the job reports its watermark condition on every transition, and the assertion is not "something was held back" but "every held_back it reports names the substation causing it." The property is asserted where it is deterministic: in evals/watermark/ offline, and in SQL against the deployed table, where no row may be published with a watermark earlier than its own interval end. On the live run that was 0 of 3,779.
There is a limit here, stated because it is real. held_back cannot be induced deterministically at capture compression — the silent substation's forty minutes of event time pass in about eight seconds of wall time at 191×, so whether a batch boundary lands inside it is a matter of alignment. It fired on this run and did not on three earlier ones. So the live occurrence is reported, not asserted, and the assertion lives where the timing is under control.
The gate refused the model I trained
This is the section I would remove if I were selling something.
Every model version lands in the SageMaker Model Registry as PendingManualApproval, with its bias report attached to the registry entry rather than filed next to it. There is no code path that registers a model any other way.
Now the finding. The gate refuses the model this repository trained, and not because a threshold happened to be tight.
Measured over a 600-meter labelled population, precision in the most deprived tercile is 1000/1000. In the least deprived tercile it is 181/1000.
The expected shape was the familiar one — a group over-policed, flagged more often and wrong more often, which shows up as precision lower in the deprived column. What came out was the opposite, and it is worse. The model is perfectly precise where deprivation is highest, and nearly useless where it is lowest. Not because it understands deprived areas better: because 66 of 66 true cases there were confirmed by an inspector, and 4 of 23 elsewhere were. Inspectors went where they had always gone. The labels are complete in one tercile and full of holes in the other.
So the model looks excellent exactly where the dispatch log is densest. It learned where inspectors went, not where tampering was. **A reviewer reading a per-group precision table would see 1000/1000 and conclude the model is fairest on the group it flags three times as often.**
And the gate would have agreed with them. It had been written as precision_least − precision_most > ceiling — the expected shape. Under this finding that expression is negative, so the gate passed a model whose label coverage differs by a factor of five between groups. It now compares the absolute difference and names the direction in its refusal.
The disparity in flagging is 2959/1000, nearly three to one. Ground truth accounts for 2880/1000 of it — tampering genuinely is more common in older installations, and that is the honest part. The residual, 78/1000, is the part the world does not explain, and it is small. An analysis that stopped at demographic parity would have reported a large disparity, a large justification, and moved on. The precision figure is what says the labels themselves are the problem.
What has not been fixed is the loop. A model trained on confirmations keeps learning the dispatch log, and no threshold in a promotion gate changes that — a gate can refuse a model; it cannot supply labels nobody collected. The only thing on the honest-mitigation list that actually breaks the loop is randomised inspection: a fraction of visits allocated independently of the model, buying unbiased labels at the price of some wasted ones. That is outside this repository, and it is written down rather than implied.
A confident number is not a correct one
Once a human has approved, a second, separate deploy puts an endpoint in front of traffic. On the first deploy there is no approved model, so the endpoint cannot exist — claim 5 as a property of the order of operations rather than as a check that could be skipped.
asked: 820,0 · answered: 0.9938…, with an event id and a timestamp. The AI Act's record-keeping article asks for the inputs as served; the capture step fails the run if it finds nothing, because an endpoint that answers and records nothing looks identical to one nobody called.Read that answer against the previous section. Asked about a meter scoring 820 in the most deprived tercile, the model is 99.4% confident. It is confident for exactly the reason the bias finding describes. The endpoint is doing its job perfectly and the answer is still not one anybody should act on — which is why the actuation path for this decision does not exist without a person, and why that is structural rather than a setting.
Two mechanisms, one contract
This is the claim nothing else in my portfolio proves, and the one most likely to be quietly fake anywhere it is claimed.
Train/serve parity means a feature served online equals the offline value for the same entity at the same instant. The trap is that the cheapest way to make that true is to have both sides call the same function — at which point you are comparing a function with itself, and the check reports green for ever, including on the day the two paths genuinely diverge.
So the contract is the only thing the two sides share. Offline is as-of SQL over Iceberg, recomputing the whole window. Online is a streaming materialiser that saw each record exactly once. Different algorithms, different engines, different failure modes.
check_parity_paths_are_independent.py fails if the two resolvers ever come to share an implementation. Without it, claim 3 degrades into the tautology above — and it would degrade silently, during an ordinary refactor, with nobody doing anything wrong.
M00001 serves energy_wh 346 at its event time. M00007 — the meter belonging to the erased subject — returns null, from the same feature group, one command later. Claim 3 and claim 6 in one pair of calls.Live, the comparison runs over three features and twenty entities, as integers, because the contract declares a scale and the ADR forbids a tolerance. Result: agreed, 0 diverged.
And here is what that sentence used to hide. For most of this project's life only one feature was actually compared. The two substation features were never served at all — their source table was a Glue catalogue entry with no writer — and an as-of query over an empty Iceberg table returns zero rows and no error. Claim 3 was true about a third of the feature set and read as though it covered all of it. check_feature_sources.py now refuses a feature pointing at a column no table anything writes actually has.
One door in this system has no key. A parity mismatch cannot be overridden by anybody, including the approver, because it means the number that trained the model and the number in production are different things — so nobody, including the person signing, knows what they would be approving. Having exactly one unopenable door is what keeps the other six controls honest; a break-glass that opens everything is a rubber stamp with extra ceremony.
The same day, delivered twice
Claim 2 is not "the pipeline is deterministic". It is that the same events — shuffled, duplicated and delivered late — publish byte-identical values with identical lineage ids.
The capture proves it the expensive way. It drives the entire generated day, then drives the entire thing again, re-shuffled with duplicates and late arrivals, and compares. That is why a thirty-minute capture costs an hour and three-quarters.
numberOfFailedCheckpoints flat at 0, backpressure at zero for the whole run. That is load-bearing rather than decorative: a restart replays from the last checkpoint, which would produce duplicates from a different cause and read as a replay defect. uptime climbs unbroken and drops when the stop job switches the application off.Underneath, the transport is doing the thing that makes replay genuinely hard.
/backfill — the three-day-late head-end, not the live stream — and they are different shapes: a flat {v, mid, ts, wh} and a nested schemaVersion 3.0. One registered union schema covers every shape a meter in this fleet may publish, and normalisation collapses them in the pure core. Lateness and schema drift are two problems, not one, and solving them in the same place is how you get a parser with a date in it.The topic carries the substation, and that is a bug I shipped and then promoted into a permanent attack. It used to be <project>/meter/<thing>/reading, which made the partition key the meter id, which gave every meter its own watermark — and a quiet substation can then never hold anything back, because there is nothing left to hold. The whole of claim 1 was quietly disarmed by a topic string. check_partition_vocabulary.py now holds the transport's labels equal to the core's own declaration, and let the transport send the meter id as the record's partition is one of the 41 planted mutations.
Erasure, and the leg it cannot reach
An erasure request has to reach the lakehouse, the offline store, the online store, every training set, and the key that makes the ciphertext readable. It cannot reach the weights of a model trained before the request, and this repository refuses to pretend otherwise.
CryptoShred, DeleteRowsPhysically, the online-store map, DeleteFromTheOfflineStore, RederiveTrainingSets and QuarantineAffectedModels, converging on CountTheLegs → EveryLegConfirmed. Read the edge condition: $.counted == 6 and $.refused == false. RefuseToCertify sits beside it in red, unentered. Note LetTheOfflineStoreSettle — a wait state, because on an earlier run a record already in flight landed 46 seconds after the DELETE.The condition counts, and that word is the fix rather than the design. It used to be a five-way AND over array positions, and a hand-counted condition cannot notice a missing leg — because the missing leg is exactly what changes the count. The offline_store leg was declared in the scope, had no branch in the state machine at all, and four of a subject's rows survived an erasure that certified. check_erasure_legs.py now holds the scope, the branches and the count equal on every push.
M00007 changes customer mid-day, so the subject owns 54 rows and their predecessor 41. After: 0 and 41. Over-deletion is a breach in the other direction and the harder one to notice, because nobody ever complains about being forgotten too thoroughly.
PendingDeletion, seven days out. Seven is the shortest AWS allows and deliberately shorter than the root key's thirty: the GDPR puts a one-month clock on responding to the request, and a key that lingers spends most of that month in a state where the data is still readable.
refused: false, counted: 6, and the sixth carrying boundary: declared with a residual window of 30 days — "Models trained before this request retain the subject statistically. They are quarantined and retrained from the shredded corpus within the residual window; crypto-shredding does not reach model weights and machine unlearning is not claimed." The system refuses to report "erased" unless every leg confirms — and refuses to imply more than it did.The certificate is not the evidence. A separate verifier asks the estate the same six questions through different services — the shred through KMS, the online store through GetRecord, the lakehouse and the training sets through Athena — because a certificate that verifies its own legs is a signature on a blank page.
Structurally incapable, not merely configured
Claim 7 is that a consequential decision about a person cannot be actuated automatically — and that the automated path is structurally incapable of it rather than switched off.
The thirty-second demonstration needs no cloud at all. Change one line in contracts/decisions/meter_anomaly.yaml — actuation: human_gated to automatic — and the contract set refuses to load:
meter_anomaly.yaml: decision 'meter_anomaly' has a significant effect on a person and declares
automatic actuation. GDPR Art. 22(1) gives a data subject the right not to be subject to a
decision based solely on automated processing that significantly affects them. This contract
does not load — the combination has no runtime representation, which is what claim 7 means by
structurally incapable.Actuation is routed by the decision contract, not by code. There is no if statement to get wrong, no flag to flip in an environment, and no runtime in which the combination exists. Both halves were exercised on the live estate: oversight queue: 20 pending, 0 actuated, then exactly one actuated, on the review of a named human. The refusal is proved on every run; what needed a live estate and a name is the positive — that a named person is the only thing that can actuate.
The other half of the same design decision is one this project could easily have got wrong in the flattering direction. Curtailment is argued high-risk under the AI Act's Annex III(2) — a safety component in the supply of electricity — including the Article 6(3) derogation, considered and rejected. The meter anomaly decision is deliberately not Annex III. It is a GDPR Article 22 problem: it is not a safety component, and asserting that it were, to make the project sound weightier, is the exact failure this portfolio exists to argue against. The two postures are enforced differently in code, not described differently in a document.
Nobody can read the lakehouse
Governance is easy to claim and hard to demonstrate, because working access looks identical whether or not the controls exist. So it is demonstrated the only way that proves anything: by being refused, as the account owner.
Insufficient Lake Formation permission(s): Required Describe on watermark_silver. And the table list on the left contains no watermark_* table either — Lake Formation filters the catalogue, so the data is not forbidden, it is absent.
repo:…/watermark:environment:deploy|destroy, through AssumeRoleWithWebIdentity. No user, no laptop, no console role-switch, and no long-lived key anywhere in the repository or the account.
make policy evaluates the whole lattice offline — 4 principals, 24 principal-resource pairs, every reachable set exact and every closed path closed — with no AWS account.Every figure this article quotes comes from a run log rather than from me querying a table, and that is the point rather than an inconvenience: the numbers were produced by a role only CI can assume, into a log nobody edits afterwards.
Every gate is attacked
A gate nobody has tried to break is a gate nobody knows works.
make gate-proof copies the repository, plants a real violation in the source, and fails unless the named gate refuses it for the right reason.
Three rules keep it a proof rather than a ritual, and each exists because the lazy version is worthless:
- Every gate must be green first. A mutation that "fails" a gate that was already failing proves nothing.
- A non-zero exit is not evidence. The named check must report the failure — otherwise a typo in the planted diff counts as a working security control, which is how a suite quietly becomes theatre.
- A mutation whose target has moved is reported
STALE, neverpassed. Refactor the code out from under a mutation and it starts silently succeeding at nothing.
The newest of them is the one that caught this article. Add a harness and leave the sentence that counts them alone plants a claim harness and touches nothing else, and the README's own check refuses it — because a document that quotes every other check was, until recently, the one artefact nothing checked. The count in this article moved from 40 to 41 while it was being written, for exactly that reason, and the screenshot above was retaken rather than captioned around.
Four of the forty-one are not hypotheticals. Round the replay's measured offset onto the interval grid, bound the telemetry read over the whole prefix instead of per substation, let the transport send the meter id as the record's partition and import a 3.11 builtin into a job that runs on Glue 4.0 are mistakes this project actually made, promoted into permanent attacks after they were fixed. A bug you fixed is a bug you can make again; a bug with a mutation guarding it is not.
Writing the mutations is also how the gaps in the harness were found: six checks had no mutation against them at all, and eleven more under scripts/ were never run by one. Coverage is mechanical now — test_every_check_script_is_run_by_a_mutation means a new check ships red until something attacks it.
secret scan 8s, lint and tests 31s, the claim gates 36s, attack our own gates 47s, core equals Flink 1m 05s and terraform and checkov 2m 21s all complete before terraform apply starts. The commit is named at the top, so what was tested and what was applied are the same ref — not "CI passed on main last night".The whole thing runs on a laptop
None of this needs a cloud to be checked. 341 tests, nine claim harnesses and 41 planted gate violations run with no AWS account, no credentials and no network — plus 37 preflight checks, terraform validate against real provider schemas, and checkov beside it.
That is not a convenience, it is the reason the claims are worth anything. All the stream logic — windowing, watermarks, deduplication, point-in-time joins, lateness, quarantine and every fallback rule — lives in a framework-free pure core: standard library only, no Flink, no boto3, no clock, enforced by a check rather than by discipline. The PyFlink job is a thin adapter that carries records and decides nothing; a separate check refuses a semantic literal in it, so a window length cannot be baked into a Flink call.
A test that needs a Flink cluster to run is a test that will not run. A second, slower tier does run the real PyFlink job on a local MiniCluster and asserts byte-identical output against the pure core — and it cannot execute on arm64 macOS at all, so it runs in CI on Linux with WATERMARK_REQUIRE_FLINK=1, which turns a missing runtime into a failure rather than a skip. A suite that quietly skips reports green for one thing less than it says.
What this is worth to a business
The expensive failure in a streaming decision system is not a wrong answer. It is an answer that arrived before the data did. A wrong forecast gets corrected by the next one. A decision taken on an incomplete window is wrong in a way that is invisible at the time, correct-looking in every log, and only discovered when somebody reconciles a month of settlement and finds it does not add up. By then every number downstream of it is suspect too.
A fallback that does not confess is a liability with a user interface. The eight throttles in the live run are the cheap part; the marker that follows them into the record is what lets an operator, a regulator or the next engineer tell a modelled decision from a conservative one six months later. Without it you have not built a fallback — you have built an unlogged mode change.
The regulatory boundary is an architectural one, and drawing it wrongly costs in both directions. Calling everything high-risk is expensive theatre; calling nothing high-risk is a filing risk. Here, one decision is argued into Annex III with the derogation considered and rejected, and another is deliberately argued out of it and into Article 22 instead — and the difference is enforced by a contract that fails to load, not by a paragraph somebody read once.
Cost discipline is a design property rather than a rule people follow. The three expensive things — Managed Flink KPUs, the Feature Store online store and any real-time endpoint — exist only inside a bounded capture. The capture's stop job runs if: always(), so a run that fails halfway still switches them off, and the destroy workflow deliberately does not require CI to pass, because the moment somebody most needs to tear an estate down is the moment something is broken.
Standby here because it has already fired once and been reversed.Two measured figures, both labelled: USD 15.83 for one full five-step sequence — two thirty-minute captures, a promotion, an endpoint served and a teardown, in a day — and USD 115.64 for the whole nine-day exercise. Both undercount. A cost allocation tag takes up to 24 hours to activate, so an estate's first hours carry no tag at all. Quoting either number without that sentence is the failure mode in this section, and the project's own design ceiling of €100 per capture is a ceiling, never quotable as an outcome.
The budget guard also fired for real, and the ceiling was the thing that was wrong. Tagged monthly spend passed a USD 110 ceiling, the action attached a deny-all policy, and the next deploy died on ecr:GetAuthorizationToken. Nothing had run away: the ceiling had been written as though it bounded one capture while the budget it enforces is monthly. It is USD 250 now, and the reasoning is recorded where the old figure's reasoning was.
What I'm not claiming
The fleet is invented and the day is synthetic — seeded, committed, and generated to contain every pathology in the scenario on purpose. The scenario describes 250,000 meters; the generated day is 40.
Claim 4 has no live proof, and the run log does not pretend it does. The capture reports 20 of 20 served values were past the 15m budget — that is a report about timing, because the comparison runs forty minutes after the drive ends. It is not a demonstration that a stale feature was refused. Freshness is proved offline, 7/7, where the clock is under control.
The savepoint-restore drill does not exist. Equivalence across a restart — cancel with a savepoint, resume, produce the same bytes over the break — needs a harness rather than an assertion, and the test is skipped. It carries a waiver id, a named owner and an expiry date, which is the difference between a gap somebody chose and a gap nobody noticed. Exceptions expire here; on expiry the finding returns and CI goes red again.
Model Monitor and Clarify are not deployed — both are closed by AWS to accounts of this class, so the argument that "the standard bias metric reports and does not vote" is made from the pipeline definition rather than from a run. The bias analysis that would have used them is computed offline and gates the promotion there instead. Also dated, also waived.
One KMS key per data subject does not scale, and the repository says so rather than finding out. Forty-one subjects here; 250,000 meters would be past the account quota and an eight-figure annual bill for encryption alone. The ADR argues the envelope design a production system needs instead of quietly shipping the toy one.
The offline-store erasure leg is correct here for a reason that would not hold in production. It waits the documented flush window, which works because nothing writes the subject once the decision layer has finished. In production the control is a write-block before the erasure, and this platform has none.
One region, one account, single-tenant. No load testing, no concurrency testing, and no continuously green integration environment. And the decision that was wrong is kept in the ledger in full, unedited, next to the one that retracts it — because an argument that was wrong is worth being able to read.
The takeaway
The hard question in a streaming system is not what to decide. It is when you are allowed to decide at all — and the honest answer to "not yet" can never be silence in a system whose failures are physical.
The code, the CI and every test behind this article are public.