What Your Health Check Endpoints Don't Tell You About Silent Model Degradation

You deploy a model. Health check passes. Life is good. Then three weeks later, a customer complains predictions are off. Your endpoint still says 200 OK. What happened?

Standard health endpoints check if the service is reachable and maybe if a dependency is alive. They don't tell you whether the model is still making good predictions. Model drift, feature drift, and concept drift happen silently. This article shows you how to detect that silent degradation without rebuilding your whole monitoring stack.

Who Needs This — and What Goes Wrong Without It

The illusion of a green health check

Your endpoint returns 200. CPU sits at forty percent. Memory is fine. The liveness probe passes every ten seconds, and Kubernetes reports everything as healthy. That feels good — until your support queue explodes with tickets about nonsensical predictions, or your business team starts asking why conversion rates dropped overnight for no obvious reason. I have seen teams spend two weeks debugging infrastructure before anyone thought to check the model output itself. The health check was green the whole time. It lied.

The catch is that traditional health probes check process liveness — not model sanity. A running Python process serving HTTP requests can return garbage predictions fifty times a second and still report itself as 'healthy'. Wrong order. Not even close. The model silently drifted, but the monitoring pipeline only cares about socket connections and response latency. That gap kills trust in production ML faster than any hardware failure I've encountered.

Real-world examples of silent failure

Here is one that stung: a recommendation model deployed on a Friday afternoon. Monday morning, the team saw engagement metrics were flat — green across the board. But the recommendations had shifted to showing winter coats in July. The model had encountered a new user segment with slightly different feature distributions, and the output distribution shifted. No errors. No crashes. Just useless predictions served at scale for three days. The business lost a full week of campaign optimization because nobody had wired a statistical check into the health endpoint.

Another case — fraud detection. The model kept flagging obvious legitimate transactions while letting actual fraud through. The ROC AUC on the monitoring dashboard barely budged, so everyone assumed things were fine. What actually happened was a subtle covariate shift in the input features that the model couldn't handle. The health check reported 'ok' because the inference server was alive and responding. I fixed this by adding a distribution comparison between live inference outputs and the validation set — caught the shift within minutes instead of days.

Why liveness probes aren't enough for ML

Liveness probes were designed for stateless applications. If the server dies, restart it. If memory leaks, kill the pod. That works for a web server. It fails for a model that quietly starts predicting poorly because its input domain drifted outside training boundaries. A 200 status code doesn't mean the prediction is useful. It only means the HTTP handler didn't crash.

Your model can be perfectly available and perfectly wrong at the same time. The health check will never tell you which one it's.

— observation from a production incident post-mortem, internal team retrospective

Most teams skip this distinction until the seam blows out. They assume that if the deployment tooling reports success, the model is doing its job. That assumption is expensive. Quick reality check — ask yourself what the recovery time would be if your model started returning random outputs right now. No alert. No dashboard change. Just silent degradation. If your answer is "we'd notice within a few hours from business metrics," you're probably underestimating the delay. Business metrics are lagging indicators. By the time they move, the damage is done. Statistical health checks on the endpoint give you leading signals — before the revenue impact, before the support tickets, before the retrospective meeting where everyone asks why nobody saw it coming.

What You Should Have in Place First

Basic Monitoring Stack Requirements

You can't monitor what you can't measure — and you can't measure without infrastructure. Before layering statistical drift detection onto your health checks, you need the plumbing in place. Start with a metrics server (Prometheus is the gold standard in Kubernetes shops) and a logging pipeline that captures every inference request and its output. No exceptions. I have seen teams try to retrofit drift detection using application logs scraped from stdout; they always hit a wall when they realize timestamps don't align or request IDs got dropped. The catch is this: your health check pings the model every few seconds, but drift detection needs historical context — meaning you need persistent storage for prediction outputs, feature vectors, and the raw response payloads. Store these in Parquet or Avro, not raw JSON. Use a time-series database (VictoriaMetrics, TimescaleDB) for the aggregated metrics; keep the raw logs in object storage.

Most teams skip this: you must also instrument the pipeline around the model, not just the model container. Latency percentiles, error codes, input size distributions — these are early indicators of degradation that raw accuracy metrics miss. A model that suddenly receives twice as many requests with malformed fields is not healthy, even if its output JSON parses correctly. That hurts. Set up at least three dashboards: one for resource health (CPU, memory, GPU utilization), one for inference throughput and error rates, and one for the prediction-value distribution over time. The third one is where your drift detection will feed, but you need the first two to know whether a degradation signal is real or a symptom of infrastructure chaos.

Access to Inference Logs and Prediction History

You need a replayable record of what your model predicted, for every request, going back at least two weeks. Not aggregated averages — per-request prediction values, confidence scores, and the raw input (or a hash of it). The tricky bit is that most health check endpoints return only a binary alive/dead status or a static response like {'status': 'ok'}. That tells you nothing about the model's behavior drifting. I have debugged incidents where the health check passed every probe while the model silently collapsed into predicting the same class for all inputs — a degenerate behavior that returns valid JSON and a correct HTTP 200 status. Without prediction history, you can't detect this until a user complains.

Design your logging schema to capture the following per request: model version identifier, input dimensionality (hash or compressed vector), the top-3 prediction probabilities (or regression values for non-classification tasks), and a monotonic request counter. Store this in a columnar format partitioned by date and model version. You don't need to keep every row forever — two weeks of rolling data gives you enough baseline for distributional comparisons, and you can downsample older data to hourly aggregates. That said, if you're under compliance (financial services, healthcare), check retention requirements before deleting anything.

Understanding Your Model's Expected Behavior

Before you can detect that something is wrong, you must define what "normal" looks like. This is harder than it sounds. Most teams assume the training data distribution is the baseline — but training data often differs from production data in subtle ways (sampling bias, pre-processing changes, seasonal shifts). Instead, establish a reference window from the first three weeks of production inference. Compute the mean, variance, and percentile boundaries for your predictions. Use these as your expected behavior, not the training set statistics. A quick reality check: if your training data was curated to remove outliers, production will immediately violate those boundaries — and that doesn't mean the model is broken.

Document what conditions should not trigger an alert. For example: if a retail demand model sees a spike during Black Friday, that's expected drift — not degradation. Build a calendar of known event windows (holidays, product launches, API deprecations) and exclude them from statistical tests or adjust the baseline dynamically. Wrong order here causes alert fatigue. I recommend maintaining a simple YAML file that lists known shift events per model, updated by the team owning that model. This file feeds into your drift detector as an exclusion filter. It's not elegant — but it works.

‘Sixty percent of model failures in production are preceded by a change in the input distribution that the health check doesn't detect.’

— Common pattern from production MLOps postmortems, paraphrased from internal reviews at three different tech companies

Augmenting Your Health Check with Statistical Tests

Adding distribution checks to /health

A standard /health endpoint returns a binary alive-or-dead signal. That's fine for disk failures. For model degradation, it's useless. The tricky bit is what happens between a green status and a production incident. I have seen teams add a second endpoint — /health/detailed — that runs a quick distribution comparison on every 100th prediction batch. You don't need a separate service for this; a single-sided Kolmogorov–Smirnov test against a stored reference tensor costs under 50 milliseconds on modern hardware. The catch: you must store the reference distribution before you deploy. Most teams skip this. They collect baselines after the model is already in production, and by then the data pipeline has shifted twice.

'A production model that always returns predictions is not necessarily a healthy model. It's often a model that learned to guess the mean.'

— platform engineer, internal post-mortem notes

Using KS-test or Chi-square for feature drift

The Kolmogorov–Smirnov test works for continuous features. For categorical fields — think country codes or device types — a Chi-square goodness-of-fit test is simpler and more stable. But here is the trade-off: statistical tests are sensitive to sample size. With 10,000 rows, trivial differences become statistically significant. With 100 rows, real drift hides. I usually set a practical significance threshold — a minimum effect size of 0.2 for KS, a Cramér's V above 0.1 for chi-square — and only then raise a flag. One pitfall: people forget that the reference distribution itself ages. A reference gathered during Black Friday looks pathological in January. Rotate your baseline every two weeks. Automate that rotation, or the check will cry wolf and your team will ignore it.

Tracking prediction distribution over time

Feature drift detection is necessary. Prediction drift detection gets you closer to the actual damage. Log the softmax outputs (or raw scores for regression) as a 1D histogram in 10 bins. Compare each new batch against the same bin counts from the training set using a simple Jensen–Shannon divergence. JS divergence is less common than KL divergence, but it's bounded between 0 and 1, so it behaves well as a single-number metric you can pipe into Prometheus. We fixed a silent degradation case last quarter by catching a 0.15 JS divergence spike — the model had started predicting the majority class for 95% of requests. The /health endpoint was green. The business lost four weeks of ad-revenue attribution. That hurts.

One more note: don't run these checks on every request. Sample — 200 consecutive predictions every 5 minutes is often enough. And if your latency budget is tight, offload the histogram comparison to a background goroutine or a sidecar container. The main endpoint stays fast; the statistical checks run in the margin. Wrong order? Then you waste compute on a dead model. Not yet? You miss the drift entirely.

Tooling and Setup: Kubernetes, MLflow, and Custom Endpoints

Health check libraries and frameworks

Most teams start with a Flask endpoint that returns 200 OK — that catches hard crashes but misses the slow bleed. For Python-based services, I reach for prometheus_client paired with a custom metrics registry. You expose two endpoint families: a fast liveness probe (does the process live?) and a heavier readiness probe that actually loads a sample batch through the model and checks latency percentiles. The catch is that readiness probes that run inference on every kubelet poll will drag down throughput. We fixed this by splitting the work — liveness on a 5-second interval, readiness every 60 seconds, and the statistical drift checks (from section 3) on a separate background thread that pushes results to a Prometheus gauge. The tricky bit is locking: if your drift computation blocks the health handler, the pod gets killed during a legitimate re-training window. Use threading.Lock with a timeout, or better, push drift scores into a sidecar container that owns the /healthz endpoint.

Integrating drift metrics into Prometheus

Raw drift scores are useless sitting in a log file. You need them in Prometheus — and from there into Alertmanager or Grafana. The pattern I use: a Histogram metric called model_prediction_distribution that records each prediction’s confidence percentile, plus a Gauge for the KS-test statistic between the last 1000 predictions and the training baseline. That sounds fine until you hit cardinality explosion — every unique model version or input feature creates a new time series. Most teams skip this: they tag by model_name and version only, never by raw feature names. A concrete gotcha: if your model accepts 300 features and you tag each one, Prometheus will silently drop metrics after a few hours. Limit drift monitoring to the top-10 features by importance, or aggregate via a weighted mean. Quick reality check—Prometheus isn’t built for per-request histograms at high traffic; sample every 10th request instead.

Canary deployments and shadow scoring

Drift detection works best when you compare two running models side-by-side. In Kubernetes, this means a canary deployment behind a service mesh like Istio or Linkerd. Route 5% of traffic to the new model, collect prediction distributions for both versions, and compare them on a Grafana dashboard before you roll out fully. The shadow-scoring pattern goes further: you run the new model in a separate pod that logs predictions but never serves them. That way you catch silent degradation before any user sees bad output. What usually breaks first is the data pipeline — shadow pods sometimes fetch stale feature stores or use a different preprocessing library. I have seen a team spend two weeks debugging a drift alert that turned out to be a column-ordering mismatch between training and production. Validate that both models receive identical inputs by hashing the request payload and checking for divergence in the sidecar.

'A health check that only tests for process liveness is a car with airbags but no brake lights.'

— overheard at a KubeCon lightning talk, paraphrased from memory

When Your Constraints Are Tight: Lightweight Alternatives

Stateless checks with precomputed baselines

Full distribution comparisons eat memory and latency. I have seen teams burn 200ms per request on a Kolmogorov–Smirnov test they didn't need. The lighter move: precompute a baseline from training data — mean, variance, and maybe the 5th and 95th percentiles — then store them as a tiny JSON blob. Your health check loads that blob once at startup; every inference call becomes a simple bounds check against those numbers. That's two comparisons per feature, not a histogram rebuild. The catch is staleness — your baseline drifts silently if the data domain shifts gradually. Set a weekly cron to recompute those values from the last 10,000 production samples, and you get a 95% solution for a 5% cost. Most teams skip this step until the seam blows out.

Using percentile-based thresholds

Mean-and-stddev checks break when your prediction distribution has long tails — think latency values or price predictions. A single outlier warps the standard deviation, triggering false alarms at 3 AM. Percentile thresholds are more robust here: flag anything below the 1st percentile or above the 99th. Precompute those cutoffs offline — once, not per request. "But what about concept drift that moves the entire distribution right?" That's a real risk. However, in tight environments where you can't run a streaming divergence test, percentile bounds catch the extreme shifts that actually harm users — a model that suddenly predicts negative prices, or a classifier output that jumps from 0.3 to 1.0. Wrong order? Not yet. You lose a day if you let that slide, but you catch the catastrophic failures in milliseconds.

'The cheapest monitoring is the one that fires only when something is actually on fire — not when the wind changes direction.'

— paraphrased from a production engineer after three false-alarm pages in one night

Batch vs. streaming trade-offs

Streaming checks — one inference, one decision — feel responsive but amplify noise. A single bad image can temporarily spike your confidence scores; a streaming check flags it, wakes an on-call engineer, and the spike vanishes thirty seconds later. Batch checks smooth that noise out. Group five minutes of predictions, compute the median confidence, and compare against a rolling baseline. The trade-off: you detect drift five minutes later. In low-latency pipelines (sub-50ms API calls), that delay is acceptable; in fraud detection it's not. What usually breaks first is the batch window itself — teams set a fixed 100-sample window, then traffic drops at 3 AM and the check never fires because it never fills. Use time-based windows (300 seconds) with a minimum-sample floor, or the check lies dead on the wire. That hurts.

One concrete trick I have used: run two lightweight checks simultaneously — a streaming percentile check that issues a warning log (no pager), and a five-minute batch check that actually pages. The warning log catches early symptoms; the batch check prevents false alarms from transient noise. Resource cost: about 4KB of counters per model endpoint. No histograms, no expensive computations. Quick reality check — this setup won't catch subtle covariate drift that creeps in over weeks. But if your constraints are truly tight, catching 80% of meaningful degradation with near-zero overhead beats a perfect system that never gets deployed because it feels too heavy. Choose the check you will actually run.

Pitfalls and Debugging: When the Check Fails or Lies

False Positives from Seasonality

Your augmented health check fires an alert at 2:14 AM. The pager goes off. You scramble, pull the dashboard — and nothing is wrong. The model predictions look fine. What happened? Most likely, your statistical test swallowed a seasonal pattern whole. Weekend traffic dips. Holiday shopping spikes. A marketing blast that drove thousands of bot-like queries. The test flagged a distribution shift, sure — but it was a *known* shift, not degradation. I have seen teams wire up a Kolmogorov-Smirnov test against last week's data and then spend three days chasing Saturday-morning noise. The fix isn't to abandon the test. It's to pin your reference window to the same hour or day-of-week, not a rolling 24-hour chunk. Or use a multiplicative seasonal adjustment — quick reality check: if your feature “page views” doubles every Sunday, your threshold better double too. Otherwise you train your team to ignore alerts. That hurts more than no alert at all.

“I spent a month tuning thresholds only to discover Monday morning looked like a model crash every single week.”

— Senior MLOps engineer, after a postmortem

Alert Fatigue from Too-Sensitive Tests

The catch is that statistical rigour cuts both ways. A two-sample t-test with p

Debugging a Failing Health Check Without Logs

Your check fails. The endpoint returns a 503. But inference logs are empty — the model container restarted before it could flush its metrics. Now what? Most teams skip this: instrument a pre-stop hook that dumps the last 100 input-output pairs to a sidecar volume before the pod dies. That single change turned a three-hour “why did it crash?” into a ten-minute “feature X was missing”. Without logs, you're guessing. With them, you replay the failing batch locally. Another pitfall: health checks that test against stale deployment versions. Kubernetes liveness probes often hit the old pod while the new one is still initializing. You get a false failure, the rollout stalls, and someone manually restarts the wrong service. We fixed this by adding a readiness gate that compares the model version in the endpoint metadata against the expected tag from the deployment manifest. Mismatch? Block traffic. Simple. Not pretty. But it stopped the blame-game cold.

Frequently Asked Questions About Model Health Monitoring

Is one health check enough?

Short answer: no. Long answer: one health check catches one kind of failure — usually that the process is running and can return 200. That tells you your model container didn't crash. It tells you nothing about whether the predictions you're serving are still accurate, or whether the distribution of incoming data has shifted silently. I have seen teams run a single `/health` ping for months, convinced everything was fine, while their model drifted so far that customer-facing scores became effectively random. The catch is that health checks check the *server*, not the *model*. You need at least two layers: one for liveness and readiness (the standard Kubernetes probes), and one for prediction quality (statistical drift detection). They answer different questions. Don't conflate them.

How often should you run drift detection?

That depends on your traffic velocity and the cost of a bad prediction. If your model serves a few hundred requests an hour, running a full statistical test every ten minutes is wasteful — you'll trigger false alarms on small sample noise. Instead, batch your predictions into hourly windows and run a two-sample Kolmogorov–Smirnov test or a population stability index against your training baseline. Quick reality check — if your model serves millions of requests a day, you can afford to run drift detection every five minutes, but you'll want a sliding window (last 1,000 predictions) rather than a fixed time interval to avoid lagging behind sudden shifts. The trade-off: faster detection means more noise, slower detection means you catch drift after it has already hurt your users. Start with a 15-minute cadence, then tune based on how many false positives you can tolerate. What usually breaks first is not the detection frequency — it's the baseline. Teams forget to update the reference distribution after retraining. That hurts.

Can you retrofit this to existing deployments?

Yes — and you should, even if your deployment is already in production. The beauty of a custom health endpoint is that it's just another route in your inference server. You don't need to rebuild your container or restart all traffic. You add a `/v2/health/model` endpoint that exposes a small summary — say, the mean prediction, the percentage of null outputs, and a PSI score compared to yesterday's distribution. That's it. Most teams skip this because they think they need a full observability stack first. Wrong order. You can start with a two-line Python script that writes a JSON file every hour, then mount that file into your container and serve it through a lightweight HTTP handler. I fixed a silent degradation at an ad-tech company by adding exactly that — we caught a vendor-side schema change within three hours instead of three weeks. The pitfall: retrofitting means your existing health checks won't fail when the new endpoint returns bad data. You must wire it into your alerting manually. Annoying, but doable in an afternoon.

“The model didn't crash. The predictions just got dumber, slowly, every day for a month.”

— engineer who added drift detection after the fact, not before

What if the statistical test itself is noisy?

Then your alerting will be noisy, and your team will start ignoring it. That's a real pitfall. The fix is not to abandon drift detection — it's to add a deadband or a minimum effect size. Don't trigger an alert just because the p-value dropped below 0.05. Use a threshold that matters for your business: if the mean prediction shifts by less than 5%, don't page anyone; log it and move on. Another pattern I have used: run two tests in parallel — one that catches distribution shifts (sensitive) and one that catches performance drops (specific). Only alert when both fire. That cuts false alarms by roughly half. Still, no test is perfect. You will have days where the input distribution looks fine but the model silently fails because a feature that was always zero suddenly becomes non-zero. That's a feature engineering issue, not a monitoring one. Different problem, different fix.