An MLOps Audit Checklist: Sixteen Evidence-Based Questions

An MLOps Audit Checklist: Sixteen Evidence-Based Questions

Audits work best when they ask questions that can be answered with evidence rather than opinions. This checklist applies that approach to MLOps, the set of engineering habits that gets a trained model into production and keeps it correct afterwards. You can run it in an afternoon against any team that serves predictions, and each item tells you what proof to ask for.

The scoring idea comes from the reference entry at the end of the article, which cites a well-known production-readiness rubric for machine learning. That rubric is a good model for how to be strict: it contains 28 tests in four groups, awards half a point for a manual test and a full point for an automated one, and takes the lowest section total as the overall score. The lowest section sets the grade because a system is only as ready as its weakest area. A score of zero is described in the rubric as more of a research project than a productionised system. Use the same logic below: do not average, look for the weakest block.

Block A: can you reproduce what is running?

  1. Is every training run tied to a record? Evidence: for the model in production today, produce the code commit, a hash of the data, the parameters and the environment description within a few minutes. If any one is missing, mark the block as failing.
  2. Is the rerun tolerance written down? Evidence: a number or range for how far a rerun’s metrics may differ. The Atlas notes that reproducibility is about metrics falling within tolerance, since some libraries do not promise identical output between releases.
  3. Is there a versioned registry with an owner? Evidence: a named person or team, and a way to tell candidates from the live version using tags or aliases, not a folder naming convention.

Block B: do training and serving agree?

  1. Is there one definition per input feature? Evidence: a single code path imported by both training and serving, rather than two reimplementations.
  2. Are served values logged? Evidence: a sample of what the live model received, kept long enough to compare with later outcomes.
  3. Is there a parity replay with a zero threshold? Evidence: a job that pushes the same records through both paths. The reference implementation described in the Atlas treats the mismatch rate as a metric that must be exactly zero; a small non-zero value signals two diverging pipelines.

If you are auditing a team that has a feature store, check that its documentation is read correctly. The Atlas quotes the tool Feast as being “not a general purpose data pipelining system”, and it notes that point-in-time correctness depends on a specific timestamp filter being switched on. A feature store helps only when it is configured the way its documentation says.

Block C: how does a new version get approved?

  1. Is the test set frozen and split by time? Evidence: the date of the split and proof that no later data leaked backwards.
  2. Does the gate compare more than the headline score? Evidence: a report showing overall quality, important slices, latency and schema checks together. In the Atlas’s worked lab, a candidate improved overall by 0.009 while a segment fell by 0.029; the gate demanded an uplift of at least 0.005 and tolerated no segment losing more than 0.02, so the candidate was turned away until that segment climbed back to 0.779.
  3. Was the latency limit applied at a high percentile? Evidence: the same lab used a 95th-percentile limit of 50 milliseconds. Averages hide slow tails.

For a sense of how much evidence matters, the Atlas also cites a survey on data leakage that traced mistakes in 22 papers to errors affecting 294 papers in 17 fields. Leakage is easy to commit and hard to see, which is why the audit asks for the split date and not merely a verbal assurance.

Block D: how is the live model watched?

  1. Are all three layers covered? Evidence: separate alerts for the service, the data and the quality of predictions.
  2. Do alert thresholds come from your own history? Evidence: a note on how each limit was calibrated. The Atlas points out that vendor defaults are frequently mistaken for truth. Its reference setup alerts when a feature-distribution score passes 0.2, but flags the value as something to calibrate on your own data, and it warns that the commonly quoted 0.1 and 0.25 bands lack academic support.
  3. Is there a freshness alarm on the monitoring itself? Evidence: for example, an alert when the statistics job has not run for more than 26 hours, and a pager rule for schema violations that persist for 15 minutes. A dashboard that silently stopped updating is worse than none.

Be sceptical about dashboards that show a single dataset-level drift verdict. The Atlas observes that one popular tool’s default only declares drift when more than half of the columns drift, which could hide a critical column moving on its own. Ask which columns matter most and whether they have their own alert.

Block E: what happens when something goes wrong?

  1. Is the response to an alert diagnostic first? Evidence: a runbook that separates pipeline disagreement, upstream defects and real drift before anyone starts a retrain.
  2. Is rollback done by alias or traffic switch? Evidence: a demonstrated switch, with the previous version still loadable. Rolling back by retraining is listed by the Atlas as an anti-pattern.
  3. Is there a non-ML fallback and a rehearsal date? Evidence: the date of the last drill; the Atlas suggests quarterly.
  4. Is there a list of downstream consumers? Evidence: the list itself, plus access control on outputs, so that a retirement does not break a team nobody remembered.

Running the session

Book two hours with the people who built the pipeline and two who consume its output. Ask for evidence on screen, not descriptions from memory. Log each question as a row in a small table: the proof shown, the person who showed it, and the date. Where a team answers “we usually do that”, record it as manual, because habits that depend on someone remembering are the first to lapse during a busy release week. The table doubles as the baseline for your next audit, so you can see whether the weakest block is actually moving.

Anti-pattern spot checks

Beyond the sixteen questions, spend ten minutes looking for habits that the Atlas lists as common traps. The first is deployment-only thinking, where the project plan ends on the day the endpoint goes live and nothing is budgeted for monitoring or retirement. The second is the belief that quick wins are free: the entry borrows the phrase “Changing Anything Changes Everything” to remind readers that a small tweak to one input can shift behaviour everywhere else in a tangled system. The third is comparing unlike windows, for instance judging this week’s data against a holiday period and calling the gap drift.

A fourth trap is subtler. Some label-free estimators of model quality are attractive because they need no outcomes, yet the Atlas notes that one such method documents that it will not work when the underlying relationship has changed. If the team leans on a label-free score, ask under which conditions it stops being trustworthy and what replaces it.

Scoring your results

For the scoring itself, give each question half a point if the evidence exists only as a manual procedure and a full point if it is automated, then total each block separately. Your overall score is the lowest block total, following the rubric’s logic. It will feel harsh the first time. That is the point: the weak block is where an incident will come from.

When you finish, write the fixes as a short list ordered by block, with the lowest block first, and agree who owns each one. Our own note on a digital services roadmap for new clients uses the same sequencing approach, and the blueprint for digital growth planning is a useful companion for putting dates against findings.

For the full definition, the scope boundaries and the lifecycle stages behind these questions, see the MLOps entry in the Ops Atlas. And if the audit finds almost nothing in place, take the entry’s modest advice to start small: a single reproducible pipeline, a registry that keeps lineage, and a way back, widened only once people actually use them.