Reproducibility is something you design in

Run the same query through your AI twice, and you may get two different answers. Most people meet this by accident and file it as a quirk. In a regulated setting it is not a quirk. It is the ground giving way, because a result you cannot reproduce is not a result. It is an opinion the machine happened to hold that day.

It happens because these systems are non-deterministic by default. Sampling introduces variation on purpose. The model version changes underneath you when the provider updates it. The retrieved context shifts as the underlying documents change. The same question asked a month apart is not really the same question to the system, because the system is no longer the same system.

That matters because reproducibility is the spine of evidence. You cannot stand behind a decision you cannot run again. When an inspector, or a court, or your own internal investigation asks the obvious question, would it do that again, and can you show me, the honest answer for most systems today is that we do not know. That is not a survivable place to be standing when the output went into something that mattered.

Designing reproducibility in is not mysterious, but it is deliberate. Pin the model version. Do not float on whatever latest happens to mean this week. Log the exact inputs, the full prompt, the retrieved context, and the settings that affect how the answer was sampled, so the run can be recreated rather than guessed at. Snapshot the corpus the retrieval drew from, because tomorrow's corpus will answer differently and you will have no way to prove what yesterday's said. Capture enough that you, or someone who has never met you, can rebuild the run and get the same result, on demand, on purpose.

And it has to be done from the start, because there is no retrofitting a model version that no longer exists. The moment the provider rolls the model forward, the old behaviour is gone, and with it any chance of reproducing what you did while it was live. You cannot go back and pin a version after it has already been retired out from under you.

Determinism is not the natural state of these systems. It is something you impose, on purpose, because you decided that being able to prove what happened was worth more than the small convenience of always running the newest thing. Most teams have not made that decision. They have simply not noticed that they were quietly making the other one.