“Okay… this is a cool demo, but how do we even trust this output if it’s different each time?” That was the feedback I got on the quick POC I built to show an AI agent's functionality. While the feedback was 100% valid, we had very different expectations for what “trust” should look like at the POC stage. Yet if you have worked in data long enough, especially in the data quality space, then this conversation makes you feel like Bill Murray waking up all over again in Groundhog Day. Like AI today, I’ve had similar conversations about output accuracy, whether it’s traditional machine learning or analytics—but what’s different now is that everyone is a data practitioner when they use AI tools.
Real quick! A huge shoutout to Astronomer for sponsoring today’s article! Please consider supporting the Scaling DataOps Newsletter by clicking the link below to sign up for their upcoming webinar on September 10th: Best Practices for Pipeline Failure Recovery 👇🏽👇🏽👇🏽


Back in the old days of pre-2023, you couldn’t just build an AI application with a few API calls and a lot of misplaced confidence. No, that was reserved for the nerds that most in the company avoided because they (read: me) wouldn’t shut up about p-values and training data. Back then, you had to source the data yourself, clean it, develop a model, argue with another department why they don’t want 100% accuracy, and then somehow convince an engineer to move it from a Jupyter Notebook into production. This gauntlet, while fun, was a huge barrier to entry for most engineers and companies considering building an AI application or data product.
That is no longer the case, as people who don’t even know what a for-loop is can now build quite advanced (and useful!) AI applications through prompts alone. While a vast difference remains between a vibe-coder and true agentic engineering, the once-siloed worlds of software and data are now colliding. Even more interesting, software teams are now grappling with problems my peers on the data side have spent decades developing tools and techniques to manage. To be completely blunt, I’m unaware of any other time when software teams were incentivized by the business (i.e., substantial resource investment) to care about the problems data teams typically face, and I think it’s one of the best opportunities for data teams to bridge the divide1.
Non-determinism: It’s a feature, not a bug.
If we are being real, LLMs are essentially a token slot machine where the house inference provider always wins. I throw in a markdown file, sprinkle in “don’t make any mistakes”, and then click the enter button over and over again hoping I get the desired outcome. After some choice words with my agent and a hole burned through my employer’s AI budget, it can confidently provide a software script I probably could have written myself… albeit taking much longer to complete than the AI.

That last point on time-to-complete points to one of AI’s largest strengths— its ability to generalize under ambiguity. AI has essentially opened up a class of problems that were always technically possible to automate but operationally infeasible: workflows dominated by long-tail edge cases.
Specifically, non-determinism is one cost of this newly unlocked flexibility. The reason so many domain-specific workflows have not been automated before AI, especially for knowledge work, is that the sheer number of edge cases meant some poor engineering team had to account for all of them in the code—and God forbid an edge case changed. In other words, unless your leadership had unlimited budget and patience, this type of work was rarely prioritized.
This flexibility comes with a tradeoff, though. When you stop explicitly encoding every possible path through a problem, you also lose some of the deterministic guarantees traditional software gives you. Thus, the question becomes less about eliminating uncertainty and more about understanding whether the uncertainty you’re accepting is appropriate for the problem you’re solving.
Since using AI, I haven’t been able to shake its parallels to the classic machine learning problem of multi-armed bandits as described by Wikipedia below:
…the multi-armed bandit problem… is named from imagining a gambler at a row of slot machines (sometimes known as “one-armed bandits“), who has to decide which machines to play, how many times to play each machine and in which order to play them, and whether to continue with the current machine or try a different machine.[3]
More generally, it is a problem in which a decision maker iteratively selects one of multiple fixed choices (i.e., arms or actions) when the properties of each choice are only partially known at the time of allocation, and may become better understood as time passes. A fundamental aspect of bandit problems is that choosing an arm does not affect the properties of the arm or other arms.
…In the problem, each machine provides a random reward from a probability distribution specific to that machine, that is not known a priori. The objective of the gambler is to maximize the sum of rewards earned through a sequence of lever pulls.[5][6] The crucial tradeoff the gambler faces at each trial is between "exploitation" of the machine that has the highest expected payoff and "exploration" to get more information about the expected payoffs of the other machines.
The uncanniness isn't just my AI-generated image above; I argue this excerpt describes a useful mental model of how AI agents operate today. While the analogy doesn’t map perfectly onto every component of an agent, the broader mental model does: repeated decisions under incomplete information where outcomes are probabilistic and the objective is to improve expected performance over time.
The agentic harness, model selection, and various other tools in our arsenal are all methods to optimize the “probability distribution” of LLM outputs and “maximize the sum of rewards” of a desired business outcome. Furthermore, I see parallels between “exploration” (e.g., AI agents retrieving context) and “exploitation” (e.g., AI agents taking action on the operator’s behalf). This is why I never agreed with software engineers who bemoaned the non-determinism of LLMs as their unredeemable flaw. Even more so, I argue that moving into the world of data and AI as a software engineer requires updating their priors on what’s acceptable.
Thankfully, many skills and techniques from other fields can help engineers navigate this new domain. Specifically, we already have fields of study built around repeated decisions where outcomes are uncertain, distributions aren’t fully known beforehand, and the objective is not to eliminate uncertainty but to make better decisions despite it.
But what about high-risk work?
Here’s the thing… my disagreement with software engineers isn’t rooted in me believing they are wrong. If anything, they're correct and doing exactly what they are hired to do: build and maintain a reliable codebase to solve business problems. Instead, it highlights a fundamental difference in how data teams and software teams think and approach solving problems. The disagreement isn’t really about whether reliability matters. It’s about when uncertainty becomes unacceptable and what you do before you reach that threshold.
Data teams come into the meeting talking about all the opportunities with data and AI, while their software engineering colleagues hear a list of experimental features that are bound to break and be their problem while on-call. Their pushback is a good thing. Both sides reflect each stakeholder’s comfort with uncertainty.
Now, I won’t argue that some problems absolutely cannot rely on AI. If you are sending billions of dollars to Mars on a rocket, I hope the calculations come from a seasoned professional, and not an AI agent hell-bent on hacking Hugging Face. But most of the work in the software industry isn’t rocket surgery! Even more surprisingly, some of the most regulated industries know nothing but uncertainty as their default and have wrestled with exactly this problem for a long time.
I’ve had a bunch of twists and turns in my career, but my foundation as a data professional is rooted in my time at Stanford Medicine, where I learned experimental design for clinical research (first as a student and then as a research analyst). This world can be summed up as the following— humans are random entities running wild, and we somehow have to show proof that giving those wild humans a pill will 1) not kill them, 2) not make them worse, and 3) hopefully improve their quality of life.
This work is literally life or death; yet through all the uncertainty and randomness, we can still reason about causality and make decisions even when uncertainty remains high. Understanding that doctors don’t have all the answers, but rather were trained to balance risk and uncertainty to optimize their patients’ health, was honestly my ultimate takeaway from grad school and strongly informs how I build with AI agents today.
All models are wrong, but some are useful.
I’m sorry, but we have to talk about statistics. To be fair, I warned you at the beginning of this article about my affinity for p-values, and now you are stuck at the water cooler with no one to save you from my pontifications. I first heard the quote, “all models are wrong, but some are useful,” in my public health modeling class, where we learned to create simulation models to show the impact of diseases and/or health interventions on a population.2

Me at my company’s happy hour.
Specifically, we learned how to build these models to support decisions under high uncertainty and limited budget (the sad truth is that governments don’t have enough resources to save everyone… this space can get dark). Under these constraints, every single model you create will absolutely be wrong, as you can’t capture everything (those random humans are running WILD!), and thus the goal is to instead make them useful for the decision being made.
One of my favorite examples of this is sampling. Say, for instance, you want to know the preferred toothpaste brand of every adult in the United States (9 out of 10 dentists’ recommendations be damned). Theoretically, you could survey every single adult in the United States, but that would be insanely expensive and time-consuming.3 So, you instead listen to statisticians and use proper sampling methods to construct your sample and power calculations to determine how many people you need to survey. The model is wrong, but it’s still very useful given that the price of fully complete correctness is infeasible.
Building AI agents despite uncertainty.
Looking back, my stakeholder was asking the right question: “How do we trust an output if it’s different each time?” We differed on what evidence of trust should look like. For a probabilistic system, trust doesn’t have to mean producing an identical answer every time. It can mean understanding how the system behaves across repeated trials: how often it succeeds, where it fails, how severe those failures are, and whether that variability is acceptable for the workflow. Thus, the unit of evaluation changes from a single output to a distribution of system behavior. So the question isn’t simply, “Can I make this system produce the same answer every time?” It’s, “Can I characterize the system’s behavior well enough to know when I can rely on it, and when I can’t, given my accepted tradeoffs?”
To me, this level of uncertainty was fine for a POC where we were just determining whether the work was worth investing more in. On the other hand, that uncertainty felt like an outsized risk for my stakeholder, who needed more proof to justify reallocating resources. This isn’t unique to my situation, as every company is going through this dance as it tries to figure out the actual ROI of its AI investments.
Ultimately, I got the POC to move to the next stage, but it required a lot of evidence:
What are the strengths and limitations of AI agents, and how does that map to our business offering?
How can we evaluate if an AI agent is reliable enough for a given workflow?
How do we compare the results of one agent over another as we iterate on the solution?
Why is the problem being solved business-critical?
What are the tradeoffs of solving this problem with and without AI agents?
Swap the words “AI agents” with “ML models” in the list above, and we are right back into the conversations we had in the 2010s.
Much of the evidence I created came from my clinical research and data scientist days. For example, you can repeatedly simulate an agentic workflow and capture the logs and metadata to create a distribution of various statistics and outcomes. In other words, quantify an AI agent's uncertainty given its settings, system prompt, and explicit task instructions.
Using the mental model of clinical research, my AI agent is a “human subject,” the LLM’s settings are the “subject’s demographics,” and the prompt or provided tool is the “intervention” (e.g., the pill that’s not a placebo). Suddenly, the black box of AI agents fits into a mental model that’s been around since the 19th century and has a plethora of tools for reasoning about causality under uncertainty.
Non-determinism isn’t AI’s weakness… It’s just your blind spot if you are not used to these types of constraints when solving such problems. Software engineering traditionally gives us ways to constrain systems, and statistics gives us ways to reason about systems whose behavior varies. Companies increasingly need both as AI agents mature and their surface area grows amongst critical systems.
Scaling DataOps Newsletter is reader-supported. Get the next issue in your inbox.

Found my content interesting and want to learn more from me? I have four courses on LinkedIn Learning teaching you the fundamentals of data quality and data engineering best practices. With 40K+ students and hundreds of five-star reviews for my hands-on coding courses, I’m confident you will learn valuable data skills and have fun doing it!
Feel free to check them out here:
Footnotes
1 I’ve written heavily on this topic on other Substacks, and this was a core theme of the O’Reilly book I co-authored.
2 Huge shoutout to my professor, Dr. Sanjay Basu, whose class convinced me to pursue data science and was my first introduction to coding. The class was based on his book Modeling Public Health and Healthcare Systems, and I still reference it today.
3 For context, the 2020 US Census cost the government about $14B to conduct.