Applied AI Architecture - 1. The Illusion of the AI System

There's a growing recognition in applied AI that most of what breaks in production isn't the model. It's everything around it. The illusion that a language model can act as an entire system has produced a steady flow of half-functional prototypes, costly rewrites, and compliance headaches.

The data reveals a stark value gap. While industry surveys suggest the vast majority of enterprise developers are exploring AI agents, a mere fraction report achieving meaningful value from their initiatives. This chasm between enthusiasm and execution isn't a failure of the models themselves, but of the architectural assumptions surrounding them. The reason is simple: reliability and traceability. When a system has to answer to customers, auditors, or regulators, unpredictability stops being an interesting research problem and becomes a material risk.

A Pattern of Failure

The failure stories are surprisingly consistent across domains. A healthcare startup builds a clinical documentation assistant that works perfectly in demos - capturing patient notes, suggesting diagnoses, formatting reports. Then they connect it to their EHR system. The model starts inventing medication dosages. It hallucinates patient histories that don't exist in the database. When a doctor questions an output, no one can trace which parts came from the model versus which came from actual records. The logs show only that a prompt went in and text came out.

A legal tech company builds a research assistant that summarises case law and generates citations. In testing it's flawless, but in production it starts citing cases that don't exist - complete with plausible case names, dates, and legal reasoning. A junior associate uses one of these hallucinated citations in a brief and the opposing counsel catches it. Now there's a malpractice review, the firm has pulled the tool, and the vendor is facing questions about their validation process they can't answer. The model's output was unconstrained as implemented. The system around it had no way to verify, constrain, or audit what it produced.

What looked like intelligence in a notebook was really a fragile composition of unconstrained calls with no strong contracts between them. Debugging sessions feel like exorcisms; when the behaviour changes, no one can prove why. That's a governance nightmare, not an innovation story.

Returning to Architecture

The underlying problem isn't randomness, though that's how most practitioners describe it. Even at temperature 0, where sampling should be deterministic (i.e. same input, same output), LLM output remains fundamentally unstructured. A call to llm.generate(prompt) returns text - just text. That text might be valid JSON. It might be malformed JSON. It might be prose that ignores the format entirely. It might be a refusal. You don't know until you parse it, and by then you're in production with user data.

This isn't a theoretical concern. Temperature 0 doesn't guarantee identical outputs across API calls due to inference batching, floating-point precision, and infrastructure variance. But even if it did - even with perfect determinism - the output is still an unconstrained string with no contract. No schema. No type guarantees. No semantic verification. The string 'patient_age: 45' and '{"patient_age": 45}' and 'The patient is forty-five years old' are all valid responses. None are interchangeable in a system that needs structured data.

Recent APIs have added structured output guarantees - JSON mode, schema enforcement - which is progress. But guaranteed JSON syntax is just the first layer. A valid JSON object can still contain hallucinated medication names, incorrect dosages, or fabricated case citations. The architecture challenge isn't just getting structured output; it's wrapping that output in validation, versioning, logging, and governance that makes it production-ready.

When people say LLMs are ‘non-deterministic’, they mean unpredictable. The architecture we need makes them bounded.

The breakthrough in applied AI will not come from more powerful models, but from more disciplined software architecture. By treating LLMs as components, we move the uncertainty to the edge of our systems. Each model call becomes a self-contained, observable event, wrapped in the deterministic scaffolding of validation, versioning, logging, and retries.

This is the same conceptual move that made unreliable networks usable. Before TCP, we had UDP - a protocol that threw packets into the void with no guarantees. Packets arrived out of order, or not at all. Building reliable applications on UDP was possible but painful, requiring every developer to reinvent acknowledgment schemes and retry logic. TCP wrapped that unreliable substrate in a deterministic layer: sequence numbers, checksums, acknowledgments, retransmission. The underlying network was still unreliable. The interface was not.

You couldn't build a bank on UDP and you can't build regulated systems on raw LLM calls.

The analogy has limits. TCP succeeded because it solved a protocol-level problem with a protocol-level solution. Packet integrity is context-free - a checksum works the same for banking data and video streams. But LLM reliability isn't a protocol problem. The HTTP request succeeds. The JSON is syntactically valid. The problem is semantic: the content is wrong, unverifiable, or doesn't match your domain requirements.

That's why the solution lives at the application layer. What constitutes a valid clinical note isn't something a universal protocol can determine - it requires medical schemas, drug databases, and business logic. Model providers are solving their part: structured outputs guarantee syntax. Your architecture must guarantee semantics. The validation, versioning, and governance logic is necessarily bespoke, just as your business logic is bespoke.

This isn't a regression to pre-TCP chaos. It's recognising that semantic correctness is an application concern, not an infrastructure concern. Instead of calling llm.generate(prompt) and hoping the output is valid, you write something like:

result = validate(
  llm.generate(prompt, version="gpt-4-0125"), 
  schema=PatientNote
)

Now you get either a typed, validated object or a logged validation error. The raw model output is still unconstrained, but the system interface is reliable. You can version the model. You can version the schema. You can retry on failure. You can observe every input and output. You can prove to an auditor exactly what happened.

Here’s what deterministic scaffolding looks like in practice and how this shift transforms how systems behave under pressure. Consider a clinical workflow with three LLM calls in sequence: one extracts structured data from a physician's voice note, another validates it against the patient's existing record for contradictions, and a third generates the formatted documentation.

In an agent-based architecture, these three calls are chained with hope. The first call outputs unstructured text. The second call ingests that text and maybe catches an error, maybe doesn't. The third call formats whatever it receives. When something goes wrong - and it will - you have three unconstrained components with no boundaries between them. The logs show three prompts and three responses. That's not enough to debug. It's definitely not enough to satisfy a regulator.

In a component-based architecture, each call is wrapped:

extraction = validate(
  llm.generate(voice_note, version="gpt-4-0125"),
  schema=ClinicalExtraction
)

validation = validate(
  llm.generate(f"Check {extraction} against {patient_record}", version="gpt-4-0125"),
  schema=ValidationResult
)

documentation = validate(
  llm.generate(f"Format {extraction} as progress note", version="gpt-4-0125"),
  schema=ProgressNote
)

Now each call has a contract. If extraction fails schema validation, the workflow stops. The log captures: prompt version, model version, input hash, output rejected at schema validation, specific fields that failed. You can retry with a more constrained prompt. You can route to human review. You can prove in an audit exactly where the failure occurred and what the system did in response.

When an auditor asks 'how did the system arrive at this recommendation?' you show them: prompt version 2.3, model gpt-4-0125, input schema validated at 2024-10-15T14:23:11Z, output passed validation, three fields extracted with confidence scores logged, all attributions preserved in metadata. Now you have proper evidence.

From Existential Threat to Engineering Problem

This architectural shift transforms hallucinations from an existential threat into something you can catch and handle. Semantic validation - checking whether a medication exists, whether a case citation is real - becomes an explicit step with its own validation wrapper and failure handling. It makes system behaviour reproducible, auditable, and compliant by design. The same logs that allow you to debug a failure are the same logs an auditor needs to verify compliance with GDPR or the EU AI Act. In this sense, good architecture is good governance.

The shift from agent-centric experiments to component-based systems is not a philosophical stance. It's a practical necessity. An ‘agent’ implies autonomy, decision-making, and emergent behaviour - all of which are incompatible with the deterministic guarantees production systems require. A component, by contrast, has a defined interface, explicit failure modes, and observable behaviour.

This doesn't mean LLMs become less useful. It means they become usefully constrained. A classifier component that routes support tickets doesn't need to be creative. It needs to return one of seven valid categories, or fail explicitly. A summariser component that generates executive briefs doesn't need to improvise. It needs to produce summaries that match a schema, respect a token budget, and preserve attributions to source documents.

The architecture determines what's possible. When you treat an LLM as a system, you inherit its failure modes. When you treat it as a component, you control them.

Next: Component Anatomy

This is the first in a series exploring applied AI architecture for practitioners. The next post will examine how to wrap unconstrained model calls into reliable components: what it looks like in practice, how to version it, how to test it, and how to integrate it into systems that already exist. After that, we'll look at observability - how to instrument LLM calls so that debugging doesn't feel like divination.

When the novelty of autonomous agents fades, what remains is the software. In that world, large language models have finally found their place: as extraordinary components inside ordinary, reliable systems. This series is about the architecture of those systems.