Building a Medical AI Agent in 3 Days: Architecture, Stack, and Lessons

A doctor needs to know whether two medicines on a patient's list interact, and which source says so. A general chatbot answers that question fluently but cannot show where the answer came from. A medical AI agent is software that answers such questions from verified medical sources and can point to the document or record behind each answer.

This build log covers a working medical AI agent built in three days. It walks through the data model, the retrieval layer that keeps answers grounded, the interface, and what a prototype like this does and does not prove. The agent answers from medicine leaflets and a drug interaction graph, and every answer can be traced back to one of them.

What is a medical AI agent?

A medical AI agent is software that answers clinical questions by choosing a tool, retrieving verified medical data, and grounding its answer in it. It decides which source fits each question, such as a document search or a database query. When its sources hold no answer, it is designed to say so instead of guessing.

Conversational AI in healthcare is the wider category. It covers symptom checkers, scheduling assistants, patient chatbots, and clinician assistants that work in natural language. A healthcare chatbot usually follows scripts or answers from one model's training data. An agent adds tools it can call, such as a vector search or a graph query, and a loop that decides which tool fits the question in front of it.

Healthcare chatbot development and agent development also differ in where the work goes. A scripted chatbot is mostly conversation design. An agent is mostly data work: preparing sources, deciding which questions each source may answer, and testing that routing sends each question to the right place.

Agentic AI in healthcare pushes that loop further, with AI agents in healthcare that plan several steps or take actions in other systems. The agent in this build is deliberately narrow. It answers medication questions from leaflets and interaction questions from a graph, and it takes no actions. The World Health Organization makes the same point in its guidance on large multi-modal models, asking developers to build for well-defined tasks.

That guidance, published in January 2024, also lists documented risks of false, inaccurate, biased, or incomplete statements from these models. A narrow agent does not remove those risks. It makes them easier to find, because every answer traces back to a leaflet or a graph query that a person can check. Broader machine learning uses are covered in the guide to reasons to implement machine learning in healthcare.

An agent is also the wrong tool for some jobs. When a question has one fixed answer in one system, such as a dose lookup in an existing drug database, a direct query is cheaper and easier to validate. Agents earn their complexity when questions arrive in free language and must be routed to different sources, the kind of applied work described on the artificial intelligence page.

Day 1: The data model and the knowledge graph

The idea came from home. My wife is a practicing doctor, and she kept raising the same problem: AI chat tools gave answers that sounded confident but were not medically accurate or evidence-based. I wanted an assistant that answers only from structured, verified medical data, and says so when the data is not there.

I started by asking GPT-4 how to build a medical AI agent that only answers from the specific medical documents I give it, then turned that conversation into an architecture. LangChain and LangGraph would define the agent's logic. Qdrant would store the leaflets as searchable vector embeddings, and Neo4j would hold drug-to-drug interactions as a knowledge graph.

Before any medical data went in, I tested tool switching on financial queries, such as the price of Bitcoin. If the agent could route a price question to the right tool, it could route a medication question too. A Swagger-powered API let me test logic flows before building any interface.

Then came the knowledge core. I deployed Qdrant and Neo4j on an Azure virtual machine and dusted off an old Excel file of drug interaction data from my medical research days. I wrote scripts to parse that file and load it into Neo4j, so each interaction became something the agent could query directly instead of reading about.

In parallel, I scraped the medicine leaflets, the summaries of product characteristics known locally as RCPs, from the national medicines agency's website. I wrote a parser for them and stored the text in Qdrant for semantic search. By the end of the day, the agent had two kinds of knowledge: text it could search by meaning, and interactions it could look up exactly.

Drug interactions suit a graph because they are relationships, not passages of text. A question about two medicines becomes a lookup of the link between them, which returns a record or returns nothing. Leaflets suit a vector store because their answers sit in paragraphs, where the wording of a question rarely matches the wording of the source. Keeping the two stores apart also keeps each answer's source obvious.

Day 2: How does a medical AI agent control hallucinations?

A medical AI agent controls hallucinations mainly through its data: it answers only from retrieved sources and says so when they are silent. A vector store handles text such as leaflets, where meaning matters more than exact wording. A knowledge graph handles facts that need an exact answer, such as whether two drugs interact.

Retrieval measurably helps. A 2025 meta-analysis of 20 studies in the American Medical Informatics Association's journal found that retrieval-augmented generation improved model performance on biomedical tasks, with a pooled odds ratio of 1.35. The same review cites a preliminary study in which GPT-4 Turbo extracted table data with only 16 percent accuracy, a real concern for leaflets full of dosing tables.

RAG in healthcare works well when the answer sits in a paragraph of text, such as a contraindication or a storage condition in a leaflet. It struggles with questions that combine records, such as checking every pair in a list of five medicines. That second kind needs a structure that can be queried exactly, which is the job the graph does in this agent.

The agent runs on two LangChain tools. process_medication_query handles general drug questions against the leaflets, and process_interaction_query checks the graph for interactions. Both accept full natural-language questions rather than keywords, so a clinician can ask the way they would ask a colleague.

An agent with two tools also needs reliable routing. A side-effect question belongs to the leaflet search, and a question naming two drugs belongs to the graph. When routing fails, a correct source answers the wrong question, a failure as risky as a hallucination.

Vector search over the leaflets worked smoothly. The graph did not. The first version let the model generate Cypher queries on the fly, and any small schema change or naming inconsistency broke the logic completely. The build switched to LangChain's GraphCypherQAChain, which turns a natural-language question into a Cypher query against the graph's schema.

In 2026, GraphCypherQAChain lives in the langchain-neo4j package, and it will not run until the developer sets an explicit flag accepting that it can make dangerous requests. The LangChain reference recommends database credentials scoped to only the permissions the chain needs. For a medical knowledge graph, that means a read-only account, because a model that writes queries can also write harmful ones.

Swapping the model is simple in this design. The agent reaches its data only through tools, so a newer model can replace the one used in the build without touching the graph or the leaflets. The tools, the schema, and the database permissions carry the design, and they are what a 2026 rebuild would keep. Model names date quickly, and tool boundaries do not.

Hallucination control, then, is mostly a data design problem. A graph returns an interaction or returns nothing, and an empty result lets the agent say the data holds no record. Prompts help only on top of sources that can say no.

Day 3: The interface and going live

Day 3 gave the agent a face. I found an open-source React project built on Next.js under the MIT license, which already used WebSockets for real-time communication. It looked great out of the box. With GPT's help, I generated the Python backend that connects the WebSocket to the agent. Then I debugged the authentication flow and message broadcasting by hand until everything synced.

For the rest of the backend, I used Supabase, and its built-in AI assistant set up authentication, chat history storage, feedback collection, and admin tools. I barely touched the Supabase dashboard. Then I pushed the code to GitHub, cloned it on the Azure VM, configured NGINX, added SSL with Certbot, and launched.

A single virtual machine suits a prototype and little else. Production needs redundancy, separate hosting for the databases and the application, and a staging environment where changes are tested before they reach users. None of that was needed to answer the three-day question, and all of it is needed before clinicians depend on an agent. It also needs monitoring that alerts someone when a database or model API stops responding.

As a final test, I added two new document types: clinical guidelines and operational protocols. The agent parsed them and answered questions about them right away. That moment was deeply satisfying, and it came at the end of three focused days of work.

By the end of day 3, the agent was running on the Azure VM, answering medication questions from the leaflets and interaction questions from the graph. It was a prototype built in three focused days, and the next section explains exactly what that word means here.

What does a 3-day prototype prove, and what does it not?

A three-day medical AI agent proves that the architecture works end to end: retrieval, graph queries, tool routing, and a live interface. It does not prove clinical accuracy, safety, security, or regulatory readiness. Those require validation work that takes months, and that gap is where most of the cost of a real product sits.

Clinical validation is the first gap. A prototype has been checked by its builder, not tested against a set of questions whose answers clinicians have reviewed. Nobody has yet measured how often it is wrong or how it fails. That measurement has to exist before a clinician relies on the agent, and it has to be repeated whenever the data or the model changes.

In practice, validation starts with a question set. Clinicians write realistic questions, including some the sources cannot answer, and agree on the correct response to each. The agent is scored on accuracy, on whether it cites the right source, and on whether it declines when it should. That last score is easy to overlook, and a confident wrong answer is where real harm starts.

The data in this build is a plain example of the second gap. The interaction records came from an old research spreadsheet, not from a maintained, licensed interaction database, and the leaflets were scraped once. A production agent needs current, versioned, licensed sources, plus a process that reloads a leaflet when the medicines agency updates it. None of that fits inside three days, and none of it is optional.

Security is the third gap. Supabase authentication is fine for a prototype, and it is not enough for patient data. A production agent needs role-based access, an audit log of every question and answer, and read-only database accounts. If it handles protected health information for a covered entity, HIPAA applies, which means a business associate agreement and an architecture built to HIPAA requirements.

On our healthcare projects, we apply the controls this prototype skipped. Development runs against anonymized data. Production access is limited to named engineers on the project, logged, and revoked when the engagement ends. Patient data stays in infrastructure the client controls, in the region its regulator requires.

Regulation is the fourth. The Food and Drug Administration reissued its Clinical Decision Support Software guidance on January 29, 2026. Software for healthcare professionals falls outside the device definition only if it meets four criteria, one of which is that the clinician can independently review the basis for a recommendation. Decision support aimed directly at patients is generally regulated as a device.

The WHO guidance also warns about automation bias, where clinicians or patients overlook errors they would otherwise catch because the output comes from software. A cited source helps only if someone reads it. Interface choices, such as showing the source beside every answer and marking any answer that has none, decide whether citations work in practice. In a clinical tool, that design work belongs in the first sprint.

These gaps are normal, and naming them is the point of a prototype. Three days answered the technical question of whether the parts fit together. The clinical, security, and regulatory questions are answered only by the slower work that follows, on healthcare builds where data has legal boundaries and every action needs an audit trail. A build log that stops at the demo skips the part that decides real use.

What were the hard parts of the build?

In a three-day medical AI agent build, the hard parts were generated database queries, real-time interface wiring, and hallucination control. The language model itself was not the problem. Each difficulty sat where the model met a system with strict rules: a graph schema, a WebSocket protocol, or a clinician's need for a traceable answer.

I assumed the graph would be the easy part. It wasn't. Letting the model write Cypher freely made the agent depend on a schema I was still changing, and after two frustrating hours of debugging I scrapped that approach. Finding GraphCypherQAChain, which maps natural-language questions to Cypher automatically, was a huge relief.

The interface taught a smaller version of the same lesson. The React template looked finished, but real-time syncing with a Python backend needed exact message handling. GPT and Supabase generated most of the plumbing. I still had to debug authentication and message broadcasting by hand before the pieces worked together. Generated code helped a lot there, and it was not perfect.

Hallucination needed constant care. Even with carefully selected documents, I had to test and adjust prompts so the agent would not go off-script or invent information when it was unsure. That tuning is still not perfect. It is far ahead of a general chatbot with no sources behind it, and that difference is the reason the project exists.

My main takeaway is that building an agent now depends less on raw coding skill than on domain knowledge and clear thinking. AI-generated code is helpful, not perfect, so expect to debug. Retrieval over curated medical data is powerful, and the agent still needs clinical validation before anyone relies on it. The article on the impact of OpenAI on app development covers generated code more broadly.