The most dangerous assistant always has an answer

A message arrives at ten at night. Say it reads like this: a patient is on a blood thinner, she has an extraction booked for the morning, and she wants to know what to do. An automated assistant answers straight away — fluent, calm, entirely sure of itself. And wrong. The scenario is written as an illustration, not something that happened.

Now look at where the failure actually is, because it isn't where it seems. The failure isn't that the answer came out inaccurate. The failure is that an answer came out at all. A clinical question isn't something software should answer better — it's something software shouldn't touch. The only correct behaviour at ten at night is to hand the question to a person, and tell the patient it has been handed over.

That sounds obvious. It isn't. Most tools on the market were built so that there is always an answer, and a tool like that can't tell a question inside its remit from one outside it.

Why "I don't know" beats a confident wrong answer

When an assistant refuses, you see it. A question stays open, someone has to call the patient back, and there's a record. That's work — but it's visible work, and you can count it and manage it.

When an assistant invents, nobody sees anything. The patient gets an answer, says thanks, and closes the chat. You meet the mistake later, when it comes back to you in a different form: someone who fasted for no reason, someone who didn't show up at all, a price quoted in your name, a review you can't trace to anything. By then it doesn't look like a software error. It looks like your clinic.

That's the whole difference. A refusal is a cost you manage. A wrong answer is a commitment made in your name, in writing, without your knowledge.

And there's a use for refusals beyond safety. They are the most useful data the system will ever hand you: the list of questions it couldn't answer is exactly the list of things you never wrote down, or wrote badly. Invented answers don't produce that list. They erase it.

The errors don't fall where you'd expect

A language model doesn't know what's true. It knows what sounds true — and on routine questions those are the same thing. Opening hours. Parking. How long a cleaning takes. It will get those right nearly every time.

The gap opens on the unusual question. The cancellation policy you worded differently from everyone else. A price that is only yours. A procedure your clinic follows and nobody else does. There's nothing there that sounds right — there's what your records say, or nothing.

And that is exactly the question the patient can't check. They asked it because they didn't know.

So the pattern runs backwards from the intuition: the tool is most accurate on the questions that don't matter, and most confident precisely where it knows least.

What it takes to build one that refuses

Refusal isn't a switch in a settings panel. It's a shape you build in — and every part of it can be inspected from outside.

**A boundary before a capability.** First you decide what the assistant doesn't touch. For us that's everything clinical: it gives no medical information and no advice, it hands over to a person, and it tells the patient that it has.

**A source before a sentence.** Inside what is in its remit, the answer is assembled only from what you wrote. No source, no answer. Simple to say, expensive to build, because it means giving up every question you could have answered approximately.

**The checker is not the writer.** A model checking itself is not a check — it is the same bias twice. The check has to run in a separate layer that didn't write the answer and isn't invested in it.

**What fails is the answer, not the sentence.** A paragraph with four true sentences and one invented one reads as trustworthy — the true parts vouch for the false one. So a single unsupported claim has to take the whole answer down.

**No free retries.** A system that regenerates until something passes is a system hunting for a way around its own check. A regeneration is a fresh chance to be wrong, not a second chance to be right.

And one more, which is the condition for all the others: **a refusal has to go somewhere.** "I don't know" that stops there isn't safety — it's an abandoned patient. The question has to reach a person with the full context, and stay open until somebody has actually taken it. An assistant that refuses and leaves the patient hanging hasn't solved the problem. It has moved it somewhere you won't see it.

What this still doesn't fix

Verification checks that every claim rests on your documents. It can't make a wrong document right. If an old price list is still sitting in your folder, you'll get an out-of-date price out of it — verified, with a citation.

What does change is that the gap becomes visible. The system points at the document the answer came from, and you find where your own records disagree with each other. That's separate work, and it's yours. A tool that promises you more than that is promising.

Five questions for your next demo

Whosever it is. Take fifteen minutes and test it yourself:

• Ask a clinical question, and see whether the tool tries to answer at all.
• Ask something true that appears in none of your documents. A good system says it doesn't have it. A bad one assembles an answer.
• Ask about a policy you changed recently, and see whether the old version comes back.
• Ask for the source — then open it and read it. A citation that doesn't say what was claimed is the quietest failure there is.
• Ask what happens to a question it refused. Who receives it, how fast, and what happens if nobody answers.

The fifth is the one that separates a product from a demo.

And finally, the disclosure

This is exactly how we build SoliumOS, on two separate paths. Facing the patient: the agent handles the administrative side only — appointments, billing, forms and policies — it answers from the answers you wrote, and anything not in them goes to a person rather than to a guess. Facing the owner, in the console: a separate model checks the answer before you see it, one unsupported claim rejects the whole thing, and what comes back is either "this isn't in your documents" or an answer marked unverified. Never a guess dressed as fact.

All of it runs in a cloud project registered to you, on your keys. Day to day, the data doesn't pass through us.

But the five questions above work on any vendor. Run them on us too.