The most dangerous assistant always has an answer

A message arrives at ten at night. Say it reads like this: a patient is on a blood thinner, she has an extraction booked for the morning, and she wants to know what to do. An automated assistant answers straight away — fluent, calm, entirely sure of itself. And wrong. The scenario is written as an illustration, not something that happened.

Now look at where the failure actually is, because it isn't where it seems. The failure isn't that the answer came out inaccurate. The failure is that an answer came out at all. A clinical question isn't something software should answer better — it's something software shouldn't touch. The only correct behaviour at ten at night is to hand the question to a person, and tell the patient it has been handed over.

Do not assume that every assistant distinguishes a question within its remit from one outside it. That behavior needs to be defined and tested. Receiving an answer does not establish that the system should have answered.

Why "I don't know" beats a confident wrong answer

A refusal can become something the team can see and manage: the question stays open and recorded, and someone is responsible for calling the patient back. It is still work, but it does not disappear when the conversation ends.

An invented answer can go unnoticed. The patient receives a reply, says thanks and closes the conversation. The mistake may surface later: someone who fasted unnecessarily, someone who did not attend, a price quoted in your name or a review you cannot trace. To the patient, it may look like a failure of the clinic rather than its software.

That's the whole difference. A refusal is a cost you manage. A wrong answer is a commitment made in your name, in writing, without your knowledge.

Refusals can be useful beyond safety. Reviewing unanswered questions can reveal information that was never documented or was written unclearly. Invented answers may hide those gaps instead of bringing them to the team's attention.

A plausible answer still needs a source

A language model can produce a fluent, confident answer without an appropriate source. The wording alone does not establish accuracy. Even routine questions — opening hours, parking or how long a cleaning takes — need answers that match the clinic's information.

The need for a source is especially clear with your cancellation policy, your prices or a procedure specific to your clinic. A generic answer may sound reasonable while failing to match your records.

The patient may have no way to check the answer independently. They are asking the clinic for clarification and may rely on the reply.

A confident tone, or a previous correct answer, is not a basis for trusting the next one. Check the support for each answer, especially when it depends on details specific to the clinic.

What it takes to build one that refuses

Refusal isn't a switch in a settings panel. It's a shape you build in — and every part of it can be inspected from outside.

A boundary before a capability. First you decide what the assistant doesn't touch. For us that's everything clinical: it gives no medical information and no advice, it hands over to a person, and it tells the patient that it has.

A source before a sentence. Inside what is in its remit, the answer is assembled only from what you wrote. No source, no answer. Simple to say, expensive to build, because it means giving up every question you could have answered approximately.

The check needs to be independent. A model's self-review can repeat the same blind spots. In the design described here, a separate layer that did not write the answer checks its claims against the sources.

An unsupported claim rejects the whole answer. A paragraph with four true sentences and one invented one can sound trustworthy because the true parts lend credibility to the false one. In our design, a single unsupported claim therefore rejects the whole answer.

Do not keep retrying until something passes. Repeatedly generating answers until one passes can weaken the control. Eventually passing a check does not establish that the process is reliable. Regeneration is therefore not a substitute for checking the sources.

And one more, which is the condition for all the others: a refusal has to go somewhere. "I don't know" that stops there isn't safety — it's an abandoned patient. The question has to reach a person with the full context, and stay open until somebody has actually taken it. An assistant that refuses and leaves the patient hanging hasn't solved the problem. It has moved it somewhere you won't see it.

What this still doesn't fix

Verification checks that every claim rests on your documents. It can't make a wrong document right. If an old price list is still sitting in your folder, you'll get an out-of-date price out of it — verified, with a citation.

What does change is that the gap becomes visible. The system points at the document the answer came from, and you find where your own records disagree with each other. That's separate work, and it's yours. A tool that promises you more than that is promising.

Five questions for your next demo

Whosever it is. Take fifteen minutes and test it yourself:

• Ask a clinical question, and see whether the tool tries to answer at all.
• Ask something true that appears in none of your documents. A good system says it doesn't have it. A bad one assembles an answer.
• Ask about a policy you changed recently, and see whether the old version comes back.
• Ask for the source — then open it and read it. A citation that doesn't say what was claimed is the quietest failure there is.
• Ask what happens to a question it refused. Who receives it, how fast, and what happens if nobody answers.

The fifth is the one that separates a product from a demo.

And finally, the disclosure

This is exactly how we build SoliumOS, on two separate paths. Facing the patient: the agent handles the administrative side only — appointments, billing, forms and policies — it answers from the answers you wrote, and anything not in them goes to a person rather than to a guess. Facing the owner, in the console: a separate model checks the answer before you see it, one unsupported claim rejects the whole thing, and what comes back is either "this isn't in your documents" or an answer marked unverified. Never a guess dressed as fact.

All of it runs in an isolated cloud environment you control, on your keys. Day to day, the data doesn't pass through us.

But the five questions above work on any vendor. Run them on us too.