How an answer is made.
Six steps run on every lookup. The answer is a sentence from the policy, not a sentence about it — and the last two steps exist so that a confident wrong answer is not one of the outputs.
- 01Ask
- 02Retrieve
- 03Expand
- 04Quote
- 05Verify
- 06Refuse
Six steps, every lookup.
§ 01-
01 — Ask
The question, with identifiers stripped
Staff ask in plain language. Direct identifiers — SIN, health number, phone, email, postal code, account IDs — are stripped in the browser and again on the server. Staff enter the policy question, not the person.
question → scrubbed query -
02 — Retrieve
Keyword retrieval over the corpus
Policy language is keyword-heavy, so retrieval runs BM25 keyword search over the indexed passages and ranks the candidates against the query. A vector-similarity and rerank layer is in development and is not live.
query → ranked passages -
03 — Expand
One hop along the cross-references
Eligibility is a chain of gates. The sections the top hits point to through parsed “see Section X” links are pulled in as well, so the whole gate set surfaces instead of one lexical match read in isolation.
top hits → + referenced sections -
04 — Quote
The answer is a span, not a summary
What comes back is drawn from the retrieved passages themselves. No model writes, edits, or extends a quoted line; ranking layers only choose among verbatim passages, and optional plain-language layers are separate and ship off.
passages → quoted spans -
05 — Verify
Exact-substring check, then metadata by identifier
Every quote is machine-checked to be an exact substring — whitespace normalized — of the source that was ingested. A quote that does not match is dropped rather than shown. The link and the effective date are then attached from that passage’s own metadata.
span → checked · cited -
06 — Refuse
Below the threshold, nothing is returned
If the top match falls below the relevance threshold, or no quote survives verification, the result is a refusal and an escalation — not the nearest thing, not a confident guess.
below threshold → escalate
A quote that doesn’t match its source is dropped.
§ 02It has to exist, character for character
Every returned span is compared against the passage it claims to come from. If it is not an exact substring of a real, ingested passage, it is dropped before it reaches the screen. A paraphrase cannot pose as a quote.
The link and the date are looked up, not written
Once a quote is verified, its section identifier, source link and effective date are
read from that passage’s own metadata by identifier and attached. There is no
path by which a citation points somewhere the quote did not come from.
“The value of the combined liquid assets of all household members cannot exceed three months total Core Benefits.”
# each returned span, checked against the passage it came from verify(span) → exact substring of passage 4127 · keep → attach section, link, effective date verify(span) → not found in any ingested passage · drop # nothing survives that the source does not literally say
The most valuable answer is “not covered.”
Below the relevance threshold, the result is no matching policy — escalate. A tool that always answers teaches staff to stop checking. This one is built so that silence is a legitimate output, and so that a gap in coverage looks like a gap instead of an answer.
I can’t find policy on that — check the manual.
The threshold is calibrated on an evaluation set so that out-of-scope questions land here. It would rather send you to the binder than return a passage that isn’t there.
Questions about policy, not about people.
§ 04-
Scrubbing
Identifiers stripped, both sides
SIN, health number, phone, email, postal code, account IDs and labelled lines are removed in the browser before the query is sent, and again on the server. Free text can still carry a name, so staff enter the policy question, not the person.
-
The log
The scrubbed question, not a client file
Each lookup records the scrubbed question, the section identifiers returned, and a hash of each quote — enough to show the same answer again, with direct identifiers already removed and no client file attached.
-
Reconstruction
An answer given in March, shown again in November
Because the log pins the sections and the quote hashes, the basis of a past answer can be reconstructed on appeal without holding the details of the person it concerned.
-
Access
Per-worker access, short retention
Even structured situational facts can be re-identifying in rare combinations, so access is per-worker and retention is deliberately short.
If a model is added for reranking or a plain-language gloss, use with real files stays gated behind concrete prerequisites — a completed Privacy Impact Assessment; a data-processing agreement that settles retention and training on inputs; queries that carry no direct identifiers; and the engine and version logged on every call.
We check the policy. You make the decision.
We return the governing rule and how the stated facts line up against it. We do not render the determination. Dollar figures follow the same discipline as quotes: amounts are read from the publisher’s own rate schedule, carry their own effective date and source page, and are never machine-written. The result is evidence for a staff member’s judgment, not a substitute for it.
What is measured before a corpus ships.
§ 06Four measures
A golden set of questions, verified by a subject-matter expert and seeded from what frontline staff actually ask, is run against each corpus before it is switched on.
- Retrieval recallDid the governing section appear among the candidates at all?
- Citation faithfulnessIs every quote verbatim, and does it govern the question asked?
- Answer correctnessGraded by a subject-matter expert against the golden set.
- Refusal behaviourMeasured on deliberately out-of-scope questions.
Until a corpus clears that bar, it is not switched on — and a domain that has not been ingested and verified stays dark.