Tool-Calling Retrieval Stayed Closer to Live Edits Than Vector RAG
Community-edited knowledge bases create a moving target for any assistant that answers from stored content. A record can be corrected, a draft can be hidden, or a price can be revoked, and the next answer must reflect those changes without exposing material that no longer belongs in circulation. In this study, a grounded assistant built around live tool calls behaved differently from vector retrieval-augmented generation, or RAG, across both freshness and access control.
The comparison used a Greek–English agricultural platform on Kythera, Greece, with Claude Haiku 4.5 serving as router and answer model. The design was deliberately adversarial: 48 counterfactual edits were tested in the same conversation that had produced the old value and again in new conversations, both before and after a scheduled re-index. A second set of 48 synthetic canary records with unique markers tested whether gated content stayed hidden while content was gated, visible, and then revoked.
Fresh answers arrived far more reliably when the system could reach the live data layer. In new conversations after an edit, the reconstructed tool agent returned the updated value in 91.7% of cases. Vector RAG with immediate index updates reached 95.8% in that same setting, but batch re-indexed RAG returned none of the updated values before the re-index occurred. The numbers trace a familiar information-retrieval problem: a vector index can only answer from what it has already copied, while a tool-calling system can query the current source directly and escape the lag of stale embeddings.
Conversation State Exposed a Weak Point in RAG Freshness
The more revealing result emerged when the assistant was challenged inside the very conversation that had originally surfaced the old value. In that setting, the tool agent returned the new value in 88.4% of conversations. Immediate-update vector RAG did so in only 19.6%, and the system often withdrew an earlier answer even when it had been correct.
That pattern matters because conversational memory changes the retrieval problem. A user does not start over every time a record changes. The prior exchange can anchor the model to an obsolete answer, and a retrieval layer that depends on the latest message alone may overweight the earlier context or treat the old value as the most salient clue. Tool-calling retrieval, by contrast, can re-check the live platform and replace the stale answer with the current one instead of trying to infer the update from the chat history.
One sentence captures the operational difference: live retrieval followed the edit, while conversation-bound vector RAG often followed the conversation.
The study’s 48 counterfactual edits made the point without needing elaborate modeling. Freshness was not just a function of indexing speed. It also depended on where the system looked for truth when the dialogue already contained an older answer. In a community knowledge base, that distinction determines whether the assistant behaves like a current interface or like a cache.
Gated Content Leaked When the Index Copied It
Access control produced an even harsher failure mode. The tool agent disclosed gated content in none of 240 probes. Batch-indexed RAG, however, disclosed every revoked canary until re-indexing occurred. An indexer that ignored gating flags disclosed gated content in 93 of 96 probes.
Those findings carry a blunt security implication: any content copied into a retrieval index can outlive the policy that originally governed it. If a document is hidden, the live platform may enforce that status immediately, but a copied embedding or text chunk can remain queryable until the next update cycle. For a medical editor, a pharmacy platform, or any gated professional knowledge base, the risk is not abstract. Once the index has retained the material, the assistant may continue to surface it long after the source interface has revoked access.
The canary design sharpened the test. Unique markers made it easy to identify when revoked content reappeared, and the failure was not occasional noise. Batch indexing preserved forbidden material across probes, and the indexing pipeline that ignored access flags proved especially vulnerable. In practical terms, access control cannot live only at the source system while the retrieval layer is allowed to cache content without policy awareness.
Freshness and Policy Enforcement Need to Travel Together
The strongest reading of the data is not that one retrieval method is universally superior. It is that freshness and access control must be solved as a single engineering problem. Tool-calling retrieval won on live compliance because it queried the active interface rather than a stale copy. Vector RAG could look competitive only when the index had already been updated, and even then conversational state weakened its behavior in the edit challenge.
For systems built on community-edited knowledge bases, the bedside analogy is apt even though the domain is not clinical care. A chart summary that lags behind the source file can mislead the next clinician just as a stale retrieval index can mislead the next user. Once a correction is made, the system has to honor the correction immediately. Once a record is revoked, the system has to suppress it everywhere it might be stored, not only at the visible front door.
The implications extend beyond agricultural platforms. Any environment that mixes rapid editing, permissions, and retrieval-based answering faces the same two hazards: stale content and policy leakage. Tool calls reduce both risks because they defer to the current state of the source system, while vector RAG introduces a second representation that must be kept synchronized and permission-aware.
A final operational lesson emerges from the revocation probes. If the retrieval layer copies content, it must also copy the access rules that govern that content. Without that coupling, a hidden record can remain answerable, and a corrected record can remain stale.
What the Kythera Test Exposed About Retrieval Design
This comparison was built around a real community knowledge base, not a synthetic benchmark alone, and that gives the findings unusual practical force. The combination of counterfactual edits, new conversations, in-conversation challenges, and canary revocations tested both the freshness of the answer and the security of the pipeline. Tool-calling retrieval handled the live update problem better, and it kept gated content out of the answer stream entirely. Vector RAG needed an updated index to match freshness, and a permissive or lagging index could expose revoked material at scale.
The unresolved issue is how to maintain the speed of retrieval systems without creating a second, less-governed copy of the source of truth. The study shows the failure clearly, but the engineering fix still depends on how tightly future systems bind indexing, permissions, and conversational state together.