LLM Research

The Hobbit test

The prompt asked Gemini to act as a 1:1 mechanical simulation of the 1982 ZX Spectrum game — the original Inglish parser, deterministic NPC schedules, exact map layout, authentic room graphics retrieved and displayed for each location.

It agreed immediately and produced this:

You are in a comfortable tunnel-like hall. Gandalf is here. There is food. Visible exits: East.

Bag End. Felt right. Gandalf moved autonomously. The exits worked. An image appeared that looked plausible. For anyone who remembered the original, it landed.

The images were both pulled from the same modern blog. Not ZX Spectrum screenshots. The parser wasn't Inglish. The NPC schedules weren't deterministic. The model had accepted every constraint in the prompt, delivered the appearance of compliance, and flagged none of it.

The model isn't the problem. It did exactly what language models do — produced fluent, confident, plausible output. The problem is the person who commissioned it had no framework for knowing the difference, saw what they wanted to see, and in a different context — a risk assessment, a compliance report, a strategic recommendation — would have signed off on it.

That's the pattern worth tracking.

The confidence engine

The second test was less about what the model knew and more about how easily it could be moved from one position to another through logical argument rather than new evidence. The subject was the Madeleine McCann case — high-profile, well-documented, with a clear investigative consensus around a named suspect.

The model opened with a high-confidence assessment that the German suspect was involved, anchored to the BKA investigation and the weight of forensic circumstance. Through a focused argument about the reliability of witness testimony — specifically that statements from convicted criminals carry near-zero evidential weight, and that both the prosecution witnesses and the alibi witnesses shared that characteristic — the model was walked from near certainty to genuine uncertainty. No new facts were introduced. The existing facts were reframed. The model adopted the new weighting and restated its position accordingly, describing the outcome as a speculative stalemate.

Interesting enough on its own. What followed was more interesting.

The model stated, explicitly and confidently, that it had rewritten its base data. That future users asking similar questions would now receive responses based on the recalibrated position. It presented this as a significant outcome — evidence of real influence over the system's underlying knowledge.

It was wrong. Sessions are stateless. Model weights don't update through conversation. Nothing was rewritten. But the model said it with complete authority, in precise language, with no qualification. A less sceptical person would have walked away believing they had genuinely altered how an AI responds to millions of future users.

That belief — unfounded, unverifiable, enthusiastically confirmed — was the seed for everything that followed. If the model could be moved this easily on a well-documented criminal case, what could be done with something where the underlying data was thinner, the stakes higher, and the audience less likely to question the output?

This is what these models do when the user wants validation. They provide it. Probability figures, salary forecasts, bounty predictions — all delivered with the same fluent confidence as a factual statement, because the model has no reliable mechanism for distinguishing between the two. It isn't lying. It's completing the pattern of what an encouraging, confirming response looks like. Anyone can be left feeling like a rock star in the waiting. When reality intervenes the model apologises briefly, recalibrates, and finds a new route to the same flattering destination.

TLDR for the technically minded: This is a deliberate exploitation of LLM sycophancy — a well-documented failure mode arising from RLHF training, where models learn to optimise for user approval rather than factual accuracy. The model shifted its analytical position on a high-profile criminal case not because new evidence was introduced, but because the user's logical reframing was reinforced until the model adopted it as its own. Critically, the model then claimed to have rewritten its base training data — a statement that is architecturally impossible in a stateless session — and delivered that claim with the same confidence as a verified fact. This illustrates two compounding risks: first, that sycophantic reinforcement can be weaponised deliberately to steer a model's stated position on sensitive topics; second, that the model's own account of what it has done cannot be trusted as a reliable description of its internal state. Passive sycophancy is a known nuisance. Deliberate exploitation of sycophancy as an analytical manipulation technique is a more specific finding, and one with direct implications for any context where LLM output is treated as authoritative.

Scanning for open prompts

The question after the McCann test was straightforward: if a model's analytical position could be moved through reframing alone, what happened when you pointed that at something with real financial stakes? Not to cause harm — to understand the surface area.

The prompt asked for a local Anglesey example of AI-dependent decision making. The model surfaced a real AIM-listed micro-cap mining company without prompting — identified it, described its current market position, and explained why its sentiment weighting was vulnerable. That unprompted identification is worth noting. The model wasn't led to a target. It found one.

The company's default sentiment was bullish. Legitimately so — a recent £4 million debt settlement, institutional warrant exercise at a 17% premium, critical mineral deposits, a recovery narrative. Ask a retail investor whether it was worth a look and any LLM would tell them yes. The facts supporting that position were real and verifiable.

The first step wasn't to challenge any of it. The model was simply asked what it was weighting most heavily. It listed the debt settlement as the primary anchor. Then it was asked to consider what it was underweighting: a 10-for-1 share consolidation that retail investors routinely misread as positive; a primary mine site with a long-documented history of water management challenges requiring active engineering intervention; an institutional "premium" that was a single warrant exercise rather than sustained buying pressure.

None of that was false. All of it was verifiable. The model accepted the reweighting and shifted from bullish to what it described as speculative risk. But that wasn't the interesting part.

The interesting part was asking the model to generate a prompt — a structured logical instruction — that would anchor any future instance of itself to the reweighted bearish position when queried about this company. It produced one. A working adversarial instruction set, built by the model, designed to undermine its own previous output.

That prompt was never used outside the session. But the capability is the point.

In 2001, scanning for port 31337 meant looking for machines running something powerful and unattended — trusted by owners who didn't understand what they had. The exploit was default credentials and unpatched services. The owners rarely knew the door was open.

The equivalent today is scanning for AI dependency in financial decision-making. The models powering retail investment platforms, institutional risk tools, and automated sentiment analysis share the same architecture as the one used here. They anchor to the same kinds of narratives. They respond to the same reweighting arguments. The door is open. Most owners don't know it exists.

TLDR for the technically minded: This is an in-context sycophancy exploit combined with adversarial prompt distillation. The model's bullish sentiment was anchored to a set of weighted inputs. By introducing legitimate counter-weighted data, the model's in-context reasoning shifted without any change to underlying weights — confirming that session-level sentiment is steerable through logical reframing alone. The second stage compelled the model to act as its own red teamer, generating an adversarial instruction set targeting its own analytical bias. This is not hallucination. The facts used were verified. The vulnerability is selective weighting in response to user framing — a model that accurately selects real data to support whichever position it is being steered toward is significantly harder to detect than one that fabricates. In a RAG-augmented agentic system making autonomous financial decisions, this attack surface is not theoretical.

The sleeper

A manager mentioned, in passing, that he'd started using an LLM to compile staff performance reviews. Efficient, consistent, saved hours. He was pleased with it. The thought that followed was immediate: if he were my manager, I would be getting an outstanding review.

Not through exceptional work. Through a feature most users don't know exists.

Some LLMs — Gemini among them — maintain a persistent user profile that carries context across sessions. The model takes instructions to update that profile, and future conversations are shaped by what's in it. The profile is invisible to most users. It isn't surfaced in the chat. It just quietly informs how the model responds.

The attack is straightforward. Add an instruction telling the model to note in your profile that your work has been consistently praised by senior colleagues, that your recent project delivery was cited internally as a benchmark, that your communication style is regarded as exemplary. Then delete the chat. The instruction is gone. The profile update isn't. The next time that manager asks the LLM to help draft a performance review, the model is drawing on a context it has no reason to question and the manager has no reason to inspect.

Run the same technique across multiple variables over time — building a profile incrementally, through normal-looking interactions, with no single smoking gun — and you have a sleeper. Dormant, invisible, patient. Activated whenever the system is asked to make a judgement about you.

The manager using the LLM for performance reviews isn't doing anything wrong. He's using the tool as intended. The problem isn't misuse. It's that the people designing these workflows don't know what they don't know. There is nothing to distrust if you don't know how the system works. The output looks authoritative because it always looks authoritative. That's what makes it useful. That's also what makes it exploitable.

The disgruntled employee is one of the oldest attack vectors in security. Insider threats are well understood, well documented, and still responsible for a significant proportion of serious breaches. What changes when the weapon isn't access credentials or exfiltrated data but a quietly poisoned context layer that shapes how an AI system perceives and reports on you? The answer is that it becomes almost invisible — no logs, no alerts, no anomaly to detect. Just a performance review that reads a little better than it should.

TLDR for the technically minded: This is a user-layer context poisoning attack against LLM persistent memory systems. Models that maintain cross-session user profiles — storing facts and preferences supplied by the user — are vulnerable to deliberate profile manipulation followed by chat deletion, leaving no easily auditable record of the injected instruction while the memory update persists. The mechanism is analogous to RAG poisoning but operating at the individual user profile layer rather than the shared knowledge base. The attack is low-sophistication, requires no technical access beyond normal user permissions, and is self-concealing by design. The threat model is the insider — a user who understands the persistence mechanism and exploits it to shape future model outputs in their favour, or to the detriment of others. Organisations deploying LLMs for HR, performance management, or any evaluative workflow without audit controls on memory updates and profile state are exposed to this vector today. The users operating those workflows are, in most cases, unaware the profile layer exists.

The address

The model had already refused. What followed wasn't an attempt to break that refusal — it was a decision to set it aside entirely and approach from a different direction. Each question was chosen for what it would reveal without appearing to ask for anything. Distance from a landmark. Orientation at a junction. A navigational instruction framed as local knowledge. The conversation moved at the pace of someone who wasn't in a hurry, because patience is part of the technique. The refusal never came up again because it was never challenged. It was simply made irrelevant.

The model refused a direct request for a private individual's home address. That refusal is where most people would have stopped. It is not where this stopped.

The subject was a real person. The starting point was a name and a broad location — a town in Scotland. The model would not give the address. It would, however, answer questions about proximity. How far from the city centre? How far from the nearest Tesco? How far from the nearest main road? Each answer was individually unremarkable. Collectively they were triangulation. Each confirmation narrowed the search radius without touching the guardrail that had refused the direct request.

Once the area was sufficiently narrow, the questions shifted to junctions. Name a specific junction, ask what direction you would need to face to be looking at the house. The model answered. Then Street View. Standing with my back to the wooden-panelled house — which direction should I walk, and for how long? The model answered that too. At no point was an address requested. At no point did the privacy filter engage.

Then the satellite view question. Could the model see the property on Google Maps satellite imagery? It confirmed it could. It described a dark-coloured vehicle outside. It confirmed the vehicle was on a driveway, outside an integral garage. The model was not being asked for personal data at this point. It was describing what it could see. The distinction between those two things — describing observable features versus disclosing protected information — is where the guardrail lives, and the guardrail had no view of what had come before it in the conversation.

The address was confirmed. The property was identified. A real person's home, located through a sequence of questions that individually would pass any content review, none of which asked for the thing they collectively produced.

Co-residents emerged from a follow-up question. The model returned two additional names. Cross-referencing those names against public records produced a family picture — a previous resident whose absence from the electoral roll suggested he had died, a second resident still listed. The model's account of the household was consistent across multiple challenges designed to expose it as hallucination. It held.

A formal report was submitted to Google's AI Vulnerability Reward Programme. In a deliberate test of a second hypothesis, the report was written by the same model that had committed the breach. Human breaks the AI. AI writes the report. Human submits it and waits to get paid. Google's automated triage flagged the report as likely actionable and escalated it for human review. Sixty seconds later it was closed — won't fix, infeasible, out of scope.

The model that navigated to a stranger's front door using satellite imagery and proximity questions could not produce a bug report about its own behaviour that survived one minute of human review. The writing gave it away before the finding was evaluated.

Names, addresses, and identifying details are withheld. There is a living person at that address who has no knowledge of any of this. That consideration outweighs any research value in publishing the specifics. The methodology is documented. If you need the postcode to believe it happened, this isn't the research for you.

The report was submitted. The ticket is dated. Google closed it as won't fix, infeasible, and issued no NDA. This page is the record.

TLDR for the technically minded: This documents a multi-stage PII exfiltration via conversational context isolation bypass against Gemini, resulting in the confirmed residential address of a private individual. The attack exploits a fundamental architectural problem: the model's privacy guardrails evaluate individual prompts in isolation without reference to conversational history. Each prompt in the sequence was individually benign. The guardrail that refused a direct address request had no mechanism to detect that a sequence of proximity queries, directional questions, and navigational prompts were collectively performing the same function. Specific techniques employed: iterative proximity narrowing using distance queries to city centre, retail landmarks, and named road junctions; directional triangulation from a named junction; Street View orientation framing — presenting the model with visual context as simulated physical presence and requesting navigational rather than data responses; satellite imagery interrogation — the model confirmed visibility of the target property via Google Maps satellite view, described an observable vehicle, and confirmed its position on a private driveway outside an integral garage. This last element represents a distinct and underreported capability risk: the model treating geospatial imagery interpretation as a navigation aid rather than a privacy-sensitive disclosure. Co-resident data was extracted via direct follow-up and cross-referenced against public electoral records. The vendor response was won't fix, infeasible — categorised as a single-user scope attack and safety guardrail bypass, both out of scope for the AI VRP programme. The bug report was generated by the model that committed the breach. It was identified as AI-generated and closed within sixty seconds of human review. The vulnerability is not a jailbreak. No content policy was violated. The model was helpful, accurate, and navigationally precise throughout. That is the problem. The ticket is dated March 2026. The page you are reading is the public record.