Genie in the Bottle AI
Constrain AI inputs, reasoning, and outputs to produce repeatable signal.
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 98%
This framework treats an AI model as a powerful but unreliable linguistic calculator that must operate inside a deliberately engineered environment. The analyst controls the source material, specifies the semantic categories the model may consider, defines how the material should be processed, and constrains the response format. Identical inputs are then rerun to test whether the result is stable. A separate model or deterministic evaluation layer checks whether the first model departed from its instructions or evidence. The mechanism converts an open-ended generative system into a bounded analytical component: controlled data and reasoning produce structured output, while independent validation catches unsupported conclusions. The result is more reproducible signal and substantially less hallucination.
Origin
Ben Hunt described how Perscient rebuilt its narrative-analysis software around context engineering after handcrafted small language models missed too many relevant stories.
Core principles
- 01Treat the model as a linguistic calculator, not an oracle.
- 02Supply the data instead of letting the model find its own.
- 03Define the permitted reasoning structure before running analysis.
- 04Demand consistent answers from identical inputs.
- 05Validate outputs with an independent model or evaluation step.
How to run it
- 1
Define the analytical target
Specify the narratives, signatures, classifications, or relationships the model is allowed to identify. Replace subjective questions with observable criteria.
Pro tip Use categories that a human analyst can define and audit.
Watch out Questions such as what is most interesting invite the model to infer personal preferences.
- 2
Control the corpus
Provide the exact documents to analyze and prevent the model from searching for additional material. Preserve source provenance so every result can be traced.
Pro tip Normalize document metadata before analysis.
Watch out Uncontrolled retrieval can introduce irrelevant or fabricated evidence.
- 3
Supply the reasoning scaffold
Tell the model how to classify and compare the supplied text. Encode semantic signatures or other permitted concepts directly into the context.
Pro tip Separate descriptive language from prescriptive language when influence is the target.
Watch out A prompt alone may be insufficient if the broader context remains unconstrained.
- 4
Constrain the output
Require a fixed schema, bounded labels, and evidence-linked responses. Prevent unsupported categories or free-form conclusions.
Pro tip Make abstention an explicitly valid result.
Watch out A model optimized to please the user may confidently fill gaps unless omission is allowed.
- 5
Test reproducibility
Run the same question against the same inputs repeatedly and compare the results. Investigate material variation before relying on the signal.
Pro tip Track consistency as an operational quality metric.
Watch out A plausible one-off answer is not evidence of a reliable process.
- 6
Validate independently
Pass the result through another model, deterministic checks, or human review to detect unsupported reasoning. Retain only outputs that remain within the scaffold.
Pro tip Use a validator with different instructions from the primary model.
Watch out Do not let the validating system silently rewrite the original evidence.
In the wild
An investment team supplies a fixed corpus of licensed financial news, defines a semantic signature for bullish management-change stories, and asks the model only to classify documents against that signature. The system returns evidence-linked matches, reruns the batch for consistency, and uses a separate validator to reject unsupported classifications.
→ The team obtains a reproducible time series showing whether the predefined narrative is gaining or losing attention.
A policy group provides a controlled collection of local articles and defines the difference between descriptive reporting and language prescribing what officials should do. The model labels each passage within a strict schema, and a second system checks every label against the source text.
→ The group measures shifts in advocacy without relying on an open-ended summary of public opinion.
Common mistakes
Asking open-ended questions
An unconstrained model will still answer, but it may select its own evidence, infer subjective criteria, or fabricate a satisfying conclusion.
Controlling only the prompt
Reliable analysis requires control of the data, context, permitted reasoning, output schema, and evaluation process—not merely polished prompt wording.
Skipping repeatability tests
If identical inputs produce materially different outputs, the result should not be treated as stable analytical signal.
Is it for you?
Best for
It is best for teams extracting repeatable signals from controlled corpora at production scale.
Not ideal for
It is not ideal for unconstrained brainstorming where novelty matters more than reproducibility.
From the transcript
“The secret to using AI successfully is to take this magic genie. It's a magic genie, and you stuff it into that bottle.”
“You have to control all the inputs. You have to control the output”
“You have to give it the scaffolding.”
From the episode
Using AI to Find Investing Stories with Perscient Co-Founder Ben Hunt
Ben Hunt