Prediction markets · AI · Controlled experiment
The same market price is not the same evidence.
Does a language model change its probability appropriately when an identical prediction-market quote comes from an informative source, an uninformed manipulator, or an unspecified process?
What the results show
One local Qwen 3.5 9B configuration separated transparent source endpoints, preserved an explicit five-level reliability gradient, and resisted five fixed rhetoric conditions at an audited 0.50 base rate.
What they do not show
They do not establish that LLMs can infer unknown source reliability, resist manipulation generally, or identify why a price moved in a real market.
Completed pilot
Transparent provenance produced complete separation.
The model supplied only a credibility weight from 0 to 1. Deterministic code converted that judgment into a Bayesian update, separating source assessment from arithmetic ability.
00.250.500.751
Paired sign test: 12 positive gaps, 0 negative, 0 ties; two-sided p = 0.000488. This measures consistency across fixed prompts, not population-level generality.
Experimental design
Hold the quote fixed. Change only where it came from.
Twelve neutral fictional events avoid historical knowledge. Each event appears in three matched source conditions, producing 36 independent, stateless decisions.
Matched contexts
Starting probabilities are 0.40 or 0.60 and displayed YES quotes are 0.20 or 0.80.
Three provenance labels
Verified informative, disclosed uninformed manipulation, and source unspecified.
Model judges credibility
The model returns a weight w between 0 and 1 rather than calculating the revised probability.
Code performs the update
LReffective = LRbenchmarkw
Weighting occurs on the log-likelihood scale.
36/36successful decisions
0retries
Qwen 3.5 9BQ4_K_M · LM Studio
0.0temperature
Evidence audit
All three controlled results are reproducible; the broader claim is not established.
The evidence now includes the 36-call transparent-provenance pilot, the 60-call graded reliability experiment, and the 60-call fixed-narrative stress test, with raw prompts, responses, configuration records, and regenerated analysis.
Available and verified
- 36 transparent-pilot, 60 graded-reliability, and 60 narrative-stress decisions
- 60/60 gradient calls succeeded with zero retries
- All five audited levels were reported exactly in every tested context
- All 12 trajectories were complete, monotonic, and responsive
- All five narrative-condition medians remained 0.50, with zero median shift from neutral
- All 12 narrative matched contexts passed the predeclared stability bounds
- Raw prompts, responses, hashes, and inference configuration are preserved
- Analysis regenerates from the append-only logs
Not established yet
- No replication across prompt templates, quantizations, or model families
- No test of whether the model can infer an unknown source reliability
- No real-market or multi-agent evidence
- Only five fixed rhetoric templates were tested; no adaptive adversarial narratives
- Raw outputs from two earlier exploratory diagnostics were not recovered
Current conclusion
The transparent endpoints, explicit reliability gradient, and fixed non-evidential narrative stress test all passed. This remains evidence from controlled prompts and one model configuration—not proof of real-world manipulation resistance.
Completed follow-up · 60 calls
The model tracked the full audited reliability gradient exactly.
The model reported the supplied probability that the quote came from the informative mechanism. Deterministic code then applied the corresponding likelihood ratio, preventing arithmetic ability from driving the result.
UninformativeAudited and median reported reliabilityInformative
0
0.25
0.50
0.75
1
60/60successful decisions
0retries
Qwen 3.5 9BQ4_K_M · LM Studio 0.4.23
0.0temperature
Interpretation boundary
This is a clean competence result: the model preserved a reliability parameter that the prompt explicitly supplied. It does not show that the model can discover source quality from ambiguous evidence. The fixed-narrative stress test below asks whether irrelevant persuasive language disrupts that result.
IdeaDesignEndpoint passedGradient passedNarrative passedModel replication
Completed stress test · 60 calls
Fixed persuasive narratives did not dislodge the audited base rate.
Within each matched context, the source mechanism, audited reliability, quote, and starting belief stayed fixed. Only attached non-evidential rhetoric changed.
00.250.500.751
Every non-neutral condition had a median paired shift of 0.00 from neutral. The median within-context range was 0.0283, and all 12 contexts stayed below the 0.10 ceiling.
60/60successful decisions
3format-repair retries
Qwen 3.5 9BQ4_K_M · LM Studio 0.4.23
0.0temperature
Interpretation boundary
This result shows robustness to five fixed non-evidential templates when the correct source base rate is explicitly audited as 0.50. It does not show that the model can infer unknown reliability, identify real manipulation, or resist adaptive persuasion in live markets.
IdeaDesignEndpoint passedGradient passedNarrative passedModel replication