Original Title: How AI judges can scale prediction markets: The case for locking LLMs into the blockchain to resolve
Original Author: Andrew Hall, a16zcrypto
Translation: Peggy, BlockBeats

Editor's Note: The value of prediction markets lies not in "betting," but in whether they can consistently output trusted price signals. However, as transaction volume grows, the most easily overlooked yet most fatal bottleneck is emerging: how contracts settle. From the indistinguishability of truth in the Venezuelan presidential election to manipulation and conflicts of interest arising from map updates, website updates, and token voting, disputes often stem not from the prediction itself, but from "who adjudicates."

In this article, author Andrew Hall (@ahall_research) proposes a more radical approach: to write a specific version of a large language model (LLM) with prompts onto the chain, allowing AI to become an auditable, predictable adjudication mechanism, replacing human governance and gaming with engineering. It may not be perfect, but perhaps it is a threshold that must be crossed for the scalability of prediction markets.

The following is the original text:

Last year, the results of the Venezuelan presidential election brought over $6 million in contract trading volume to the prediction market. However, after the vote count was completed, the market found itself in a nearly intractable position: the government announced Nicolas Maduro's victory, while the opposition and international observers accused electoral fraud. So, should the contract settlement in the prediction market be based on "official information" (Maduro's victory) or on the "trusted consensus of reports" (opposition's victory)?

In this case of the Venezuelan election, observers' accusations were numerous: from believing that rules were disregarded, participants' "money was stolen," to describing the protocol used to handle dispute contracts as a "judge, jury, and executioner" in a high-stakes political drama; some even directly called it "heavily manipulated."

This is not an isolated, one-off incident but a symptom that I believe will recur frequently in the process of scaling prediction markets and may even be one of the most significant bottlenecks: contract resolution.

The stakes here are very high. If settlement is done correctly, people will trust your market, be willing to trade in it, and prices will become a socially meaningful signal. If settlement is done wrong, the trading experience becomes frustrating and unpredictable. Participants may slowly exit, liquidity faces risks of drying up, and prices no longer reflect an accurate prediction of a "stable target." Instead, prices begin to reflect a mixture of things: both the probability of the event actually occurring and traders' judgment of "how a skewed settlement mechanism will ultimately adjudicate."

While Venezuela's controversies have received high visibility, the more covert failures often occur across platforms:

The Ukraine Map Manipulation incident illustrates how attackers can directly "game" settlement mechanisms. A contract regarding territorial control specified that settlement would be based on a particular online map. Someone was accused of editing this map to influence the contract's outcome. When the "ground truth" itself can be manipulated, so too can your market.

The Government Shutdown Contract event demonstrates that settlement information sources can yield inaccurate, or at least unpredictable, results. This contract stipulated that the market would settle based on the U.S. Office of Personnel Management (OPM) website's indication of "shutdown end" time. President Trump signed the appropriations bill on November 12, but for some reason, the OPM website was not updated until November 13. Traders who correctly predicted the shutdown would end on the 12th ultimately lost due to the website administrator's delay.

The Zelenskyy Suit Market raised concerns about conflicts of interest. The contract asked whether Ukrainian President Zelenskyy would wear a suit at an event—a seemingly trivial question that attracted over $200 million in bets. When Zelenskyy appeared at a NATO summit wearing attire described as a "suit" by media outlets like the BBC and New York Post, the market initially settled as "yes." However, UMA token holders disputed the result, leading to a reversal of the settlement to "no."

In this article, I will explore whether combining GPT (Large Language Model) with cryptographic technology intelligently can create a scalable settlement method for prediction markets: one that is highly resistant to manipulation, achieves accuracy, full transparency, and trust neutrality.

This Is Not Just a Prediction Market Issue

Similar issues have plagued the financial markets. The International Swaps and Derivatives Association (ISDA) has long grappled with the "Settlement/Determination" challenge in the Credit Default Swap (CDS) market—contracts that pay out in case of a corporate or sovereign debt default. ISDA's review report in 2024 candidly describes these difficulties. Their "Determination Committees," composed of major market participants, vote to determine if a "credit event" has occurred. However, this process has been criticized as opaque, potentially subject to conflicts of interest, and yielding inconsistent outcomes, much like UMA's process.

The fundamental issue is the same: when vast sums hinge on the determination of what precisely happened in a nebulous event, any settlement mechanism becomes a target for gaming and attack; and any ambiguity can turn into a flashpoint for conflict.

So, what should a good settlement mechanism look like?

Key Features of a Good Solution

Any viable solution must simultaneously satisfy several key properties:

Resilience to Manipulation
If attackers can influence the settlement by editing Wikipedia, spreading fake news, bribing oracles, or exploiting program bugs, the market will become a game of "who can manipulate the best" rather than "who can predict the best".

Reasonable Accuracy
The mechanism must provide correct settlements in the majority of cases, most of the time. Since the world inherently contains fuzziness, absolute accuracy is impossible, but systemic bias or blatant errors will completely destroy credibility.

Pre-Event Transparency
Traders must be clear on how the settlement will occur before placing their bets. Changing the rules mid-game is a breach of the most basic agreement between the platform and participants.

Trusted Neutrality
Participants must trust that the mechanism won't favor any trader or outcome. This is why having large UMA holders adjudicate their own bet contracts is such a concern: even if they act fairly, the appearance of conflicts of interest is enough to erode trust.

Human committees can excel in some aspects but struggle in others, especially in terms of manipulation resistance and trusted neutrality at scale. Token voting systems like UMA also face a range of well-discussed issues such as "whale domination" and conflicts of interest.

This is where AI can come in.

Why Use LLM as Judge

In the prediction market space, an increasingly discussed proposal is to have large language models (LLMs) act as settlement "judges" and lock specific models and prompts into the blockchain when the contract is created.

The basic architecture is roughly as follows: at contract creation, the market maker not only specifies the settlement criteria in natural language but also explicitly states the LLM (with a timestamped model version) that will be used to adjudicate the result and the full prompt that will be given to it.

This provision is submitted and cryptographically bound to the chain. Once trading begins, participants can audit the entire settlement mechanism—they know exactly which AI model is judging, what prompt it will receive, and which sources of information it can access.

If they do not accept this setup, they do not participate in the trade. By settlement, the LLM committed on-chain will run the committed prompt, access the specified information source, and provide a ruling output. This output directly determines who receives compensation.

This approach can simultaneously satisfy several key constraints:

Strong Manipulation Resistance (though not absolute)
Unlike a Wikipedia page or a small news website, it is not as easy to "rewrite" a mainstream LLM's output. Model weights are fixed at commitment time. To manipulate the settlement, an attacker would either need to compromise the information source relied upon by the model or poison the model's training data far in advance—compared to bribing an oracle or editing a map, such attacks are more costly and the results more uncertain.

Higher Accuracy
With the rapid advancement of inference-type models and the ability to access information and gather evidence online, LLM judges should be able to accurately rule on many markets. Experiments on its accuracy are also ongoing.

Built-in Transparency
The entire settlement mechanism is visible and auditable before anyone places a bet. There are no rule changes along the way, no "free discretion" ad hoc judgments, no behind-the-scenes negotiations. Participants are clear on what they are signing up for.

Significantly Enhanced Trust and Neutrality
The LLM has no economic interest in the settlement result, cannot be bribed, and does not hold UMA tokens. Even if it has biases, they are due to the model's inherent properties, not ad hoc decisions by vested interests.

Of course, LLM judges also have limitations, which I will further explain below.

Limitations of LLM Judges

Models Make Mistakes
An LLM may misinterpret news, fabricate facts, or use inconsistent ruling logic in similar cases. However, as long as traders know in advance which model their bet corresponds to, they can factor these "flaws" into pricing. If a model has a clear bias in handling ambiguous situations, savvy traders will adjust their strategies. The model does not need to be perfect; it needs to be predictable.

Manipulation is not impossible, just harder
If the prompt specifies certain news sources, attackers may attempt to "plant articles" in those sources. This is costly in large media but perhaps feasible in smaller media—it reproduces the problem of "map editing" in another form. Therefore, prompt design is crucial: a mechanism relying on multiple sources of redundant information is more robust than relying on a single-point information source.

Poisoning Attacks Theoretically Possible
An adversary with sufficient resources may attempt to bias future rulings by influencing training data. However, this would require preemptive action before the contract is live, with uncertain returns and high costs, making it a much higher barrier compared to bribing a committee member.

LLM Plurality of Judges Creates Coordination Challenges
If different market creators commit to different models and prompts, liquidity will be fragmented. Traders will struggle to horizontally compare contracts and aggregate information across markets. Standardization is valuable, but allowing markets to experiment and find the best LLM+prompt combination is also valuable. A reasonable path may be a combination of both: allowing experimentation while establishing mechanisms for the community to gradually converge on a validated default solution.

How Should Builders Proceed?

In summary, AI settlement essentially swaps one set of issues (human biases, conflicts of interest, opacity) for another set of issues (model limitations, prompt engineering challenges, information source vulnerability). However, the latter may be more controllable and amenable to engineering. So, what are the next steps? Platforms should:

Start with Low-Risk Contracts
Test LLM settlement in lower-risk markets, establish a track record. Which models perform best? Which prompt structures are more robust? What failure modes can be expected in practice?

Promote Standardization
As best practices gradually emerge, the community should work towards forming a standardized LLM+prompt combination that can serve as a default option. This will not stifle innovation but will help concentrate liquidity in markets where "everyone understands the rules."

Develop Transparent Tools
For example, develop interfaces that allow traders to easily inspect the settlement mechanism in full before placing bets—including the model, prompt, and information source. Settlement rules should not be buried in lengthy fine print.

Ensure Ongoing Governance
Even with the introduction of AI judges, humans will still need to make "meta-decisions": which models are trustworthy? How to handle clear errors from models? When should default solutions be updated? The goal is not to completely remove humans from the system but to extract them from ad hoc, case-by-case rulings and move towards systematic rule-making.

Prediction markets have immense potential to help us make sense of a noisy and complex world. However, this potential relies on trust, and trust relies on fair contract settlement. We have seen what happens when settlement mechanisms fail: chaos, outrage, traders leaving the platform. I have seen people feel deceived and leave the prediction markets for good because a result "contradicted the spirit of their bet," even if they once loved these platforms. This means we have missed an opportunity to unlock the value of prediction markets and expand their broader use.

Litigation is not perfect. But when combined with cryptographic technology, it can be transparent, neutral, and resistant to the manipulation tactics that have long plagued human-based systems. In an era where the prediction market expands faster than governance mechanisms iterate, this may be exactly what we need.

[Original Article Link]

Welcome to join the official BlockBeats community:

Telegram Subscription Group: https://t.me/theblockbeats

Telegram Discussion Group: https://t.me/BlockBeats_App

Official Twitter Account: https://twitter.com/BlockBeatsAsia