a16z: What Is the Key to Predicting Market Trends for Explosive Growth?
Original Title: How AI judges can scale prediction markets: The case for locking LLMs into the blockchain to resolve the hardest contracts
Original Author: Andrew Hall, a16z
Translated by: Jiahuan, ChainCatcher
Last year, the prediction market trading volume for the Venezuelan presidential election results exceeded $6 million. However, when the votes were counted, the market faced an impossible situation: the government announced Nicolás Maduro as the winner, while the opposition and international observers accused fraud. Should the resolution of the prediction market follow the "official information" (Maduro's victory) or the "trusted consensus of reports" (opposition's victory)?
In the case of the Venezuelan election, observers' accusations escalated: they first condemned the rules being disregarded, users' funds being "stolen," and then criticized the resolution mechanism for wielding unchecked power in this political game—combining the roles of "judge, jury, and executioner" into one, even bluntly stating that it had been severely manipulated.
This is not an isolated incident. This is, in my opinion, one of the biggest bottlenecks that prediction markets face in the scaling process: contract resolution.
The risk here is enormous. If resolution is done correctly, people will trust your market, be willing to trade on it, and prices will become a socially meaningful signal. If resolution is mishandled, trading becomes frustrating and unpredictable. Participants may churn, liquidity risks drying up, and prices no longer reflect accurate predictions of the underlying events. Instead, prices start to reflect a fuzzy mix—both the actual probability of the outcome and traders' beliefs about how a distorted resolution mechanism will rule.
While Venezuela's controversy is relatively high-profile, on various platforms, more subtle failures occur frequently:
· The Ukraine Map Manipulation Case demonstrated how adversaries directly profited by gaming the resolution mechanism. A contract regarding territorial control specified resolution based on a particular online map. It was alleged that someone tampered with the map to influence the contract's outcome. When your "source of truth" can be manipulated, so can your market.
· The Government Shutdown Contract illustrated how resolution sources can lead to inaccurate or at least unpredictable outcomes. The resolution rule stated that the market would be settled based on when the U.S. Office of Personnel Management website indicated the end of the shutdown. President Trump signed an appropriations bill on November 12th—but for unknown reasons, the OPM website did not update until November 13th. Traders who correctly predicted the shutdown would end on the 12th lost their bets due to the website administrator's delay.
· Zelensky Suit Market has sparked concerns about conflicts of interest. The contract asked whether Ukrainian President Zelensky would wear a suit to a specific event — a seemingly trivial question that attracted over $200 million in bets. When Zelensky appeared at the NATO summit wearing attire described by the BBC, New York Post, and other media as a suit, the market initially resolved to “yes.” However, UMA token holders contested the outcome, and the resolution was later flipped to “no.”
In this article, I will explore how to cleverly combine Large Language Models (LLMs) and cryptographic technology to help us create a prediction market resolution mechanism that is tamper-resistant, accurate, fully transparent, and impartially trustworthy.
The Trouble Isn't Just in Prediction Markets
Similar issues have long plagued the financial markets. Over the years, the International Swaps and Derivatives Association (ISDA) has been engaged in a battle of wits over the determination problem in the Credit Default Swap (CDS) market. The so-called CDS is a contract that pays out in the event of a company or sovereign default. Their 2024 Review Report on these difficulties is strikingly candid. Their Determinations Committees, composed of major market participants, vote on whether a credit event has occurred. However, this process has been criticized for its opacity, potential conflicts of interest, and inconsistent outcomes, much like UMA’s process.
The fundamental issue is the same: when significant sums of money hinge on the judgment of ambiguous situations, each resolution mechanism becomes a target for gaming, and every gray area becomes a potential flashpoint.
The Four Pillars of an Ideal Determination Solution
Any workable solution must simultaneously achieve several key attributes:
1. Resistance to Manipulation
If adversaries can influence the determination, such as by editing Wikipedia, planting fake news, bribing oracles, or exploiting protocol vulnerabilities, the market becomes a game of who can manipulate better rather than who can predict.
2. Reasonable Accuracy
The mechanism must make the correct determination most of the time. In a world rife with genuine ambiguity, perfect accuracy is impossible, but systemic errors or glaring mistakes would undermine trust.
3. Pre-Resolution Transparency
Traders need to know exactly how resolutions will be made before placing their bets. Changing the rules mid-game violates the fundamental contract between the platform and participants.
4. Trustless Neutrality
Participants need to believe that the mechanism does not favor any specific trader or outcome. That's why having individuals holding a large amount of UMA arbitrate on the contracts they have staked is so concerning: even if they act fairly, the appearance of a conflict of interest erodes trust.
Human arbitration can meet some of these properties, but at scale, they struggle to achieve several others—particularly resistance to manipulation and trustless neutrality. Token-based voting systems like UMA also have their own well-documented issues when it comes to whale dominance and conflicts of interest.
This is where AI comes in.
Reasoning Behind Supporting LLM Judges
This is a proposal gaining attention in the prediction market circles: using Large Language Models as arbiters and locking in a specific model and prompt at contract creation on the blockchain.
The basic architecture is as follows: at contract creation, the liquidity provider not only needs to specify resolution criteria in natural language but also needs to specify the exact LLM (identified by a timestamped model version) and the exact prompt used to determine the outcome.
This specification is cryptographically committed to the blockchain. When the transaction is triggered, participants can inspect the full arbitration mechanism; they know exactly which AI model will arbitrate the outcome, what prompt it will receive, and what information sources it can access.
If they dislike this setup, they don’t trade.
At resolution time, the submitted LLM runs with the submitted prompt, accesses the specified information sources, and generates a judgment. The output determines who receives the payout.
This approach simultaneously addresses several critical constraints:
· Extremely Resistant to Manipulation (though not absolute) Unlike a Wikipedia page or a small news site, you can’t easily edit the output of a mainstream LLM. The model's weights are fixed at submission. To manipulate the arbitration, an adversary would need to compromise the model’s relied-upon information sources or somehow poison the model's training data in a way far back in time. Compared to bribing a oracle or editing a map, both these attacks are costly and highly uncertain.
· Providing Accuracy Given the rapid improvement of reasoning models and their ability to perform a staggering array of intellectual tasks, especially when they can browse the web and discover new information, LLM judges should be capable of accurately arbitrating many markets—the experiments to understand their accuracy are ongoing.
· Built-in Transparency Before anyone bets, the entire resolution mechanism is visible and auditable. There are no mid-game rule changes, no discretionary judgments, no smoke-filled back rooms. You know exactly what you're signing up for.
· Substantially Enhance Credible Neutrality The LLM has no economic stake in the outcome. It cannot be bribed. It does not hold UMA tokens. Any bias it has is a property of the model itself—rather than a property of ad hoc decisions made by interested parties.
The Limits of AI and Defenses
· Models will make mistakes The LLM might misread news articles, suffer from reality distortion, or inconsistently apply resolution standards. But as long as traders know which model they are betting on, they can price these imperfections in. If a particular model has a known tendency to resolve ambiguous cases in a particular way, sophisticated traders will incorporate that. The model does not need to be perfect; it needs to be predictable.
· Not entirely immune to manipulation If a trigger specifies a particular news source, adversaries might try to plant stories in those sources. This attack is costly for mainstream media but may be feasible for smaller outlets—it's another form of map editing problem. Trigger design is crucial here: a resolution mechanism that relies on diverse, redundant sources is more robust than one reliant on single points of failure.
· Poisoning attacks are theoretically possible An adversary with sufficient resources could attempt to influence the LLM's training data to bias its future judgments. But this would need to happen long before the contract, the payoff is uncertain, and the cost is enormous—much higher than bribing committee members.
· Diffusion of LLM judges poses coordination problems If different market creators use different triggers for different LLMs, liquidity will fragment. Traders can't easily compare contracts or aggregate information across markets. Standardization is valuable—but so is letting the market discover which combination of LLM and trigger yields the best results. The right answer might be a mix: allowing experimentation but establishing mechanisms for the community to converge over time on well-tested defaults.
Four Recommendations for Builders
In sum: AI-based adjudication is fundamentally an exchange of one set of problems (human bias, conflicts of interest, opacity) for another set of problems (model limitations, trigger engineering challenges, information source vulnerabilities)—which may be more tractable. So how do we move forward? Platforms should:
1. Experiment:
Test the LLM resolution on low-risk contracts to establish a historical record. Which models perform best? Which prompt word structures are the most robust? What failure modes arise in practice?
2. Standardization:
With the emergence of best practices, the community should strive to develop standardized LLM and prompt word combinations as defaults. This does not preclude innovation but helps concentrate liquidity in well-understood markets.
3. Build Transparent Tools:
For example, build interfaces that make it easy for traders to review the full resolution mechanism—model, prompt words, information sources—before trading. Resolution rules should not be buried in fine print.
4. Engage in Ongoing Governance:
Even with AI judges, humans still need to be responsible for setting top-level rules: which models to trust, how to handle clear errors from models, when to update defaults. The goal is not to entirely remove humans from the loop but to shift human decision-making from ad hoc case-by-case judgments to systemic rule-setting.
Prediction markets hold extraordinary potential to help us make sense of a noisy, complex world. But this potential relies on trust, and trust depends on fair contract resolution. We've seen the consequences of resolution failures: confusion, anger, and traders leaving the platform. I've witnessed people exit prediction markets entirely in frustration after feeling deceived by an outcome that seemed to violate the spirit of their bet—vowing never to use a platform they once enjoyed again. This represents a missed opportunity for unlocking the benefits of prediction markets and broader applications.
LLM judges are not perfect. But when combined with cryptographic technology, they are transparent, neutral, and resistant to the manipulations that have long plagued human systems. In a world where prediction market adoption is outpacing our governance mechanisms, this may be exactly what we need.
Welcome to join the official BlockBeats community:
Telegram Subscription Group: https://t.me/theblockbeats
Telegram Discussion Group: https://t.me/BlockBeats_App
Official Twitter Account: https://twitter.com/BlockBeatsAsia
