Perceive Beating AI News Flash: AI memory company Engram teamed up with legal AI company Harvey for an experiment. They first created a virtual law firm with 1 billion Tokens, nearly 10,000 documents, and 266 legal issues. Then they had Qwen3.8-27B pre-train by "learning" this knowledge. Part of the knowledge was directly embedded into the model's parameters, while the rest was organized into notes for on-demand retrieval.
The results showed that this 27B miniature model achieved an average score of 67% in 250 legal tasks, outperforming all benchmarked models. The cost per single query was approximately $0.13, only 1/10 to 1/24 of Claude Opus 4.8's high-intensity reasoning. Furthermore, in a stricter overall accuracy rate evaluation, it led Opus 4.8 by 30% to 25%.
Prior to training, without access to search tools, Qwen was asked specific questions about the law firm, and its accuracy was only 4.7%. After training, this accuracy increased to 72.6%. The post-training model also became more adept at information retrieval, with 99.2% of tasks initiating with targeted searches.
However, the experiment still used a virtual law firm. As real company data changes daily, the continuous updating of the model's memory remains the biggest challenge.
