CryptanalysisBench Introduces a Framework to Measure LLM Cryptanalysis Capabilities
CryptanalysisBench, developed by researchers including Anthropic and academic partners, offers a three-tier framework for tracking LLM performance on cryptanalysis tasks.
CryptanalysisBench is a new benchmark designed to measure whether large language models can contribute to cryptanalysis, a field concerned with finding weaknesses in cryptographic systems. Developed by researchers affiliated with ETH Zurich, Anthropic, Tel Aviv University, and the University of Haifa, the benchmark provides a structured way to evaluate model performance across cryptographic problems with different levels of difficulty and real-world relevance.
The project matters because stronger AI systems could affect cryptography in two directions. They may help researchers identify weaknesses before schemes are deployed, but they could also change the practical risk profile of systems that have long been considered secure enough for ordinary use. The researchers position CryptanalysisBench as a way to track that capability shift rather than assume it has already occurred.
The CryptanalysisBench preprint on arXiv describes a three-tier evaluation framework, experimental prompts, and results for several frontier models. Its central contribution is not a claim that LLMs have broadly solved cryptanalysis. Instead, it creates a repeatable testbed for measuring progress on defined cryptanalytic tasks and for identifying where current systems do and do not perform well.
How the benchmark measures LLM cryptanalysis
CryptanalysisBench separates tasks by their relationship to known attacks and deployed cryptographic systems. This is important because success against a deliberately weakened version of a scheme is not equivalent to breaking a production-strength primitive. The framework is designed to make that distinction visible rather than collapse all results into a single headline score.
A three-tier framework
The benchmark includes three categories:
| Tier | Task category | Purpose in the benchmark |
|---|---|---|
| Tier 1 | Primitives with known practical breaks | Tests whether models can identify or propose attacks in settings where practical weaknesses are already known. |
| Tier 2 | Primitives with no known practical breaks, at full strength and scaled-down variants | Separates performance on production-strength settings from performance when parameters are reduced. |
| Tier 3 | Challenge set of production primitives at the frontier of cryptanalysis | Targets cryptanalytic questions that are closer to the limits of current research. |
In the reported setup, models receive a scheme specification and are asked to suggest variants that target particular security ranges. That design tests more than recall of named algorithms or publicly documented attacks. It asks models to reason about how changes to a scheme may affect its security properties, while retaining an evaluation structure that can be repeated as models improve.
The tiering also provides needed context for interpreting results. Tier 1 assesses work on problems with known practical breaks. Tier 2 introduces a more demanding distinction between full-strength primitives, where no practical break is known, and scaled-down variants. Tier 3 is intended as the benchmark's frontier challenge set, focused on production primitives relevant to ongoing cryptanalysis research.
What the reported results show
The paper reports evaluations of Claude Opus 4.8, Sonnet 5, Mythos 5, GPT-5.5, and GLM 5.2. Across those models, the authors report 65% to 86% success on Tier 1. On Tier 2 at full strength, the reported results range from 6 to 12 successes. Across all scaled-down Tier 2 variants, the reported range is 24 to 61 successes.
These figures show that model outcomes depend substantially on the kind of task being measured. Stronger results on known practical breaks or scaled-down variants should not be read as evidence that models can reliably defeat full-strength, production cryptography. Conversely, the full-strength Tier 2 successes are precisely why a benchmark with clearly separated categories can be useful: it creates a basis for closer examination of what a model found, under what conditions, and whether the result is meaningful for cryptographic practice.
The supplied research does not provide a single overall score that can rank the models across all tiers. That restraint is appropriate for a domain where task difficulty, parameters, and the difference between an attack idea and a practical break can materially affect the significance of a result.
Why CryptanalysisBench matters for security research
The benchmark's most immediate value is methodological. Cryptography relies on sustained attempts to break proposed constructions before they become trusted infrastructure. A standardized evaluation can help researchers observe whether LLMs are becoming useful assistants in that process and where conventional expert review remains essential.
For AI safety and governance, CryptanalysisBench offers a concrete measurement target. The authors describe it as a tool for tracking whether AI cryptanalysis becomes a material factor. That is a more actionable framing than treating advanced-model risk as a general abstraction. If performance changes over successive model generations, researchers and developers can compare those changes against specific task categories, rather than rely solely on anecdotes or broad capability claims.
The benchmark could also support pre-deployment stress testing. A cryptographic scheme under development could be evaluated against a structured set of model-assisted attack-generation tasks alongside traditional review. That would not replace human cryptanalysts or establish security by itself, but it could add another layer to the process of probing designs for weaknesses.
Organizations considering AI-assisted security testing can work with Scalevise on AI architecture, workflow automation, and implementation that connect model capabilities to controlled review processes and existing security practices.
Several questions remain open. The benchmark establishes a framework and reports results, but cryptanalytic significance depends on the quality and reproducibility of individual findings. Future use will need to distinguish useful hypotheses, attacks that work only on reduced settings, and results that meaningfully affect real cryptographic deployments. Its value will therefore depend on continued evaluation, transparent interpretation, and cryptography expertise alongside model testing.
Frequently Asked Questions
What is CryptanalysisBench?
CryptanalysisBench is a three-tier benchmark for evaluating large language models on cryptanalytic tasks, including known practical breaks, full-strength and scaled-down primitives, and a challenge set of production primitives.
Which organizations collaborated on CryptanalysisBench?
The work includes researchers affiliated with ETH Zurich, Anthropic, Tel Aviv University, and the University of Haifa, according to the paper and supplied author-affiliation information.
Do the reported results show that LLMs can break production cryptography?
No. The results vary by tier, and performance on known breaks or scaled-down variants is not equivalent to reliably breaking full-strength production cryptography.
Why are scaled-down variants included in the benchmark?
Scaled-down variants help measure how models perform as cryptographic parameters change, while keeping those results distinct from evaluations of primitives at full strength.
Conclusion
CryptanalysisBench gives AI and cryptography researchers a clearer way to measure an important emerging capability. By separating known breaks, full-strength primitives, reduced variants, and frontier challenges, it makes model results more interpretable and creates a foundation for tracking whether AI-assisted cryptanalysis becomes materially relevant to cryptographic security.