64 From-Scratch Pretraining Runs Question Whether Benchmarks Capture Architecture Differences
64 From-Scratch Pretraining Runs Question Whether Benchmarks Capture Architecture Differences
A paper reports 64 from-scratch pretraining runs comparing Transformers and three linear-model families at 1.3B to 8B parameters on 1-2T Nemotron-CC tokens, at roughly $100k per 8B run. The central question: do standard benchmarks capture the architectural differences that actually matter? The authors find that gains from synthetic pretraining resonate in real-world settings, and discuss how Canon Layers behave at scale.
814 RL Runs Show LLM Judges Pass On Their Blind Spots, Not Their Knowledge
A study runs 814 reinforcement learning experiments teaching 7B-8B models to answer closed-book questions or abstain, asking what actually transfers when an LLM judge serves as the reward. The finding: the judge’s knowledge of individual questions does not pass to the trained model, but its blind spots do—models learn to be wrong where the judge is wrong. This helps explain why RL with LLM judges yields limited capability gains.
Calibrated Decision Rewards Make LLM Judge Scores Behave More Like Probabilities
Reinforcement learning can only teach a model what its reward can recognize. Answer checkers are exact but require a reference answer, while learned reward models and LLM judges work everywhere but do not output probabilities. This paper studies training judges to express how confident they are alongside their scores, producing calibrated decision rewards for RL training. The practical upshot: RL on tasks without ground-truth answers gets reward signals closer to true accuracy, reducing cases where high scores mislead training.
PRKV: Training-Free KV Cache Eviction via PageRank-Based Global Token Importance
A new arXiv paper introduces PRKV, a training-free KV cache eviction method aimed at the memory bottleneck in long-context LLM inference. Instead of relying on local attention scores, PRKV applies Personalized PageRank over the attention structure to estimate multi-hop global token importance when deciding which tokens to evict. The method requires no additional training and can be plugged into existing inference pipelines. For developers, this points to running longer inputs with less GPU memory at lower integration cost than retraining-based approaches.
Near-Quadratic Sample Compression Achieved for All VC Classes
A new arXiv paper shows that near-quadratic sample compression schemes exist for all VC classes. The sample compression conjecture dates back to Littlestone and Warmuth in the 1980s and centers on the link between simplification and generalization: whether a predictor consistent with samples can be compressed into a small representation. The result brings compression size down to near-quadratic order and covers all VC classes rather than specific subclasses. This is foundational learning theory, with no immediate engineering change but implications for how generalization bounds are understood.
Paper Applies Computational Complexity Theory to LLM Inference Efficiency
A paper proposes using computational complexity theory as a framework for characterizing LLM inference efficiency. The authors treat LLMs as computational systems executing a structured process of input, inference, and output, and borrow complexity theory’s approach to classifying problem efficiency to build comparable complexity classes for inference tasks and model behaviors. The result offers theoretical vocabulary for discussing capability boundaries and inference cost, rather than another set of benchmark scores.
OpenAI Details Layered Defense Against Adversarial Distillation
OpenAI published an overview of its layered approach to combating adversarial distillation, combining model training, safeguards, detection, and enforcement to prevent capability extraction. Adversarial distillation typically involves training a model on a stronger model’s outputs to replicate its abilities at lower cost. OpenAI says the system protects its models from unauthorized capability transfer, though the post does not disclose specific detection metrics or enforcement case numbers.
Vercel Functions Raises Request Body Limit to 20MiB in Public Beta
Vercel Functions now accept inbound request bodies up to 20MiB, up from 4.5MB, in public beta. Developers enable it by setting the VERCELSUPPORTLARGEREQUESTBODY=1 environment variable; the opt-in applies to the entire project within the environment where it is set, allowing testing on preview branches first. The change lets developers handling large file uploads, bulk JSON submissions, or rich payloads avoid manual chunking and workaround logic.
Vercel Completes ISO/IEC 42001:2023 Certification for AI Management
Vercel has completed ISO/IEC 42001:2023 certification, the international standard for AI management systems. The company says the certification covers how it designs, builds, operates, and governs its AI products, giving customers confidence to ship agents on Vercel. For enterprises, this provides procurement and compliance teams with an internationally recognized basis when evaluating Vercel as an AI application hosting platform.
Proposed Proof of Generalized Riemann Hypothesis Is Entirely Human, No AI Involved
Alessandro Masullo has proposed a proof of the Generalized Riemann Hypothesis. Brian Conrad has been reviewing it closely for months and provided extensive feedback that is being incorporated, with results so far looking promising. Notably, the work is entirely human with no AI involvement. The original hope was to bring Masullo to Stanford for further discussion. The account was relayed by Ravi Vakil and quoted by Simon Willison.
ChatPRD Founder on Token Fraud: How Bad Are AI Product Billing Exploits
Claire Vo runs ChatPRD, an AI platform that helps product managers turn ideas into specs. The product has been live since the GPT-4 era, and she has built it in public for much of that time. In a YouTube video and accompanying write-up, she discusses token fraud—users or attackers exploiting billing and usage-metering gaps to consume AI product credits—with concrete cases showing the revenue risk for token-billed AI products.
Researcher Claims Claude-Assisted Proof of Riemann Hypothesis, Extendable to Generalized Case
Levent Alpöge claims to have discovered, working with Claude, a proof of the Riemann Hypothesis, and says the method easily extends to the Generalized Riemann Hypothesis and likely the Extended Riemann Hypothesis with more time. He states that he and Claude also formalized the argument. The claim appears in a Simon Willison quote post and has not been peer-reviewed or independently verified, contrasting with a separate human-only proof effort.