On October 28, 2026, I attended the inaugural Bench Summit, hosted by Google DeepMind, Kaggle and Zapier. It was a one-day event, but a very well-organized one, with three threads running through it:
- Frontier evaluation, led by DeepMind
- Real-world benchmarks and infrastructure, from Kaggle and Zapier
- Startups, mostly focused on data and RL environments for mid- and post-training of frontier language models
Below are the notes I found most interesting, from the talks, the panels and my conversations with people at the event. Please be aware of the fact that the frontier moves fast, so some of these points may be out of date in a few months, or even a few weeks.
Gemini 4 Carbon launch
The event opened with DeepMind announcing its new frontier model, Gemini 4 Carbon, at the same moment it went public. I found the reasoning behind the name interesting, "Carbon turns into diamond under pressure," as they showed it taking the top spot on the Artificial Analysis Intelligence Index v5.0 with a score of 59, ahead of Claude Opus 5.5 at 58.
Unfortunately, the model isn't open to the general public yet. Access is currently limited to selected cyber defenders through Google's Fairwind Program. My guess is that an upcoming Fable 5.5 or the next version of Astra could catch up before Carbon reaches everyone.
How DeepMind evaluates the frontier
The next talk was about evaluating the frontier. DeepMind (and, I'd guess, the other frontier labs) splits evaluation across the model and harness development lifecycle into two parts: internal evals and external benchmarks.
- What internal evals target: either their consumer and enterprise products, or futuristic scenarios where models have to go beyond knowledge retrieval and memorization.
- Hill-climbing vs. release evals: both are drawn from the same task pools. Several release evals can be combined into a hill-climbing set, which is used to measure checkpoints throughout mid- and post-training.
- Difficulty matters: reading between the lines, eval difficulty also plays a role during training, and DeepMind uses curriculum learning to handle it.
- Verticals: about 20% of the final push in post-training and in the eval suite comes from verticals such as legal and health. Most tasks in these verticals are not strictly verifiable, so they work with vendors for their domain knowledge and intuition (I also heard this called "taste").
- Grading the process, not just the result: roughly speaking, this comes down to checks and assertions on the agent's reasoning traces rather than grading only the final result. Sometimes a correct process without a perfect final result can even be labelled a positive outcome. This is called methodology and best-practices grading.
To sum up, DeepMind looks at evals along three axes during the model lifecycle:
- Real-world use cases
- Eval quality, plus better tooling for traces
- Runtime cost and feasibility of inference
On feasibility: if a hill-climbing eval takes more than 12–24 hours to run, it is probably not a good choice for iterative development, as it will slow down mid- and post-training.
Zapier real-world benchmarks panel
After DeepMind, Zapier's benchmark team held a panel with Greg Kamradt, president of the ARC Prize Foundation, the organization behind the famously challenging ARC-AGI benchmark. The panel focused on where real-world benchmarks are right now, and where they are heading.
The main takeaway from this session was that we are entering a phase where short operational tasks, especially in verifiable domains, are nearly solved. The next phase is shifting expectations and benchmarks toward long-horizon, hypothesis-driven, experimental tasks.
- Zapier is retiring AutomationBench 1 and moving to version 2. It will have fewer, higher-quality long-horizon tasks, and a single task can take up to a week to complete and evaluate.
- The next ARC-AGI benchmark will also focus on long-horizon research and experimentation. They trim both tails of the difficulty distribution and keep problems that humans solve at a rate between 20% and 80%.
Read the traces, not just the outcomes
Another big takeaway from this panel, and from the event and my conversations in general, is that to iterate and build better evals, we should really look at agent traces rather than focusing only on outcomes.
One example from ARC-AGI-3: dropped into novel games in an isolated environment, GPT-5.5 kept mapping them onto familiar ones like Tetris or Breakout. According to the panel, GPT-6 Astra instead built its own plan and notation to work out the gameplay before it started playing.
ARC published an analysis of these failures and split them into three modes:
- Correct local effect, wrong world model: the model notices what an action does, but builds the wrong theory of the game.
- Wrong level of abstraction: "this looks like Tetris."
- Solving without a lesson: the model succeeds but doesn't carry the lesson forward.
The benefits of trace analysis extend to versioning the evaluation process, too. Both Zapier and ARC are actively retiring saturated tasks and running regression analysis on the failure points to build new iterations on top of them.
Reading traces also helps with understanding when LLM-as-a-judge systems are needed and what they add. I met engineers and benchmark developers who discovered problems in their evaluation simply by reading the traces the judge produced. In those cases, poor harness engineering had led the judge to miss crucial domain-specific and intuitive criteria in its reasoning.
Engineering takeaways from the startup demos
The rest of the summit was quick demos and presentations from Kaggle engineers and from several small and medium-sized startups and solo founders (SF has a lot of them). Topics ranged from evaluation and benchmarking platforms to rubric design, reward hacking in RL environments, sandboxing, network isolation, and cheating detection and gamification. Here are the engineering points I found most interesting:
- Environment isolation: some evaluation failures happen when the data, the agent and the grader share the same environment and the agent can reach the grader or the reference answers. Isolating the grader so nothing crosses that boundary mitigates this.
- Simulation inside the sandbox: many benchmarks run inside simulations, both to evaluate models fairly and for reproducibility. For example, a simulation can include synthetic web search, so the model under evaluation can't find the ground truth on the web.
- Inference and scaling: evaluation is both a training and an inference problem. Slow and unreliable environments are one of the main limits on scaling RL training and hill-climbing, or even just on benchmarking long-horizon research tasks.