AI·OpenAI released GPT-5.6 in three tiers—Sol (flagship), Terra, and Luna—boosting benchmarks in Agent's Last Exam (56.3 score), math, coding, security, and research while slashing token costs and adding Max/Ultra reasoning modes.
Search
What are you looking for?
16 results for “benchmarks”
AI·A user shares benchmarks of dual RTX 3090 and Titan RTX cards running llama.cpp with Qwen3.6-27B at 180k context, comparing tensor and pipeline parallel modes for PCIe transfer efficiency.
AI·The post shares practical recipes and benchmarks for running LLMs on popular consumer hardware, with a new website listing hardware-filtered setups and user-voted usage stats.
AI·Hidden Decoding expands each token into n independent streams during continued pretraining of a fixed Transformer backbone, using separate embedding tables and retaining intermediate KV caches for latent computation; Stream-Factorized Attention limits cross-stream mixing to reduce costs. Experiments show frontier 80B and 617B MoE models improve on all benchmarks over matched baselines.
AI·MASTE: A Multi-Agent Pipeline for Zero-Shot Aspect Sentiment Triplet Extraction
AI·Reports EPYC 9374F CCD benchmarks using ik_llama.cpp; finds limited decoding gains vs older 9135 at low thread counts.
AI·UniClawBench is the first capability-driven benchmark for proactive agents, using live Docker containers and closed-loop multi-agent evaluation to assess five foundational skills across 400 real-world tasks.
AI·The paper introduces IdeaGene-Bench (IG-Bench) to benchmark AI systems on scientific lineage reasoning and lineage-grounded idea generation.
AI·CausalDS benchmark evaluates causal reasoning in LLM data-science agents bridging symbolic and data-analysis approaches.
AI·Single-Rollout Asynchronous Optimization is an agentic RL method that replaces group-wise sampling with single-rollout updates and adds token-level clipping to improve stability and performance on coding and reasoning benchmarks.
Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a…
We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written t…
Algorithms & Theory
The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-…
Large Audio-Language Models (LALMs) are increasingly integrated into daily applications, yet their generative biases remain underexplored. Existing speech fairness benchmarks rely on synthetic speech and Multiple-Choice Questions (MCQs), both offering a fragmented view of fairness. We propose VIBE, a framework that evaluates generative bias through open-ended tasks such as personalized recommendations, using human-recorded speech. Unlike MCQs, our method allows stereotypical associations to man…
Share what your favorite models are right now and why. Given the nature of the beast in evaluating VLMs (untrustworthiness of benchmarks, immature tooling, intrinsic stochasticity), please be as detailed as possible in describing your setup (at least hardware and inference engine) nature of your usage (what applications, how much, personal/professional use) tools/frameworks/prompts etc. Rules Onl…