AI·OpenAI released GPT-5.6 in three tiers—Sol (flagship), Terra, and Luna—boosting benchmarks in Agent's Last Exam (56.3 score), math, coding, security, and research while slashing token costs and adding Max/Ultra reasoning modes.
Search
What are you looking for?
26 results for “benchmark”
AI·A user shares benchmarks of dual RTX 3090 and Titan RTX cards running llama.cpp with Qwen3.6-27B at 180k context, comparing tensor and pipeline parallel modes for PCIe transfer efficiency.
AI·The author benches a quad 5060Ti setup for Qwen3.6-27B code generation using MTP, reporting 608 tokens/s prefill and 52.2 tokens/s decode at 256k context on a cost-effective build.
AI·The post shares practical recipes and benchmarks for running LLMs on popular consumer hardware, with a new website listing hardware-filtered setups and user-voted usage stats.
AI·VultronRetriever family of models is released on Hugging Face, topping MTEB leaderboard classes including VultronRetrieverPrime-8B as global #1, with offline iPhone Q&A and embedding demo.
AI·Hidden Decoding expands each token into n independent streams during continued pretraining of a fixed Transformer backbone, using separate embedding tables and retaining intermediate KV caches for latent computation; Stream-Factorized Attention limits cross-stream mixing to reduce costs. Experiments show frontier 80B and 617B MoE models improve on all benchmarks over matched baselines.
AI·MASTE: A Multi-Agent Pipeline for Zero-Shot Aspect Sentiment Triplet Extraction
AI·User compares 5090 at 600W full compute and prompt processing vs shunt-modded 6000 PRO MaxQ water-cooled at 300-600W.
AI·LEXIC pushes gaze-only reading comprehension prediction on EyeBench with lightweight language-model-free conditioning and injected complexity.
AI·Follow-up post details data-engineering optimizations that scaled a SQLite/FTS5 patent database from 3.5M to 5.36M records while classifying them with Nemotron 9B on an RTX 5090.
AI·Questions costs and viability of aftermarket SXM2 boards for two V100 GPUs, comparing to consumer GPU setups.
AI·Reports EPYC 9374F CCD benchmarks using ik_llama.cpp; finds limited decoding gains vs older 9135 at low thread counts.
AI·Qwen3.6 35B-A3B Q8_0 model (no KV quant) generates full HTML flight simulator in single prompt and is praised as stronger than Q4_K_M on GPU due to better CPU performance.
Scientific ideas rarely start from a blank page. They inherit mechanisms, repair known limitations, and recombine pieces of earlier work, much like biological genomes. Current benchmarks still say little about whether AI systems can follow this inheritance structure. We present IdeaGene-Bench (IG-Bench), a benchmark for scientific lineage reasoning and lineage-grounded idea generation. IG-Bench is organized around the IdeaGene framework: each paper or proposal is represented as a set of minimal…
The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it dif…
Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templat…
A new analysis from OpenAI reveals issues in SWE-Bench Pro, a popular coding benchmark, raising concerns about reliability and accuracy in evaluating AI models.
What it is Talos-XII is a CLI simulator for the gacha system in Arknights: Endfield. Rather than sampling from a static probability table, it trains a small set of neural nets to model environment uncertainty and pull-decision policy, then uses them to answer questions a static table can’t easily express — e.g. “as a F2P player, what’s my probability of getting the rate-up unit on free currency a…
Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a…
We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written t…
Introducing GeneBench-Pro, a new benchmark testing AI performance in genomics, biology, and scientific research using complex, real-world datasets.
Introducing LifeSciBench, an expert-authored, expert-reviewed benchmark for evaluating how AI systems handle real-world life science research tasks and decisions.
Algorithms & Theory
The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-…
Large Audio-Language Models (LALMs) are increasingly integrated into daily applications, yet their generative biases remain underexplored. Existing speech fairness benchmarks rely on synthetic speech and Multiple-Choice Questions (MCQs), both offering a fragmented view of fairness. We propose VIBE, a framework that evaluates generative bias through open-ended tasks such as personalized recommendations, using human-recorded speech. Unlike MCQs, our method allows stereotypical associations to man…
Share what your favorite models are right now and why. Given the nature of the beast in evaluating VLMs (untrustworthiness of benchmarks, immature tooling, intrinsic stochasticity), please be as detailed as possible in describing your setup (at least hardware and inference engine) nature of your usage (what applications, how much, personal/professional use) tools/frameworks/prompts etc. Rules Onl…