AI·LLM-as-judge scores fluctuate across evaluators even with fixed candidate responses, treating the issue as measurement validity. Across four datasets, scaling Qwen3 from 1.7B to 32B parameters yields only limited adjacent gains, while MiniMax M2-to-M2.7 API upgrades show none; stronger judges reduce but do not eliminate position and verbosity biases, with repeated juries offering little benefit under correlated errors.
Search
What are you looking for?
16 results for “dataset”
AI·This position paper reviews LLM-driven formal theorem provers and argues that current systems function mainly as solvers for well-defined problems, not as research agents capable of discovering new theorems or resolving open conjectures at the frontier. It identifies key limitations in datasets, exploration, tools, and collaboration and proposes a roadmap for AI4Math systems to support genuine mathematical research.
AI·ICDAR 2026 HIPE-OCRepair competition evaluated LLM-assisted post-correction of noisy OCR from 17th-20th century multilingual (EN/FR/DE) historical newspapers and books. Four teams used zero-shot to fine-tuning approaches; results show significant error reduction but recurring over-correction on low-noise inputs, with a public dataset and evaluation framework released.
AI·The paper introduces PLURAL, a large-scale value-focused preference dataset grounded in the Integrated Values Survey across 92 countries to improve LLM representation of diverse non-Western value systems.
AI·The paper presents a cost-efficient human-LLM collaborative annotation framework to construct the EspanSt stereotype dataset for non-English languages and underrepresented cultures.
AI·TypeProbe recovers type representations from hidden states of pretrained code models using parallel Java and Python datasets.
Hi everyone, A year ago I began pre-training language models exclusively on 1800’s London data. Recently I have completed my largest dataset ever, containing 40B tokens or 160GB of 1800-1875 english data from England and the United States. I will soon train a 2B parameter model on it, but for now I’ve trained a 500M parameter evaluation model on a 5B token sample. I have also fine tuned the eval …
In this work, we present Canvas360, a two-stage framework for in-context panoramic generation that combines geometry-aware pretraining with downstream task-specific fine-tuning. To address the lack of large-scale, high-quality training data tailored to in-context panoramic tasks, we propose Canvas360Dataset, a collection of 1M high-quality paired panoramic samples for style transfer, inpainting, outpainting, and editing, enabling effective supervision across diverse in-context generation scenar…
Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templat…
I am doing some work with cell type classification, where I have 4.3 million cells and 512 features (condensed embeddings from the encoder of a transformer). The broader goal is to implement a contextual bandit for augmenting the training set of the dataset, as it is currently imbalanced, and rare cell type classification is poor when I tried a baseline logistic regression classifier. Dataset: Fe…
Humans can navigate an unfamiliar city and gradually form a coherent spatial mental map spanning tens of square kilometers. Can AI build spatial representations at a comparable scale? Although recent foundation models have advanced scene reconstruction and embodied intelligence, scaling to entire cities remains an open challenge, primarily due to the lack of city-scale data. To bridge the gap, we introduce WildCity, a real-world multimodal dataset collected by autonomous fleets traversing compl…
Most safety alignment work treats "detect the attack" as a text classification problem — does the prompt contain language the model's safety guardrails should catch. That assumption breaks down for LLM agents with real tool access. Here's a concrete case: take a known, public security vulnerability (a CVE), work out the sequence of tool calls that would exploit it, then have an LLM rewrite that a…
Introducing GeneBench-Pro, a new benchmark testing AI performance in genomics, biology, and scientific research using complex, real-world datasets.
Doing a bachelor thesis on fine-grained car classification (telling apart VW Golf generations from listing photos). Simple setup: frozen encoder → embeddings → weighted k-NN. On my small dataset (175 train / 132 test): I thought maybe it was a cosine vs euclidean thing, but my embeddings are L2-normalized so both give the same ranking. Tried both, DINOv2 stays at 41%. I get that SigLIP was traine…
We're happy to release MIRA, a collaboration between General Intuition, Kyutai, and Epic Games. Mira was trained on 10k hours of synthetic Rocket League data. The model has 5B parameters and runs for 4 players at 20 fps on a single B200. We've released a playable online demo, an in-depth technical report as well as a 1k hour dataset of 4-players gameplay: Demo: https://mira-wm.com Technical repor…