What are you looking for?

UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

AI·The paper proposes UltraX, a method for refining large-scale pre-training data using adaptive programmatic editing to improve LLM quality when scaling data yields diminishing returns.

Training an LLM from scratch on 1800's texts (160GB dataset)

Hi everyone, A year ago I began pre-training language models exclusively on 1800’s London data. Recently I have completed my largest dataset ever, containing 40B tokens or 160GB of 1800-1875 english data from England and the United States. I will soon train a 2B parameter model on it, but for now I’ve trained a 500M parameter evaluation model on a 5B token sample. I have also fine tuned the eval …