other · Zenodo (CERN European Organization for Nuclear Research) · 2026

DataTrove: large scale data processing

Penedo, Guilherme, Kydlíček, Hynek, Cappelli, Alessandro, Wolf, Thomas, Sasko, Mario

Cited by 1Open access

Abstract

DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt commonly used processing blocks with a framework to easily add custom functionality. DataTrove processing pipelines are platform-agnostic, running out of the box locally or on a slurm cluster. Its (relatively) low memory usage and multiple step design makes it ideal for large workloads, such as to process an LLM's training data.

Cite this paper

APA 7

Penedo, G., Kydlíček, H., Cappelli, A., Wolf, T., & Sasko, M. (2026). DataTrove: large scale data processing. Zenodo (CERN European Organization for Nuclear Research). https://doi.org/10.5281/zenodo.11038903

MLA 9

Penedo, Guilherme, et al. “DataTrove: large scale data processing.” Zenodo (CERN European Organization for Nuclear Research), 2026, https://doi.org/10.5281/zenodo.11038903.

Chicago

Penedo, Guilherme, Hynek Kydlíček, Alessandro Cappelli, Thomas Wolf, and Mario Sasko. 2026. “DataTrove: large scale data processing.” Zenodo (CERN European Organization for Nuclear Research). https://doi.org/10.5281/zenodo.11038903.

Harvard

Penedo, G. et al. (2026) ‘DataTrove: large scale data processing’, Zenodo (CERN European Organization for Nuclear Research). Available at: https://doi.org/10.5281/zenodo.11038903.

BibTeX

@misc{penedo2026datatrove,
  title = {{DataTrove: large scale data processing}},
  author = {Penedo, Guilherme and Kydlíček, Hynek and Cappelli, Alessandro and Wolf, Thomas and Sasko, Mario},
  journal = {Zenodo (CERN European Organization for Nuclear Research)},
  year = {2026},
  doi = {10.5281/zenodo.11038903}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.