article · ArXiv.org · 2025

NotSoTiny: A Large, Living Benchmark for RTL Code Generation

Razine Moundir Ghorab, Emanuele Parisi, Cristian Gutierrez, Miquel Albertí-Binimelis, Miquel Moreto, Dario García-Gasulla, Gökçen Kestor

Cited by 0Open access

Abstract

LLMs have shown early promise in generating RTL code, yet evaluating their capabilities in realistic setups remains a challenge. So far, RTL benchmarks have been limited in scale, skewed toward trivial designs, offering minimal verification rigor, and remaining vulnerable to data contamination. To overcome these limitations and to push the field forward, this paper introduces NotSoTiny, a benchmark that assesses LLM on the generation of structurally rich and context-aware RTL. Built from hundreds of actual hardware designs produced by the Tiny Tapeout community, our automated pipeline removes duplicates, verifies correctness and periodically incorporates new designs to mitigate contamination, matching Tiny Tapeout release schedule. Evaluation results show that NotSoTiny tasks are more challenging than prior benchmarks, emphasizing its effectiveness in overcoming current limitations of LLMs applied to hardware design, and in guiding the improvement of such promising technology.

Cite this paper

APA 7

Ghorab, R. M., Parisi, E., Gutierrez, C., Albertí-Binimelis, M., Moreto, M., García-Gasulla, D., & Kestor, G. (2025). NotSoTiny: A Large, Living Benchmark for RTL Code Generation. ArXiv.org.

MLA 9

Ghorab, Razine Moundir, et al. “NotSoTiny: A Large, Living Benchmark for RTL Code Generation.” ArXiv.org, 2025.

Chicago

Ghorab, Razine Moundir, Emanuele Parisi, Cristian Gutierrez, Miquel Albertí-Binimelis, Miquel Moreto, Dario García-Gasulla, and Gökçen Kestor. 2025. “NotSoTiny: A Large, Living Benchmark for RTL Code Generation.” ArXiv.org.

Harvard

Ghorab, R.M. et al. (2025) ‘NotSoTiny: A Large, Living Benchmark for RTL Code Generation’, ArXiv.org.

BibTeX

@article{ghorab2025notsotiny,
  title = {{NotSoTiny: A Large, Living Benchmark for RTL Code Generation}},
  author = {Ghorab, Razine Moundir and Parisi, Emanuele and Gutierrez, Cristian and Albertí-Binimelis, Miquel and Moreto, Miquel and García-Gasulla, Dario and Kestor, Gökçen},
  journal = {ArXiv.org},
  year = {2025}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.