preprint · arXiv (Cornell University) · 2024

Adaptive Decoding via Latent Preference Optimization

Shehzaad Dhuliawala, Ilia Kulikov, Ping Yu, Aslı Çelikyılmaz, Jason Weston, Sainbayar Sukhbaatar, Jack Lanchantin

Cited by 1Open access

Abstract

During language model decoding, it is known that using higher temperature sampling gives more creative responses, while lower temperatures are more factually accurate. However, such models are commonly applied to general instruction following, which involves both creative and fact seeking tasks, using a single fixed temperature across all examples and tokens. In this work, we introduce Adaptive Decoding, a layer added to the model to select the sampling temperature dynamically at inference time, at either the token or example level, in order to optimize performance. To learn its parameters we introduce Latent Preference Optimization (LPO) a general approach to train discrete latent variables such as choices of temperature. Our method outperforms all fixed decoding temperatures across a range of tasks that require different temperatures, including UltraFeedback, Creative Story Writing, and GSM8K.

Cite this paper

APA 7

Dhuliawala, S., Kulikov, I., Yu, P., Çelikyılmaz, A., Weston, J., Sukhbaatar, S., & Lanchantin, J. (2024). Adaptive Decoding via Latent Preference Optimization. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2411.09661

MLA 9

Dhuliawala, Shehzaad, et al. “Adaptive Decoding via Latent Preference Optimization.” arXiv (Cornell University), 2024, https://doi.org/10.48550/arxiv.2411.09661.

Chicago

Dhuliawala, Shehzaad, Ilia Kulikov, Ping Yu, Aslı Çelikyılmaz, Jason Weston, Sainbayar Sukhbaatar, and Jack Lanchantin. 2024. “Adaptive Decoding via Latent Preference Optimization.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2411.09661.

Harvard

Dhuliawala, S. et al. (2024) ‘Adaptive Decoding via Latent Preference Optimization’, arXiv (Cornell University). Available at: https://doi.org/10.48550/arxiv.2411.09661.

BibTeX

@misc{dhuliawala2024adaptive,
  title = {{Adaptive Decoding via Latent Preference Optimization}},
  author = {Dhuliawala, Shehzaad and Kulikov, Ilia and Yu, Ping and Çelikyılmaz, Aslı and Weston, Jason and Sukhbaatar, Sainbayar and Lanchantin, Jack},
  journal = {arXiv (Cornell University)},
  year = {2024},
  doi = {10.48550/arxiv.2411.09661}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.