preprint · arXiv (Cornell University) · 2024

COSBO: Conservative Offline Simulation-Based Policy Optimization

Cited by 0Open access

Abstract

Offline reinforcement learning allows training reinforcement learning models on data from live deployments. However, it is limited to choosing the best combination of behaviors present in the training data. In contrast, simulation environments attempting to replicate the live environment can be used instead of the live data, yet this approach is limited by the simulation-to-reality gap, resulting in a bias. In an attempt to get the best of both worlds, we propose a method that combines an imperfect simulation environment with data from the target environment, to train an offline reinforcement learning policy. Our experiments demonstrate that the proposed method outperforms state-of-the-art approaches CQL, MOPO, and COMBO, especially in scenarios with diverse and challenging dynamics, and demonstrates robust behavior across a variety of experimental conditions. The results highlight that using simulator-generated data can effectively enhance offline policy learning despite the sim-to-real gap, when direct interaction with the real-world is not possible.

Cite this paper

APA 7

Kargar, E., & Kyrki, V. (2024). COSBO: Conservative Offline Simulation-Based Policy Optimization. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2409.14412

MLA 9

Kargar, Eshagh, and Ville Kyrki. “COSBO: Conservative Offline Simulation-Based Policy Optimization.” arXiv (Cornell University), 2024, https://doi.org/10.48550/arxiv.2409.14412.

Chicago

Kargar, Eshagh, and Ville Kyrki. 2024. “COSBO: Conservative Offline Simulation-Based Policy Optimization.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2409.14412.

Harvard

Kargar, E. and Kyrki, V. (2024) ‘COSBO: Conservative Offline Simulation-Based Policy Optimization’, arXiv (Cornell University). Available at: https://doi.org/10.48550/arxiv.2409.14412.

BibTeX

@misc{kargar2024cosbo,
  title = {{COSBO: Conservative Offline Simulation-Based Policy Optimization}},
  author = {Kargar, Eshagh and Kyrki, Ville},
  journal = {arXiv (Cornell University)},
  year = {2024},
  doi = {10.48550/arxiv.2409.14412}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.