Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

preprint · arXiv (Cornell University) · 2022

Adversarial Counterfactual Environment Model Learning

Xionghui Chen, Yang Yu, Zhengmao Zhu, Zhihua Yu, Zhenjun Chen, Chenghe Wang, Yinan Wu, Hongqiu Wu, Rongjun Qin, Ruijin Ding, Fangsheng Huang

Cited by 6Open access

Abstract

A good model for action-effect prediction, named environment model, is important to achieve sample-efficient decision-making policy learning in many domains like robot control, recommender systems, and patients' treatment selection. We can take unlimited trials with such a model to identify the appropriate actions so that the costs of queries in the real world can be saved. It requires the model to handle unseen data correctly, also called counterfactual data. However, standard data fitting techniques do not automatically achieve such generalization ability and commonly result in unreliable models. In this work, we introduce counterfactual-query risk minimization (CQRM) in model learning for generalizing to a counterfactual dataset queried by a specific target policy. Since the target policies can be various and unknown in policy learning, we propose an adversarial CQRM objective in which the model learns on counterfactual data queried by adversarial policies, and finally derive a tractable solution GALILEO. We also discover that adversarial CQRM is closely related to the adversarial model learning, explaining the effectiveness of the latter. We apply GALILEO in synthetic tasks and a real-world application. The results show that GALILEO makes accurate predictions on counterfactual data and thus significantly improves policies in real-world testing.

Cite this paper

APA 7

Chen, X., Yu, Y., Zhu, Z., Yu, Z., Chen, Z., Wang, C., Wu, Y., Wu, H., Qin, R., Ding, R., & Huang, F. (2022). Adversarial Counterfactual Environment Model Learning. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2206.04890

MLA 9

Chen, Xionghui, et al. “Adversarial Counterfactual Environment Model Learning.” arXiv (Cornell University), 2022, https://doi.org/10.48550/arxiv.2206.04890.

Chicago

Chen, Xionghui, Yang Yu, Zhengmao Zhu, Zhihua Yu, Zhenjun Chen, Chenghe Wang, Yinan Wu, et al. 2022. “Adversarial Counterfactual Environment Model Learning.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2206.04890.

Harvard

Chen, X. et al. (2022) ‘Adversarial Counterfactual Environment Model Learning’, arXiv (Cornell University). Available at: https://doi.org/10.48550/arxiv.2206.04890.

BibTeX
@misc{chen2022adversarial,
  title = {{Adversarial Counterfactual Environment Model Learning}},
  author = {Chen, Xionghui and Yu, Yang and Zhu, Zhengmao and Yu, Zhihua and Chen, Zhenjun and Wang, Chenghe and Wu, Yinan and Wu, Hongqiu and Qin, Rongjun and Ding, Ruijin and Huang, Fangsheng},
  journal = {arXiv (Cornell University)},
  year = {2022},
  doi = {10.48550/arxiv.2206.04890}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.

Related papers