Abstract
An issue concerning the use of deep reinforcement learning (RL) agents is whether they can be trusted to perform reliably when deployed, as training environments may not reflect real-life environments. Anticipating instances outside their training scope, learning-enabled systems are often equipped with out-of-distribution (OOD) detectors that alert when a trained system encounters a state it does not recognize or in which it exhibits uncertainty. There exists limited work conducted on the problem of OOD detection within RL, with prior studies being unable to achieve a consensus on the definition of OOD execution within the context of RL. By framing our problem using a Markov Decision Process, we assume there is a transition distribution mapping each state-action pair to another state with some probability. Based on this, we consider the following definition of OOD execution within RL: A transition is OOD if its probability during real-life deployment differs from the transition distribution encountered during training. As such, we utilize conditional variational autoencoders (CVAE) to approximate the transition dynamics of the training environment and implement a conformity-based detector using reconstruction loss that is able to guarantee OOD detection with a pre-determined confidence level. We evaluate our detector by adapting existing benchmarks and compare it with existing OOD detection models for RL.
Original language | English |
---|---|
Title of host publication | Special Track on AI Alignment |
Editors | Toby Walsh, Julie Shah, Zico Kolter |
Publisher | Association for the Advancement of Artificial Intelligence |
Pages | 12452-12460 |
Number of pages | 9 |
Edition | 12 |
ISBN (Electronic) | 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 157735897X, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978, 9781577358978 |
DOIs | |
Publication status | Published - Apr 11 2025 |
Externally published | Yes |
Event | 39th Annual AAAI Conference on Artificial Intelligence, AAAI 2025 - Philadelphia, United States Duration: Feb 25 2025 → Mar 4 2025 |
Publication series
Name | Proceedings of the AAAI Conference on Artificial Intelligence |
---|---|
Number | 12 |
Volume | 39 |
ISSN (Print) | 2159-5399 |
ISSN (Electronic) | 2374-3468 |
Conference
Conference | 39th Annual AAAI Conference on Artificial Intelligence, AAAI 2025 |
---|---|
Country/Territory | United States |
City | Philadelphia |
Period | 2/25/25 → 3/4/25 |
Bibliographical note
Publisher Copyright:Copyright © 2025, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
ASJC Scopus Subject Areas
- Artificial Intelligence