We develop an efficient Human-AI Copilot Optimization method (HACO), which incorporate human into the Reinforcement Learning (RL) training loop to boost the learning efficiency and ensure the safety. Sidestepping the requirement of complex reward engineering, HACO injects the human knowledge extracted from partial demonstration into the proxy value function by Offline RL technique. On the other hand, entropy regularization and intervention minimization are used for encouraging exploration and saving human budget respectively. The comprehensive experiments show the superior sample efficiency and safety guarantee of the proposed method.

The experiments conducted on MetaDrive Simulator show the efficient training as well as the low safety violation, when compared with RL, Offline RL, Imitation Learning and human-in-the-loop baselines. Results are reported in the Table 1. Compared to other data-hungry baselines, HACO utilizes less transitions and achieve highest success rate. Also, with the human protection, HACO yields only 30.14 total safety violations in the whole training process, two orders of magnitude less than other RL baselines, even though HACO access neither the cost or reward signal. In the human-in-the-loop paradigm, another concern is the expensive human budget consuming. As shown in Fig. 2 B., expensive human budget decreases along with the training.