Performance analysis of hierarchical reinforcement learning framework for stochastic space logistics

Yuji Takubo, Hao Chen, Koki Ho

    Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

    Abstract

    This paper analyzes a hierarchical reinforcement learning architecture for long-term space campaign designs, which can consider the stochastic parameters of mission scenarios and future influence of the space infrastructure that is deployed in the earlier mission for resource utilization. In the hierarchical framework, we have three levels of the decision-making process: vehicle design via value function approximation (i.e., vehicle design agent), infrastructure deployment mission planning via a reinforcement learning algorithm (i.e., infrastructure deployment agent), and space transportation scheduling via mixed-integer linear programming; these three levels are used iteratively to find the optimal mission design and vehicle/infrastructure sizing which minimize the total campaign cost. Additionally, an asynchronous pre-training phase prior to the dual agent learning phase is introduced, where each agent respectively pre-learns the sub-optimal policy beforehand so that the agents can run efficiently in the dual agent learning phase. The framework overcomes the difficulty in solving a robust design solution for space campaign design under uncertainty, and it is also flexible enough to incorporate various reinforcement learning algorithms. As a case study, the framework is applied to a set of lunar space campaign scenarios with potential resource utilization capabilities. Also, representative state-of-the-art reinforcement learning algorithms are integrated into this framework for comparison. The results show that the deterministic Actor-Critic reinforcement learning algorithms outperform other tested algorithms for the considered space campaign design.

    Original languageEnglish
    Title of host publicationAccelerating Space Commerce, Exploration, and New Discovery Conference, ASCEND 2020
    StatePublished - 2020
    EventAccelerating Space Commerce, Exploration, and New Discovery Conference, ASCEND 2020 - Las Vegas, United States
    Duration: 16 Nov 202019 Nov 2020

    Publication series

    NameAccelerating Space Commerce, Exploration, and New Discovery Conference, ASCEND 2020

    Conference

    ConferenceAccelerating Space Commerce, Exploration, and New Discovery Conference, ASCEND 2020
    Country/TerritoryUnited States
    CityLas Vegas
    Period16/11/2019/11/20

    Fingerprint

    Dive into the research topics of 'Performance analysis of hierarchical reinforcement learning framework for stochastic space logistics'. Together they form a unique fingerprint.

    Cite this