A Simulation Based Framework for Synthesizing Action-Selection Policies for Autonomous Unmanned Surface Vehicles

PI: Prof. S K Gupta

A major issue in the development of increased autonomy for robotic vehicles such as USVs is the time and expense of developing the software necessary to handle a large variety of missions and all the variations in the encountered environments. This is a challenging task and requires writing hundreds of thousands of lines of code by human programmers.

We have developed a new approach for developing planning software that operates autonomous USVs. This new approach takes advantage of the significant progress that has been made in virtual environments (VE) and automated synthesis. The basic idea behind our approach is as follows. The USV explores the VE by trying different moves. USV moves are simulated in the VE and evaluated based on their ability to make progress toward the mission goal. If a successful action is identified as a part of the exploration, then this action will be integrated into the policy, driving the USV.

We anticipate that there may be portions of the mission, where exploration alone will not be adequate to discover the right decision rules. In such cases, two additional approaches are utilized to make progress in acquiring the right policy. The first approach involves seeding the system with the policy employed by humans to solve a challenging task. The second approach is to restrict the action space based on some type of feasibility criteria.

We focus specifically on automated generation of action selection policy used for blocking the advancement of an intruder boat towards a valuable target. The first major component of our approach is development of a physics-based metamodel. High fidelity simulation of USV is computing-intensive and cannot be used for discovering decision rules or trees used in planning. We have developed a meta-model by conducting off-line simulations of the USV in the sea. This simulation accounts for wave and USV interactions.

We have developed a mission planning system whose main part is an evolutionary module for evolving action selection policy decision trees. We used this system to automatically generate decision trees expressing blocking policy for the USV. This means that instead of automatically generating a program composed of low-level controller actions, we generate a program represented as a decision tree that consists of high-level navigation commands as building blocks together with conditionals and other program constructs.

We have developed a VE-based visualization system, which serves as an emulator of the real USV environment and allows human players to play against each other or against the computer. In the game, the player controlling the intruder boat must reach a protected target, while the player controlling the USV must block and delay the intruder as long as possible. The game can be played on two computers over a network. In addition to offering basic gaming capabilities, the visualization system provides collision detection and basic physics to the objects in the scene. Our results show that the policy automatically discovered by the system performs at par with the policy designed by an experienced human programmer.

Virtual Environment for Simulation

Simple mission: USV protects area containing the target