LEARNING TO SCHEDULE CONTROL FRAGMENTS FOR PHYSICS-BASED CHARACTER SIMULATION AND ROBOTS USING DEEP Q-LEARNING

Deep-Q-learning control-fragment scheduling for robots/physics sim.

The disclosure provides an approach for learning to schedule control fragments for physics-based virtual character simulations and physical robot control. Given precomputed tracking controllers, a simulation application segments the controllers into control fragments and learns a scheduler that selects control fragments at runtime to accomplish a task. In one embodiment, each scheduler may be modeled with a Q-network that maps a high-level representation of the state of the simulation to a control fragment for execution. In such a case, the deep Q-learning algorithm applied to learn the Q-network schedulers may be adapted to use a reward function that prefers the original controller sequence and an exploration strategy that gives more chance to in-sequence control fragments than to out-of-sequence control fragments. Such a modified Q-learning algorithm learns schedulers that are capable of following the original controller sequence most of the time while selecting out-of-sequence control fragments when necessary.

BACKGROUNDField of the Invention

Embodiments presented herein relate to the field of physics-based character simulation and robot control. More specifically, this disclosure presents techniques for learning to schedule control fragments for virtual characters and robots using deep Q-learn ing.Description of the Related Art

With a robust control system, physics-based characters and robots can, in theory, be controlled to respond naturally to actions of a user and changes in their environment. For a variety of behaviors, it is difficult to design such controllers that are responsive to user input and robust to environmental disturbances. One approach for simplifying the control design for complex motions is to track a reference motion, thereby leveraging the natural style and strategies contained in motion capture data. For example, motion capture data from a skateboarder riding a skateboard may be tracked in a simulation of a virtual character riding a skateboard. However, in a robust tracking control system, the tracking reference may need to be rescheduled (e.g., by synchronizing the simulation with the tracking reference on specific events such as ground contact and by planning transitions among multiple behaviors) in order to respond to environmental perturbations and user interactions. For example, in response to a bump in the road, the character riding the skateboard may fall over unless the tracking reference being used is rescheduled, or a transition is mad