- Number
- 10929743
- Published
- 2021-02-23
- Filed
- 2016-09-27
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Liu; Libin, Hodgins; Jessica Kate
- CPC
- G06N3/006; G06N3/008; G06N3/0499; G06N3/08; G06N3/092
- Verdict
- Medium Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Deep-RL control scheduling for physics-based character/robot simulation.
Abstract
The disclosure provides an approach for learning to schedule control fragments for physics-based virtual character simulations and physical robot control. Given precomputed tracking controllers, a simulation application segments the controllers into control fragments and learns a scheduler that selects control fragments at runtime to accomplish a task. In one embodiment, each scheduler may be modeled with a Q-network that maps a high-level representation of the state of the simulation to a control fragment for execution. In such a case, the deep Q-learning algorithm applied to learn the Q-network schedulers may be adapted to use a reward function that prefers the original controller sequence and an exploration strategy that gives more chance to in-sequence control fragments than to out-of-sequence control fragments. Such a modified Q-learning algorithm learns schedulers that are capable of following the original controller sequence most of the time while selecting out-of-sequence control fragments when necessary.
Background
BACKGROUND Field of the Invention (1) Embodiments presented herein relate to the field of physics-based character simulation and robot control. More specifically, this disclosure presents techniques for learning to schedule control fragments for virtual characters and robots using deep Q-learning. Description of the Related Art (2) With a robust control system, physics-based characters and robots can, in theory, be controlled to respond naturally to actions of a user and changes in their environment. For a variety of behaviors, it is difficult to design such controllers that are responsive to user input and robust to environmental disturbances. One approach for simplifying the control design for complex motions is to track a reference motion, thereby leveraging the natural style and strategies contained in motion capture data. For example, motion capture data from a skateboarder riding a skateboard may be tracked in a simulation of a virtual character riding a skateboard. However, in a robust tracking control system, the tracking reference may need to be rescheduled (e.g., by synchronizing the simulation with the tracking reference on specific events such as ground contact and by planning transitions among multiple behaviors) in order to respond to environmental perturbations and user interactions. For example, in response to a bump in the road, the character riding the skateboard may fall over unless the tracking reference being used is rescheduled, or a transition is made t
Claims
1. A computer-implemented method for controlling motion, the computer-implemented method comprising: segmenting one or more tracking controllers that maintain a time-indexed reference trajectory of at least part of a specified motion, wherein the one or more tracking controllers are segmented into a reference sequence of control fragments and based on a time interval; training, by a simulation application when executed by one or more computer processors, a neural network for rescheduling the reference sequence of control fragments into a different sequence, based on a state of simulation, the specified motion, and a reward function; using the trained neural network to reschedule the reference sequence of control fragments into a different sequence at runtime to account for the specified motion changing due to a physics-based interaction between a character and an environment; and executing, by the simulation application, the rescheduled sequence of control fragments to control the character to perform the changed motion, wherein the character is selected from a virtual character and a physical robot. ||
9. A non-transitory computer-readable medium storing a simulation application executable to perform operations for controlling motion, the operations comprising: segmenting one or more tracking controllers that maintain a time-indexed reference trajectory of at least part of a specified motion, wherein the one or more tracking controllers are segmented into a reference sequence of control fragments and based on a time interval; training, by the simulation application when executed by one or more computer processors, a neural network for rescheduling the reference sequence of control fragments into a different sequence, based on a state of simulation, the specified motion, and a reward function; using the trained neural network to reschedule the reference sequence of control fragments into a different sequence at runtime to account for the specified motion changing due to a physics-based interaction between a character and an environment; and executing, by the simulation application, the rescheduled sequence of control fragments to control the character to perform the changed motion, wherein the character is selected from a virtual character and a physical robot. ||
17. A system for controlling motion, the system comprising: one or more computer processors; and a memory including a simulation application executable by the one or more computer processors to perform operations comprising: segmenting one or more tracking controllers that maintain a time-indexed reference trajectory of at least part of a specified motion, wherein the one or more tracking controllers are segmented into a reference sequence of control fragments and based on a time interval; training a neural network for rescheduling the reference sequence of control fragments into a different sequence, based on a state of simulation, the specified motion, and a reward function; and using the trained neural network to reschedule the reference sequence of control fragments into a different sequence at runtime to account for the specified motion changing due to a physics-based interaction between a character and an environment; and executing the rescheduled sequence of control fragments to control the character to perform the changed motion, wherein the character is selected from a virtual character and a physical robot.