Your vehicle will know you’re going to drive onto the road ahead of you to get a head start, or a robot-assisted surgeon will be able to anticipate their next small movement. That’s the state of the art in today’s high-tech AI world: Anticipating Human Motion Machine Learning (ML).
This is not predictive science fiction; this is hard science that combines computer vision, deep learning, and behavior science to create models that precisely predict human movement. Its applications are revolutionizing everything from industrial robots to autonomous vehicles and public safety systems, and are making our worlds smart and safe.
The Challenge: Why Humans are Hard to Predict
Weather is unpredictable, but humans more so. Our behavior is governed by a random mix of conscious decision, habit responses, and other influences:
- Intent vs. Action: A person can intend to look at a street sign (intent) without stepping onto the sidewalk into the street for a second or two (action). ML needs to distinguish between a passing thought and a decision physical action.
- Context is Everything: It’s all a matter of context what a movement is. Slowing down by a person walking towards a curb is not slowing down by a person walking towards an empty beach.
- High Dimensionality: Human movement is nearing the very trajectory of dozens of joints (hip, knee, ankle, etc.) in three-space, one second at a time.
The models need to handle huge amounts of dynamic data (sensor data, acceleration, velocity, video) and predict them a little into the future with ultra-low latency.
The Models: Deep Learning for Prediction
Solutions for prediction used to rely on simple physics-based models. New solutions rely on complex deep learning models, parameterized to find temporal (time-dependent) correlations:
1. Recurrent Neural Networks (RNNs) and LSTMs
These were the first workhorses. Recurrent Neural Networks (RNNs), and its extended cousin, Long Short-Term Memory (LSTM) Networks, are best suited for sequence data. They “remember” what the movement steps in the sequence have been thus far, so they know the correct cue to propose the next pose given the movement thus far. They need to forecast short-term high-granularity motion like reaching for something.
2. Generative Models (GANs)
More recently, Generative Adversarial Networks (GANs) have been developed. A GAN uses two adversarial networks: a Generator to generate possible future sequence of motion, and a Discriminator that tries to determine if such sequence is real or generated. The adversarial process pushes the Generator to generate highly realistic, naturally structured, and diverse predictions, eliminating the issue where an action can have multiple reasonable outcomes (e.g., stopping or going).
3. Transformer Models
Pre-trained on natural language processing (NLP) tasks, Transformer models are becoming popular. They are especially well-suited to modeling long-range dependencies—what one was doing a last 10 seconds will determine what they are doing now. They can read the entire input sequence in a single pass and are especially well-suited to modeling fine, multi-person relationships (e.g., following a crowd).
Applications: Making the World Anticipatory
With the ability to accurately map human movement, directly onto more efficient, safer systems:
- Autonomous Vehicles (AVs): Most critical use case. AVs must predict pedestrians, cyclists, and other drivers’ behaviors. A model with prediction of a child running out with a ball onto the road in 1.5 seconds is crash avoidance vs. hard brake.
- Human-Robot Collaboration (Cobots): There are human-robot collaboration (Cobots) in factories and operating rooms. Predictive motion enables the robot to decelerate, move out of the tool’s path, or even halt the instant that a human operator enters its shared workspace so that there is maximum safety without any loss of productivity.
- Public Safety and Surveillance: In a public space like train stations or airports, ML can be trained on crowd movement patterns to identify panic, stampede, or violence tendencies, and the authorities can take suitable action well in advance.
- Geriatric Nursing and Healthcare: Home motion detection identifies abnormal movement (e.g., loss of balance or abnormal stand-to-sit transition) and automatically sends alerts to caregivers before a fall occurs.
By allowing machines to predict what we will do, not just are we making smart technology, but we’re bringing into the fabric of modern existence a new level of sensing security and expectation.