A team of autonomous driving researchers at the Massachusetts Institute of Technology is developing systems to perceive and identify nearby objects and to understand social interactions in traffic, as part of a new research initiative with Toyota.

The MIT AgeLab will build and analyze new perception and motion-planning technologies for automated vehicles in partnership with the Toyota Collaborative Safety Research Center (CSRC), according to MIT. The new initiative, called CSRC Next, is part of a five-year-old ongoing relationship with Toyota. MIT AgeLab is a unit within the MIT Center for Transportation and Logistics.

Bryan Reimer, a research scientist at MIT AgeLab, has been leading the first phase of projects with Toyota CSRC. He manages a multidisciplinary research team focused on understanding how drivers respond to the growing complexity of the modern operating environment, MIT explained in a released statement.

Reimer and his team studied the demands of modern in-vehicle voice interfaces and found that they draw drivers’ eyes away from the road to a greater extent than expected. His study eventually contributed to the instrumentation redesign in the current Toyota Corolla and the upcoming 2018 Toyota Camry.

Reimer and his team are also developing and building prototypes of hardware and software systems that can be integrated into vehicles to detect everything about the state of the driver and the external environment. These prototypes are designed to work with cars with minimal levels of autonomy and with cars that are fully autonomous, MIT said. 

Computer scientist and team member Lex Fridman is leading a group of seven computer engineers who are working on computer vision, deep learning, and planning algorithms for semi-autonomous vehicles. The application of deep learning is used for understanding both the world around the car and the human behavior inside it.

“The vehicle must first gain awareness of all entities in the driving scene, including pedestrians, cyclists, cars, traffic signals, and road markings,” Fridman said. “We use a learning-based approach for this perception task and also for the subsequent task of planning a safe trajectory around those entities.”

Fridman and his team, now engaged in the next phase of the project with Toyota CRSC, set up a stationary camera at a busy intersection on the MIT campus to automatically detect the micro-movements of pedestrians as they make decisions about crossing the street. Using deep learning and computer vision methods, the system automatically converts the raw video footage into millisecond-level estimations of each pedestrian’s body position. The program has analyzed the head, arm, feet and full-body movement of more than 100,000 pedestrians. 

Fridman’s research also focuses on the world inside the car. 

“Just as interesting and complex is the integration of data inside the car to improve our understanding of automated systems and enhance their capability to support the driver,” Fridman explained. “This includes everything about the driver’s face, head position, emotion, drowsiness, attentiveness, and body language.” 

With Toyota and other partners, the team is exploring the use of cameras positioned to monitor the driver, as well as methods to extract all those driver state factors from the raw video and turn them into useable data.

“What’s innovative about Lex’s work is that it uses state-of-the-art methods in computer science and artificial intelligence to study the complexities of human intent grounded in large-scale, real-world data,” Reimer said.

Toyota CSRC Director Chuck Gulash said the research “leverages the AgeLab’s expertise in computer vision, state detection, naturalistic data collection and deep learning to focus on the challenges and opportunities of autonomous vehicle technologies.”

When asked how the research collaboration would affect future automotive technology, Gulash said it will contribute to better computer-based perception of a vehicle’s environment as well as social interactions with other road users.

“What is unique about the AgeLab’s work is that it brings together advanced computer science with a human-centered perspective on driver behavior,” Gulash said. “As with all CSRC projects, output from the AgeLab’s effort will be openly shared with industry, academia and government to contribute to future safe mobility.”