Benefits of Naturalistic Studies in Automotive Research
Whether designing new products or evaluating the competition, in an ideal world we would always discover exactly what our customers need. Like biologists in the field researchers would study customers “in their natural habitat” to gain critical user insights before making decisions that could either launch a new business successfully or cost millions from a failed product experience.
Unfortunately, it’s not always easy to do. Those challenges cause too many companies to avoid the cost and complexity of usability studies which can provide unique customer insights. Through advances in video capture, mobile internet and machine learning, Pulse Labs has made it more efficient and effective than ever before.
What Is Usability?
Usability is more than just how easy a product is to learn or use. We can think of it being influenced by four components: the user, the technology, the task, and the context. Of those four, the only one we can truly control is the technology that makes up our product. The user is independent and certainly beyond our control, though they are guided through the design and product interface. The task, what the user is trying to accomplish, may be something we’ve planned for, still we know each user will approach their own tasks differently. Finally, the context, meaning those elements like temperature, noise, distraction, lighting, make up the environment for a product that has been designed well for the lab but fails in the real world.
Naturalistic data, observing users in their personal environment as opposed to a simulated or lab setting, is critical to gain real understanding of their behavior. Only naturalistic studies, which test the intersection of the user, the technology, the task, and the context simultaneously can provide the insights needed for real innovation.
While this is important to your average mobile app or smart home device, with automobile usability this is even more important. Think for a minute about everything competing for a driver’s attention: passengers, traffic, road conditions, weather, fatigue, mobile phones, maps and infotainment systems. It's no wonder that human performance and behavior contribute to over 90% of crashes.
Automotive manufacturers understand this. In speaking with a major European OEM recently they described the innovation struggle as the industry accelerates and becomes more defined by a car’s software than its hardware. The “5 year marathon” in new model designs has been pushed into “2 year sprints”. Cars that didn’t change once they left the factory now receive major feature upgrades through over the air (OTA) software updates regularly. Consumers expect new features and experiences throughout their vehicle’s life. New interface technologies like voice and gestures that are replacing knobs and switches, mean consumers must navigate increasingly complex systems without becoming distracted.
Advanced in-vehicle infotainment and user assistants, thanks to high-speed mobile networks, add to the demand from usability researchers to truly visualize user behavior. Additionally, in-vehicle sensors and telematics connectivity mean that naturalistic user behavior can be matched to the exact state of the vehicle. By combining recorded vehicle data with video of the driver and passengers, and adding external data regarding traffic, weather, and other conditions, it’s possible to form a comprehensive view of the automotive user experience unlike anything previously available.
NHTSA and FHWA were key players in establishing the importance of studies incorporating Naturalistic Driving as a vital element of driver based data collection […] based on the recognition that Naturalistic Driving studies were the most effective and accurate means of obtaining relevant and meaningful data directly applicable to safety issues the Agencies were attempting to address [...] facilitated by the increasing prevalence of in-vehicle technology and NHTSA’s efforts directed at making significant improvements in techniques and technology for unobtrusively collecting data on driver behavior and performance in a naturalistic setting.
Meters, Labs, Simulations, and Surveys
Web and mobile applications have for years relied on metering user clicks and activity paths as a shortcut to understanding consumer behavior. Metering activity in this way can show some insights, especially when aggregated across hundreds or thousands of user sessions, but it lacks the ability to match interactions with emotions. No “big data” click path analysis can highlight whether a button was clicked in anticipation or frustration. When those emotions influence whether the driver maintains control at 60 MPH on a rain-slick road, the need for seeing real world interactions instead of click-streams is obvious.
Because automotive systems need to be tested before vehicles begin production, manufacturers rely on lab and simulation environments to mimic future vehicles and observe users. That’s certainly valid, especially in the earliest days of technologies that aren’t ready for deployment on the road. Simulators also offer the unique benefit of being able to repeat test conditions across a wide variety of users. Despite that there is still a shortage of research validating that simulator testing demonstrates behavior that transfers to on the road performance.
Even the most advanced simulation lacks the fidelity of the real world. Sun, rain, rough roads, and challenging traffic conditions can be approximated but never fully duplicated. The usability context, as discussed, means that many of the most common distractions such as passenger interaction, incoming calls or confusing road directions, fatigue, and emotional factors like traffic frustration or anxiety about reaching a destination, aren’t possible to recreate during lab testing. Using new VR technologies to attempt to create a high-resolution environment within the lab does not work for all subjects as many can be impacted by “simulator sickness” that will impact concentration and reaction time.
Surveys are another common tool where users are asked to gauge their satisfaction with a feature or product experience. Again, these can add value, but they are low fidelity data collection methods. Surveys often suffer from low customer click-thru and completion rates. The data can be biased due to the time lapse between the user’s experience and when the survey is delivered or completed. Survey design also means that at best you’ll only get answers to the questions you ask, not data on problems researchers don’t yet know exist.
Simulated and surveyed experiences have value but cannot provide the full view of users in the real world. To reach a complete understanding of the automotive user, naturalistic studies, like those provided by Pulse Labs, are essential.
So Why Aren’t More Naturalistic Studies Done?
If naturalistic studies produce the most comprehensive and accurate results, then why don’t usability researchers use them more frequently? One easy answer is cost. While some studies can be done with an observer in the vehicle, the best testing is done with camera outfitted cars that can capture user interactions with the vehicle as well as eye movement and attention.
Outfitting vehicles with cameras is also something many research teams aren’t equipped to do. While proper installation takes time, video analysis takes even longer. Historically videos captured during naturalistic studies had to be viewed at close to real time to pinpoint key user behaviors. That means one hundred hours of captured test video would take several workdays for the research team to sort through, then edit and put into a form that could be presented to product owners for discussion.
Limited usability test budgets mean that companies fall back to simulations and surveys instead of getting the best real-world data possible. Instead of learning from their customers, they get biased data and a best guess. That’s where Pulse Labs turns guesswork into real insight for leading automotive companies and Tier 1 suppliers.
How Pulse Labs Makes Automotive Naturalistic Studies Possible
The Pulse Labs platform addresses all these issues by making real-world naturalistic studies efficient and effective for automotive usability researchers. Only Pulse Labs supports integration of multiple camera feeds along with telematics data from the vehicle for a 360° view of the user, the technology, the task, and the context. Once captured Pulse Labs’ unique usability AI applies machine learning to scan videos to recognize key moments of human and machine interaction. This advanced feature speeds up video analysis and allows researchers to focus on learning and insight, not shifting through hours and hours of unimportant footage.
Integrating the qualitative, naturalistic data with quantitative user feedback is also a strength of the Pulse Labs platforms. Users can add their own feedback elements to the system’s identified highlights. This provides a powerful combination of observed behavior with real user commentary for the most complete test results possible.
The Pulse Labs platform is designed to work for both private and production panels. If researchers already have panelists identified or wish to limit observations to vehicles not available to the public, Pulse Labs can outfit private test fleets. For competitive analysis or testing feedback from OTA updates to existing vehicles Pulse Labs has complete panel recruitment and management capabilities.
Pulse Labs gives usability researchers the kinds of tools they need to test, analyze, and report on today's and tomorrow’s complex in-vehicle interfaces. In fact, these challenges become even more important as autonomous vehicles are introduced to our roads and freeways. Consumers will struggle to trust autonomous vehicles if the interface is unresponsive or overly complex. Autonomous vehicles can increase safety and remove imperfect human driving but only if occupants accept that the interface gives them proper feedback and sufficient control.
Partner with Pulse Labs Today
Great usability is critical to product acceptance and in today’s hyper-competitive world, brand loyalty. Consumers demand every product is as easy to use as the phone in their pocket. Without continuously testing your own vehicles, and your competitor’s products, it’s impossible to succeed. Vehicles that are easy to use and integrate the best features of voice, touch and tactile interfaces will not only improve driver performance and safety but increase long term brand loyalty as well. Partnering with Pulse Labs means real-world, naturalistic usability research can become a critical component for automotive industry success today and into the autonomous, connected, and electric future.