A test plan must identify defects and reflect real-world complexity, including its unpredictability
Vehicles are complicated systems operating in complex environments, requiring more than a single philosophical approach to testing. This technical evaluation requires a comprehensive testing strategy, focusing on five key approaches: compliance, combinatorial, stochastic, extreme and attack testing. Each method targets different types of defects and failures.
Effective testing will consider the product (hardware/software), the application, the environment and cybersecurity. The approaches we choose will depend on the associated risks and available resources.
Compliance testing is crucial to product development and quality assurance. It ensures that a product adheres to specified standards, regulations or requirements (often called specifications). Standards and regulations can originate from outside the company (for example, government or industry bodies such as SAE International), while product requirements and specifications often come from the customer or development entity.
The testing relies on documented requirements – clearly defined standards or regulations the product must adhere to. It is essential to identify which standards are applicable, including specific sections or paragraphs that apply to the product. This must be coupled with access to relevant compliance documentation, such as industry standards or legal regulations (FMVSS 101, for example).
The vehicle is not subjected to one single stimulus at any given time but to myriad parameters and the magnitude of those inputs (for example, elevated temperature while under a specific vibration profile). We use two- and three-factor variable combinatorial testing, and this will impact the test planning.
Combinatorial testing involves systematically exploring the interactions between different factors to identify potential issues and ensure robust product performance. Factors in technical products include hardware components, software configurations, environmental conditions and user behaviors. Combinatorial testing is particularly valuable when dealing with complex systems in which numerous variables can influence product behavior.
The real world is random, and it is advantageous to replicate this randomness in product development. To do so, we use stochastic testing, which is rooted in probability theory and introduces an element of randomness into the testing process. This is especially important when testing software or embedded products as, from experience, the sequence of events to which the product is subjected can evoke latent failures.
Although it is true that we can never imagine and test every possibility, not exploring the product’s viability in the face of this randomness is analogous to putting one’s head in the sand. Automation can help replicate randomness in testing.
Exploratory testing is a hands-on, unscripted approach in which testers actively explore the application, leveraging their domain knowledge and intuition. Instead of predefined test cases, testers navigate the system, identifying defects and gaining a deeper understanding of their intricacies.
The test engineer’s creativity is important here because some of the tests may not be scripted but instead rely on the engineer’s intuition and understanding of how customers will use the product, and interactions within the system. One of the primary limitations to product specifications is that it is impossible to articulate all desired behaviors and all the unwanted product behaviors in a timely document.
Vehicles have databuses and Bluetooth elements, which create possibilities for manipulation. It has probably been more than a decade since I heard of a vehicle being hacked and the ABS manipulated. Attack mode testing goes beyond conventional testing and into the security realm by actively emulating the tactics of malicious actors. This involves ethical hacking, penetration testing and vulnerability assessments to identify and rectify potential weaknesses in vehicle systems.
Experience suggests it is easy to fall into the trap of focusing or limiting product testing to requirements- or compliance-based approaches. There are times when this is the necessary approach, but on its own it will lead to field failures and customer disruption that would have otherwise been predictable. Simply understanding the workings of the product in a benign environment tells us nothing about it, relegating this learning to post-production, where it comes at a higher cost through warranty expenses, customer dissatisfaction and litigation. It is time to think beyond a single testing approach and be guided by what is at stake (risk), and the time and talent available to explore and learn.
Jon M Quigley is a seasoned automotive test engineer with over 20 years of experience at both suppliers and OEMs, including more than a decade each at Volvo and Volvo Trucks. He has also authored nearly 20 books on product development and management
