Shift-left your safety-critical software testing with test automation
September 05, 2017
The trend in major safety-critical projects shows exponential growth in total costs, with software being a larger portion of the total year-over-year development budget.
We are in a cost crisis with safety-critical software. The increased functionality that’s now required has grown beyond the ability to pay for its development. For example, the Boeing 787 program required 6.5 million lines of code, which cost $4 billion to design, develop, and test.
The trend in major safety-critical projects shows exponential growth in total costs, with software being a larger portion of the total year-over-year development budget. The next big aerospace program will likely be unaffordable using the same techniques used in previous programs. So, what can we do?
Safety certification and the required test and verification required is a large portion of the software development budget. Beginning software test earlier in the development lifecycle while leveraging automation pays big dividends in cost, risk, and schedule, as seen in Figure 1.
[Figure 1 | Cost of software development per lines of code in commercial aviation projects]
Where bugs are created and found, and the cost impact
Unsurprisingly, most defects are introduced at the beginning of a project, even before the first line of code is written. Most bugs are found and fixed during testing, but a good percentage (as much as 20 percent) are discovered during operation, after the product has shipped. In certified systems, this either means an extremely expensive fix-test-re-certify cycle or operator workarounds for the problem (Figure 2).
[Figure 2 | The percentage of defects introduced and detected during different phases of development]
Defects become exponentially more expensive to find and fix over the length of the project. Worst case is fixing defects after a product has shipped to customers. Post-deployment defect fix costs are conservative and don’t include damage to your brand and liability from safety incidents in the field (Figure 3). Clearly, the goal is to move the defects detected and fixed to earlier in the lifecycle – in other words, to “shift left.” In addition, it’s desirable to reduce the number of defects making it through to the customer (a reality in every domain).
[Figure 3 | Relative cost to find and fix bugs during each development phase. During requirements and design is the baseline (1x) and the least costly place to fix defects.]
The role of test automation in shift-left
The safety-critical software industry recognizes the need to change the way things are done. Too many projects try to reinvent the wheel, and certifying new software is time-consuming and expensive.
The growth in connectivity and functionality of new products means methods need to change. A huge part of any safety-critical project is testing, and automation is absolutely necessary to achieve safety, security, and quality goals. Here’s an example of how test automation tools support new software development methods and increase testing and documentation productivity:
Supporting agile and iterative development methods: The problems with the waterfall method are understood and many teams use more modern development methods to improve quality and safety. Test automation is an important part of any iterative development method, since test suites are run on each new iteration of a module, component, etc. Test automation supports these methods with repeatable automated tests, providing reporting at various levels for each test but also accumulative results over time. Dynamic analysis tools are critical to detecting runtime errors that are difficult to detect, and static analysis plays an important role in detecting defects before testing starts.
Supporting software inspections: Inspections are a good practice to remove defects early in the development lifecycle. Inspections mean reviewing everything, not just source code. For example, inspecting requirements and design is critical in preventing the leading source of bugs in the system (Figure 2, again). Many bugs are literally designed into the system. Tools play a lesser role at this stage, but do enhance the effectiveness of code reviews. Automated unit testing, dynamic error detection, and static analysis provide greatly-improved error detection in a project’s early coding stages. The results of automated testing can be presented in code reviews, thus decreasing the dependence on manual error detection and allowing more time for detecting incorrect requirements and design decisions.
Increasing testing productivity: Manual testing is tedious and less repeatable. Results collection can be ad hoc and errors can be missed despite “correct” results. Achieving required code coverage, which varies based on safety standards and criticality of the project, is difficult to track. Test automation not only makes testing less tedious and repeatable, but the reporting capabilities of advanced testing tools create important management information on the state of the project. Adding dynamic analysis, which analyzes the code when it’s running (to detect tricky runtime errors), and static analysis, which analyzes the code before it runs, increases the test tools’ bug detection capability.
Automating coding standard compliance: Many safety-critical projects require source code standards. MISRA, for example, is common in automotive software but has gained acceptance in other industries. Some standards require that code comply to a company standard that meets certain goals. In each case, manually enforcing code compliance is tedious and error prone. Static analysis tools can enforce compliance and advanced tools go further by detecting errors that are beyond format violations.
Automating certification documentation: A large portion of the workload in achieving software safety certifications is in documenting processes, validation, and verification. Test automation decreases the cost of documenting test results and coverage analysis significantly.
Accelerating re-use of third-party software: A key strategy to increase productivity is software re-use. Ideally, already-certified components can be used to reduce the development costs for these subunits. Projects need to make use of COTS (commercial off the shelf) software and possibly open source and other source code. Automating the evaluation of this software with static and dynamic analysis tools decreases the risk of using these components.
Improving quality, safety, and security: Even strict testing regimens can miss critical errors. Code coverage alone isn’t enough to ensure proper behavior in the case of security attacks or multithreaded code, for example. Static analysis tools can detect errors in source code without running a specific test and can find bugs such as security vulnerabilities that are hard to discover in unit or system testing. Dynamic analysis tools can detect errors in running code during tests that might be reflected in test results (e.g., a slow memory leak). Fuzzing and penetration testing during system testing can find bugs missed during normal operating conditions. In totality, the additional defects and security vulnerabilities found by state-of-the-art tools help reduce cost, risk, and many of the 20% or so bugs that make it into production.
What affect does shift-left have?
It’s clear that something must be done about solving the issues so clearly shown in Figure 2. Too many defects are being introduced, and staying undetected, in the beginning of the lifecycle, and too many are left in the product when it’s in customers’ hands. Adopting new development methods, reusing components, leveraging COTS and open source, and tool automation are all key steps to improving development productivity.
[Figure 4 | The graph shows an improved development process that shifts the detection of bugs and security vulnerabilities earlier in the lifecycle.]
We know from Figure 3 that costs rise significantly at each development phase. Figure 5 shows the comparison of costs to fix defects in the traditional method versus the method shown in Figure 4. Finding and fixing bugs earlier, unsurprisingly, costs less than fixing them later. In the situation presented here, the overall cost difference is about 40% in favor of the shift-left approach.
[Figure 5. The graph shows the relative cost to fix bugs in the traditional approach versus the shift left approach. Even with the same number of total defects, early detection reduces costs significantly.]
The importance of certified tool chains and qualification assistance
Using automated tools in safety-critical projects requires trust in the tools themselves. The onus is on the product manufacturer to have confidence that the processes and tools used to create the software meet the standard requirements. Tool vendors can help by having tools certified by safety standards bodies before sale to the manufacturers or, in cases where no such pre-certification is possible, provide qualification assistance. They can then use the tool vendor’s certification evidence in their own certification submission and reduce the effort needed. For example, Parasoft C/C++test has been certified by TÜV SÜD to be qualified for safety-related software development according to IEC 61508 and ISO 26262 standards.
In some software safety standards, such as DO-178B and DO-178C, certification is done on a system level and individual tools and software aren’t certified independently. In these cases, the tools vendor provides qualification kits and assistance in terms of documentation and professional services, greatly reducing the cost and effort needed to qualify the tools for use on the project.
