Beyond TÜV: A Path to High-Criticality Tool Qualification
August 16, 2022
Blog
Today’s software-based, safety-critical systems depend on certified software tools and processes for development. For many applications, tool qualification is a necessary step in ensuring the tool chain produces quality code to fulfill the needs of applicable safety standards. In many cases, the use of TÜV certified tools is sufficient but there’s an increasing number of very high-stakes applications where the functional safety standards demand more.
For critical applications where TÜV certification falls short, the ISO 26262 and DO-330 standards offer a viable path to qualification. By understanding how other standards fail to meet qualification expectations and exploring how ISO 26262 and DO-330 can be adapted beyond their original intended industries, development teams can plan a tool qualification process beyond TÜV.
Figure 1: Most functional safety standards are direct derivatives of IEC 61508 – part of the “IEC 61058 family”
The Process of Safety-Critical Tool Qualification
Tool qualification is the practice of ensuring the risk of a tool error impacting the safety of a system is acceptably low, either because the errors are few or because the function does not impact safety. From IEC 61508, “Functional safety of electrical/electronic/programmable electronic safety-related systems”:
“Examples include the deactivation of a medical infusion pump should it malfunction or the automatic activation of an overflow valve when a certain liquid or pressure level has been reached.”
Most functional safety standards define qualification processes to ensure potential errors are either avoided or detected in the application of the tool chain, but few provide any details for the most critical applications. While these standards have a common goal of building confidence and trust in the tool, there is little clarity on what development teams should actually do.
This presents a difficult challenge, as tool qualification is too much of a complex and time-consuming activity to start off on the wrong foot.
Figure 2: Classifying software such that the verification and validation activity is proportionate to risk
ISO 26262 and DO-330 Standards
ISO 26262, "Road vehicles – Functional safety", is a derivation of the functional safety standard IEC 61508 that’s specific to the automotive industry. For tool qualification, ISO 26262 requires a level of confidence that is dependent upon the circumstances of its deployment, as stated in §11.2:
“…the possibility that the malfunctioning software tool and its corresponding erroneous output can introduce or fail to detect errors in a safety-related item or element being developed; and the confidence in preventing or detecting such errors in its corresponding output.”
For most critical applications, this standard provides a path to dramatically reduce the overhead involved with qualification by allowing the evaluation of the tool based upon evidence of the application in a suitable software development process. One way that development teams take advantage of this is by using tools approved by a TÜV-certifying organization, such as the LDRA tool suite.
For the applications where standards demand more rigorous qualification in the context of the project development environment, vendors often supply tool qualification support packs (suites, packages, kits, etc.) Applied correctly, these packs can demonstrate whether the tool has been configured appropriately to provide the correct results in the tool chain and the environment in which it will be deployed.
Figure 3: Extracts from LDRA Tool Qualification Plan as provided in the DO-330 TQSP
RTCA/DO-330, “Software Tool Qualification Consideration,” was released to supplement the tool qualification requirements for airborne software development specified in the RTCA/DO‑178C standard, “Software Considerations in Airborne Systems and Equipment Certification.” Under the terms of DO-330, every project requires tool qualification regardless of the level of criticality.
The principles described in both standards can apply to other industries, and in fact, DO-330 explicitly states so in section §1.2:
“This document provides guidance for airborne and ground-based software. It may also be used by other domains, such as automotive, space, systems, electronic hardware, aeronautical databases, and safety assessment processes.”
Where Other Standards Fall Short
While industry-specific safety standards usually have demanding tool qualification requirements for the most critical applications, many do not provide specific details on how to meet them. Examples include:
- IEC 61508 – Perhaps the most surprising, since it forms the basis of the sector-specific ISO 26262, this standard has this statement in Part 3 §7.4.4.5, “Where such failure mechanisms are identified, appropriate mitigation measures shall be taken”, with no details on what these measures are.
- EN 50128, “Railway applications - Communication, signalling and processing systems” – has this statement in Section 6.7.4.2, “The justification shall include the identification of potential failures which can be injected into the tools output and the measures to avoid or handle such failures”, with no details on what these measures are.
- IEC 62304, “Medical device software - Software life cycle processes” – has multiple statements requiring tools verification with no specific instructions, including this one in §3.3, “confirmation through provision of objective evidence that specified requirements have been fulfilled”, with no guidance on what the evidence should be.
Adapting ISO 26262 and DO-330 for Other Domains
The reality is that development teams are responsible for figuring it out, by determining what constitutes adequate “assessment”, “justification”, and “verification” for standards certification. These choices aren’t easy to make, let alone defend, so it’s worthwhile to consider the case for using either ISO 26262 or DO-330 tool qualification processes for filling these gaps.
There is nothing about the qualification processes defined by either standard that makes them sector specific, and tool qualification support is available from a multitude of vendors for both. The ideal path is dependent on the characteristics of the project but it’s useful to compare five key similarities and differences between ISO 26262 and DO-330 to inform the best choice:
- Potential impact of errors: For both standards, assessing the tool confidence level must account for the potential impact of errors associated with methods and processes supported by the tool.
- Type of tool: Both standards differentiate between tools that can introduce an error into the system, such as a code generator, and tools that may fail to detect an error, such as test tools.
- Confidence requirements: Both standards classify confidence requirements. DO-330 expresses those as Criteria 1,2, and 3 while ISO 26262 expresses them as Tool Confidence Levels (TCL) 1,2, and 3.
- Alternative qualification methods: Both standards use tables to map software criticality to qualification methods but while DO-330 requires that test tools at risk of failing to detect errors follow its own qualification process, ISO 26262 permits the use of different qualification processes under certain circumstances.
- Test verification technique: The ISO 26262 tool qualification process is primarily one of reviews and analyses while DO-330 mandates on-target testing to demonstrate tool effectiveness and compliance for higher Design Assurance Levels (DAL).
Conclusion
Most functional safety standards require some form of project-specific tool qualification for high-stakes applications but only ISO 26262 and DO-330 offer details on how to go about doing it. Moreover, there’s nothing implicit about their defined qualification processes that restricts them to certain industries, making them ideal candidates for use in other sectors too.
Deciding between the two standards is a matter of understanding their common principles and where they differ to best fit the target application. Vendor qualification packs can streamline and reduce effort. A typical set of artifacts for a test tool would include test code appropriate for the functionality under test, expected results, and associated documentation to fulfil the objectives of the chosen standard.
