The ISO 26262 defines the fault injection test as a test method for the system integration and unit test level (ISO 26262-4 [System] Tables 5, 8, 10, 13, 15, 18; ISO 26262-5 [Hardware] Table 11; ISO 26262-6 [software] tables 10, 13).
This method has certainly a large part in the implementation of a possible error-free and therefore safe system. My focus in this blog is on an efficient implementation of this test method in practice. I will compare practices in the aerospace with those in the automotive industry.
Die ISO 26262-4 [System] beschreibt den Fault Injection Test wie folgt:
The test uses special means to introduce faults into the item. This can be done within the item via a special test interface or specially prepared elements or communication devices. The method is often used to improve the test coverage of the safety requirements, because during normal operation safety mechanisms are not invoked.
The ISO 26262-4 [System] defines the fault injection test as follows: Fault Injection Test includes injection of arbitrary faults in order to test safety mechanisms (e.g. corrupting software or hardware components, corrupting values of variables, by introducing code mutations, or by corrupting values of CPU registers).
The following points are of particular importance for the subsequent considerations:
1) Requirements based testing and fault injection tests are treated as two (different?) test
2) The ISO 26262 assumes that you need the fault injection tests to increase the test coverage of Safety Requirements.
3) Within the fault injection tests arbitrary errors are introduced into the system to proof the safety mechanisms that have been encoded in the software to examine.
Now let’s take a quick look in the aerospace industry. How does this industry deal with the issue Fault Injection Test?
In the aerospace the standards DO-178C [Software], DO-254 [Hardware] and ARP 4754 [System] represent the comparable content of ISO 26262.
The term Fault Injection Test is not used in the aerospace standards. Instead the terms “Normal Range Test Cases” and “Robustness Test Cases” are representing all aspects of the testing. The DO-178C combines these two terms with the requirements-based testing. The following two questions are central in every safety-critical aerospace project:
1) Are the Requirements verified with Normal Range Test Cases?
2) Is the Source Code robust against High-Level Requirements?
In the aerospace industry, the requirements play a very important role. It is required that the entire source code is covered by appropriate requirements. This includes the code for the behavior of the system in case of failure. In relation to safety aspects, the correct functioning of this source code part has to be proven.
The requirements will be tested on normal range and Robustness Test Cases. Whereby the Robustness Test Cases on the Requirements aimed itself to question and possibly to find gaps in the requirements. The importance of the robustness testing is indisputable, but it is also important to recognize that the determination of whether an appropriate test “Passed” or “Failed”, is not possible any more in a lot of cases. In not a few cases, namely lack the expected system behaviour. Ideally, testers and system specialists together discuss the problematic cases. The result of the discussion is then always a determination of the correct system behaviour. A documentation of the results, is the definition / modification of requirements.
If you follow this procedure, this means that the relevant Robustness Test Cases also verify Requirements.
If one follows the logic in the aerospace industry, then fault injection tests contain both Normal range tests and partly Robustness tests. In any case, there is a corresponding requirement for each test.
If we take now the aerospace view and interpret the ISO 26262, with respect to the Fault Injection Test then some exciting result appear:
- A clear test result, including test results of fault injection tests, you can get only if the expected result is known. Requirements define this expected result. If you follow this logic, the following question arises: Where is the difference between a fault injection test and a requirements-based test in the ISO 26262? The fault injection testing is then “just” a subset of requirements-based testing.
- If fault injection tests are not available, there is definitely a lack of test coverage of Safety Requirements. Each Safety Requirement needs to be 100% verified. From this point of view, fault injection tests are not anything special.
- From the perspective of aerospace also “arbitrary” errors result in a system always in a defined system behavior. In this definition “arbitrary” equated with “a mistake from a defined (= known) list of errors”.
The fault injection test at the end simply represents a subset of all requirements-based tests. Nevertheless it is an important subset! Experienced Tester are needed to get good Fault Injection Test and equally good test environments are to be provided for this kind of tests.
In the automotive industry, the practical application of requirements engineering is a discipline that has been rolled out only in recent years. In many current projects which do have an ASIL classification, the number of requirements is still not very high. The ISO 26262, for sure, also had this pragmatic aspect in view, when the fault injection test was defined. The fewer requirements there are the more sense makes the definition of such a test method.
Another aspect is that fault injection testing can be used to validate the safety mechanisms in a early project phase. The arguments in the aerospace industry are strongly influenced by the need to proof the compliance to the certification authority. Often the early stages of projects tend to come a bit too short.
In summary, it can be said that fault injection tests represent an important test method. Best advantage of this method can be made, if requirements are available for the expected system or software behaviour.
I’ll be glad to help you also with any specific questions about your project . Send an email to: martin.heininger [at] heicon-ulm.de
An overview of the services can also be found on the HEICON Homepage.