||Testing for Produc: Testing for Production
Executing The Reliability Growth Test
The reliability growth test supplements the design reliability effort but does not take its place. Similarly, it is too expensive to be a quality screen. Test articles should be constructed with quality parts (rescreened for 100 ppm or less defect rate). Equipment should be subjected to environmental stress screening, when appropriate, prior to reliability growth testing to find and eliminate workmanship defects.
Test Reviews: Test Readiness Review. A test readiness review should be held approximately one week prior to the start of the reliability growth test to ensure all test items and supporting elements are ready to begin the test. The review should include the following as a minimum:
- results of latest reliability predictions
- status of design
- results of previous tests
- review of all open problems and failures
- availability of approved test procedures
- readiness status of test equipment and test chambers
Test Reviews: Monthly Status Reviews. After the start of the reliability growth testing, monthly tracking meetings should be held to review test progress, addressing at a minimum:
- current reliability growth assessments and projections based on test results and illustrated by reliability growth plots
- open failures and schedule for correction
- results of failure analyses and corrective action recommendations
- effective of previously implemented corrective actions
- results of other tests which might affect reliability growth hardware
Test Reviews: Weekly Status Reviews. After the start of reliability growth testing, test progress should be briefed during Weekly Status Reviews during Program Manger's meetings.
Test Reviews: Special Status Reviews. When the reliability growth plot shows a growth rate less than the minimum required, Special Status Reviews should be held to discuss options for correcting this situation, including the possibilities of more aggressive failure analysis and corrective action or suspension of the reliability growth test and a major redesign effort.
Failure Actions. Upon the occurrence of a failure, the equipment under test should be repaired and placed back on test. The failed parts, circuit card, etc. can then be analyzed off-line without adversely affecting the test schedule. When ready, corrective actions should be taken on all equipment under test as logical block changes which must be tested adequately to ensure the original failure mode was corrected and no new failure modes introduced.
ASSESSING RELIABILITY GROWTH
Because of the relatively formal control associated with reliability growth testing, it is better suited to quantification than other elements of the TAAF process, even though demonstrating reliability achievement is not the goal of reliability growth testing. Quantitative questions such as "Where are we?" (currently achieved reliability) and "How rapidly are we progressing?" (growth rate) are outcomes of reliability growth testing. It is important to understand:
- the most important models for growth,
- limits associated with interpretation of data, and
- the kinds of decisions that the results of growth assessment can support.
Models for Growth. The only direct, model-independent measure of current reliability is the lapsed time between the two most recent failures, but the uncertainty associated with a sample of one is so large as to make the measure worthless. Therefore, it is necessary to take the accumulated in reliability growth testing into account, which in turn requires choosing a model for the growth process.
Both the AMSAA and the Duane methods model reliability growth as a power function of test time. The Duane model, with its heavy reliance on graphic techniques, is well suited for quick (and "dirty") analyses and for detection of discontinuities in the growth process. The AMSAA model is essential for determination of confidence bounds and for objectivity in parameter estimation. It should always be used in connection with contractual quantification requirements.
Although both models imply continuity of growth, which
requires that testing be suspended after each failure until the corresponding
fix has been implemented in all test items, discontinuous growth is the norm
in the real world.
Although such discontinuities interrupt the smooth theoretical progress of reliability growth versus test time, it is common - and reasonable - practice to make estimates as if a smooth power function were the appropriate model unless there is clear evidence that test phases need to be analyzed separately.
Limits Associated with Interpretation of Data. Reliability growth produces inherently very "noisy" test data, even under ideal conditions. For instance, after five failures, 10% of the MTBF estimates can exceed the true value by factors greater than 2.6, and 10% at less than one half of the true value. Even at the thirtieth failure (implying a fairly lengthy test) the factors can be approximately 1.4 and 0.7 respectively.
Growth rates are also difficult to measure accurately. They can be in error by a factor of three or more in either direction for reasonable test times. It is possible for the growth to appear as negative or worsening, when the true trend is positive, and vice versa. The possibility for errors varies with growth rate, being relatively much smaller for higher growth rates. Thus, estimated reliability and estimated growth aid your judgment - but they do not substitute for it.
Types of Decisions. The types of decisions you may need to make for which growth assessment is an aid include:
- Early termination of reliability growth testing on the basis of compelling evidence (on the growth curve) that reliability objectives have been reached early.
- Early termination of reliability growth testing when quantitative and qualitative results (i.e., types and patterns of failures) indicate a need for major redesign (followed by a TAAF restart) rather than a series of fixes.
- Extension of reliability growth testing when there is both substantial doubt the objectives have been reached and confidence that growth will continue (e.g., when there is evidence of growth, but at a lower rate than planned).
- Extension of reliability growth testing when additional time is needed to verify the effectiveness of the most recent fixes.
- Test multiple, identical units in parallel to increase the opportunity of identifying failure modes.
For more information on the topics above, see
The Tri-Service Technical Brief for TAAF Implementation (January 1989)
MIL-STD-785 Task 102, Task 103, Task 104
MIL-STD-781D Task 103, Task 202
MIL-HDBK 781 Section 4.3