Because testing is a major cost and schedule driver, adequate planning is essential long before the start of any testing. Test planning between subcontractors, the prime contractor, and the government should start with program initiation. To ensure a successful integrated test program, close coordination is required between the government, the prime contractor, and all subcontractors.
DoD Directive 5000.3 requires the preparation of a Test and Evaluation Master Plan (TEMP). The TEMP is a broad plan relating test objectives to required system characteristics and critical issues, and is a top level document used at major milestone reviews to assess the adequacy of planned test and evaluation. The TEMP normally covers only government-required tests, and does not provide a sufficient level of detail to identify contractor and subcontractor tests. In an attempt to control the test program at the contractor and subcontractor level, contracts may contain requirements for the submittal of individual test plans for government approval. If an integrated test plan is not required, these individual test plans may not be reviewed for duplicate or missing test activities resulting in an inefficient and costly test program.
The prime contractor should be responsible for the preparation and updating of an Integrated Test Plan (ITP). To develop an efficient and well-coordinated integrated test program, the prime contractor and all subcontractors should jointly participate in the preparation of the ITP. The ITP should include all developmental tests to be performed by the prime contractor and all subcontractors at both the system and subsystem levels. The ITP should be a detailed working-level document which will aid in identifying risk as well as duplicate or missing test activities, and will provide for the most efficient use of test facilities and test resources. In developing the ITP, the purpose and time phasing of each individual test should be carefully examined. Unnecessary tests should be eliminated and test schedules should be adjusted to provided sufficient time for retest, should failure occur. The proper sequencing of tests is necessary to ensure completion of required lower-level subcontractor tests prior to the start of prime contractor tests.
During Development Test and Evaluation (DT&E), the contractor and the government normally conduct separate, dedicated tests,. In many instances these separate test periods result in redundant testing, testing which is not user oriented, lack of continuity in the contractor's development program, and a lack of cooperation between contractor and government personnel. In order to increase the efficiency of DT&E, the government should participate in some of the contractor's testing. This will help eliminate redundant testing, reduce the length of DT&E phases, provide more user-oriented test results, and result in a more mature system for Operational Test and Evaluation (OT&E).
Most test schedules are planned to support the major milestone reviews that occur during the development of a weapon system. The tests are planned to provide positive (successful) test results for presentation at the milestone reviews, in order to obtain approval for the project to proceed to the next milestone. This leads to a test philosophy in which passing tests is the main objective of the test program, rather than considering the engineering need for the test or the technical information provided by the test results. As a result of this philosophy, test schedules tend to be success-oriented, many times resulting in schedule slippage due to the need for retest or a lack of test assets.
As test programs progress, many tests will disclose a need for redesign and retest. In some instances, only a minor correction and verification test will be required. In other cases, the corrective actions many be extensive and require significant retest. If test schedules have not allowed sufficient time for redesign and retest, changes and retesting may be delayed until production equipment is available. If the changes prove incorrect and additional redesign is required, production units have to be retrofitted and a large number of Engineering Change Proposals (ECPs) may be required during the early phases of the production program. Also due to the sequential nature of some tests, the performance of certain tests may be delayed until production, possibly resulting in additional ECPs.
Test schedules should be properly phased primarily on the basis of engineering considerations, rather than strictly milestone-oriented. The purpose or objective of each test should be considered as well as the interrelation of various tests with each other. Since the start of certain tests may be dependent upon the completion of others, critical tests should be identified and provisions made for schedule slippage due to needed redesign and retest. In certain cases, critical test schedules can be accelerated by providing more test assets or additional test facilities. This strategy can provide significant leverage to reduce the overall development test schedule. Milestone reviews can then be planned on the basis of realistic test schedules. More engineer-oriented test results showing design strengths and weaknesses should be presented at design reviews. The review should discuss design weaknesses and how they have been or will be corrected. The overall success of a carefully integrated test program will result in a minimum of resources applied to testing and the elimination of a costly ECP or retrofit program during production.