||TR 1-99: DoD Acquisition Management Metrics
Chapter 3: Methodology
Concept and Approach
As an underlying concept
to this study, we used only official data available to DoD principals and staff
at the Defense Acquisition Board (DAB) Milestone (MS) II and III
meetings. The ratio-nale was that acquisition decisions were based largely on
this data, whether it was complete or incomplete, clear or hazy, right or
wrong, with regard to estimates stated. We therefore limited our data sources to
the program SARs for the years the program was in EMD, and the Initial Operational
Test and Evaluation (IOT&E) reports of the Service OTAs and the Beyond Low
Rate Initial production (BLRIP) independent evaluations issued by the
The current research is a follow-on effort to the original effort. So
that newer data would be compatible with the original data, we felt the original
criteria for cost, schedule and performance success must be maintained.
Experience has shown us no problems have occurred using this approach.
For purposes of this research study, the success of a DoD acquisition
program will be cat-egorized in one of five ways. Either the program is Very
Successful, Successful, Fairly Successful, Marginally Successful, or Not
Successful. The subjective description of these categories follows. For the
original report, only the portion of the definition relating to cost or schedule
overruns apply. The remainder of the definitions apply to the current follow-on
Very Successful (Score of 5)
There are few, if any, system shortcomings. The MS II program
budget and program schedules were essentially adhered to.
The DOT&E MS III BLRIP report was positive. The Service IOT&E/OPEVAL report
was positive, namely effective and suitable without caveat. (If not suitable,
the deficiencies could be corrected without major impactó i.e., no SAR breach.)
Successful (Score of 4)
The ADM from MS
II and MS III DABs was straightforward. There were system shortcom-ings. The MS
II program budget and schedule were slipped, but not by more than 30 percent in
cost and 12 months in schedule.
The DOT&E MS III BLRIP
report was positive. The Service IOT&E/OPEVAL (Operational Evaluation) report was
positive. The overall evaluation was effective and suitable, with perhaps a few
marginally suitable parameters.
Fairly Successful (Score
The ADM from the MS II and MS III DABs contained problem
statements. The program shortcomings were listed; a few could be critical. The
MS II program budget and schedule had to be revised, but were within 45
percent of the MS II budget and no more than 18 months behind the MS II schedule.
The DOT&E MS III BLRIP report contained
a few negative comments. The Service IOT&E/OPEVAL report could be marginally
effective and marginally suitable.
(Score of 2)
The ADM from the MS II and MS III DABs indicated
major performance and suitability prob-lems existed. The program probably would
be canceled on the basis of performance to date, but other external factors are
being considered. The MS II program budget and schedule was revised more than
once, and is now up to 60 percent overrun in cost and two years behind the
The exit criteria of the MS II ADM were not
completely met. An outcome of the MS III DAB would be to delay entry into full
The DOT&E MS III BLRIP
report was marginally effective and/or marginally suitable. The Service IOT&E/OPEVAL
recommended, at best, the system was potentially effective and potentially
Not Successful (Score of 1)
The ADM from the MS II DAB reluctantly
approved the continuation of the program into EMD, or held the program in
the Demonstration/Validation phase. The MS II budget is over 60 percent overrun, and the
program is more than 2 years behind schedule. A DOT&E BLRIP report will say it is
not effective and not suitable. This category would also include programs that
have in fact been terminated.
For programs that have not had their MS
III DAB review as yet, their success will be judged on the general approach
discussed herein, and on the available official documentation.
and Schedule criteria are objective; they use, in general, the standard
DoD decrements for SAR reporting: 15 percent in cost growth and 6 months in
schedule slippage. Performance evaluation is subjective, but controlled. Originally
all performance success ratings were assigned by one researcher. Most recently,
the performance success rating is to be assigned by a three-member
Subject-Matter Expert (SME) Panel of T&E Department faculty members in accordance with
established rating criteria and initial standardization procedures.
A standardized approach to present the data has been established. The
major division in presenting the data is general management metrics followed
by operational test metrics. Examples of the former include cost and
schedule overruns and of the latter, comparisons between OTA and DOT&E evaluations.
Next within each category we list the overall metric for the years covered
(1980-1996). Then the first sub-set of the metric is by year group (1980-1988,
1989-1992, 1993-1996), and then by individual Service.
In reviewing this
data, particularly bar graphs, the reader is reminded that overrun data is the
inverse of success data. For example, a bar graph showing a large overrun would
be relatively tall. The associated success rating for this cost or schedule
parameter would be relatively short.
The database is maintained on an Excel
spreadsheet with columns representing parameters recorded, and rows
representing the number of programs entered to date. Currently about 8-12 new
programs are added each year. The first 12 columns are called primary data,
that is, the identification of
programs and their EMD cost, schedule, and performance results. The remaining
columns are expanded data, required to arrive at the program success ratings.
Taxonomy of weapon system types was created for use within the study and
||Electronic-Electronic Warfare-Airborne |
|| Electronic-Comm/Nav/Radar-Airborne |
|| Electronic-Comm/Nav/Radar-Ground |
|| Missile/Munition-Airborne |
|| Missile/Munition-Ground |
Consistent with the idea that all data should be that which are available
to OSD-level decision makers, cost data for all systems were collected from
the system SARs. As the period of interest was EMD, the RDT&E and Procurement
costs were collected for the period from MS II through MS III. Planned costs
were those reported in the SAR following MS II; actual costs were those reported
in the SAR following MS III.
In theory, one could go to the program SARs
and retrieve, rather directly, the planned and actual cost of EMD. In practice,
the retrieval of costs from the SARs is not simple. For one thing, costs are
reported in several places in the SAR: for example, sections 11, 13, 14 and 16.
For another, costs are reported in both Base Year and in Then Year dollars; the
inflators for each appropriation are not always explicit. And yet another, the
costs are reported by fiscal year; EMD need not start or end in conjunction with
fiscal years. This leaves the analyst with the problem of determining how much
of a particular fiscal year's costs should be allocated to EMD for the database.
Base Year vs. Then Year . In principle, the use of Base Year figures will
remove the effect of inflation on the figures, making it easier to perform
analyses. In the case of the EMD database, the EMD costs are used as a ratio of
planned to actual. For most programs, the same inflators will be used in both
sets of figures, thereby yielding the same results from either Base Year or Then
Year figures. For those programs in which actual EMD fiscal years were
significantly longer than planned, when inflation might have had an effect on
the ratios, use of either figure will still identify a significant overrun. In
addition, several programs changed their Base Year for SAR-reporting during EMD.
This creates additional difficulties in using Base Year figures for the
database. For all the above reasons, this analysis used Then Year costs.
SAR Section . Sections 11, 13 and 14 do not separate costs by fiscal year;
it is not possible to determine costs for EMD alone from these figures.
Section 16 is the sole portion of the SAR which records costs by fiscal year. Start
of EMD (MS II), found in section 9, determines the earliest fiscal year to
use. If MS II is in the middle of a fiscal year, costs can be prorated month
for month. MS III determines the end of EMD. If MS III is in the middle of
a fiscal year, the end of IOT&E can be used to determine the end of EMD
expenditures. With the above adjustments, section 16 figures were used for
the SAR cost data.
RDT&E Costs . For some programs
it is difficult to determine what RDT&E costs during the period of EMD
actually were incurred in development of the program. In particular, when
the program includes several models, or includes preplanned product
improvements, the program can incur RDT&E costs supporting these others projects during
the period that the basic development is in EMD. An excellent example of this
is the Joint Tactical Information Distribution System (JTIDS) program, in
which class 2H/2M development was initiated before class 2 completed MS III. SAR data can not
isolate the costs of the basic program, except in those cases where the program
office identified the parts of the project separately in the SAR. Additionally,
for programs with preplanned product improvements, total cost figures (such as
are in sections 11, 13 or 14) will not define the basic development. In those
instances where cost definition was not clear, best estimates of allocation to
the basic development, year by year, were made.
Procurement Costs .
The SARs do not separately identify the cost of LRIP production. The
best approximation is to use all procurement costs for those years before MS
III; this is predicated on the legal constraint against using full-rate
production funds before that decision. Procurement costs also have the same
separation problem as discussed for RDT&E above, for programs with several models; the
same solution was used.
RDT&E vs. Procurement
. There remains a significant problem in determining the costs
to be used in calculating EMD Cost Overrun. For any program with more than
a few LRIP items, the Procurement costs in EMD will be significantly
(in some cases overwhelmingly) larger than RDT&E costs. This results in the program
being scored for EMD Cost Overrun predominately by Procurement costs. The problem
is that a large number of LRIP items may indicate a very bad program (they
need to continue LRIP to keep the line warm while the program keeps trying
to pass IOT&E, for example). A large number of LRIP items may also indicate
a very good program (testing in EMD has been so successful the services demand
the item in the field before MS III, for example). Thus,
overwhelming RDT&E with Procurement EMD Costs produces equivocal data. For this reason,
EMD costs for RDT&E and Procurement were entered separately in the
database, with RDT&E costs alone being used for the EMD Cost Overrun
calculation. Analysis of Procurement costs may still be performed, if desired, since
the data are present in the database. It should be noted that a
sensitivity analysis was performed on the data from the systems reported in 1995.
The result was that use of RDT&E costs alone produced the same EMD Cost
Success factor, within a score 1 more or less, in 13 out of 14 programs (see Appendix