By definition, this study concerns
itself with the Engineering and Manufacturing Development (EMD) phase
exclusively, since that is where LRIP resides.
The data for this
study were restricted to that contained within the Selected Acquisition Report
(SAR) for the years in which the program was in EMD, and the IOT&E and BLRIP
operational test reports. This was an early, deliberate design decision. The
rationale was we wanted to use only the data available to the decision makers
at the time the decision was made.
In some instances, the data within
the SARs were incomplete and in some cases the type of data contained within
the SARs varied between programs. This is not considered unusual given the
elapsed time since some of the programs had their DAB meetings and the
dynamics of the process.
SAR reports are required for
Acquisition Category I (ACAT I) programs only, whereas the programs within the
database are for all programs that have completed EMD and are on the DOT&E
Oversight List. Therefore it was not possible to obtain cost and schedule data
for some programs.
Even for programs with SARs it was not
always easy, or even possible, to extract cost data for the program which was
initiated at MS II. Several programs were changed significantly during EMD, to
the point that the cost figures for the initial program were lost in the
actual reported costs. One example is the JTIDS, which shows the difficulties
in abstracting cost data from the SARs of programs which have
undergone significant restructuring. As initiated at MS II in 1981, this was a joint Army/Air
Force demonstration program for class 2 terminals only, with a follow-on if
the demonstration was successful. The Navy had its own program. In 1985 the
Navy was directed to use the Air Force terminals. Also in 1985, the Army
redirected its program to develop 2M (data only) terminals. In 1986 the Air Force
added 2H terminals, initially for the E-3A only, with other platforms to be
added later. One result of all this restructuring was that MS IIIA in 1989 was
only for Navy/Air Force class 2 terminals. Throughout EMD the SAR cost data were
revised and reordered to reflect these programmatic decisions. Comparing
planned costs for EMD to actual costs, as well as comparing planned and actual
schedule information, using only the SAR, was virtually impossible under these
circumstances. In some cases a best-effort set of figures was derived; in
others, no reasonably comparable set of figures was possible and no data were
The degree to which software must be developed for a system
is another important variable, which would affect program success. We had
planned to include this as a variable, but discovered that the SARs contained
very little data in this regard.
Program concurrency— the overlapping
of program development and program production phases— is a concern. This study
generated little data relevant to concurrency; although again, program
concurrency intuitively would have a major impact on program EMD success.
The current research may be constrained by the decision to use the
original research methodology. The reason for this decision has been explained
The basic intent of this research is to estimate the degree of
accuracy by program managers between their planned and their actual cost,
schedule, and performance figures. External influences, not necessarily under
the control of the program manager, would confound this analysis. Probably
some degree of external influence, usually attributed to Congress, does exist.
J. Ronald Fox in his 1988 13 book listed other potential external inputs, such
as: "… the DoD often sets optimistically low cost estimates for proposed
programs…. Only much later does it become clear that these programs will
require more money than originally estimated" 14 or "… "cut insurance" is
another budgetary technique. In this case… DoD inflates the cost of selected
programs to ensure that even after expected congressional cuts, desired
funding levels will remain intact," 15 and …" There is simply no need to lie;
one only has to be optimistic." 16 Any degree of situational estimating would
have an impact on pure estimating accuracy.
Recently there have been
major changes, improvements, etc. to the "5000 Series Acquisition Regulations"
which regulate the mainstream DoD weapon systems development this research
covers. This includes the large number of major improvements commonly called
Acquisition Reform (AR) and pilot programs such as Advanced Concept Technology
Demonstrator (ACTD). The effect and relation of these new initiatives on our
research is not discussed.
the established DoD acquisition system for ACAT I programs, the only
oversight documents required on a routine basis by the Office of the Secretary
of Defense (OSD) are the SARs and DOT&E BLRIP evaluation.
validity of this study depended greatly on how important the data contained
within the SARs was to the actual decisions made at DAB meetings. It was
assumed the information was central and vital to the decision process.
This study is the only known research that attempts to combine
and evaluate EMD cost and schedule together with performance as measured by
OTA and DOT&E operational tests during EMD.
Also this study attempts to
identify program EMD costs only. The SAR, using development and production
estimate baselines, contain cost to date, and cost to complete for the entire
program. It is assumed you cannot easily extract EMD costs only from the SARs.