3.0 Technical Basis

Software risk evaluation involves extracting risk information from the expertise available on a given software development project, and converting this risk data into coherent decision making information. Similar data transfer and transformation challenges are addressed in another field of study concerned with the formalization of decision making information - the knowledge engineering field. For this reason, the SEI Risk Program chose to analyze lessons-learned in knowledge engineering and apply the best-fit practices when developing the SRE model.

From this analysis of knowledge engineering practices, many techniques commonly employed in the development of expert systems were found to be applicable in an evaluative software risk assessment. An expert system is defined as "a computer program that performs a task normally done by an expert or consultant and which, in so doing, uses captured, heuristic knowledge" [Dym]. The process of populating an expert system, called knowledge acquisition, involves "the transfer and transformation of problem-solving expertise (i.e., knowledge about a particular domain, understanding of domain problems, and problem solving skills) from a knowledgeable person to a computer representation" Mulvehill.

Of the several knowledge acquisition methods employed by practitioners, protocol analysis is the most common. Protocol analysis is an approach to the study of complex human behavior, including performance of complex tasks, interactions among people, and interactions with devices such as computers [Poltrack]. Unlike empirical methods frequently used by psychologists and human factors engineers which are computationally well-defined and rigorous, protocol analysis is actually a collection of practices that are adapted to the particular circum- stances of the research. For the purposes of this document, the common steps in protocol analysis are defined as follows:

• record - transcribe all of the subject's spoken protocols via available data collection channels (e.g., notetaking, audio/video recordings)
  

• group - segment the raw data into meaningful units using appropriate categorization techniques (e.g., summary, decomposition, classification)
  

• format - transliterate segmented data into analysis vocabulary (e.g., maintain content and context, while reducing variability of expression)
  

• codify - develop a cognitive model of the subject's behavior based on the processed data (e.g., cause-effect relationships, chronology)
  

• compute - execute the encoded cognitive model to simulate human performance
  

• refine - test and update the model by comparing the model's behavior with the human protocol
  

The SRE model described in this document applies protocol analysis as a technique for implementing the steps in the Risk Management Paradigm in support of a software development program evaluation. Figure 3-1 illustrates how the steps in the Protocol Analysis Model (lower graphic) are implemented in support of the Risk Management Paradigm (upper graphic). Note that the upper graphic is the same as Figure 3-1 , depicted as linear rather than as circular. Although not shown explicitly, risk communication is implicitly assumed to be present in all steps.

The shading in Figure 3-1 signifies that this initial version of the SRE methodology addresses only the risk identification and risk analysis steps in the Risk Management Paradigm. In terms of the Protocol Analysis Model, this first version of the SRE methodology addresses the following steps: record, group, format, and part of codify. Risk planning, tracking, and control are all areas of intense research within the SEI Risk Program and will be addressed in future versions of the SRE methodology.