An empirical study on the knowledge acquistition process for expert systems

Expert systems, a form of artificial intelligence, are computer programs that enhance the problem-solving and decision-making performance of users. The power of these systems is dependent on the knowledge that is extracted from the experts. The process of extracting expertise is called knowledge acquisition. As a process, knowledge acquisition involves eliciting, analyzing and interpreting the knowledge that a human expert uses when solving a particular problem and then transforming this knowledge into a suitable machine representation. Several knowledge elicitation techniques have been reported in the literature. Despite the rapid development of techniques for knowledge acquisition, there has been little effort involved in evaluating the effectiveness of the techniques.

The objective of this research was to determine how the knowledge source (domain expert), task features (domain), knowledge engineer (expert system developer) and knowledge acquisition technique affect the effort needed to develop a knowledge base, the time spent to develop a knowledge base and the quality of the elicited knowledge base. Nine librarians and six pilots (experts) and nine graduate university students (knowledge engineers) served as subjects in this study. Two types of domains were investigated, for which knowledge bases were created for each domain: (1) an aircraft flight maneuver task and (2) a librarian material selection task. In the experiment, each person serving as an expert participated in the study once using each of the three techniques: (1) interview, (2) verbal protocol and (3) concept mapping with three different knowledge engineers. Each knowledge engineer used each of the three techniques by conducting knowledge acquisition session with experts for each task. Time to elicit knowledge from an expert, time to analyze elicited knowledge, accuracy and completeness of the knowledge base, expert workload, and knowledge engineer workload were measured at the end of each knowledge acquisition session. Results of ANOVAs showed a significant effect of task type only for the completeness measure and not for all other measures. In addition, results of ANOVAs showed that the quality of the knowledge base was more dependent on the expert's and the knowledge engineer's personal characteristics and performance than a selected knowledge acquisition technique. However, the time spent on either the knowledge elicitation or knowledge analysis sub-tasks of the knowledge acquisition process was dependent on the knowledge acquisition technique. The verbal protocol technique shortened the time spent on knowledge elicitation, whereas the concept mapping technique shortened the time spent on analysis. From the point of the knowledge engineer, it is concluded that the concept mapping technique is the most efficient technique and requires the least effort spent On the other hand, the domain experts were split, with the librarians viewing the concept mapping technique as the most efficient and the pilots viewing the verbal protocol technique as the most efficient method. Overall, the selection of the knowledge engineer and expert is as important as the selection of a knowledge acquisition technique at least with these two tasks. Although there were not any quality or time differences using the different techniques, the concept mapping technique is recommended based on this study, as compared to both the interview and verbal protocol techniques as a means of reducing the knowledge engineer's workload.