We set out a protocol for giving a computerised medical decision system the label of expert-system meaning that its responses are of the same standard as those of an expert. We apply it to the validation of an expert-system, MENINGE, which is applied to diagnosis of meningitis in children. The proposed protocol can be widely used since it doesn't refer to the existence of correct responses for therapy or diagnosis, and since it involves simple statistical computations. First this validation rests on a comparison of agreement among experts and among physicians new to medical practice. Secondly, the agreement of the system with the experts is compared to the agreement of the system with physicians new to medical practice. In order to perform this comparison we recommend a standard agreement measure, the choice of which is an important issue in this paper.
Keywords: Agreement; Convex functions; Entropy; Expert-system; Kappa-coefficient.