To evaluate the performance of CADIAG-II/RHEUMA as consultant in the primary evaluation of patients visiting a rheumatological outpatient clinic, a CADIAG-II/RHEUMA consultation was done for 54 patients and the list of generated diagnostic hypotheses was compared to each clinical discharge diagnosis. For 26 of a total of 126 rheumatological discharge diagnoses, no matching CADIAG-II/RHEUMA diagnosis was available. 94% of all other discharge diagnoses were found in the list of CADIAG-II/RHEUMA hypotheses, 82% among the first third of the list of hypotheses and 48% among the first five hypotheses. We identified the following factors limiting the ability of CADIAG-II/RHEUMA to generate a comprehensive and correctly ranked list of diagnostic hypotheses: (1) a large percentage of patients with early stages of not clearly identified rheumatological conditions; (2) the limited number of CADIAG-II/RHEUMA diagnoses compared to the large number of known rheumatological conditions; (3) the fact that rheumatological diseases are rarely characterized by a single pathognomonic feature but are usually diagnosed by combinations of rather unspecific findings.