Recent evidence demonstrates that the quality of raters' applied clinical skills is directly related to study outcome. As such, the training and evaluation of raters' clinical skill in administering symptom-rating scales is essential before being certified to rate patients in clinical trials. This study examined a novel approach to rater training and certification that focused on both conceptual knowledge and applied skills. Forty-six raters (MDs = 14; PhDs = 7; MA = 5; BA/LPN/RN = 20) in a large multicenter depression study went through a 2-step Hamilton Rating Scale for Depression (HAMD) certification process: didactic training, administered online via an interactive Web tutorial, and live, applied training, where raters interviewed depressed patients while being remotely observed via 3-way teleconference. Raters' applied skills were evaluated using the Rater Applied Performance Scale (RAPS), designed specifically to evaluate critical rater behaviors associated with good clinical interviews. Raters received feedback immediately following the interviews; those receiving a failing score were given 2 more opportunities to pass. Each subsequent session was accompanied by feedback, and was conducted by a different trainer, who was blind to the results of the previous session as well as to which session number it was, to avoid bias. Raters who failed on the third attempt were excluded from rating patients in the trial. All training and testing occurred prior to the startup meeting. Results found a significant improvement pre-to-post Web training in raters knowledge of scoring conventions, P < 0.001. On the applied component, raters' RAPS scores improved significantly on the second attempt following feedback, from 9.05 to 11.58, P < 0.001, and from their second to their third session (from 9.00 to 11.00, P = 0.033. Three raters failed all 3 attempts and were excluded from the study. Results support the efficacy of the approach in improving both conceptual knowledge and applied interviewing skill.