This study explored the modulatory effects of nitric oxide and thromboxane A2 on contractions to ergonovine and methylergonovine in human coronary arteries. To elucidate the different role of nitric oxide synthase in the response to the ergot alkaloids, the serotonin (5-HT) receptors involved in nitric oxide synthase in the response to the ergot alkaloids, the 5-HT receptors involved in nitric oxide release and the contraction of the vascular smooth muscle were characterized with more selective 5-HT-receptor agonists and antagonists. Rings of human coronary arteries from explanted hearts were suspended in organ chambers for isometric tension recording. After testing for contractile (potassium chloride, 60 mM) and endothelial function (substance P, 10(-8) M), respectively, they were exposed to ergot alkaloids or other agonists in the absence or presence of U 46619 (10(-9) M), or nitro-L-arginine (10(-4) M), or both. Ergonovine and methylergonovine were comparable, weak vasoconstrictors in untreated preparations. Contractions to ergonovine were augmented by U 46619, but not by nitro-L-arginine. Contractions to methylergonovine were augmented only by combining U 46619 and nitro-L-arginine. Serotonin and methylergonovine, but not ergonovine, elicited endothelium-dependent, nitric oxide-mediated relaxations. Nonselective 5-HT(1B/1D)-receptor stimulation caused both contractions and relaxations; selective 5-HT1B stimulation caused relaxations only. In the human coronary artery, contractions to ergonovine are not dependent on NO release but are synergistically augmented by thromboxane. Methylergonovine causes similar effects on the vascular smooth muscle, but contractions are inhibited by the release of NO from the endothelium. The 5-HT receptor on the endothelium appears to be different from the receptor on the vascular smooth muscle, which mediates the contractile response to the ergot alkaloids.