CARM1 (coactivator-associated arginine methyltransferase 1), which belongs to type I PRMTs (protein arginine methyltransferases), is a potential therapeutic target for treatment of multiple cancers. In this study, we first identified several hit compounds against CARM1 by structure-based virtual screening (IC50 = 35.51 ± 6.68 to 68.70 ± 8.12 μM) and then carried out chemical structural optimizations, leading to six compounds with significantly improved activities targeting CARM1 (IC50 = 18 ± 2 to 107 ± 6 nM). As a compound with an ethylenediamino motif, the most potent inhibitor, ZL-28-6, also exhibited potent inhibition against other type I PRMTs. Compared to the type I PRMT inhibitor from our previous work (DCPR049_12), ZL-28-6 showed increased potency against CARM1 and decreased activity against other type I PRMTs. Moreover, ZL-28-6 showed better antiproliferation activities toward a series of solid tumor cells than DCPR049_12, indicating its broad spectrum of anticancer activity. In addition, cellular thermal shift and Western blot assays validated that ZL-28-6 could target CARM1 in cells. Taken together, the inhibitor we identified could serve as a potent probe for studying CARM1's biological functions and shed light on the future design of novel CARM1 inhibitors with stronger activities and selectivities.