Incorporating protein flexibility is a major challenge for docking-based virtual screening. With an increasing number of available crystal structures, ensemble docking with multiple protein structures is an efficient approach to deal with protein flexibility. Herein, we report the successful application of a docking-based virtual screen using multiple crystal structures to discover novel inhibitors of lymphoid-specific tyrosine phosphatase (LYP), a potential drug target for autoimmune diseases. The appropriate use of multiple protein structures allowed a better enrichment than a single structure in the recovery of known inhibitors. Subsequently, an optimal ensemble of LYP structures was selected and used in docking-based virtual screening. Eight novel LYP inhibitors (IC50 ranging from 7.95 to 56.6 μM) were identified among 23 hit compounds. Further studies demonstrated that the most active compound B15 possessed some selectivity over other protein phosphatases and could effectively up-regulate TCR (T cell receptor)-mediated signaling in Jurkat T cells. These novel hits not only provided good starting points for the development of therapeutic agents useful in autoimmune diseases but also demonstrated the advantages of choosing an appropriate ensemble of protein structures in docking-based virtual screening over using a single protein conformation.