Digits backward (DB) is a widely used neuropsychological measure that is believed to be a simple and effective index of the capacity of the verbal working memory. However, its neural correlates remain elusive. The aim of this study is to investigate the neural correlates of DB in 299 healthy young adults by combining voxel-based morphometry (VBM) and resting-state functional connectivity (rsFC) analyses. The VBM analysis showed positive correlations between the DB scores and the gray matter volumes in the right anterior superior temporal gyrus (STG), the right posterior STG, the left inferior frontal gyrus and the left Rolandic operculum, which are four critical areas in the auditory phonological loop of the verbal working memory. Voxel-based correlation analysis was then performed between the positive rsFCs of these four clusters and the DB scores. We found that the DB scores were positively correlated with the rsFCs within the salience network (SN), that is, between the right anterior STG, the dorsal anterior cingulate cortex and the right fronto-insular cortex. We also found that the DB scores were negatively correlated with the rsFC within an anti-correlation network of the SN, between the right posterior STG and the left posterior insula. Our findings suggest that DB performance is related to the structural and functional organizations of the brain areas that are involved in the auditory phonological loop and the SN.