Older adults may be more vulnerable to negative health effects from alcohol as they age. Distress and adverse neighborhood conditions that provoke distress may influence drinking behavior. Using baseline data from the Baltimore Memory Study, a cohort study of adults aged 50-70 years living in 65 Baltimore City neighborhoods, we investigated the association between neighborhood psychosocial hazards (NPH) and the number of binge drinking days in the past month among non-abstainers (N = 645). We used negative binomial regression with generalized estimating equations to estimate the relative number of binge drinking days per month associated with a one standard deviation increase in NPH score. Residing in neighborhoods with more psychosocial hazards was independently associated with more binge drinking for females, but not for males. For females, each one standard deviation increase in NPH score was associated with a 1.52 relative risk of binge drinking (95 % confidence interval, 1.10, 2. 10) in the adjusted model. The findings were robust to a sensitivity analysis in which we used the average number of drinks per drinking occasion as an alternative outcome. Our findings provide evidence linking adverse neighborhood conditions with alcohol consumption in non-abstaining late middle-aged women, and suggest that late middle-aged men and women may have different reactions to adverse residential neighborhoods.