The objective of this large population-based cross-sectional study was to evaluate the association between smoking, passive smoking, alcohol consumption, and hearing loss. The study sample was a subset of the UK Biobank Resource, 164,770 adults aged between 40 and 69 years who completed a speech-in-noise hearing test (the Digit Triplet Test). Hearing loss was defined as speech recognition in noise in the better ear poorer than 2 standard deviations below the mean with reference to young normally hearing listeners. In multiple logistic regression controlling for potential confounders, current smokers were more likely to have a hearing loss than non-smokers (odds ratio (OR) 1.15, 95 % confidence interval (CI) 1.09-1.21). Among non-smokers, those who reported passive exposure to tobacco smoke were more likely to have a hearing loss (OR 1.28, 95 %CI 1.21-1.35). For both smoking and passive smoking, there was evidence of a dose-response effect. Those who consume alcohol were less likely to have a hearing loss than lifetime teetotalers. The association was similar across three levels of consumption by volume of alcohol (lightest 25 %, OR 0.61, 95 %CI 0.57-0.65; middle 50 % OR 0.62, 95 %CI 0.58-0.66; heaviest 25 % OR 0.65, 95 %CI 0.61-0.70). The results suggest that lifestyle factors may moderate the risk of hearing loss. Alcohol consumption was associated with a protective effect. Quitting or reducing smoking and avoiding passive exposure to tobacco smoke may also help prevent or moderate age-related hearing loss.