The nature of data obtainable from the commercial smartphone - bolstered by a translational model emphasizing the impact of social and physical zeitgebers on circadian rhythms and mood - offers the possibility of scalable and objective vital signs for major depression. Our objective was to explore associations between passively sensed behavioral smartphone data and repeatedly measured depressive symptoms to suggest which features could eventually lead towards vital signs for depression. We collected continuous behavioral data and bi-weekly depressive symptoms (PHQ-8) from 131 psychiatric outpatients with a lifetime DSM-5 diagnosis of depression and/or anxiety over a 16-week period. Using linear mixed-effects models, we related depressive symptoms to concurrent passively sensed behavioral summary features (mean and variability of sleep, activity, and social engagement metrics), considering both between- and within-person associations. Individuals with more variable wake-up times across the study reported higher depressive symptoms relative to individuals with less variable wake-up times (B [95% CI] = 1.53 [0.13, 2.93]). On a given week, having a lower step count (-0.16 [-0.32, -0.01]), slower walking rate (-1.46 [-2.60, -0.32]), lower normalized location entropy (-3.01 [-5.51, -0.52]), more time at home (0.05 [0.00, 0.10]), and lower distances traveled (-0.97 [-1.72, -0.22]), relative to one's own typical levels, were each associated with higher depressive symptoms. With replication in larger samples and a clear understanding of how these components are best combined, a behavioral composite measure of depression could potentially offer the kinds of vital signs for psychiatric medicine that have proven invaluable to assessment and decision-making in physical medicine. Clinical Trials Registration: The data that form the basis of this report were collected as part of clinical trial number NCT03152864.