We extend the formalism of self-consistent Green's function theory to include three-body interactions and apply it to isotopic chains around oxygen for the first time. The third-order algebraic diagrammatic construction equations for two-body Hamiltonians can be exploited upon defining system-dependent one- and two-body interactions coming from the three-body force, and, correspondingly, dropping interaction-reducible diagrams. The Koltun sum rule for the total binding energy acquires a correction due to the added three-body interaction. This formalism is then applied to study chiral two- and three-nucleon forces evolved to low momentum cutoffs. The binding energies of nitrogen, oxygen, and fluorine isotopes are reproduced with good accuracy and demonstrate the predictive power of this approach. Leading order three-nucleon forces consistently bring results close to the experiment for all neutron rich isotopes considered and reproduce the correct driplines for oxygen and nitrogen. The formalism introduced also allows us to calculate form factors for nucleon transfer on doubly magic systems.