RNA polymerase II initiates from low complexity sequences so cells must reliably distinguish "real" from "cryptic" promoters and maintain fidelity to the former. Further, this must be performed under a range of conditions, including those found within inactive and highly transcribed regions. Here, we used genome-scale screening to identify those factors that regulate the use of a specific cryptic promoter and how this is influenced by the degree of transcription over the element. We show that promoter fidelity is most reliant on histone gene transactivators (Spt10, Spt21) and H3-H4 chaperones (Asf1, HIR complex) from the replication-independent deposition pathway. Mutations of Rtt106 that abrogate its interactions with H3-H4 or dsDNA permit extensive cryptic transcription comparable with replication-independent deposition factor deletions. We propose that nucleosome shielding is the primary means to maintain promoter fidelity, and histone replacement is most efficiently mediated in yeast cells by a HIR/Asf1/H3-H4/Rtt106 pathway.