Different assemblies of accessory proteins with clathrin are critical for transporting precisely various cargos between intracellular compartments. GGA proteins are adaptors for clathrin-mediated intracellular trafficking, connecting other accessory and cargo proteins to clathrin-coated vesicles. Both binding to the GAE domain of GGA protein yGGA2 in Saccharomyces cerevisia, Ent3 and Ent5 are involved in different trafficking pathways. Ent5 is ubiquitous and localized in a manner independent of yGGA2, and Ent3 functions preferentially through yGGA2. Not known are the sources of these differences. Here we show not all acidic-phenylalanine motifs in Ent3/5 are active for yGGA2_GAE domain binding. Two of the three acidic-phenylalanine motifs from Ent3 can bind to the yGGA2_GAE domain, while only one of the two motifs from Ent5 can bind. We also determined the crystal structure of the yGGA2_GAE domain at 1.8 A resolution. Structural docking and mutagenesis analysis shows inactive motifs in Ent3 and Ent5 repel yGGA2_GAE binding through disfavored residues at positions 1 and 3. These results suggest accessory proteins may fine-tune the GGA adaptor dependence by adjusting their non-acidic-phenylalanine residues, thus contributing to the distinct role of Ent3 and Ent5 in trafficking.