We have recently shown that graphene is unsuitable to replace metals in the current-carrying elements of metamaterials. At the other hand, experiments have demonstrated that a layer of graphene can modify the optical response of a metal-based metamaterial. Here we study this electromagnetic interaction between metamaterials and graphene. We show that the weak optical response of graphene can be modified dramatically by coupling to the strong resonant fields in metallic structures. A crucial element determining the interaction strength is the orientation of the resonant fields. If the resonant electric field is predominantly parallel to the graphene sheet (e.g., in a complementary split-ring metamaterial), the metamaterial's resonance can be strongly damped. If the resonant field is predominantly perpendicular to the graphene sheet (e.g., in a wire-pair metamaterial), no significant interaction exists.