The main environmental concerns about genetically modified (GM) crops are the potential weediness or invasiveness in the crop itself or in its wild or weedy relatives as a result of transgene movement. Here we briefly review evidence for pollen- and seed-mediated gene flow from GM crops to non-GM or other GM crops and to wild relatives. The report focuses on the effect of abiotic and biotic stress-tolerance traits on plant fitness and their potential to increase weedy or invasive tendencies. An evaluation of weediness and invasive traits that contribute to the success of agricultural weeds and invasive plants was of limited value in predicting the effect of biotic and abiotic stress-tolerance GM traits, suggesting context-specific evaluation rather than generalizations. Fitness data on herbicide, insect, and disease resistance, as well as cold-, drought-, and salinity-tolerance traits, are reviewed. We describe useful ecological models predicting the effects of gene flow and altered fitness in GM crops and wild/weedy relatives, as well as suitable mitigation measures. A better understanding of factors controlling population size, dynamics, and range limits in weedy volunteer GM crop and related host or target weed populations is necessary before the effect of biotic and abiotic stress-tolerance GM traits can be fully assessed.