Not all cotton material is equal as you point out. I find organic cotton makes no difference for me personally.
Many cotton fabrics are processed with wrinkle free adhesive chemicals, sizing, spray starch, fire retardant and Scotch Guard. I am allergic to all these additives. Also as grovercleveland mentioned many are allergic to dyes, blue dye is a proven allergen, but China as he explained adds metals and or other elements to some colors. Adding metals is inexpensive and gives a better long lasting color. It's a fact that many painters decades ago ended up with Parkinson's or Alzheimer's because of exposure to lead. Mental imparement happened to children who chewed paint at an early age. That suspected green shirt should be tossed Many chemicals can't be washed out and if mixed with other clothing can contaminate them.
I am not allergic to petroleum, God bless Vaseline, which is pure petroleum and has been my go to when all else failed, but I am allergic to the adhesive additives, as proven in my 5 Day Extended Patch Test.
If the cotton feels soft, wrinkles and needs ironing, that's wonderful for me!