The medium would probably be a little more versatile and a bit closer in measurements to my small UHF and madras.
I’d imagine you’d get a relaxed, sweatshirt for, that you could fit a tee under, as opposed to a snug long-sleeved tee fit.
My regular short sleeved tshirts are more of a relaxed fit as it is. They're all roughly around 40" chest, and I have a 34" chest.
Does the material have much stretch to it? My biggest concern with the small was actually the opening. The opening measures around 35.2", and my hips around where the opening would be are 33" before adding the pants and belt to the mix.