^What Dan said.
I think the rule of thumb is, go for your UHF size if you want to layer, go one down from UHF for a more fitted look over a single shirt/tee etc.
I use mine as my winter coat in northern Norway so wear it over tee+UHF or henley+vest combos'. It's an XL which is also my IH shirt size. It doesn't look baggy, but would do over just a tee.