My size won't be that much of a help, as the vest is IMO too big for me.
I don't own any IH, but I wear 16 in FW shirts, FW jackets in between 38 and 40.
It does depend on how you want to wear the vest, I can wear it over thicker jackets but i find it too bulky over only a denim jacket. I think I would need a 38 for a snug fit, and a 40 for a more relaxed fit.