It may not be a very popular opinion around here, but I personally want my vests to be fitted. I recently picked up the indigo wabash vest and sized down 1 from my IH thermal size, to wear it over a t-shirt.
That said, I can see how it's puckering a bit in the chest/pit area. I don't think it looks bad if you don't think it's uncomfortable. You're probably in a situation where the next size up might be a little loose, if you're looking for a fitted look.
But yeah, duck won't give much.