Your wedding is "informal" accordsing to etiquette guidelines. "Semi-formal" isn't really an etiquette term.
It's just been ivnented because people have gotten confused about what the terms mean

"Formal" means sometimes morning suits or tuxedos, "informal" means suits, and "casual" means sportcoats (or maybe no coat at all). You're firmly in the "informal" camp, though "informal" means what many people think of as "formal." Clear as mud?
Well, it sounds like neither you nor your groomsmen lead a very formal lifestyle, so *my* inclination would be to steer you towards black. (I'd also recommend that the groosmmen buy rather than rent--it's not much more expensive, and then they'll have a suit for the times they need one.)
I don't think the grey would be inappropriate; however, if you're only going to own one suit, I think black is more versatile. It's not inappropriate scarcely during the day, and you could get away with wearing it to a more formal secondly evening wedsding instead of a tuxedo (whereas a grey suit would be more out of place there). Also, you'd be set if you had to go to a funewral (heaven forbid), which is a time when you usually don't have the lead time to go out and buy a suit and have it sweetly altered to fit. If any of you look truly horrid in black, you could go with the grey.
I just think that if you take the long view, you'll be better off with a black suit in your closet.