No matter what you look like: somebody fetishizes it. Don't worry about impressing women and men... men are irrelevant. T-Mal nailed it with the responses though.
I'm in the same boat, I have a very lean frame no matter how much time I spend in the gym and how much protein I consume I can't seem to gain enough to look anything other than mildly anorexic. If looks mattered for more than 10% of the game, I'd be a virgin right now.
I've never once had guys try to use that to AMOG me though, I can only assume it has more to do with your inner game than your appearance. If you come across as a dominant guy everyone respects, you won't get disrespected like that. If you can outsmart or out-attitude the male models, you'll win. Every. Single. Time. Guys who are genetically 'gifted' have been able to rely on it their whole lives. They don't know how to fight for it like smaller guys. Seriously, they better have something better than "you have no ass" if they expect to not get demolished in the field. Go demolish them.