Dude, there are books written on flirting.
Banter is always a good bet. Light fun conversation displaying personality (intelligence values interests) and wit (observational associative humor) often using teasing comments.
For example of good Banter watch the TV show How I Met Your Mother. Not for the entertainment value but for the interactions between the friends as a tutorial on how to Banter!
As for the brother, just be cool man. If the girl is into you she'll bring him around your interaction and he'll give you the nod. Which can be a straight forward acceptance, a quick appraisal and exit, or an ironic warning (he'll be smiling and joking, but is also partly serious).
If he says stay away or quit bothering his sister seriously, stay away.
Honestly the best thing you can do right now is to approach a hundred women this week and stop obsessing about gym girl. She'll see your interest waning and try to rekindle it.