Don't take it personally. I get this a lot from loud men and it doesn't bother me at me all but then I only have mild social anxiety.
From his perspective you are quite compared to him.
Also he may have said it as a way to encourage you to talk or to bring you out of your shell a little bit.
Or he could have just been an *******, I wasn't there so I can't tell
I've a friend who is seriously quite to the the point I forget she's even there but she still fits in well and no one every picks on her or tells her she's too quite.
I do wish though that it was easier to talk to her and i never know what to say :(
But she is rather sweet and lovely.
So even if your quite people will still see your an awesome person
in my experience the loudest in the group is the most insecure, they get their attention and validation from people around them.
So just because someone acts confident sometimes it can just be a show, to hide who they really are or how insecure the feel.
One thing I will add, is men love banter so don't be afraid to talk back to them. sometimes light hearted insults is their way of bonding.
Well done for going