Blue, given your past this can be more traumatic with it digging up past traumas. That being said perhaps this will fade as you process it?
I agree with Carys, if you work in situations where this can happen there should be awareness about how to control such situations. Not only to proect yourself but to protect the person who may not understand they are crossing a boundary that, with another person outside this environment, may be intepreted as a potential criminal act bringing damaging consequences to the person (and obviously the person harassed) in dealing with intrusive questioning.
In this instance your employer needs to be supporting you. Do they know about your past? They should be training you to handle such situations now if they have failed to in the past.
With the person having learning difficulties it sounds like they are treating as someone who is unaware of what they have done as you would do with a child, talk to them to prevent is happening in the future. So, the person may just respond to this now knowing it's wrong. You could also talk about your future contact with this person if you are uncomfortable and ask them to make adjustments as the environment should be safe for you and him/them. Not only you but he could otherwise do the same to others there including those with their own learning difficulties.
Obviously if this has been someone they judge not so unaware of their actions it would be a criminal matter.
You say yourself you know it is being blown up into a bigger issue due to the past. It caught you off guard and confrontations can be hard for anyone not trained/experienced in handling them.
Have you spoke to your partner about this? Perhaps you need some hugs, some pampering, some distraction, a bit of reassurance from your best friend and a little time to let go of the emotions it has stirred up.
I hope you feel a bit better in the morning.