There is always a "them". Going in the queue means you always push someone else back. There will be others who may not need them because doctors are performing tests to rule things out. But there may be many who need them.
For instance, my mum had a skin problem her GP spent about 6 months trying different treatments on. He had no idea what it could be and had to refer her to a dermatologist for more specialised treatments. She had to wait something like 8 months for that appointment and she really did have something that only a dermatologist could treat, which was backed up by the doctor who was actually excited about it as it was a bit of a rarity in the UK.
My dad is currently waiting a couple of months for tests on his heart for possible angina. Months.
That is the reality a doctor sees.
This is why the non anxious get angry with hypochondriacs.
I'm not sure what that doctor thinks is going to help. IT may help with reassurance but those of us on here, who understand the disorder and have seen how yours manifests, know that's just kicking the can down the road. But he gets to push the cost onto your GP and local services...it's not coming out of his budget
A doctor suggesting tests, even to rule out, is like waiving a big stand of cakes at Johnny Vegas. I realise he mentioned this but it's just planting another seed in your head. I wouldn't be surprised if your GP doesn't secretly roll their eyes at that.
You don't have to tell me, I'm hurtling towards 50!