Just because the onset of your dizziness wasn't sudden does not mean it's not ear related! Sometimes I think they're too quick to label things anxiety rather than something else.
Do you notice what's happening to you when it comes on or when it gets worse? Looking at those patterns can help you figure out what's wrong. There is usually something that sets the dizzy sensations off, although it may be hard to notice it at first.
I had labs a couple years ago, and in the week or two before it I noticed that I was feeling a bit funny, like looking up and down in my job made me feel slightly off balance. I had my first ever panic attack because I felt so weird walking down the street one day (which apparently can be one of the first signs of labs). When it came on fully I had about 6 weeks of really bad dizziness, but never full spinning vertigo, it was more like that sensation that you get when you've been spinning around for a minute and then lie down - like a feeling of dizziness inside my head rather than full room spinning vertigo. My balance was also off really badly, I couldn't really move around very much.
It resolved itself, but ever since then when I get sick I often tend to get the vertigo back too. Then at the end of last November, after sleeping really badly on my neck, the full blown dizziness came back again for a couple of weeks and ever since I've been struggling with being off balance and frequent bouts of vertigo throughout the day.
At first they told me it was just labs again, and I had to wait for it to pass. Then they thought it might be my sinuses so we tried all kinds of things to get them to calm down and eventually they did, but the dizziness still hadn't resolved. So I'm waiting to see an ENT and in the mean time my doctor recommended I make an appointment with a physiotherapist who specialises in Vestibular Rehabilitation Therapy (VRT) which has been an absolute godsend!!
The physio has explained so much of the weird stuff that's been happening to me over the last few years - he thinks the first bout of labs may have damaged my balance system and every time there is a slight weakness in my body my balance gets weak and that's why I get all wobbly again. Knowing this explains so much to me about why I've gotten dizzy over the years.
Through the testing he did we discovered that my balance is not too bad, but it's my vision that is causing me most of the problems. We discovered that most of the triggers of vertigo for me are things like looking up and down while I'm looking from my computer to my keyboard, trying to focus on things while I'm walking or moving, even wearing different types of shoes can affect how dizzy I feel!
Stress and anxiety make it 100 times worse, because the muscles in my neck get tight and affect my ears which can set me off balance, but being off balance also makes my neck muscles tight so it's a bit of a vicious circle lol. I actually think that I have TMJ and that when this flares up it badly (usually from stress) it affects my ear too - but I have to wait for the ENT to diagnose this properly. So you can see how anxiety would make this problem worse for someone, but it's not the source of the problem!
So he's tailored treatment of these VRT exercises to help me retrain the signals going from my eyes to my brain and help me get back in focus and balance. It's not easy but I'm slowly seeing improvement and hope to be back in balance in a couple of weeks. I still need to see the ENT to get a proper diagnosis, but this work I'm doing with the VRT will benefit me for the rest of my life. Now that I understand it a lot more I'm less anxious about it and have a strong hope that it will resolve.
If it helps you to figure this out for yourself here's a little list of the symptoms I get, to see if an inner ear disorder may be something that you have too: off balance (feels like I'm drunk), vertigo, blurry/fuzzy vision (usually when moving), difficulty looking up and down repeatedly, difficulty reading and using computers, swaying/rocking sensation when still, tinnitus, nausea, head/neck/shoulder aches, feeling of fullness in ears, crackling sound in ears, brain fog, feeling overwhelmed and weird in crowds or places where there is a lot of visual stimulation (like looking at the stones on the beach or looking around in a shop), major fatigue, difficulty concentrating and gets worse with concentration/stress/sickness/low blood sugar/any type of weakness.
As for the TMJ the symptoms I get are ones like jaw & facial pain that radiates to my ear, headaches, sinus problems, teeth gringing and clenching, and sometimes a clicking jaw.
Take a good look at your own symptoms and see if you can figure out the patterns that make your own problems worse. Then perhaps see if you can find someone in your area who does VRT or ask your doctor about it. If you really feel it's not your anxiety then it's worth investigating further to help you solve the problem.
I've found this website really informative and helpful to understand what's happening:
http://vestibular.org/
Sorry if this was a long ramble about myself, I just thought it might help you figure out if you had a similar problem! x