I imagine that the increased risk is very small, and that it only applies to women who've been on the drug for years and years. If you're only taking it as a means to help you recover from anxiety I'm sure that it will be fine! Working your way back to managing the anxiety without the drug is a good aim, the lifestyle changes above will really help you.
But if you're worrying about it talk to your doctor about an alternative.
And maybe try one or two of the things I mentioned above to combat this panic, it's a good place to start