Re: Do I just give up and take the medication?
In a word, yes. I take Prozac as well (60 mg) and must admit it was hellish waiting for it to kick in. However the side effects were temporary. Could also ask your Dr. about other SSRI options like Lexapro and Zoloft. I was on Lex for 10 years before it faded out and I transitioned to Prozac. For me Lexapro was great.
Your post says it all, panic is unfortunately taking a toll on you currently and was gone when you were on medication. To me the answer is clear, but obviously I am pro-medication.
If you do decide to go back on a SSRI, keep with it. The transition may be tough but temporary and you know the relief they can provide. Best wishes.
__________________
What works for one person may not work for all.
Always discuss medical concerns with your Dr.