What is my problem?

Hi, I need help identifying one of my conditions because I cant find anything similar online. My issue is my struggle to work outside of my normal hours. I'm happy to work mon-fri 9-5. If I get asked to out of hours work my mind works itself into a frenzy and it can keep me awake at night trying to think of a way out of it. I really have no idea why im like this. Growing up I never wanted a paper round or a Saturday job because that was my free time to do what I wanted. In my job I occasionally did out of hours work years back from home over the internet but the anxiety and tension I felt about it got worse and worse over time. Now it feels like a phobia. It is so frustrating feeling you are getting close to a panic attack about something but you've really no idea why. If I could find out what class of condition it was it would at least point me in the right direction. Any advice would be greatfully recieved.
FYI I work in IT. I guess part of my worry is something going wrong and a task that should of taken 1 hour ends up taking the rest of your day as you try and fix what went wrong. When things take longer than expected really raises my tension levels.
Strangely I also struggle to travel for work which might be Agrophobia. The odd thing though is that it is less of an issue if the travel is for personal reasons. I'm not sure if this travel thing is related to working out of hours as well. I dont know if Agrophobia can be time based as well.

Hi, Many thanks for the reply. I'm glad to hear you got your life back on track by becoming self employed. It's nice to know im not the only one who finds working outside hours so difficult. If I try googling the subject I find nothing which made me think I was the only one. I just dont get why I find it so hard. I envy others who can work shifts and weekends without a second thought.
I was wondering how you got a diagnosis? I've found the NHS unhelpful. I've a host of mental health problems. On the talking side I've only been offered counciling or those classes where you learn to relax more etc. The only vague diagnosis I've had is from the short chat I had with my GP years back who said depression and generalized anxiety. I've never mentioned the working out of hours thing as it wasnt an issue until more recently. You'd think you'd need to have 4-5 sessions with a psycotherapist to get a diagnosis. Even if you have a diagnosis how will that help in the long term. I have a mortgage to pay and im very scared issues like this will cause me to lose my house. I simply dont know what to do. No tablets seem to help. I've given up on the NHS and any private help would be very expensive and time consuming.