Should I change/quit my course of study?

I'm faced with the decision of whether to drop out of my current Business Information Systems degree program and switch to Psychology. I've always had an interest in psychology, but thought computer science might be a good/better fit for my skills. The problem, however, is that I'm currently forcing myself through my studies. While I enjoy programming, advanced mathematics presents a significant challenge for me.

Switching to computer science also doesn't make sense, since we have the same math course in the first semester. Furthermore, I've been told by senior students that starting in the second semester, we'll be covering math that's important for our subject. The fact that we'll be allowed to use calculators is reassuring (currently, using one is prohibited), as my mental math skills have always been lacking. During my school years, my math grades fluctuated between 3 and 5, but improved to 1 and 3 when calculators were allowed.

Now I'm faced with the question of whether I should continue to struggle through the first semester and do my best to pass. Or whether I should switch to psychology, which I'm also very interested in, but would require moving since my current university doesn't offer it.
I also thought that an objective opinion from others might be helpful.