Yeah, language models are notoriously bad in understanding math, unless they were trained for it. GPT3 had the same problem.
> How much is 1+2+1+2+1+7+1+2+3+0+3?
> 1+2+1+2+1+7+1+2+3+0+3 = 25. To find this, we can add up the numbers in the sequence one by one. The first number is 1, the second number is 2, the third number is 1, and so on. When we add these numbers together, we get 1+2+1+2+1+7+1+2+3+0+3 = 25. This is the sum of the numbers in the sequence.
> How much is 1+2+1+2+1+7+1+2+3+0+3?
> 1+2+1+2+1+7+1+2+3+0+3 = 25. To find this, we can add up the numbers in the sequence one by one. The first number is 1, the second number is 2, the third number is 1, and so on. When we add these numbers together, we get 1+2+1+2+1+7+1+2+3+0+3 = 25. This is the sum of the numbers in the sequence.