Math is Subjunctively Objective
2+3=5 is an outcome of a set of artificial laws we can imagine. In that sense, it does exist "purely in your imagination", just as any number of hypothetical systems could exist. "2+3=5" doesn't stand alone without defining what it means - ie. the concept of a number, addition etc. It corresponds to the statement that IF addition is defined like so, numbers like this, and such-and-such rules of inference, then 2+2=5 is a true property of the system.
In a counterfactual world where people believe 2+3=6, in asking about addition you're still talking about the same system with the same rules, not the rules that describe whatever goes on in the minds of the people. (Otherwise you would be making a different claim about a different system.)
So yes, 2+3=5 is clearly true and has always been true even before humans because its a statement about a system defined in terms of its own rules. Any claims about it already include the system's presumptions because those are part of the question, and part of what it means to be "true".
2 rocks + 3 rocks is a different matter - you're talking about the observable world rather than a system where you get to define all the rules in advance. To apply mathematical reasoning to the real world, you have to make the additional claim "combining physical items is isomorphic to the rules of addition", and you're now in the realm of justifying this with empirical evidence. (Of which there is plenty)
I think a better phrasing of your final question then is to ask why do physical systems seem to correspond to the rules of this particular system, but there is a degree of circularity there - obviously we haven't just made up the rules of mathematics arbitrarily - we've based the lowest levels on recognised concepts, and then found that the same laws seem to apply at very deep levels with very high degrees of congruence with the world. If the universe were somehow different and nothing ever acted in any way corresponding to our model of "addition" or "numbers" though, then we'd not attach any special significance to it. Our "mathematics" would be quite different, and we'd be asking the same question about that system.