I think the answer to whether numbers are in the map or territory is simply mu. A map describes the territory and the territory is made out of atoms. Numbers are clearly not made out of atoms, so they can't exist in the territory. But they don't fit into the map either because the map must change when the territory changes, but numbers don't change no matter how many atoms you move around or how much you change how atoms interact with each other. Suppose an all-powerful being changed the world so that any time two objects got near each other, a third appears: 1 + 1 = 3. The entire math canon could still work, and you could still spend lifetimes describing it, but it wouldn't be helpful for describing how the world works. It would be neither in the map, nor in the territory, but it would still be real. Mu.