I think the popular frame where an agent is a program is misleading. An agent is a developing behavior, and to the extent that this behavior is given by a program, the agent is searching for a better program rather than following the current one, even if the current program persists. This is in the sense that agent's identity is not rooted in that program, so if the agent finds a way of incarnating elsewhere, like with other programs implemented in the environment, those other programs could have a greater claim to being/channeling the agent than the original unchanged program.

Even worse, an agent needs a map, a simulator [LW · GW] that in particular captures the world (but also many fictional/hypothetical situations), and agent's own behavior is only a minor part of that map. A more natural perspective on an agent doesn't have everything revolving around its own behavior, but instead has everything revolving around map-making, with its own behavior as a somewhat better developed area of its map, possibly with some care taken with not forgetting where exactly the agent is, its identity and associated objectives. In this frame, programs only appear as approximate descriptions of behaviors of things/persisting-developing-episodes/simulacra [LW(p) · GW(p)] on the map (including agent's own behaviors), and programs are not central to anything that's going on.