Comment by evoth on Sapient Algorithms · 2023-07-18T00:05:22.816Z · LW · GW

In thinking about my own "sapient algorithms", something they may be causing them to reinforce themselves and spread is the fact that in these cases, the reward of being present is greater than that of continuing on autopilot. 

If remembering to check for my wallet prevents me from forgetting it, then maybe my brain picks up on that and extends it to similar situations which could benefit from a similar presence of mind. Conversely, mindlessly opening an app on my phone historically reduces momentary boredom, so that's what I'll end up doing in that scenario unless I intentionally counteract this by inserting a more beneficial (i.e. greater reward) action in its place.

So, even if the outcome of the sapient algorithm is more unpleasant than the autopilot alternative (like in your caffeine example), perhaps the extra agency gained in the process (and recognition of the value of this agency) is a reward in itself.