Take over my project: do computable agents plan against the universal distribution pessimistically?

post by Cole Wyeth (Amyr) · 2025-02-19T20:17:04.813Z · LW · GW · 3 comments

This is a question post.

Contents

   Why you might want to do this:
  Why you might not want to do this:
None
3 comments

This post is for technically strong people looking for a way to contribute to agent foundations (more detailed pros and cons later). I have not tried as hard as usual to make it generally accessible because, you know, I am trying to filter a little - so expect to click through some links if you want to follow. One other reason to read this post is if you know more than me about the title, in which case please enlighten me / point me in the right direction. I'd particularly like to hear from an Infra-Bayesiamism expert. 

It has been shown that the average-case performance of an algorithm on samples drawn from the universal distribution is the worst-case performance of that algorithm. I think this is a sort of justification for worst-case analysis and possibly an underlying reason that randomized algorithms are powerful (their worst-case isn't as bad because they effectively mix over deterministic algorithms). More speculatively, I think this may also be the source of intuition fueling max-min style decision algorithms under "Knightian uncertainty" - what philosophers call decision under ignorance rather than risk. If you deal with Knightian uncertainty using an ignorance prior like a good Bayesian, and choose a Solomonoff/Hutter style universal distribution like a good algorithmic information theorist, worst-case style thinking should be appropriate. Even more speculatively, I think this may justify a form of Infra-Bayesianism for computationally bounded reasoners (this is beyond my expertise). 

I worked on a weak form of this claim off-and-on for a few months and then abandoned it. I seem to do my best writing/work with at least one co-author to keep me grounded and make me eventually stop generalizing/perfecting and actually wrap my projects (hopefully you in this case, my dear reader). At this point I seem unlikely to pick it up again on my own in the near future. I have some notes mostly proving (roughly) that a computable planner should use a pessimistic -> stochastic planning strategy against the universal distribution, which I would like to pass off to someone (who would then be first author). If the specific ideas/approaches/formalization in the notes seem less interesting than the general idea, I'm fine with you just kicking me from the project and not using my notes (maybe leave me an acknowledgment in your paper/post) or wrapping my approach up and then moving on to prove more powerful related results. 

 Why you might want to do this:

1. You have a strong background in statistics / probability / algorithms but haven't previously contributed to agent foundations / alignment theory and would like to. In this case, if the idea seems very unusually interesting to you, the project might be a good on-ramp. I can provide guidance for up to a few hours a week.  Also I think the expected outcome is good - I am pretty confident that at minimum we can carry my preliminary ideas through to some semi-interesting results that I will probably be able to use in the future. There also seems to be a lot of upside; the best-case outcome (which relies on you being really brilliant) would be quite useful unification of various ideas in decision theory through algorithmic information theory.    

   I believe that someone specializing in statistical estimation would have a comparative advantage over me.

2. You are an experienced researcher in e.g. IB and this sounds trivial - after glancing at my notes or just hearing this idea you immediately see how to prove a more interesting version of my claim by a technically superior method. In that case, why not write it down?

Why you might not want to do this:

1. It doesn't sound interesting to you on its own merits -> don't do it
2. I am a lowly second-year PhD student; perhaps the idea is not as good as I think. If the reason is a lack of novelty, that should be reflected in the comments.
3. My notes are currently a bit messy and (obviously) incomplete, which is why I haven't tried to publish them and why this post exists.
4. Maybe you don't want to add more authors (myself and potentially a supervisor) to whatever you end up publishing (though this can be fine as discussed above).
5. I can't promise to pay you. However, if we start a prolonged and fruitful collaboration I can probably help you find a funding source.  
6. This sounds interesting, but one of the other open problems [LW · GW] on my list sounds more interesting - in that case, please say so!

If you are interested, just comment or DM me!

Answers

3 comments

Comments sorted by top scores.