LessWrong 2.0 Reader

I do use programs that give probabilistic outputs here. See claim 1 and the setup section.

I am fairly sure that there is a version of Solomonoff induction where the programs themselves output probabilities, and it's equivalent in the limit to the version where programs output binary answers. I think it's called 'stochastic' Solomonoff induction, or something like that.

I hadn't actually realised how load-bearing this might be for the proof until you pointed it out though. Thank you.
 

Simplified the solomonoff prior is the distribution you get when you take a uniform distribution over all strings and feed them to a turing machine.

Since the outputs are also strings: What happens if we iterate this? What is the stationary distribution? Is there even one? The fixed points will be quines, programs that copy their source code to the output. But how are they weighted? By their length? Presumably you can also have quine-cycles of programs that generate each other in turn, in a manner reminiscent metagenesis. Do these quine cycles capture all probability mass or does some diverge?

Very grateful for answers and literature suggestions.

The time and space bounded SI is then approximately optimal in the sense that its total error compared to this efficient predictor, as measured by KL-divergence from the predictions made by $P^{\ast }$, will be $\le \left|P^{\ast }\right|$ bits summed across all data points.^[7] 

I think that classical Solomonoff induction gives zero posterior to any program with less than perfect prediction record? I can see why this works for Solomonoff with unbounded description length, this is solved by the DH(h) term you mention above.

But for bounded Solomonoff you have to allow some non-perfect programs to stay in the posterior, or you might be left with no candidates at all? 

Is there an easy way to deal with this?

This is not a problem if the programs are giving probabilistic outputs, but that's a different class of programs than used in classical Solomonoff induction.

When you start a new chat, you reset the memory, if I understand it correctly. Maybe you should do that once in a while. Then you may need to explain stuff again, but maybe it gives you a new perspective? Or you could write the background story in a text file, and copy-paste it to each new chat.

Could the LLM accidentally reinforce negative thought patterns or make unhelpful suggestions?

I am not an expert, but I think that LLMs are prone to agreeing with the user, so if you keep posting negative thought patterns, there is a risk that LLM will reflect them back to you.

What if the LLM gives advice that contradicts what my therapist says? How would I know what to do?

Trust the therapist, I guess? And maybe bring it up in the next session, kinda "I got an advice to do X, what is your opinion on that?".

What is the risk of becoming too dependent on the LLM, and how can I check for that?

No idea. People are different; things that are harmless for 99 may hurt 1. I don't know you.

Are there specific prompts or ways of talking to the LLM that would make it safer or more helpful for this kind of use?

Just guessing here, but maybe specify its role (not just "you are a therapist", but specifically e.g. "a Rogerian therapist"), and specify the goal you want to achieve ("your purpose is to help me get better in my everyday life" or something).

Maybe at the end ask LLM for a summary of things that you should tell your human therapist? Something like "now I need to switch to another therapist, prepare notes for him so that he can continue the therapy".

My best guess is around 2/3.

Something related that I find interesting, for people inside a company, the real rival isn't another company doing the same thing, but people in your own company doing a different thing.

Imagine you work at Microsoft in the AI research team in 2021. Management want to cut R&D spending, so either your lot or the team doing quantum computer research are going to be redundant soon. Then, the timeline splits. In one universe, Open AI release Chat GPT, in the other PsiQuantum do something super impressive with quantum stuff. In which of those universes do the Microsoft AI team do well? In one, promotions and raises, in the other, redundancy.

People recognise this instinctively. Changing companies is much faster and easier than changing specialities. So people care primarily about their speciality doing well, their own specific company is a secondary concern.

A fusion expert can expert at a different fusion company way faster and more easily than they can  become an expert in wind turbines. Therefore, to the fusion expert all fusion companies are on the same side against the abominable wind farmers. I suspect this is also true of most people in AI, although maybe when talking to the press they will be honour bound to claim otherwise.

I wonder if any of the perceived difference between fusion and AI might be which information sources are available to you. It sounds like you have met the fusion people, and read their trade magazines, and are comparing that to what mainstream news says about AI companies (which won't necessarily reflect the opinions of a median AI researcher.).

Also worth considering is that how much an "institution" holds a view on average may not matter nearly as much as how the powerful decision makers within or above that institution feel.

This is an interesting prediction, and if it is true that agents are blocked primarily because of alignment issues, this would both be an update that AI alignment is harder than I think, but also that there are stronger incentives to solve the alignment problem than I think, as well.

Many [LW · GW] people expect 2025 to be the Year of Agentic AI. I expect that too - at least in the sense of it being a big topic. But I expect people to eventually be disappointed. Not because the AI is failing to make progress but for more subtle reasons. These agents will not be aligned well - because, remember, alignment is an unsolved problem. People will not trust them enough. 

I'm not sure how the dynamic will unfold. There is trust in the practical abilities. Right now it is low, but that will only go up. There is trust in the agent itself: Will it do what I want or will it have or develop goals of its own? Can the user trust that its agents are not influenced in all kinds of ways by all kinds of parties - the labs, the platforms, the websites the agents interact with, attackers, or the governments. And it is not clear which aspect of trust will dominate at which stage. Maybe the bioggest problems will manifest only after 2025, but we should see some of this this year.

Yeah I agree that with hindsight, the conclusion could be better explained and motivated from first principles, rather than by running an experiment. I wrote this post in the order in which I actually tried things as I wanted to give an honest walkthrough of the process that lead me to the conclusion, but I can appreciate that it doesn't optimise for ease to follow.