LessWrong 2.0 Reader

Technological progress leading to ever-better, ever-more-flexible communication technology, which serves as an increasingly more efficient breeding ground for ever-more-viral memes – and since virality is orthogonal to things like "long-term wisdom", the society ends up taken over by unboundedly destructive ideas?

Thanks. I've submitted my own post [LW · GW] on the 'change our mind form', though I'm not expecting a bounty. I'd instead be interested in making a much bigger bet (bigger than Cole's 100 USD), gonna think about what resolution criterion is best.

Thanks for thinking about this, I think this is an important topic!

Inside the AI's chain-of-thought, each forward pass can generate many English tokens instead of one, allowing more information to pass through the bottleneck.

I wonder how one would do this; do you mean allow the model to output a distribution of tokens for each output position? (and then also read-in that distribution) I could imagine this being somewhere between normal CoT and latent (neuralese) CoT!

After the chain-of-thought ends, and the AI is giving its final answer, it generates only one English token at a time, to make each token higher quality. The architecture might still generate many tokens in one forward pass, but a simple filter repeatedly deletes everything except its first token from the context window.

If my interpretation of your idea above is correct then I imagine this part would look just like top-k / top-p generation like it is done currently, which seems sensible.

I'm only ~30% certain that I correctly understood your idea though so I'd love if you could clarify how this generating many tokens idea looks like!

I made the same mistake when I was young. And it is difficult in retrospect to find out exactly why. I don't remember hearing or reading this explicitly, but somehow I got the idea that first you need to figure out who is your "true love" and then you need to ask them out and... hope that the feeling is reciprocated?

Which is why I have wasted lots of time worrying about how I truly feel about some person, and when I finally felt sure this was the right choice, I got rejected and was emotionally devastated.

And when I put it like this, of course it sounds completely stupid. If you want to figure out whether someone is a good fit for you, you need to interact with them, preferably in many different situations. And you should also interact with people who maybe don't seem like a good fit but also don't have any obvious red flags, because maybe you will change your opinion after you learn more about them.

The very concept of "true love" probably needs to be thrown away, because liking/wanting another person is not necessarily a symmetric thing, so there is a high chance of getting rejected; a better concept would be something like having a pool of "potential loves", people you feel you could be happy with, and if a few of them reject you it's perfectly okay because you only need one of them anyway. (Unless you are poly.) Plus, as you meet new people, he pool of "potential loves" can grow.

Another thing I wish I had understood better is that experimenting with romantic relations is not only ethically acceptable (as long as you do not hurt other people unnecessarily) but probably also necessary. You may sincerely think that someone is a good match for you based on the data you have at the moment... but as you start dating, you find out more about the person, and maybe you learn that they are actually not a good match. It may feel like tricking them, and you may actually get accused of having tricked them, but it's just learning things that you probably could not have learned different way, e.g. because people are different in public and in private.

Rejections are an indicator of progress.

You can make the same mistake hundred times and get rejected hundred times, that is no progress.

Also, you can use an approach that works in 10% of situations, you try it 10 times, get rejected 9 times and succeed once. Here, persistence led to success, but calling it "progress" feels kinda incorrect.

each rejection is valuable new data on what to do better next time.

People are different; sometimes two people want different (or even opposite) things.

I guess it makes sense to distinguish between improving in general, and adapting to a subculture. Some things seem to be always good, except some people care about them more, and some care less. (For example, I spent a lot of time and effort improving my dancing skills -- some women were highly impressed, some didn't care at all, but I haven't met anyone who would consider dancing skills a bad thing.) Other things can be admired by some people, and hated by other people, so if you do more of that, you increase your changes with the former, and decrease your chances with the latter. Here you need to figure out which type of people you want to impress. It probably helps to be flexible, to be able to do something, but also to stop doing that, but you still risk accidentally making a wrong first impression. This is further complicated by the fact that some people are in denial about what attracts them.

In my experience, everyone already understands that cream cheese frosting is (a) sweet, and (b) delicious. I have never met anyone, nor heard of anyone, who was somehow under the impression that cream cheese frosting is in any way incongruous or weird.

In other words, as far as I can tell, the problem you are describing is completely nonexistent.

cured fish.

Why would I do that to myself? I don’t feel my sins deserve that level of punishment.

Perhaps you are not aware of the lox & cream cheese bagel sandwich, a venerable and beloved item of New York City cuisine. If you have not had this food, then you are missing out on a singular life experience, and you are spiritually impoverished by this lack. I suggest rectifying this omission forthwith.

Please don't see downvotes as rejection. People on LessWrong downvote each other like crazy all the time.

There was this post discussing whether AI are being enslaved [LW · GW] which got -46 karma. The top comment compared the post to "taking a big 💩 in public"

The post author later wrote another post [LW · GW] in a similar vein, strongly criticizing people who downvoted her last time. Somehow the hivemind changed its mind and upvoted her this time.

Don't take downvotes too seriously :)

Of course, if you're looking for more than just upvotes, but for the post to have a major effect on the community's beliefs and strategies. My pessimistic take is that a single post almost never accomplishes this. Even a great post by Eliezer Yudkowsky usually has negligible influence on the community, except when he gets very lucky.

LessWrong is full of people frustrated at their words falling on deaf ears.^[1] Here is an example, Applying superintelligence without collusion [LW · GW], by Eric Drexler. He reflects despairingly that:

At t+7 years, I’ve still seen no explicit argument for robust AI collusion, yet tacit belief in this idea continues to channel attention away from a potential solution-space for AI safety problems, leaving something very much like a void.

I think I mostly agree with this, I think things possibly get more complicated when you throw decision theory into the mix. I think it unlikely I'm being adversarially simulated in part. I could believe that such malign prior problems are actually decision theory problems much more than epistemic problems. Eg "no, I am not going to do what the evil super-simple-simulators want me to do because they will try to invade my prior iff (I would act like they have invaded my prior iff they invade my prior)".

I think there's a world trend toward more authoritarian leaders, which contributed to both these events. And it should raise our probability of e.g. Turkey or China doing something silly. But where this trend comes from, I'm not sure. It certainly predates the current AI wave. It could be due to social media making people more polarized or something. But then again there were plenty of worse dictators in history, long before social media or electricity. So maybe what's happening now is regression to the mean, and nice democracy was an anomaly in place and time.

I find this possible though it's not my median scenario to say the least. But I am also not sure I can put the probability of such a fast development below 10%. 

Main cruxes: 

I am not so sure that "automating AI research" is going to speed up development by orders of magnitude. 

My experience is that cracked AI engineers can implement any new paper / well specified research idea in a matter of hours. So speeding up the coding can't be the huge speedup to R&D.

The bottleneck seems to be:
A.) Coming up with good research ideas.

b.) Finding the precise formulation of that idea that makes most sense/works.

LLMs so far are bad at both. So I currently only see them scouring the immediate neighbourhood of existing ideas, to eke out incremental progress in the current paradigm. 

Is that enough? Is an LLM building on a base model that has a loss close to the irreducible loss AGI? I.e. does accelerating this improvement matter for the transition to AGI and superintelligence? 

I think not even the authors believe that. So they make the leap of faith that accelerated research will make a qualitative difference too. I think there are additional gaps between human cognition and LLMs beyond recursive reasoning in latent space and sample efficiency. 

Will all those gaps be closed in the next few years?