LessWrong 2.0 Reader

Does your model predict literal worldwide riots against the creators of nuclear weapons? They posed a single-digit risk of killing everyone on Earth (total, not yearly).

It would be interesting to live in a world where people reacted with scale sensitivity to extinction risks, but that's not this world.

You're encouraged to write a self-review, exploring how you think about the post today. Do you still endorse it? Have you learned anything new that adds more depth? How might you improve the post? What further work do you think should be done exploring the ideas here?

Still endorse. Learning about SIA/SSA from the comments was interesting. Timeless but not directly useful, testable or actionable.

If you want [abstractapplic]'s feedback on anything, let me know.

I have received creative feedback from abstractapplic. It was useful and made me happy. This is an endorsement.

I am down to some level of tagging along and learning together, but not a full commitment. You probably want to find someone that can make a stronger commitment as an actual study partner.

I am a year 3 student (which means I may already know some of the stuff, and that I have other courses) and timezones likely suck (UTC+8 here). We can discuss on discord @papetoast if you like.

I thought this quote was nice and oddly up-to-the-minute, from Iris Murdoch's novel The Philosopher's Pupil (1983), spoken by the character William Eastcote at a Quaker meeting:

My dear friends, we live in an age of marvels. Men among us can send machines far out into space. Our homes are full of devices which would amaze our forebears. At the same time our beloved planet is ravaged by suffering and threatened by dooms. Experts and wise men give us vast counsels suited to vast ills. I want only to say something about simple good things which are as it were close to us, within our reach, part still of our world. Let us love [LW · GW] the close things, the close clear good things, and hope that in their light other goods may be added. Let us prize innocence [LW · GW]. The child is innocent, the man is not. Let us prolong and cherish the innocence of childhood, as we find it in the child and as we rediscover it later within ourselves. Repentance, renewal of life, such as is the task and possibility of every man, is a recovery of innocence. Let us see it thus, a return to a certain simplicity [LW · GW], something which is not hard to understand, not a remote good but very near. And let us not hesitate to preach to our young people and to impart to them an idealism which may later serve them as a shield. A deep cynicism in our society too soon touches old and young, forbidding us to speak and them to hear, and making us by an awful reversal ashamed [LW · GW] of what is best. A habit of mockery destroys the intelligence and sensibility which is reverence [LW · GW]. Let us prize chastity [LW · GW], not as a censorious or rigid code, but as fastidious respect [LW · GW] and gentleness, a rejection of promiscuity, a sense of the delicate mystery of human relations. Let us do and praise those things which make for a simple orderly open and truthful [LW · GW] life. Herein let us make it a practice to banish evil thoughts. When such thoughts come, envious, covetous, cynical thoughts, let us positively drive them off, like people in the olden days who felt they were defeating Satan. Let us then seek aid in pure things, turning our minds to good people, to our best work, to beautiful and noble art, to the pure words of Christ in the Gospel, and to the works of God obedient to Him in nature. Help is always near if we will only turn. Conversion is turning about, and it can happen not only every day but every moment. Shun the cynicism which says that our world is so terrible that we may as well cease to care and cease to strive, the notion of a cosmic crisis where ordinary duties cease to be and moral fastidiousness is out of place. At any time, there are many many small things we can do for other people [LW · GW] which will refresh us and them with new hope [LW · GW]. Shun too the common malice which finds consolation in the suffering and sin of others, blackening them to make our grey seem white, rejoicing in our neighbours’ downfall and disgrace, while excusing our own failures and cherishing our own undiscovered secret sins. Above all, do not despair, either for the planet or in the deep inwardness of the heart. Recognize one’s own evil, mend what can be mended, and for what cannot be undone, place it in love and faith in the clear light of the healing goodness of God.

How could you possibly know something like that?

For example, I’m sure I’ve looked up what “rostral” means 20 times or more since I started in neuroscience a few years ago. But as I write this right now, I don’t know what it means. (It’s an anatomical direction, I just don’t know which one.) Perhaps I’ll look up the definition for the 21st time, and then surely forget it yet again tomorrow. :)

What else? Umm, my attempt to use Anki was kinda a failure. There were cards that I failed over and over and over, and then eventually got fed up and stopped trying. (Including “rostral”!) I’m bad with people’s names—much worse than most people I know. Stuff like that.

Most people do not read many books or spend time in spaces where SAT vocab words would be used at all…

If we’re talking about “most people”, then we should be thinking about the difference between e.g. SAT verbal 500 versus 550. Then we’re not talking about words like inspissate, instead we’re talking about words like prudent, fastidious, superfluous, etc. (source: claude). I imagine you come across those kinds of words in Harry Potter and Tom Clancy etc., along with non-trashy TV shows.

I don’t have much knowledge here, and I’m especially clueless about how a median high-schooler spends their time. Just chatting :)

This is actually a good use case, which fits with what gpt does well, where very cheap tokens help!

Pending some time for people to pick at it to test it's limits, this might be really good. My instinct is legal research, case law etc. will be the test of how good it is, if it does well this might be it's foothold into real commercial use that actually generates profit.

My prediction is that we will be glad this exists. It will not be "phd level", a phrase which defaces all who utter it, but it will save some people a lot of time and effort

Where I think we disagree: This will likely not elicit a Jevon's-paradox scenario where we will collectively spend much more money on LLM tokens despite their decreased cost, Killer app this is not.

My prediction is that low level users will use this infrequently because Google (or vanilla chatGPT) is sufficient, what they are looking for is not a report but a webpage and one likely at the top of their search already. Even if it would save them time, they will never use it so often that their first instinct would be deep research and not Google, they will not recognize where deep research would be better and won't change their habits even if they do. On the far end, some grad students will use this to get them started but it will not do the work of actually doing the research. Besides pay walls disrupting things and limits to important physical media, there is a high likelihood that this won't replace any of the actual research grad students (or lawyers/paralegals etc) will have to do. The number of hours they spend won't be much effected, the range of users who will find much value will be few and they probably won't use it every day.

I expect that, by token usage, deep research will not be a big part of what people use chatGPT for. If I'm wrong I predict it's because law professions found a use for it.

I will see everyone in 1 year (if we're alive) to see if this pans out!

It seems that their conclusion was that no amount of happy moments for people could possibly outweigh the unimaginably large quantity of suffering in the universe required to sustain those tiny flickers of merely human happiness amid the combined agony of a googolplex or more fundamental energy transitions within a universal wavefunction. There is probably some irreducible level of energy transitions required to support anything like a subjective human experience, and (in the context of the story at least) the total cost in suffering for that would be unforgivably higher.

I don't think the first half would definitely lead to the second half, but I can certainly see how it could.

Things the OP is concerned about like 

"What makes this transition particularly hard to resist is that pressures on each societal system bleed into the others. For example, we might attempt to use state power and cultural attitudes to preserve human economic power. However, the economic incentives for companies to replace humans with AI will also push them to influence states and culture to support this change, using their growing economic power to shape both policy and public opinion, which will in turn allow those companies to accrue even greater economic power."

This all gets easier the smaller the society is. Coordination problems get harder the more parties involved. There will be pressure from motivated people to make the smallest viable colony in terms of people, which makes it easier to resist such things. For example there is much less effective cultural influence from the AI culture if the colony is founded by a small group of people with shared human affirming culture. Even if 99% of the state they come from is disempowered if small numbers can leave, they can create their own culture and set it up to be resistant to such things. Small groups of people have left decaying cultures throughout history and founded greater empires.

Spot check regarding pedestrians, at current time RSS "rule 4" mentions:

In a crowded school drop-off zone, for example, humans instinctively drive extra cautiously, as children can act unpredictably, unaware that the vehicles around have limited visibility.

The associated graphic also shows a pedestrian. I'm not sure if this was added more recently, in response to this type of criticism. From later discussion I see that pedestrians were already included in the RSS paper, which I've not read.