LessWrong 2.0 Reader

If the untrusted model has a private and un-audited chain of thought (like o1), and that chain of thought is preserved between rounds, the untrusted model may have a very good idea of what its original command was before the auditor replaced the command.

Your "the untrusted model is stateless between queries and only sees the command history and system state" does cover this, but wanted to call out that that assumption might not hold if current architectures are indicative of the future direction of development.

links 11/18/2024: https://roamresearch.com/#/app/srcpublic/page/11-18-2024

• [[links]]
  • https://en.m.wikipedia.org/wiki/Nicholas_Roerich  I like his art; he seems to have led an interesting life
  • https://en.m.wikipedia.org/wiki/Maurice_Quentin_de_La_Tour Rococo portraits
  •  
  • https://www.zach.be/p/yc-is-wrong-about-llms-for-chip-design
    • this doesn't seem to be a coherent argument -- author claims both that LLMs will indeed be used by chip design companies to save on human labor, and that they're not very good and can only replace the more routine parts of a chip designer's work. But...this isn't "YC is wrong, there is not a startup opportunity in using LLMs for chip design!" even if it's true it's just a more measured and realistic picture of what LLMs will be doing in chip design!
      • and they even make sure to shill their own AI-for-chip-hardware startup, which they claim is working on challenges that nobody else is: https://www.normalcomputing.com/
  • https://www.programmablemutter.com/p/how-chaotic-is-trump-ii-going-to speculation about Donald Trump's foreign policy
  • https://en.m.wikipedia.org/wiki/Daniel_Defoe  i want to read his nonfiction
  • https://en.m.wikipedia.org/wiki/Kotex sanitary pads have been around since 1920
  • https://asteriskmag.com/issues/08/the-death-and-life-of-prediction-markets-at-google  the tl;dr: Google's first internal prediction market was intended to go public and stalled as the regulatory environment remained unfriendly; its successor focused on predictions about competitor activity rather than Google's own activity (which is less subject to "office politics" considerations blocking an honest assessment of Google's chances of success etc) and never intended to be public-facing, and is still in active use.
  • https://en.m.wikipedia.org/wiki/Fran%C3%A7oise_d%27Aubign%C3%A9,_Marquise_de_Maintenon
  • https://en.m.wikipedia.org/wiki/Madame_de_Pompadour  patroness of the French Enlightenment.
    • "who was that woman who was an influential mistress to the King of France, politically savvy and queen in all but name"? "which one? this just keeps happening"
  • https://nintil.com/dont-assume this is Jose Luis Ricon life advice, which unsurprisingly tells you to be less afraid of things and talk to people more.
    • shan't.
  • https://en.m.wikipedia.org/wiki/Bill_Monroe
  • https://calisphere.org/item/0bc1137c37161874ded76c71bf982409/  scary guys
  •  

It seems to me like the "more careful philosophy" part presupposes a) that decision-makers use philosophy to guide their decision-making, b) that decision-makers can distinguish more careful philosophy from less careful philosophy, and c) that doing this successfully would result in the correct (LW-style) philosophy winning out. I'm very skeptical of all three.

Counterexample to a): almost no billionaire philanthropy uses philosophy to guide decision-making.

Counterexample to b): it is a hard problem to identify expertise in domains you're not an expert in.

Counterexample to c): from what I understand, in 2014, most of academia did not share EY's and Bostrom's views.

My intuition is to get less excited by single projects (a Double Crux bot) until someone has brought them all together & created momentum behind some kind of "big" agglomeration of people + resources in the "neutrality tools" space. 

I didn't know about all the existing projects and I appreciate the resource! Concrete >> vague in my book, I just didn't actually know much about concrete examples.

Cool. Is this right? For something with a 1/n chance of success I can have a 95% chance of success by making 3n attempts, for large values of n. About what does "large" mean here?

A cool way to measure dishonesty: How many people claim to have completed an impossible five minute task.

This has since been community noted, fairly from my understanding.  

This graph is not about how many people reported completing a task in 5 minutes when that was not true, this graph shows how many people completed the whole task even though it took them more than 5 minutes (which was all the time they were getting paid for).  
 

Yep, Seth has really clearly outlined the strategy and now I can see what I missed on the first reading. Thanks to both of you!

How would you compare your ideas here to Asimov's fictional science of psychohistory? I ask because while reading this post I kept getting flashbacks to Foundation.

Models or real objects or things capture something that is not literally present in the world. The world contains shadows of these things, and the most straightforward way of finding models is by looking at the shadows and learning from them. Hypotheses is another toy example.

One of the features of models/things seems to be how they capture the many possibilities of a system simultaneously, rather than isolated particular possibilities. So what I gestured at was that when considering models of humans, the real objects or models behind a human capture the many possibilities of the way that human could be, rather than only the actuality of how they actually are. And this seems useful for figuring out their preferences.

Path-dependence is the way outcomes depend on the path that was taken to reach them. A path-independent outcome is convergent, it's always the same destination regardless of the path that was taken. Human preferences seem to be path dependent on human timescales, growing up in Egypt may lead to a persistently different mindset from the same human growing up in Canada.