LessWrong 2.0 Reader

(not a lawyer) 

My layman's understanding is that managerial employees are excluded from that ruling, unfortunately. Which I think applies to William_S if I read his comment correctly. 

Does anyone know if it's typically the case that people under gag orders about their NDAs can talk to other people who they know signed the same NDAs? That is, if a bunch of people quit a company and all have signed self-silencing NDAs, are they normally allowed to talk to each other about why they quit and commiserate about the costs of their silence?

Preface: I think this comment will be pretty unpopular here.

I think this is a very unhelpful frame for any discussion (especially so the more high-stakes it is) for the reasons that SlateStarCodex outlines in Against Bravery Debates, and I think your comment would be better with this removed.

Uh I'm saying I think henry's is better. Except for the title maybe.

Preface: I think this comment will be pretty unpopular here.


This style of thinking seems illogical to me. It has already clearly resulted in a sort of evaporative cooling in OpenAI. At a high level, is it possible you have the opposite of a wishful thinking bias you claim OpenAI researchers have? I won't go into too much detail about why this post doesn't make sense to me. as others already have. 
But broadly speaking:

• I doubt rationality gives you too much of an advantage in capabilities research, and believing this when on a site full of rationalists seems a little pretentious almost. 
   
• I also have no idea how any alignment research so far has helped capabilities in any way.  I don't even know how RLHF has helped capabilities. If anything, it's well documented that RLHF diminishes capabilities (base models can for example play chess very well). The vast majority of alignment research, especially research before LLMs,  isn't even useful to alignment (a lot of it seems far too ungrounded).  
   
• There was never a real shot of solving alignment until LLMs became realized either.  The world has changed and it seems like foom priors are wrong, but most here haven't updated. It increasingly seems like we'll get strong precursor models so we will have ample time to engineer solutions and it won't be like trying to build a working rocket on the first try. (The reasons being we are rapidly approaching the limits of energy constraints and transistor density without really being close to fooming). I don't get why this mental model is still so popular when reality clearly diverges.
  
   

Well I actually have a hunch to why, and this will probably be the least popular part of this comment. I hypothesize many holding on to the above priors don't want to let them go because that means this problem they have dedicated a lot of mental space to will seem more feasible to solve. 

If it's instead a boring engineering problem, this stops being a quest to save the world or an all consuming issue. Incremental alignment work might solve it, so in order to preserve the difficulty of the issue, it will cause extinction for some far-fetched reason. Building precursor models then bootstrapping alignment might solve it, so this "foom" is invented and held on to (for a lot of highly speculative assumptions), because that would stop it from being a boring engineering problem that requires lots of effort and instead something a lone genius will have to solve. The question that maybe energy constraints will limit AI progress from here on out was met with a maybe response [LW(p) · GW(p)], but the number of upvotes make me think most readers just filed it as an unconditional "no, it won't" in their minds.

There is a good reason to think like this - if boring engineering really does solve the issue, then this community is better off assuming it won't. In that scenario, boring engineering work is being done by the tech industry anyways, so no need to help there. But I hope if people adopt the mindset of assuming the worst case scenario to have the highest expected effects of research, they realize the assumption they are making is an assumption, and not let the mental effects consume them.
 

What are your timelines like? How long do YOU think we have left?

I know several CEOs of small AGI startups who seem to have gone crazy and told me that they are self inserts into this world, which is a simulation of their original self's creation. However, none of them talk about each other, and presumably at most one of them can be meaningfully right?

One AGI CEO hasn't gone THAT crazy (yet), but is quite sure that the November 2024 election will be meaningless because pivotal acts will have already occurred that make nation state elections visibly pointless.

Also I know many normies who can't really think probabilistically and mostly aren't worried at all about any of this... but one normy who can calculate is pretty sure that we have AT LEAST 12 years (possibly because his retirement plans won't be finalized until then). He also thinks that even systems as "mere" as TikTok will be banned before the November 2024 election because "elites aren't stupid".

I think I'm more likely to be better calibrated than any of these opinions, because most of them don't seem to focus very much on "hedging" or "thoughtful doubting", whereas my event space assigns non-zero probability [LW · GW] to ensembles that contain such features of possible futures (including these specific scenarios).

May be the most surprising answer will be Paul Valéry. He is a great poet but during one night in 1892 he decided to spend all his life in solving the mystery of intelligence and wrote extensive notebooks about it. https://collecties.kb.nl/en/koopman-collection/1951-1960/cahiers 

I think it is safe to infer from the conspicuous and repeated silence by ex-OA employees when asked whether they signed a NDA which also included a gag order about the NDA, that there is in fact an NDA with a gag order in it, presumably tied to the OA LLC PPUs (which are not real equity and so probably even less protected than usual).

But @henry [LW · GW] was involved in the explainer, so it's nice for him to get the comment notifications and the karma.

Yep! My guess is I will send one out to people who bought tickets next week, along with various spreadsheets for signing up to activities with (e.g. giving a lightning talk).

(I personally strongly prefer slack for a bunch of UI reasons including threading and especially because I always find the conversational culture on discord disorienting, though I know manifest has a community discord so it might be worth using discord.)