LessWrong 2.0 Reader

I'd go a step beyond this: merely following incentives is amoral. It's the default. In a sense, moral philosophy discusses when and how you should go against the incentives. Superhero Bias [LW · GW] resonates with this idea, but from a different perspective.

I don't understand why you would short the market if your P(Doom) is high. I think most Dooms don't involve shorts paying off?

don't see the downstream impacts of their choices,

This could be part of it... but I think a hypothesis that does have to be kept in mind is that some people don't care. They aren't trying to follow action-policies that lead to good outcomes, they're doing something else. Primarily, acting on an addiction to Steam [LW · GW]. If a recruitment strategy works, that's a justification in and of itself, full stop. EA is good because it has power, more people in EA means more power to EA, therefore more people in EA is good. Given a choice between recruiting 2 agents and turning them both into zombies, vs recruiting 1 agent and keeping them an agent, you of course choose the first one--2 is more than 1.

I don't think the assumption is highly dubious. You don't need to believe in the possibility of mind uploading or biological immortality to assume radically transformative changes in the human condition due to advanced AI. The "Neuroscience and Mind" section of Dario Amodei's (who has a formal background in biophysics) essay attempts to clearly speculate what'll happen in these areas with the helped of advanced AI (even setting aside that mind uploading is probably "possible in principle").

Even if some goals are unattainable,  AGI could still likely (as Dario speculates) drive radical advancements in areas like health, longevity, and cognitive enhancement. The point isn't to guarantee specific outcomes, but to recognise that AGI will likely push the boundaries of what we currently believe is possible and transform the world unrecognisably. And we should be mentally reflecting and preparing for that.

If k is even, then k^x is even, because k = 2n for n in $Z$ and we know (2n)^x is even. But do LLMs know this trick? Results from running (a slightly modified version of) https://github.com/rhettlunn/is-odd-ai. Model is gpt-3.5-turbo, temperature is 0.7. 

Is 50000000 odd? false
Is 2500000000000000 odd? false
Is 6.25e+30 odd? false
Is 3.9062500000000007e+61 odd? false
Is 1.5258789062500004e+123 odd? false
Is 2.3283064365386975e+246 odd? true
Is Infinity odd? true

If a model isn't allowed to run code, I think mechanistically it might have a circuit to convert the number into a bit string and then check the last bit to do the parity check.

The dimensionality of the residual stream is the sequence length (in tokens) * the embedding dimension of the tokens. It's possible this may limit the maximum bit width before there's an integer overflow. In the literature, toy models definitely implement modular addition/multiplication, but I'm not sure what representation(s) are being used internally to calculate this answer.

Currently, I believe it's also likely this behaviour could be a trivial BPE tokenization artifact. If you let the model run code, it could always use %, so maybe this isn't very interesting in the real world. But I'd like to know if someone's already investigated features related to this.

Yeah, that means you get exactly one chance to get "its own ideas" right, and no, I don't think that success is likely.

Not if you built a model that does (or on reflection decides to do) value learning [? · GW]: then you instead get to be its research subject and interlocutor while it figures out its ideas. But yes, you do need to start the model off close enough to aligned that it converges to value learning. [LW · GW]

I think not enforcing an "in or out" boundary is big contributor to this degradation -- like, majorly successful religions required all kinds of sacrifice.

I feel ambivalent about this.  On one hand, yes, you need to have standards, and I think EA's move towards big-tentism degraded it significantly. On the other hand I think having sharp inclusion functions are bad for people in a movement^[1], cut the movement off from useful work done outside itself, selects for people searching for validation and belonging, and selects against thoughtful people with other options [EA · GW]. 

I think I'm reasonably Catholic, even though I don't know anything about the living Catholic leaders.

I think being a Catholic with no connection to living leaders makes more sense than being an EA who doesn't have a leader they trust and respect, because Catholicism has a longer tradition, and you can work within that. On the other hand... I wouldn't say this to most people, but my model is you'd prefer I be this blunt... my understanding is Catholicism is about submission to the hierarchy, and if you're not doing that or don't actively believe they are worthy of that, you're LARPing. I don't think this is true of (most?) protestant denominations: working from books and a direct line to God is their jam. But Catholicism cares much more about authority and authorization. 

It feels like AI safety is the best current candidate for [lifeboat], though that is also much less cohesive and not a direct successor for a bunch of ways. I too have been lately wondering what "Post EA" looks like.

I'd love for this to be true because I think AIS is EA's most important topic. OTOH, I think AIS might have been what poisoned EA? The global development people seem much more grounded (to this day), and AFAIK the ponzi scheme recruiting is all aimed at AIS and meta (which is more AIS). ETG was a much more viable role for GD than for AIS. 

1. ^{^}

   If you're only as good as your last 3 months, no one can take time to rest and reflect, much less recover from burnout. 

There is a straightforward compmech take also. If the goal of the agent is simply to predict well (let's say the reward is directly tied to good prediction) for a sequential task AND it performs optimally then we know it must contain the Mixed State Presentation of the epsilon machine (causal states). Importantly the MSP must be used if optimal prediction is achieved.

There is a variant I think, that has not been worked out yet but we talked about briefly with Fernando and Vanessa in Manchester recently for transducers /MDPs

I think those are the meta because they have just enough space to not only give opinions but to mention reasons for those opinions and expertise/background to support the many unstated judgment calls.

Note that the essays by Altman and Amodei are popular because their positions are central beyond the others because they have not only demonstrable backgrounds in AI but lots of name recognition (we're mostly assuming Altman has bothered learning a lot about how Transformers work even if we don't like him). And that the Gladstone report got itself commissioned by at least a little piece of the government.

A Narrow Path just demonstrates in the text that you and your co-authors have thought deeply about the topic. Shorter essays leave more guesswork on the authors' expertise and depth of consideration.

That link appears not to work. I'd be quite interested in what those numbers were 10 years ago when Deepmind and the like were getting excited about DNNs, but it wasn't that interesting to the wider world, who generally didn't believe that something like what is happening could happen. 

(I believed in the theory of superintelligence, like there's an exponential that's going to go past this arbitary point eventually, and IIRC had wildly wildly too distant expectations of when it might begin to happen in any meaningful way. Just thinking back to that time makes the last couple of years shocking to comprehend.)