LessWrong 2.0 Reader

I don't think the assumption is highly dubious. You don't need to believe in the possibility of mind uploading or biological immortality to assume radically transformative changes in the human condition due to advanced AI. The "Neuroscience and Mind" section of Dario Amodei's (who has a formal background in biophysics) essay attempts to clearly speculate what'll happen in these areas with the helped of advanced AI (even setting aside that mind uploading is probably "possible in principle").

Even if some goals are unattainable,  AGI could still likely (as Dario speculates) drive radical advancements in areas like health, longevity, and cognitive enhancement. The point isn't to guarantee specific outcomes, but to recognise that AGI will likely push the boundaries of what we currently believe is possible and transform the world unrecognisably. And we should be mentally reflecting and preparing for that.

If k is even, then k^x is even, because k = 2n for n in $Z$ and we know (2n)^x is even. But do LLMs know this trick? Results from running (a slightly modified version of) https://github.com/rhettlunn/is-odd-ai. Model is gpt-3.5-turbo, temperature is 0.7. 

Is 50000000 odd? false
Is 2500000000000000 odd? false
Is 6.25e+30 odd? false
Is 3.9062500000000007e+61 odd? false
Is 1.5258789062500004e+123 odd? false
Is 2.3283064365386975e+246 odd? true
Is Infinity odd? true

If a model isn't allowed to run code, I think mechanistically it might have a circuit to convert the number into a bit string and then check the last bit to do the parity check.

The dimensionality of the residual stream is the sequence length (in tokens) * the embedding dimension of the tokens. It's possible this may limit the maximum bit width before there's an integer overflow. In the literature, toy models definitely implement modular addition/multiplication, but I'm not sure what representation(s) are being used internally to calculate this answer.

Currently, I believe it's also likely this behaviour could be a trivial BPE tokenization artifact. If you let the model run code, it could always use %, so maybe this isn't very interesting in the real world. But I'd like to know if someone's already investigated features related to this.

Yeah, that means you get exactly one chance to get "its own ideas" right, and no, I don't think that success is likely.

Not if you built a model that does (or on reflection decides to do) value learning [? · GW]: then you instead get to be its research subject and interlocutor while it figures out its ideas. But yes, you do need to start the model off close enough to aligned that it converges to value learning. [LW · GW]

I think not enforcing an "in or out" boundary is big contributor to this degradation -- like, majorly successful religions required all kinds of sacrifice.

I feel ambivalent about this.  On one hand, yes, you need to have standards, and I think EA's move towards big-tentism degraded it significantly. On the other hand I think having sharp inclusion functions are bad for people in a movement^[1], cut the movement off from useful work done outside itself, selects for people searching for validation and belonging, and selects against thoughtful people with other options [EA · GW]. 

I think I'm reasonably Catholic, even though I don't know anything about the living Catholic leaders.

I think being a Catholic with no connection to living leaders makes more sense than being an EA who doesn't have a leader they trust and respect, because Catholicism has a longer tradition, and you can work within that. On the other hand... I wouldn't say this to most people, but my model is you'd prefer I be this blunt... my understanding is Catholicism is about submission to the hierarchy, and if you're not doing that or don't actively believe they are worthy of that, you're LARPing. I don't think this is true of (most?) protestant denominations: working from books and a direct line to God is their jam. But Catholicism cares much more about authority and authorization. 

It feels like AI safety is the best current candidate for [lifeboat], though that is also much less cohesive and not a direct successor for a bunch of ways. I too have been lately wondering what "Post EA" looks like.

I'd love for this to be true because I think AIS is EA's most important topic. OTOH, I think AIS might have been what poisoned EA? The global development people seem much more grounded (to this day), and AFAIK the ponzi scheme recruiting is all aimed at AIS and meta (which is more AIS). ETG was a much more viable role for GD than for AIS. 

1. ^{^}

   If you're only as good as your last 3 months, no one can take time to rest and reflect, much less recover from burnout. 

There is a straightforward compmech take also. If the goal of the agent is simply to predict well (let's say the reward is directly tied to good prediction) for a sequential task AND it performs optimally then we know it must contain the Mixed State Presentation of the epsilon machine (causal states). Importantly the MSP must be used if optimal prediction is achieved.

There is a variant I think, that has not been worked out yet but we talked about briefly with Fernando and Vanessa in Manchester recently for transducers /MDPs

I think those are the meta because they have just enough space to not only give opinions but to mention reasons for those opinions and expertise/background to support the many unstated judgment calls.

Note that the essays by Altman and Amodei are popular because their positions are central beyond the others because they have not only demonstrable backgrounds in AI but lots of name recognition (we're mostly assuming Altman has bothered learning a lot about how Transformers work even if we don't like him). And that the Gladstone report got itself commissioned by at least a little piece of the government.

A Narrow Path just demonstrates in the text that you and your co-authors have thought deeply about the topic. Shorter essays leave more guesswork on the authors' expertise and depth of consideration.

That link appears not to work. I'd be quite interested in what those numbers were 10 years ago when Deepmind and the like were getting excited about DNNs, but it wasn't that interesting to the wider world, who generally didn't believe that something like what is happening could happen. 

(I believed in the theory of superintelligence, like there's an exponential that's going to go past this arbitary point eventually, and IIRC had wildly wildly too distant expectations of when it might begin to happen in any meaningful way. Just thinking back to that time makes the last couple of years shocking to comprehend.)

48 entities gave feedback on the Department of Commerce AI reporting requirements. 

Public comments offering feedback on BIS's proposed reporting requirements are now up! It received responses from 48 entities including OpenAI, Anthropic, and many AI safety groups.

The reporting requirements are probably one of the most important things happening in US AI policy-- I'd encourage folks here to find time to skim some of the comments.

Probably, if some AI were to recommend additional blood testing I could manage to persuade the wctual medical professionals to do it. Recent conversation went some thing like this:

Me: “can I have my thyroid levels checked pleas? And the consultant endocrinologist said he’d like to see a liver function test done next time i give a blood sample.”

Nurse (taking my blood sample and pulling my medical record up in the computer) “you take carbimazole right?”

Me: “yes”

Nurse (ticking boxes on a form on the computer) “… and full blood panel, and electrolytes…”

Probably wouldn’t be hard to get suggestions from an AI added to the list.

Some notes from the transcript:

I believe there are ways to recruit college students responsibly. I don't believe the way EA is doing it really has a chance to be responsible. I would say, the way EA is doing it can't filter and inform the way healthy recruiting needs to.  And they're funneling people, into something that naivete hurts you in. I think aggressive recruiting is bad for both the students and for EA itself.

Enjoyed this point -- I would guess that the feedback loop from EA college recruiting is super long and is weakly aligned.  Those in charge of setting recruiting strategy (eg CEA Groups team, and then university organizers) don't see the downstream impacts of their choices, unlike in a startup where you work directly with your hires, and quickly see whether your choices were good or bad.

Might be worth examining how other recruiting-driven companies (like Google) or movements (...early Christianity?) maintain their values, or degrade over time.

Seattle EA watched a couple of the animal farming suffering documentaries. And everyone was of course horrified But, not everyone was ready to just jump on, let's give this up entirely forever. So we started doing more research, and I posted about, a farm a couple hours away that did live tours, and that seemed like a reasonable thing to learn, like, a limited but useful thing.

Definitely think that on the margin, more "directly verifying base reality with your own eyes" would be good in EA circles. Eg at one point, I was very critical of those mission trips to Africa where high schoolers spend a week digging a well; "obviously you should just send cash!" But now I'm much more sympathetic.

This also stings a bit for Manifund; like 80% of what we fund is AI safety but I don't really have much ability to personally verify that the stuff we funded is any good.

The natural life cycle of movements and institutions is to get captured and be pretty undifferentiated from other movements in their larger cultural context. They just get normal because normal is there for a reason and normal is easiest.  And if you want to do better than that, if you want to keep high epistemics, because normal does not prioritize epistemics, you need to be actively fighting for it, and bringing a high amount of skill to it. I can't tell you if EA is degrading at like 5 percent a year or 25 percent a year, I can tell you that it is not self correcting enough to escape this trap.

I think not enforcing an "in or out" boundary is big contributor to this degradation -- like, majorly successful religions required all kinds of sacrifice and

What I think is more likely than EA pivoting is a handful of people launch a lifeboat and recreate a high integrity version of EA. 

It feels like AI safety is the best current candidate for this, though that is also much less cohesive and not a direct successor for a bunch of ways. I too have been lately wondering what "Post EA" looks like.

I hear that as every true wizard must test the integrity of their teacher or of their school, Hogwarts, whatever the thing is. The reason you don't get to graduate until you actually test the integrity of the school is because if you're just taking it on its own word, then you could become a villain.

You have to respect your own moral compass to be able to be trusted.

Really liked this analogy!

Which EA leaders do you most resonate with?

I would suggest that if you don't care about the movement leaders who have any steering power, you're not in that movement.

I like this as a useful question to keep in mind, though I don't think it's totally explanatory. I think I'm reasonably Catholic, even though I don't know anything about the living Catholic leaders.

Timothy: Give me a vision of a different world where ea would be better served by the by having leadership that actually was willing to own their power more 

Elizabeth: which you'll notice even holden won't do 

Timothy: yeah, he literally doesn't want the power.

Elizabeth: Yeah, none of them do. CEA doesn't want it. 

I've been thinking that EA should try to elect a president, someone who is empowered but also accountable to the general people in the movement, a schelling person to be the face of EA. (plus of course, we'd get to debate stuff like optimal voting systems and enfranchisement -- my kind of catnip)