LessWrong 2.0 Reader

I want to emphasize that I'm criticizing "AI 2027"'s projection of R&D spending, i.e. this table [LW · GW]. If companies cut R&D spending, that falsifies the "AI 2027" forecast.

In particular, the comment I'm replying to proposed that while the current money would run out in ~2027, companies could raise more to continue expanding R&D spending. Raising money for 2028 R&D would need to occur in 2027; and it would need to occur on the basis of financial statements of at least a quarter before the raise. So in this scenario, they needs to slash R&D spending in 2027- something the "AI 2027" authors definitely don't anticipate.

Furthermore, your claim may itself be false. We lack sufficient breakdown of OpenAI's budget to be certain. My estimate from the post was that most AI companies have 75% cost of revenue; OpenAI specifically has a 20% revenue sharing agreement with Microsoft; and the remaining 5% needs to cover General and Administrative expenses. Depending on the exact percentage of salary and G&A expenses caused by R&D, it's plausible that OpenAI eliminating R&D entirely wouldn't make it profitable today. And in the future OpenAI will also need to pay interest on tens of billions in debt.

My understanding is that there was a separate image model in historical vlms like flamingo but that it passed on a vector representation of the image not text

Thanks for taking time to reply!

Yes OpenAI realtime API is really cool. When speaking to realtime API, I start each sentence with two words indicating what I want it to do. It's clunky but it works. "Translate Chinese, what is the time?" "Reply Chinese, how are you?" Ideally yes I could write an app to prepend the instruction audio to each sentence.

If I had this as higher priority I'd actually want to setup this Twilio app.

He asks “How interested are you in Widgets?” He has learnt from previous job interviews that, if he answers honestly, the interviewer will think he is any of lying, insane, or too weird to deal with, and not hire him, even though this is not in the best financial interests of the company, were they fully informed.

By the standard "intentionally or knowingly cause the other person to have false beliefs", answering 'honestly' would be lying, and answering in a toned down way would not (because it maximizes the truth of the belief that the interviewer gets).

(I have successfully done Unbendable Arm after Valentine showed me in person, though he didn't explain any of the mechanics. My experience of it didn't involve visualization, but felt like placing my fingertips on the wall across the room and resolving that they'd stay there. Contra jimmy's comment [LW(p) · GW(p)], IIRC I initially held my arm wrong without any cueing.)

Strongly related: Believing In [LW · GW]. From that post:

My guess is that for lack of good concepts for distinguishing “believing in” from deception, LessWrongers, EAs, and “nerds” in general are often both too harsh on folks doing positive-sum “believing in,” and too lax on folks doing deception. (The “too lax” happens because many can tell there’s a “believing in”-shaped gap in their notions of e.g. “don’t say better things about your start-up than a reasonable outside observer would,” but they can’t tell its exact shape, so they loosen their “don’t deceive” in general.)

I feel like this post is similarly too lax on, not deception, but propositional-and-false religious beliefs.

this week's meetup is on the train to crazy town [? · GW]. it was fun putting together all the readings and discussion questions, and i'm optimistic about how the meetup's going to turn out! (i mean, in general, i don't run meetups i'm not optimistic about, so i guess that's not saying much.) im slightly worried about some folks coming in and just being like "this metaphor is entirely unproductive and sucks", should consider how to frame the meetup productively to such folks.

i think one of my strengths as an organizer is that ive read sooooo much stuff and so its relatively easy for me to pull together cohesive readings for any meetup. but ultimately im not sure if it's like, the most important work, to e.g. put together a bibliography of the crazy town idea and its various appearances since 2021. still, it's fun to do.

For the same reasons 'training an agent on a constitution that says to care about $x$' does not, at arbitrary capability levels, produce an agent that cares about $x$

Ok, but I'm trying to ask why not.

Here's the argument that I would make for why not, followed by why I'm skeptical of it right now.

New options for the AI will open up at high capability levels that were not available at lower capability levels. This could in principle lead to undefined behavior that deviates from what we intended.

More specifically, if it's the case that if...

1. The best / easiest-for-SGD-to-find way to compute corrigible outputs (as evaluated by the AI) is to reinforce an internal proxy measure that is correlated with corrigibility (as evaluated by the AI) in distribution, instead of to reinforce circuits that implement corrigibility more-or-less directly.
2. When the AI gains new options unlocked by new advanced capabilities, that proxy measure comes apart from corrigibility (as evaluated by the AI), in the limit of capabilities, so that the poxy measure is almost uncorrelated with corrigibility 

...then the resulting system will not end up corrigible.

(Is this the argument that you would give, or is there another reason why you expect that "training an agent on a constitution that says to care about $x$' does not, at arbitrary capability levels, produce an agent that cares about $x$"?)

But, at the moment, I'm skeptical of the above line of argument for several reasons.

• I'm skeptical of the first premise, that the best way that SGD can find to produce corrigible (as evaluated by the AI) is to reinforce a proxy measure.
  • I understand that natural selection, when shaping humans for inclusive genetic fitness, instilled in them a bunch of proxy-drives. But I think this analogy is misleading in several ways.
  • Most relevantly, there's a genetic bottleneck, so evolution could only shape human behavior by selecting over genomes, and genomes don't encode that much knowledge about the world. If humans were born into the world with detailed world models, that included the concept of inclusive genetic fitness baked in, evolution would absolutely shaped humans to be inclusive fitness maximizers. AIs are "born into the world" with expansive world models that already include concepts like corrigibility (indeed, if they didn't, Constitutional AI wouldn't work at all). So it would be surprising if SGD opted to reinforce proxy measures instead of relying on the concepts directly.
• We would run the constitutional AI reinforcement process continuously, in parallel with the capability improvements from the RL training.
  • AI's capabilities increase, it will gain new options. If the AI is steering based on proxy measures, some of those options will involved the proxy coming apart from the target of the proxy. But when that starts to happen, the constitutional AI loop will exert an optimization pressure on the AI's internals to hit the target, not just the proxies.

Is this the main argument? What are other reasons to think that 'training an agent on a constitution that says to care about $x$' does not, at arbitrary capability levels, produce an agent that cares about $x$?
 

Nope. Try it out. If you attempt to split the activation vector into 1050 vectors for animals + attributes, you can't get the dictionary activations to equal the feature activations $c_i\left(x\right)$, $c_i^{\prime }\left(x\right)$.

I believe the Scramblers from blindsight weren’t self aware, which means they couldn’t think about their own interactions with the world.

As I recall the crew was giving one of the Scramblers a series of cognitive tests. It aced all the tests that had to do with numbers and spatial reasoning, but failed a test that required the testee to be self aware. 


 

Oh, if you're in the business of compiling a comprehensive taxonomy of ways the current AI thing may be fake, you should also add:

• Vibe coders and "10x'd engineers", who (on this model) would be falling into one of the failure modes outlined here [LW · GW]: producing applications/features that didn't need to exist, creating pointless code bloat (which helpfully show up in productivity metrics like "volume of code produced" or "number of commits"), or "automatically generating" entire codebases in a way that feels magical, then spending so much time bugfixing them it eats up ~all perceived productivity gains.
• e/acc and other Twitter AI fans, who act like they're bleeding-edge transhumanist visionaries/analysts/business gurus/startup founders, but who are just shitposters/attention-seekers who will wander off and never look back the moment the hype dies down.