simeon_c's Shortform

@Daniel Kokotajlo [LW · GW] If you indeed avoided signing an NDA, would you be able to share how much you passed up as a result of that? I might indeed want to create a precedent here and maybe try to fundraise for some substantial fraction of it.

Do you know if there have been any concrete implications (ie. someone giving Daniel a substantial amount of money) from the discussion?'

(My track record of 0% accuracy on which messages will politically snowball is holding up very well. I'm glad that sometimes people like you say things the way you say them, rather than only people like me saying things how I say them.)

What kind of effects are you thinking about?

I was just thinking not 10 minutes ago about how that one LW user who casually brought up Daniel K's equity (I didn't remember your username) had a massive impact and I'm really grateful for them.

There's a plausible chain of events where simeon_c brings up the equity > it comes to more people's attention > OpenAI goes under scrutiny > OpenAI becomes more transparent > OpenAI can no longer maintain its de facto anti-safety policies > either OpenAI changes policy to become much more safety-conscious, or loses power relative to more safety-conscious companies > we don't all die from OpenAI's unsafe AI.

So you may have saved the world.

How did you know Daniel Kokotajlo didn't sign the OpenAI NDA and probably lost money?

Something which concerns me is that transformative AI will likely be a powerful destabilizing force, which will place countries currently behind in AI development (e.g. Russia and China) in a difficult position. Their governments are currently in the position of seeing that peacefully adhering to the status quo may lead to rapid disempowerment, and that the potential for coercive action to interfere with disempowerment is high. It is pretty clearly easier and cheaper to destroy chip fabs than create them, easier to kill tech employees with potent engineering skills than to train new ones.

I agree that conditions of war make safe transitions to AGI harder, make people more likely to accept higher risk. I don't see what to do about the fact that the development of AI power is itself presenting pressures towards war. This seems bad. I don't know what I can do to make the situation better though.

Why Putin probably won't stop with Ukraine: https://en.m.wikipedia.org/wiki/Minsk_agreements

How do you draw that conclusion from the Minsk agreements? In those, Ukraine committed to pass laws for Decentralisation of power, including through the adoption of the Ukrainian law "On temporary Order of Local Self-Governance in Particular Districts of Donetsk and Luhansk Oblasts". Instead of Decentralization they passed laws forbidding those districts from teaching children in the languages that those districts wants to teach them. 

Ukraines unwillingness to follow the agreements was a key reason why the invasion in 2022 happened and was very popular with the Russian population. Being in denial about that is not helpful is you want to help prevent wars from breaking out.

Having maximalist foreign policy goals is not the way you get peace. 

This creates a new global equilibrium where the US is no longer powerful enough to disincentivize all authoritarians regime from grabbing more land etc.

The latest illegal land grab was done by Israel without any opposition by the US. If you are truly worried about land grabs being a problem why not speak against that US position of being okay with some land grabs instead of just speaking for buying more weapons?

When I last looked a couple of months back, I found very little discussion of this topic in the rationalist communities. The most interesting post was probably this one from 2021: https://forum.effectivealtruism.org/posts/8cr7godn8qN9wjQYj/decreasing-populism-and-improving-democracy-evidence-based [EA · GW]

I supposed it's not a popular topic because it rubs up against politics. But I do think that liberal democracy is the operating system for running things like LW, EA, and other communities we all love. It's worth defending it--though what that means exactly is vague to me.

On the AI aspect I suspect we could make a small case study out of Israel's use of their AI.

I wonder if potential war is the greatest concern with regard to either loss of liberalism or sites like LW. Interesting news story on views about democratic electoral processes and public trust in them as well as trust that the more democratic form of government will accomplish what it needs to. (Perhaps a lot of reading into with that particular summary but simplist was I could express the summary.)

I've not read the report so not sure if the headline is actually accurate about election -- certainly what is reported in the story doesn't quite support the "voters skeptical about fairness of elections" headline claim. The rest does seem to align with lots of news and events over the past 5 or 10 years.

Rephrasing based on an ask: "Western Democracies need to urgently put a hard stop to Russia and China war (preparation) efforts" -> Western Democracies need to urgently take actions to stop the current shift towards a new World order where conflicts are a lot more likely due to Western democracies no longer being a hegemonic power able to crush authoritarians power that grab land etc. This shift is currently primarily driven by the fact that Russia & China are heavily rearming themselves whereas Western democracies are not.

@Elizabeth

It would also be quite terrible for safety if AGI was developed during a global war, which seems uncomfortably likely (~10% imo).

This may be likely, iirc during wars countries tend to spend more on research and they could potentially just race to AGI like what happened with space race. Which could make hard takeoff even more likely.

Glad you're keeping your eye out for these things!

It's 8 hours away from the Bay, which all-in is not that different from a plane flight to NY from the Bay, so the location doesn't really help with being where all the smart and interesting people are.

Before we started the Lightcone Offices we did a bunch of interviews to see if all the folks in the bay-area x-risk scene would click a button to move to the Presidio District in SF (i.e. imagine Lightcone team packs all your stuff and moves it for you and also all these other people in the scene move too) and IIRC most wouldn't because of things like their extended friend network and partners and so on (@habryka [LW · GW] @jacobjacob [LW · GW] am I remembering that correctly?). And that's only a ~1.5-hr move for most of them.

This debate comes from before the RSP so I don’t actually think that’s cruxy. Will try to dig up an older post.

↑ comment by simeon_c (WayZ) · 2024-04-08T18:21:29.546Z · LW(p) · GW(p)

There was a hot debate recently but regardless, the bottom line is just "RSPs should probably be interpreted literally and nothing else. If a literal statement is not strictly there, it should be assumed it's not a commitment."

I've not seen people doing very literal interpretation on those so I just wanted to emphasize that point.

Replies from: Raemon

↑ comment by Raemon · 2024-04-08T18:31:44.022Z · LW(p) · GW(p)

I currently think Anthropic didn't "explicitly publicly commit" to not advance the rate of capabilities progress. But, I do think they made deceptive statements about it, and when I complain about Anthropic I am complaining about deception, not "failing to uphold literal commitments."

I'm not talking about the RSPs because the writing and conversations I'm talking about came before that. I agree that the RSP is more likely to be a good predictor of what they'll actually do.

I think most of the generator for this was more like "in person conversations", at least one of which was between Dario and Dustin Moswkowitz:

The most explicit public statement I know is from this blogpost (which I agree is not an explicit commitment, but, I do think 

• Capabilities: AI research aimed at making AI systems generally better at any sort of task, including writing, image processing or generation, game playing, etc. Research that makes large language models more efficient, or that improves reinforcement learning algorithms, would fall under this heading. Capabilities work generates and improves on the models that we investigate and utilize in our alignment research. We generally don’t publish this kind of work because we do not wish to advance the rate of AI capabilities progress. In addition, we aim to be thoughtful about demonstrations of frontier capabilities (even without publication). We trained the first version of our headline model, Claude, in the spring of 2022, and decided to prioritize using it for safety research rather than public deployments. We've subsequently begun deploying Claude now that the gap between it and the public state of the art is smaller.

Replies from: WayZ

↑ comment by simeon_c (WayZ) · 2024-04-08T21:33:41.398Z · LW(p) · GW(p)

If you wanna reread the debate, you can scroll through this thread (https://x.com/bshlgrs/status/1764701597727416448). 

This workflow appears to work quite well. This suggests that we should collectively try to irritate Gwern into solving the alignment problem. This is a big problem, so we'll have to apply the workflow iteratively. First post a confident argument that it's impossible for specific reasons, then make similarly overconfident posts on increasingly specific arguments about how subsets of the problem are effectively impossible. The remainder will be the best available solution.

That's a pretty funny example. Lots of us could've recognized the central flaw (I'm pretty alert to intimidation by physics/math and drawing much broader conclusions than the proof allows), but it would take Gwern to disprove it so eloquently, thoroughly, and with such thorough references.

So, we should all be like Gwern. Merely devote our lives to the pursuit not just of knowledge, but well-organized knowledge that's therefore cumulative. Recognize that nobody will pay us to do this, so live cheaply to minimize time wasted making a living (and probably distortions in rationality from having goals outside of knowledge).

I don't know Gwern's story in any more detail than that, which is my recollection of what he's said about himself. (The distortions of rationality is my addition; I really need to write up my research and thinking on motivated reasoning.)

Since the scaling experiment is not yet done, it remains possible that long-horizon agency is just a matter of scale even with current architectures, no additional research necessary. In which case additional research helps save on compute and shape the AIs, but doesn't influence ability to reach the changeover point, when the LLMs take the baton and go on doing any further research on their own.

Distributed training might be one key milestone that's not yet commoditized, making individual datacenters with outrageous local energy requirements unnecessary. And of course there's the issue of access to large quantities of hardware.

They are pushing the frontier (https://arxiv.org/abs/2406.07394), but it’s hard to say where they would be without llamas. I don’t think they’d be much far behind. They have gpt-4 class models as is and also don’t care about copyright restrictions when training models. (Arguably they have better image models as a result)