Supposing the 1bit LLM paper pans out 2024-02-29T05:31:24.158Z
OpenAI wants to raise 5-7 trillion 2024-02-09T16:15:00.421Z
O O's Shortform 2023-06-03T23:32:12.924Z


Comment by O O (o-o) on China-AI forecasts · 2024-02-29T04:50:34.449Z · LW · GW

Weight them by wealth too.

Comment by O O (o-o) on Can we get an AI to do our alignment homework for us? · 2024-02-27T17:05:01.191Z · LW · GW

I don’t have the specifics but this is just a natural tendency of many problems - verification is easier than coming up with the solution. Also maybe there are systems where we can require the output to be mathematically verified or reject solutions whose outcomes are hard to understand.

Comment by O O (o-o) on Can we get an AI to do our alignment homework for us? · 2024-02-26T09:07:36.637Z · LW · GW

My intuition is that it is at least feasible to align a human level intelligence with the "obvious" methods that fail for superintelligence, and have them run faster to to produce superhuman output. 

Second, it is also possible to robustly verify the outputs of a superhuman intelligence without superhuman intelligence.

And third, there is a lot of value to be captured from narrow AI that don't have deceptive capabilities but are very good at say solving math. 

Comment by O O (o-o) on O O's Shortform · 2024-02-17T20:48:07.442Z · LW · GW

The response to Sora seems manufactured. Content creators are dooming about it more than something like gpt4 because it can directly affect them and most people are dooming downstream of that.

Realistically I don’t see how it can change society much. It’s hard to control and people will just become desensitized to deepfakes. Gpt4 and robotic transformers are obviously much more transformative on society but people are worrying about deepfakes (or are they really adopting the concerns of their favorite youtuber/TV host/etc)

Comment by O O (o-o) on Critiques of the AI control agenda · 2024-02-15T07:13:16.206Z · LW · GW

Our LLM agents can perform complex hacks like blind SQL union attacks.

SQL Union attacks are actually pretty simple and only work on poorly designed and typically old websites. Pretty much any website of the modern era sanitizes inputs to make such attacks impossible.

I have some doubts about the complex actions bit too. My impression so far is that LLMs are still pretty bad at doing long horizon tasks, that is they’re not reliable enough to use at all. SQL union attacks are the ones that seem to have 45 steps so I’m guessing those steps are mostly just guessing lots of different query structures, not really planning.

Comment by O O (o-o) on O O's Shortform · 2024-02-13T09:55:52.521Z · LW · GW

The problem is AI stocks will go up a lot even if transformative AI won’t happen (and it instead just has a lot of mundane utility). You can short treasury futures relatively easily too. I imagine the people shorting these futures will have TAI priced in before it’s obvious to us through other metrics.

Comment by O O (o-o) on O O's Shortform · 2024-02-12T19:14:15.038Z · LW · GW

30y-TIPS seems like a better fit.

Comment by O O (o-o) on O O's Shortform · 2024-02-12T08:20:14.072Z · LW · GW

It’s the estimate of real economic growth. If AGI has a good chance of happening in the next 30 years and it’s priced in, that graph should go up.

Comment by O O (o-o) on O O's Shortform · 2024-02-11T08:41:29.377Z · LW · GW

30Y-this* is probably the most reliable predictor of AI timelines. It’s essentially the markets estimate of the real economic yield of the next 30 years.

Comment by O O (o-o) on How to (hopefully ethically) make money off of AGI · 2024-02-11T08:28:37.167Z · LW · GW

They have bought longer term treasury bonds.

Comment by O O (o-o) on OpenAI wants to raise 5-7 trillion · 2024-02-10T00:16:07.781Z · LW · GW

What are some high level changes you have in mind?

Comment by O O (o-o) on OpenAI wants to raise 5-7 trillion · 2024-02-09T20:03:19.381Z · LW · GW

If you view what they’re doing as an inevitability, I think OpenAI is one of the safest players to be starting this.

Comment by O O (o-o) on OpenAI wants to raise 5-7 trillion · 2024-02-09T19:59:00.367Z · LW · GW

Wasn't there a bet on when the first 1T training run will happen? Not saying this will be it, but I don't think it's absurd to see a very large number in the next 20 years.

Comment by O O (o-o) on OpenAI wants to raise 5-7 trillion · 2024-02-09T17:00:37.624Z · LW · GW

Yeah if your product can multiply GDP by a huge amount in the long run, basically any amount of money we can produce is an insignificant sum.

Comment by O O (o-o) on Why I think it's net harmful to do technical safety research at AGI labs · 2024-02-07T21:07:35.682Z · LW · GW

Infohazards. Researching capability risks within an AI lab can inspire researchers hearing about your findings to build new capabilities.

Does research really work like this? That is, only 1 person is capable of coming across an idea? It seems usually any discovery has a lot of competitors who are fairly close. I doubt the small number of EA people choosing to or not to work in safety will have any significant impact on capabilities.

Comment by O O (o-o) on Adam Smith Meets AI Doomers · 2024-02-01T16:42:04.815Z · LW · GW

This will only work if we move past GPUs to ASICs or some other specialized hardware made for training specific AI. GPUs are too useful and widespread in everything else to be controlled that tightly. Even the China ban is being curbed with Chinese companies using shell companies in other countries (obvious if you look at sales #)

Comment by O O (o-o) on AI #49: Bioweapon Testing Begins · 2024-02-01T16:25:37.116Z · LW · GW

For the go bit, adversarial attacks exist and can even be used by humans. There are gaps in its gameplay with regards to “cyclic” attacks.

Comment by O O (o-o) on What exactly did that great AI future involve again? · 2024-01-31T09:56:57.456Z · LW · GW

Do you think there’s a pathway to immortality without AGI? We still haven’t made any more progesss on aging than the Romans did.

Comment by O O (o-o) on What exactly did that great AI future involve again? · 2024-01-31T09:54:43.076Z · LW · GW

Without AGI, no chance in our lifetimes or any lifetimes that are soon. Possibly never given dysgenic effects and declining world population.

Comment by O O (o-o) on AlphaGeometry: An Olympiad-level AI system for geometry · 2024-01-17T17:54:44.875Z · LW · GW

Is the imo grand challenge plausibly solved or solved soon?

Comment by O O (o-o) on O O's Shortform · 2024-01-08T17:26:41.583Z · LW · GW

Any rationalist analysis of who might win the 2024 U.S. presidential election?

Comment by O O (o-o) on Stop talking about p(doom) · 2024-01-03T20:01:48.793Z · LW · GW

Exactly, not very informative. For pretty much any p(doom) the range is anywhere from non-existent to very likely. When someone gives a Fermi estimate of a p(doom) between 0% and 100% they may as well be saying the p(doom) is between ~0% and ~100%. Divide any number by 10 and multiply by 10 to see this.

Comment by O O (o-o) on Dating Roundup #2: If At First You Don’t Succeed · 2024-01-02T17:41:17.195Z · LW · GW

Actually this got me thinking, to what degree does attractiveness influence fashion+grooming choices vs the other way around.

Comment by O O (o-o) on Stop talking about p(doom) · 2024-01-02T17:26:53.151Z · LW · GW

I’m still unconvinced that p(doom) numbers aren’t largely pulled out of thin air. It’s at best a fermi estimate so we can safely assume the real p(doom) numbers are within an order of magnitude of the estimate. Doesn’t bode well for most estimates as it means basically any number implies a 0-100% of doom.

Comment by O O (o-o) on O O's Shortform · 2024-01-02T06:20:52.645Z · LW · GW

Post your forecasting wins and losses for 2023.

I’ll start:


  • I thought the banking crisis was gonna spiral into something worse but I had to revert within a few days sadly
  • overestimated how much adding code execution to gpt would improve it
  • overconfident about LK99 at some points (although I bet against it but it was more fun to believe in it and my friends were betting on it)


  • tech stocks
  • government bond value reversal
  • meta stock in particular
  • Taylor swift winning times POTY
  • random miscellaneous manifold bets (don’t think too highly of these because they were safe bets that were wildly misprinted)
Comment by O O (o-o) on 2023 in AI predictions · 2024-01-02T05:03:17.311Z · LW · GW

Any new predictions for 2024?

Comment by O O (o-o) on O O's Shortform · 2023-12-20T09:27:11.499Z · LW · GW

I apply kelly criterion to all investments I control. It doesn’t take much for leverage to be worth it, excess returns of 7% and a standard deviation of 12% still imply greater than 1 leverage.

Comment by O O (o-o) on OpenAI, DeepMind, Anthropic, etc. should shut down. · 2023-12-19T16:52:21.565Z · LW · GW

Presumably these innovations were immediately profitable.

That’s not always the case. It can take time to scale up an innovation, but I’d assume it’s plausibly profitable. AGI is no longer a secret belief and several venture capitalists + rich people believe in it. These people also under stand long term profit horizons. Uber took over 10 years to become profitable. Many startups haven’t been profitable yet.

Also a major lab shutting down for safety reasons is like broadcasting to all world governments that AGI exists and is powerful/dangerous.

Comment by O O (o-o) on OpenAI, DeepMind, Anthropic, etc. should shut down. · 2023-12-19T08:20:08.704Z · LW · GW

meaning the smattering of new orgs would still advance slower collectively, and each may have more trouble getting millions/billions of funding unless the leadership are also decent negotiators

This seems to contradict history. The split up of Standard Oil for example led to innovations in oil drilling. Also you are seriously overestimating how hard it is to get funding. Much stupider and more poorly run companies have gotten billions in funding. And these leaders in the worst case can just hire negotiators.

Comment by O O (o-o) on OpenAI, DeepMind, Anthropic, etc. should shut down. · 2023-12-19T08:11:46.074Z · LW · GW

And they would start developing web apps at Microsoft or go to XAI, inflection, and Chinese labs?

Am I in crazy town? Did we not see what happened when there was an attempt to slightly modify OpenAI, let alone shut it down.

Comment by O O (o-o) on O O's Shortform · 2023-12-19T06:55:32.797Z · LW · GW

I think it’s at the very least clear for the majority of investments, leverage of 1 is suboptimal even if you assume future returns are lower and volatility is higher.

Comment by O O (o-o) on O O's Shortform · 2023-12-18T09:01:36.791Z · LW · GW

Anyone Kelly betting their investments? I.e. taking the mathematically optimal amount of leverage. So if you’re invested in the sp500 this would be 1.4x. More or less if your portfolio has higher or lower risk adjusted returns.

Comment by O O (o-o) on Are There Examples of Overhang for Other Technologies? · 2023-12-15T18:46:46.584Z · LW · GW

The history of (or lack of) nuclear exchanges seems relevant to discussing whether they will be effective at enforcing treaties.

Comment by O O (o-o) on Are There Examples of Overhang for Other Technologies? · 2023-12-15T18:16:51.066Z · LW · GW

Nuclear exchanges won’t end the world, but they will make the nations that started them forever irrelevant. If the top 100 major cities in the U.S. were wiped out, the U.S. would become the next Roman Empire for all sakes and purposes, an echo of the past. That represents an immense loss in GDP and a complete destruction of the economy.

Also you can’t have it both ways,

Either a nuclear exchange is deadly and a reason to abide a treaty (and suicidal enough that leaders won’t actually do it) or it’s not and people won’t abide by it.

Comment by O O (o-o) on Are There Examples of Overhang for Other Technologies? · 2023-12-15T18:09:48.760Z · LW · GW

I’m seeing a lot of examples in this thread of the person arguing how past examples which seemingly apply don’t really apply because of some technicality. Of course the situation doesn’t exactly apply, we aren’t talking about bioweapons in the Soviet era here. The parallels in a hypothetical AI treaty are obvious here.

A key question is has the threat of anything prompted a nuclear exchange?

The answer is no.

Has the threat of even a supposed nuclear exchange from faulty sensors prompted a nuclear exchange?


Nuclear weapons are very expensive to make, very hard to develop, only good at killing, and for all practical purposes pretty useless.

We still failed to several rogue countries from developing them. Of course many countries didn’t build them or are downsizing their stockpiles, but is that primarily because of treaties or because they’re very expensive and practically useless.

Try starting a nuclear exchange over China’s “cancer research” gpu clusters. Wonder how that will go.

Another key question is would an overhang exist. We don’t need to even compare this to jet rexords, we have evidence they exist in deep learning from the history deep learning! Many hardware upgrades lead to researchers quickly beating SoTA by just trying algorithms the hardwares let them. There is also a slower algorithmic overhang, just look at chess algos learning rates vs computes.

Maybe I’m reading between the lines too much, but a lot of these arguments would make sense if someone first decided governance is effective then decided on their positions.

Comment by O O (o-o) on Are There Examples of Overhang for Other Technologies? · 2023-12-14T21:39:49.781Z · LW · GW

I'm pretty confused by your post.  All your examples seem like good examples of overhangs but you come to the opposite conclusion. 

Comment by O O (o-o) on Is being sexy for your homies? · 2023-12-13T23:38:58.737Z · LW · GW

Personally I just simply like the idea of being an attractive person more so than the societal implications of it. It’s just an axis that I want to do well in (in addition to money, sociability, etc). A lot of that’s reinforced by what other men think, but I also (maybe even primarily) take into account what women think.

Comment by O O (o-o) on Is Bjorn Lomborg roughly right about climate change policy? · 2023-12-12T07:38:45.272Z · LW · GW

Elaborate? Assuming there were no political blockers, why is it not cost effective? Is it because the energy output of a plant is limited by how far the energy can travel, and therefore you’d need many plants?

Comment by O O (o-o) on We're all in this together · 2023-12-07T23:37:39.383Z · LW · GW

Money and power won't matter as much, but status within your social "tribe" will be probably one of the most important things to most. For example, being good at video games, sports, getting others to follow your ideas, etc. 

Comment by O O (o-o) on Google Gemini Announced · 2023-12-07T00:22:59.365Z · LW · GW

Yes those margins are narrow and probably gamed. GPT4’s paper is from the base version and it has probably received modest capabilities upgrades since. Gemini also uses more advanced prompting tactics.

Comment by O O (o-o) on Google Gemini Announced · 2023-12-06T21:31:42.280Z · LW · GW

The progress in competitive programming seems to be miscalculated in a way that makes Alpha Code 2 appear better than it is. It 

  1. Samples 1e6 solutions
  2. Of all the solutions that pass the given test cases, it picks the 10 ones with the best "score"
  3. Submits up to 10 of the solutions until one of them passes

Steps 1 and 2 seem fine, but a human competitor in one of these contests would be penalized for step 3, which AlphaCode2 appears not to be[1]. Further the training set contamination combined with the fact that these are only "easier" div2 questions, imply that the solutions could very well appear in the test set and this just reconstructs that solution near verbatim. 

In defense of AlphaCode 2, the fine-tuned scoring model that picks the 10 best might be a non trivial creation. It also seems. AC2 is more sample efficient than AC1, so it is getting better at generating solutions. Assuming nothing funky is happening with the training set, at the limit, this means 1 solution per sample. 

  1. ^

    Could be wrong, but if I am the paper should have made it more explicit

Comment by O O (o-o) on Google Gemini Announced · 2023-12-06T20:11:32.789Z · LW · GW

From a short read, capabilities seem equal to gpt4. Alpha code 2 is also not penalized for its first 9 submissions, so I struggle to see how it can be compared to humans.

Comment by O O (o-o) on Based Beff Jezos and the Accelerationists · 2023-12-06T18:26:59.613Z · LW · GW

I can see that tweet implying some trans-humanist position, not necessarily extinction. I think he is about to have a debate with Connor Leahy so it will all be cleared up.

Comment by O O (o-o) on Some quick thoughts on "AI is easy to control" · 2023-12-06T03:30:43.919Z · LW · GW

I consider this system to be superhuman, and the problem of aligning it to be "alignment-complete" in the sense that if you solve any of the problems in this class, you essentially solve alignment down the line and probably avoid x-risk,

I find this line of reasoning (and even mentioning it) not useful. Any alignment solution will be alignment complete so it’s tautological.

I think you’ve defined alignment as a hard problem, which no one will disagree with, but you also define any steps taken towards solving the alignment problem as alignment complete, and thus impossible unless they also seem infeasibly hard. Can there not be an iterative way to solve alignment? I think we can construct some trivial hypotheticals where we iteratively solve it.

For the sake of argument say I created a superhuman math theorem solver, something that can solve IMO problems written in lean with ease. I then use it to solve a lot of important math problems within alignment. This in turn affords us strong guarantees about certain elements of alignment or gradient descent. Can you convince me that the solution to getting a narrow AI useful for alignment is as hard as aligning a generally superhuman AI?

What if we reframe it to some real world example. The proof for the Riemann hypothesis begins with a handful of difficult but comparatively simple lemmas. Solving those lemmas is not as hard as solving the Reimann hypothesis. And we can keep decomposing this proof into parts that are simpler than the whole.

A step in a process being simpler than the end result of the process is not an argument against that step.

Comment by O O (o-o) on Thoughts on “AI is easy to control” by Pope & Belrose · 2023-12-04T05:43:01.784Z · LW · GW

If this is your definition of a dystopia we already live in a dystopia. You can’t make nuclear bombs without being picked up by the FBI/CIA and you’ll probably be arrested in the process. Making something illegal doesn’t define an authoritarian regime. Governments already try to stop international players from building nukes. It just lacks teeth because you can ultimately live with sanctions.

The other problem is it’s way too easy to avoid surveillance or defect in a human regime. For example, you can have a decentralized training network, claim you are training good AIIt’s also unusually easy to regulate AI training. Right now GPUs are easy to control because only Nvidia can make them. This won’t always be true. It’s also much easier to hide GPU production than making nukes because we need GPUs and CPUs for a ton of other useful things.

Theoretically an ASI could probably extrapolate attempts to use compute correctly from your internet signals. Further, if you have the benefits you want from an ASI, you have much less reason to build a 2nd one that’s possibly unaligned. “The digital god says you can’t build it” probably sounds a lot more compelling than “Joe Biden says you can’t build it”.

Comment by O O (o-o) on Thoughts on “AI is easy to control” by Pope & Belrose · 2023-12-03T18:36:07.915Z · LW · GW

The argument is probably something like superintelligence allows robust monitoring of other people’s attempts at building superintelligences, and likely could resolve whatever prisoners dilemmas people have to build them. We don’t need an authoritarian regime.

I expect an ASI can denoise easily available data enough to confidently figure out who is trying to build a superintelligence, and either stop them through soft power/argumentation or implore them to make it aligned.

I think most problems are probably easy to solve with a truly aligned superintelligence. It may come up with good solutions we haven’t even thought of.

Comment by O O (o-o) on Quick takes on "AI is easy to control" · 2023-12-03T09:47:57.812Z · LW · GW

Testing it on out of distribution examples seems helpful. If an AI still acts as if it follows human values out of distribution, it probably truly cares about human values. For AI with situational awareness, we can probably run simulations to an extent (and probably need bootstrap this after a certain capabilities threshold)

Comment by O O (o-o) on Quick takes on "AI is easy to control" · 2023-12-03T09:38:38.328Z · LW · GW

I like “deprecate”

Comment by O O (o-o) on Is OpenAI losing money on each request? · 2023-12-02T00:58:53.185Z · LW · GW

News is the cap grows 20% a year so it will really last until AGI

Comment by O O (o-o) on Is OpenAI losing money on each request? · 2023-12-01T05:51:31.462Z · LW · GW

With basically a blank check from VC, they’ll instead invest in making their models and infra more efficient/better instead of raising prices. They can run a large loss for a very long time.