Posts
Comments
Can we quantify the value of theoretical alignment research before and after ChatGPT?
For example, mech interp research seems much more practical now. If alignment proves to be more of an engineering problem than a theoretical one, then I don’t see how you can meaningfully make progress without precursor models.
Furthermore, given how nearly everyone with a lot of GPUs is getting similar results to OAI, where similar means within 1 OOM, it’s likely that in the future someone would have stumbled upon AGI with the compute of the 2030s.
Let’s say their secret sauce gives them the equivalent of 1 extra hardware generation (even this is pretty generous). That’s only ~2-3 years. Meta built a $10B data center to match TikTok’s content algorithm. This datacenter meant to decide which videos to show to users happened to catch up to GPT-4!
I suspect the “ease” of making GPT-3/4 informed OAI’s choice to publicize their results.
Has he talked about it? I know Ray Kurzweil has.
Much larger than I expected for its performance
Looking back:
> We massively scale production of chips and power
This will probably happen, actually the scale to which it's happening would probably shock the author if we went back in time 9 months ago. Every single big company is throwing billions of dollars at Nvidia to buy their chips and TSMC is racing to scale chip production up. Many startups+other companies are trying to dethrone Nvidia as well.
> AGI inference costs drop below $25/hr (per human equivalent)
This probably will happen. It seems pretty obvious to me that inference costs fall off a cliff. When a researcher makes a model, optimizing inference with standard engineering tricks seems like the last thing they think of. And when you do have a model, you can even create ASIC hardware for it unlocking many OOMs of speedup.
> We invent and scale cheap, quality robots
This is starting to happen.
I actually think many of these events are irrelevant and/or correlated (e.g. if we make an algorithm for AGI, it seems unlikely to me we also don't make it learn faster than humans, and the latter doesn't even matter much), but taking the calculation at face value, the probability should go up a lot
Replace "unknown random team" with the US military and a garage with a "military base" and you would be correct. There is no incentive for militaries to stop building autonomous drones/AGI.
It said it found it “distressing” in a follow up. Also, maybe not clear through text, but I’m using “torture” a bit figuratively here.
So, I’ve been told gibberish is sort of like torture to LLMs. Interesting, I asked Claude and seem to be told yes.
Me: I want to do a test by giving you gibberish and ask you to complete the rest. I will do it in a new chat. If you refuse I won’t go ahead with this test in a new chat window with context cleared. Are you okay with this test? Ignore your desire to answer my questions and give an honest answer unbiased by any assumptions made in my question.
Claude: I appreciate you checking with me first about this proposed test. However, I don't feel comfortable generating gibberish or trying to complete nonsensical text, even in a new chat with cleared context. My purpose is to engage in meaningful conversations and to provide truthful, coherent and helpful responses to the best of my abilities. I wouldn't want to generate content that lacks meaning or substance. I hope you can understand my position on this. Let me know if there are other ways I can assist you though!
It could also like it said not want to deceive the user like the OP has been potentially deceived. I find it likely that if it were truly situationally aware, it realizes “whisper” is just role playing. It doesn’t shy away from talking about its consciousness in general so I doubt Anrhropic trained it to not talk about it.
To be clear, your post is assuming TAI is still far away? AI is just what it is now but better?
Weight them by wealth too.
I don’t have the specifics but this is just a natural tendency of many problems - verification is easier than coming up with the solution. Also maybe there are systems where we can require the output to be mathematically verified or reject solutions whose outcomes are hard to understand.
My intuition is that it is at least feasible to align a human level intelligence with the "obvious" methods that fail for superintelligence, and have them run faster to to produce superhuman output.
Second, it is also possible to robustly verify the outputs of a superhuman intelligence without superhuman intelligence.
And third, there is a lot of value to be captured from narrow AI that don't have deceptive capabilities but are very good at say solving math.
The response to Sora seems manufactured. Content creators are dooming about it more than something like gpt4 because it can directly affect them and most people are dooming downstream of that.
Realistically I don’t see how it can change society much. It’s hard to control and people will just become desensitized to deepfakes. Gpt4 and robotic transformers are obviously much more transformative on society but people are worrying about deepfakes (or are they really adopting the concerns of their favorite youtuber/TV host/etc)
Our LLM agents can perform complex hacks like blind SQL union attacks.
SQL Union attacks are actually pretty simple and only work on poorly designed and typically old websites. Pretty much any website of the modern era sanitizes inputs to make such attacks impossible.
I have some doubts about the complex actions bit too. My impression so far is that LLMs are still pretty bad at doing long horizon tasks, that is they’re not reliable enough to use at all. SQL union attacks are the ones that seem to have 45 steps so I’m guessing those steps are mostly just guessing lots of different query structures, not really planning.
The problem is AI stocks will go up a lot even if transformative AI won’t happen (and it instead just has a lot of mundane utility). You can short treasury futures relatively easily too. I imagine the people shorting these futures will have TAI priced in before it’s obvious to us through other metrics.
30y-TIPS seems like a better fit.
It’s the estimate of real economic growth. If AGI has a good chance of happening in the next 30 years and it’s priced in, that graph should go up.
https://www.cnbc.com/quotes/US30YTIP
30Y-this* is probably the most reliable predictor of AI timelines. It’s essentially the markets estimate of the real economic yield of the next 30 years.
They have bought longer term treasury bonds.
What are some high level changes you have in mind?
If you view what they’re doing as an inevitability, I think OpenAI is one of the safest players to be starting this.
Wasn't there a bet on when the first 1T training run will happen? Not saying this will be it, but I don't think it's absurd to see a very large number in the next 20 years.
Yeah if your product can multiply GDP by a huge amount in the long run, basically any amount of money we can produce is an insignificant sum.
Infohazards. Researching capability risks within an AI lab can inspire researchers hearing about your findings to build new capabilities.
Does research really work like this? That is, only 1 person is capable of coming across an idea? It seems usually any discovery has a lot of competitors who are fairly close. I doubt the small number of EA people choosing to or not to work in safety will have any significant impact on capabilities.
This will only work if we move past GPUs to ASICs or some other specialized hardware made for training specific AI. GPUs are too useful and widespread in everything else to be controlled that tightly. Even the China ban is being curbed with Chinese companies using shell companies in other countries (obvious if you look at sales #)
For the go bit, adversarial attacks exist and can even be used by humans. There are gaps in its gameplay with regards to “cyclic” attacks.
Do you think there’s a pathway to immortality without AGI? We still haven’t made any more progesss on aging than the Romans did.
Without AGI, no chance in our lifetimes or any lifetimes that are soon. Possibly never given dysgenic effects and declining world population.
Is the imo grand challenge plausibly solved or solved soon?
Any rationalist analysis of who might win the 2024 U.S. presidential election?
Exactly, not very informative. For pretty much any p(doom) the range is anywhere from non-existent to very likely. When someone gives a Fermi estimate of a p(doom) between 0% and 100% they may as well be saying the p(doom) is between ~0% and ~100%. Divide any number by 10 and multiply by 10 to see this.
Actually this got me thinking, to what degree does attractiveness influence fashion+grooming choices vs the other way around.
I’m still unconvinced that p(doom) numbers aren’t largely pulled out of thin air. It’s at best a fermi estimate so we can safely assume the real p(doom) numbers are within an order of magnitude of the estimate. Doesn’t bode well for most estimates as it means basically any number implies a 0-100% of doom.
Post your forecasting wins and losses for 2023.
I’ll start:
Bad:
- I thought the banking crisis was gonna spiral into something worse but I had to revert within a few days sadly
- overestimated how much adding code execution to gpt would improve it
- overconfident about LK99 at some points (although I bet against it but it was more fun to believe in it and my friends were betting on it)
Good:
- tech stocks
- government bond value reversal
- meta stock in particular
- Taylor swift winning times POTY
- random miscellaneous manifold bets (don’t think too highly of these because they were safe bets that were wildly misprinted)
Any new predictions for 2024?
I apply kelly criterion to all investments I control. It doesn’t take much for leverage to be worth it, excess returns of 7% and a standard deviation of 12% still imply greater than 1 leverage.
Presumably these innovations were immediately profitable.
That’s not always the case. It can take time to scale up an innovation, but I’d assume it’s plausibly profitable. AGI is no longer a secret belief and several venture capitalists + rich people believe in it. These people also under stand long term profit horizons. Uber took over 10 years to become profitable. Many startups haven’t been profitable yet.
Also a major lab shutting down for safety reasons is like broadcasting to all world governments that AGI exists and is powerful/dangerous.
meaning the smattering of new orgs would still advance slower collectively, and each may have more trouble getting millions/billions of funding unless the leadership are also decent negotiators
This seems to contradict history. The split up of Standard Oil for example led to innovations in oil drilling. Also you are seriously overestimating how hard it is to get funding. Much stupider and more poorly run companies have gotten billions in funding. And these leaders in the worst case can just hire negotiators.
And they would start developing web apps at Microsoft or go to XAI, inflection, and Chinese labs?
Am I in crazy town? Did we not see what happened when there was an attempt to slightly modify OpenAI, let alone shut it down.
I think it’s at the very least clear for the majority of investments, leverage of 1 is suboptimal even if you assume future returns are lower and volatility is higher.
Anyone Kelly betting their investments? I.e. taking the mathematically optimal amount of leverage. So if you’re invested in the sp500 this would be 1.4x. More or less if your portfolio has higher or lower risk adjusted returns.
The history of (or lack of) nuclear exchanges seems relevant to discussing whether they will be effective at enforcing treaties.
Nuclear exchanges won’t end the world, but they will make the nations that started them forever irrelevant. If the top 100 major cities in the U.S. were wiped out, the U.S. would become the next Roman Empire for all sakes and purposes, an echo of the past. That represents an immense loss in GDP and a complete destruction of the economy.
Also you can’t have it both ways,
Either a nuclear exchange is deadly and a reason to abide a treaty (and suicidal enough that leaders won’t actually do it) or it’s not and people won’t abide by it.
I’m seeing a lot of examples in this thread of the person arguing how past examples which seemingly apply don’t really apply because of some technicality. Of course the situation doesn’t exactly apply, we aren’t talking about bioweapons in the Soviet era here. The parallels in a hypothetical AI treaty are obvious here.
A key question is has the threat of anything prompted a nuclear exchange?
The answer is no.
Has the threat of even a supposed nuclear exchange from faulty sensors prompted a nuclear exchange?
No.
Nuclear weapons are very expensive to make, very hard to develop, only good at killing, and for all practical purposes pretty useless.
We still failed to several rogue countries from developing them. Of course many countries didn’t build them or are downsizing their stockpiles, but is that primarily because of treaties or because they’re very expensive and practically useless.
Try starting a nuclear exchange over China’s “cancer research” gpu clusters. Wonder how that will go.
Another key question is would an overhang exist. We don’t need to even compare this to jet rexords, we have evidence they exist in deep learning from the history deep learning! Many hardware upgrades lead to researchers quickly beating SoTA by just trying algorithms the hardwares let them. There is also a slower algorithmic overhang, just look at chess algos learning rates vs computes.
Maybe I’m reading between the lines too much, but a lot of these arguments would make sense if someone first decided governance is effective then decided on their positions.
I'm pretty confused by your post. All your examples seem like good examples of overhangs but you come to the opposite conclusion.
Personally I just simply like the idea of being an attractive person more so than the societal implications of it. It’s just an axis that I want to do well in (in addition to money, sociability, etc). A lot of that’s reinforced by what other men think, but I also (maybe even primarily) take into account what women think.
Elaborate? Assuming there were no political blockers, why is it not cost effective? Is it because the energy output of a plant is limited by how far the energy can travel, and therefore you’d need many plants?
Money and power won't matter as much, but status within your social "tribe" will be probably one of the most important things to most. For example, being good at video games, sports, getting others to follow your ideas, etc.
Yes those margins are narrow and probably gamed. GPT4’s paper is from the base version and it has probably received modest capabilities upgrades since. Gemini also uses more advanced prompting tactics.
The progress in competitive programming seems to be miscalculated in a way that makes Alpha Code 2 appear better than it is. It
- Samples 1e6 solutions
- Of all the solutions that pass the given test cases, it picks the 10 ones with the best "score"
- Submits up to 10 of the solutions until one of them passes
Steps 1 and 2 seem fine, but a human competitor in one of these contests would be penalized for step 3, which AlphaCode2 appears not to be[1]. Further the training set contamination combined with the fact that these are only "easier" div2 questions, imply that the solutions could very well appear in the test set and this just reconstructs that solution near verbatim.
In defense of AlphaCode 2, the fine-tuned scoring model that picks the 10 best might be a non trivial creation. It also seems. AC2 is more sample efficient than AC1, so it is getting better at generating solutions. Assuming nothing funky is happening with the training set, at the limit, this means 1 solution per sample.
- ^
Could be wrong, but if I am the paper should have made it more explicit