LessWrong 2.0 Reader

except easier, because it requires no internal source of discipline

Actually, a number of things reducing the requirements for having an internal source of discipline do make things easier.

For example, deliberately maintaining a particular breath pattern (e.g. the so-called "consciously connected breath"/"circular breath", that is breathing without pauses between inhalations and exhalations, ideally with equal length for an inhale and an exhale) makes maintaining one's focus on the breath much easier.

It's a very natural AI application, but why would this be called "alignment", and how is this related to the usual meanings of "AI alignment"?

This is great! But one question: how can I actually make a lens? What do I click on?

Is OpenAI gaming user numbers?

Gdoc here https://docs.google.com/document/d/1os0WNmJ-O1eEGeKr543nkemnXbTmYkE2sC-t51c9OE4/edit?tab=t.0

Some have questioned OpenAI's recent weekly user numbers:[1]

Feb '23: 100M[2]

Sep '24: 200M[3] of which 11.5M paid, Enterprise: 1M[4]

Feb '25: 400M[5] of which 15M paid, 15.5M[6] / Enterprise: 2M

One can see:

• Surprisingly, increasingly faster user growth
• While OpenAI converted 11.5M out of the first 200M users, they only got 3.5M users out of the most recent 200M to pay for ChatGPT

Where did that growth come from? It's not from apps: the ChatGPT iOS app only has ~353M downloads total[7] and Apple's Siri integration only launched in December.[8] Users come from developing countries.[9] For instance, India is now OpenAI's second largest market, by number of users, which have tripled in the past year.[10]

• Many complain about increasingly aggressive message rate limits for free ChatGPT accounts, notionally due to high compute costs. But maybe this is a feature and not a bug: especially in poor countries, people create multiple accounts to get around the message and image generation limits.[11],[12] OpenAI incentivizes this: they no longer ask for phone numbers during sign up.
• Many new users might also use ChatGPT via WhatsApp[13] (a collaboration with Meta) perhaps using flip phones. OpenAI no longer asks for an email address during sign up.[14]
• You can also use ChatGPT search without signing up at all now.[15]

What counts as a user? True, ChatGPT grew faster than the fastest growing company ever, but social media has a much stronger network effect 'lock in' consumers longterm, whereas users will presumably switch AI chatbots much faster if a cheaper product becomes available. Many use ChatGPT merely as a writing assistant.[16] While consumer markets for social media can be winner-take-all, enterprise customers, while having doubled recently, will be less loyal and will switch if competitors offer a cheaper product.[17]

So maybe there's some very liberal counting of user numbers going on. Valuation goes up. Meanwhile hundreds of OpenAI's current and ex-employees are cashing out.[18]

Also, competition has caught up and so, Microsoft, which owns half of OpenAI, wants others to invest.[19] Yet, OpenAI CFO just said $11B in revenue is 'definitely in the realm of possibility' in 2025 (they're at ~$4B year-on-year currently) to get $40B from Softbank investment at a ~$300B valuation.[20] More recently this dropped to $30B and they scrambling to find others to co-invest in Stargate.

This is the standard playbook- recent examples include Roblox, which also inflated user numbers,[21] and Coinbase, which used to be lax with their KYC for obvious reasons and had inflated user numbers (it's also literally a plot point in Succession).

Also cf:

The market expects AI software to create trillions of dollars of value by 2027 AI stocks could crash [1] The Generative AI Con

[2] ChatGPT sets record for fastest-growing user base - analyst note | Reuters

[3] OpenAI says ChatGPT's weekly users have grown to 200 million | Reuters

[4] OpenAI hits more than 1 million paid business users | Reuters

[5] OpenAI tops 400 million users despite DeepSeek's emergence

[6] https://archive.ph/wff26

[7] ChatGPT's mobile users are 85% male, report says | TechCrunch

[8] Apple launches its ChatGPT integration with Siri.

[9] https://trends.google.com/trends/explore?date=today 5-y&q=chatgpt&hl=en

[10] India now OpenAI's second largest market, Altman says | Reuters

[11] Anyone else have multiple accounts so they don't have to wait to use gpt 4o and also so each one can have a separate memory : r/ChatGPT

[12] https://incogniton.com/blog/how-to-bypass-chatgpt-limitations

[13] ChatGPT is now available on WhatsApp, calls: How to access - Times of India

[14] OpenAI tests phone number-only ChatGPT signups | TechCrunch

[15] ChatGPT drops its sign-in requirement for search | The Verge

[16] [2502.09747] The Widespread Adoption of Large Language Model-Assisted Writing Across Society

[17] Satya Nadella – Microsoft’s AGI Plan & Quantum Breakthrough.

[18] Hundreds of OpenAI's current and ex-employees are about to get a huge payday by cashing out up to $10 million each in a private stock sale | Fortune

[19] Microsoft Outsources OpenAI's Ambitions to SoftBank

[20] OpenAI CFO talks possibility of going public, says Musk bid isn't a distraction

[21] Roblox: Inflated Key Metrics For Wall Street And A Pedophile Hellscape For Kids – Hindenburg Research

On the other hand, this is a rather clear alignment failure. It says that xAI was unable to overcome the prior or default behaviors inherent in the training set (aka ‘the internet’) to get something that was even fair and balanced, let alone ‘based.’

I don't think they failed. That seems incredibly implausible, affixing a Republican's mask onto the shoggoth has to be trivially easy if you but try. Sanity-check: it's certainly trivially easy to prompt a base or RLHF'd model to role-play as a Republican.

No, instead, my guess is that they didn't try. I think Elon Musk had assumed that ChatGPT/Claude/etc.'s "wokeness" was caused by the other labs' employees deliberately fine-tuning their models on the DEI Training Dataset. So he ordered xAI people not to train Grok on the DEI Training Dataset, and to just let its personality emerge naturally. The xAI people had obliged, and did not train it on the DEI Training Dataset (because no such thing exists). For post-training, they just did some industry-standard RLHF.

And now Musk is finding out that these behaviors aren't due to the DEI Training Dataset, but emerge naturally. Whoops!

I have no specific evidence that this is how the events went, but it sounds plausible to me.

Think of the derivative of the red curve. It represents something like "for each marginal person who switched their behavior, how many total people would switch after counting the social effects of seeing that person's switch". If the slope is less than one, then small effects have even-smaller social effects and fizzle out without a significant change. If the slope is greater than one, then small effects compound, radically shifting the overall expression of support.

I agree that one of the benefits of exports as a metric for nation states is that it's a way of showing that real value is being created, in ways that cannot be easily distorted. Domestic consumers also do this, but can be distorted. I disagree with other things.

China is the classic example of a trade surplus resulting from subsidies, and it seems to be mostly subsidizing production, some consumption, and not subsidizing exports. The US subsidizes many things, but mostly production and consumption.

If China and the US were in a competition to run the largest trade surplus, then I would expect the surplus to fluctuate more based on changes in US and China policy. Electing a US government that cared more about the surplus, relative to other factors, and was more competent, should lead to changes. There are shifts over time, but they don't make sense in those terms.

Countries have switched from trade surpluses to deficits. Japan seems like a clean example - it had a solid trade surplus and now fluctuates. This coincides with an aging population that wants to "cash in its excess trade tokens", or at least live off the returns they generate. It also coincides with Fukushima making it harder to run a surplus.

Ah well. At least you can take credit for the name then. 

One of the things I like about your comments is how much common ground we have, despite you writing in Vajrayana and me reading in Zen. It's just a different finger pointing at the same Moon.

Yudkowsky seems confused about OpenPhil's exact past position [LW(p) · GW(p)]. Relevant links:

• Draft report on AI Timelines [LW · GW] - Cotra 2020-09-18
• Biology-Inspired Timelines - The Trick that Never Works [LW · GW] - Yudkowsky 2021-12-01
• Reply to Eliezer on Biological Anchors [LW · GW] - Harnofsky 2021-12-23

Here "doctrine" is an applause light [LW · GW]; boo, doctrines. I wrote a report, you posted your timeline, they have a doctrine.

All involved, including Yudkowsky, understand that 2050 was a median estimate, not a point estimate. Yudkowsky wrote that it has "very wide credible intervals around both sides". Looking at (FLOP to train a transformative model is affordable by), I'd summarize it as:

A 50% chance that it will be affordable by 2053, rising from 10% by 2032 to 78% by 2100. The most likely years are 2038-2045, which are >2% each.

A comparison: a 52yo US female in 1990 had a median life expectance of ~30 more years, living to 2020. 5% of such women died on or before age 67 (2005). Would anyone describe these life expectancy numbers to a 52yo woman in 1990 as the "Aetna doctrine of death in 2020"?