Posts
Comments
Something bouncing around my head recently ... I think I agree with the notion that "you can't solve a problem at the level it was created".
A key point here is the difference between "solving" a problem and "minimising its harm".
- Solving a problem = engaging with a problem by going up a level from which is was createwd
- Minimising its harm = trying to solve it at the level it was created
Why is this important? Because I think EA and AI Safety have historically focussed (and has their respective strengths in) harm-minimisation.
This applies obviously the micro. Here are some bad examples:
- Problem: I'm experiencing intrusive + negative thoughts
- Minimising its harm: engage with the thought using CBT
- Attempting to solve it by going meta: apply meta cognitive therapy, see thoughts as empty of intrinsic value, as farts in the wind
- Problem: I'm having fights with my partner about doing the dishes
- Minimising its harm: create a spreadsheet and write down every thing each of us does around the house and calculate time spent
- Attempting to solve it by going meta: discuss our communication styles and emotional states when frustration arises
But I also think this applies at the macro:
- Problem: People love eating meat
- Minimising harm by acting at the level the problem was created: asking them not to eat meat
- Attempting to solve by going meta: replacing the meat with lab grown meat
- Problem: Unaligned AI might kill us
- Minimising harm by acting at the level the problem was created: understand the AI through mechanistic interpretability
- Attempting to solve by going meta: probably just Governance
I think it is good to have some ratio of upvoted/agreed : downvotes/disagreed posts in your portfolio. I think if all of your posts are upvoted/high agreeance then you're either playing it too safe or you've eaten the culture without chewing first.
Hi! I think in Sydney we're ~ 3 seats short of critical mass, so I am going to reassess the viability of a community space in 5-6 months :)
It sounds like you're saying you feel quite comfortable with how long you wait, which is nice to hear :)
I'm pretty confident that a majority of the population will soon have very negative attitudes towards big AI labs. I'm extremely unsure about what impact this will have on the AI Safety and EA communities (because we work with those labs in all sorts of ways). I think this could increase the likelihood of "Ethics" advocates becoming much more popular, but I don't know if this necessarily increases catastrophic or existential risks.
Not sure if this type of concern has reached the meta yet, but if someone approached me asking for career advice, tossing up whether to apply for a job at a big AI lab, I would let them know that it could negatively affect their career prospects down the track because so many people now perceive such as a move as either morally wrong or just plain wrong-headed. And those perceptions might only increase over time. I am not making a claim here beyond this should be a career advice consideration.
More thoughts on "80,000 hours should remove OpenAI from the Job Board"
- - - -
A broadly robust heuristic I walk around with in my head is "the more targeted the strategy, the more likely it will miss the target".
Its origin is from when I worked in advertising agencies, where TV ads for car brands all looked the same because they needed to reach millions of people, while email campaigns were highly personalised.
The thing is, if you get hit with a tv ad for a car, worst case scenario it can't have a strongly negative effect because of its generic content (dude looks cool while driving his 4WD through rocky terrain). But when an email that attempts to be personalised misses the mark, you feel it (Hi {First Name : Last Name}!).
Anyway, effective altruism loves targeting. It is virtually built from the ground up on it (RCTs and all that, aversion to PR etc).
Anyway, I am thinking about this in relation to the recent "80,000 hours should remove OpenAI from the Job Board" post.
80k is applying an extremely targeted strategy by recommending people work in some jobs at the workplaces creating the possible death machines (caveat 1, caveat 2, caveat 3, caveatn).
There are so many failure modes involved with those caveats that makes it virtually impossible to not go catastrophically poorly in at least a few instances and take the mental health of a bunch of young and well-meaning EAs with it.
[IMAGE] there is something about the lack overlap between these two audiences that makes me uneasy. WYD?
I think an assumption 80k makes is something like "well if our audience thinks incredibly deeply about the Safety problem and what it would be like to work at a lab and the pressures they could be under while there, then we're no longer accountable for how this could go wrong. After all, we provided vast amounts of information on why and how people should do their own research before making such a decision".
The problem is, that is not how most people make decisions. No matter how much rational thinking is promoted, we're first and foremost emotional creatures that care about things like status. So, if 80k decides to have a podcast with the Superalignment team lead, then they're effectively promoting the work of OpenAI. That will make people want to work for OpenAI. This is an inescapable part of the Halo effect.
Lastly, 80k is explicitly targeting very young people who, no offense, probably don't have the life experience to imagine themselves in a workplace where they have to resist incredible pressures to not conform, such as not sharing interpretability insights with capabilities teams.
The whole exercise smacks of nativity and I'm very confident we'll look back and see it as an incredibly obvious mistake in hindsight.
A piece of career advice I've given a few times recently to people in AI Safety, which I thought worth repeating here, is that AI Safety is so nascent a field that the following strategy could be worth pursuing:
1. Write your own job description (whatever it is that you're good at / brings joy into your life).
2. Find organisations that you think need thing that job but don't yet have it. This role should solve a problem they either don't know they have or haven't figured out how to solve.
3. Find the key decision maker and email them. Explain the (their) problem as you see it and how having this role could fix their problem. Explain why you're the person to do it.
I think this might work better for mid-career people, but, if you're just looking to skill up and don't mind volunteering, you could adapt this approach no matter what stage you're at in your career.
TIL that the words "fact" and "fiction" come from the same word: "Artifice" - which is ~ "something that has been created".
When I was ~ 5 I saw a homeless person on the street. I asked my dad where his home was. My dad said "he doesn't have a home". I burst into tears.
I'm 35 now and reading this post makes me want to burst into tears again. I appreciate you writing it though.
I'm pleased to announce the launch of a brand new Facebook group dedicated to AI Safety in Wellington: AI Safety Wellington (AISW). This is your local hub for connecting with others passionate about ensuring a safe and beneficial future with artificial intelligence / reducing x-risk.To kick things off, we're hosting a super casual meetup where you can:
- Meet & Learn: Connect with Wellington's AI Safety community.
- Chat & Collaborate: Discuss career paths, upcoming events, and training opportunities.
- Share & Explore: Whether you're new to AI Safety or a seasoned expert, come learn from each other's perspectives.
- Expand the Network: Feel free to bring a friend who's curious about AI Safety!
Join the group and RSVP for the meetup here
https://www.facebook.com/groups/500872105703980
one of my more esoteric buddhist practices is to always spell my name in lower case; it means I am a regularly reminded of the illusory nature of Self, while still engaging in imagined reality (Parikalpitā-svabhāva) in a practical way.
When AI Safety people are also vegetarians, vegans or reducetarian, I am pleasantly surprised, as this is one (of many possible) signals to me they're "in it" to prevent harm, rather than because it is interesting.
Hey mate thanks for the comment. I'm finding "pretty surprised" hard to interpret. Is that closer to 1% or 15%?
Hi Ann! Thank you for your comment. Some quick thoughts:
"I would consider, for the sake of humility, that they might disagree with your assessment for actual reasons, rather than assuming confusion is necessary."
- Yep! I have considered this. The purpose of my post is to consider it (I am looking for feedback, not upvotes or downvotes).
"They also happen to have a have a p(doom from not AGI) of 40% from combined other causes, and expect an aligned AGI to be able to effectively reduce this to something closer to 1% through better coordinating reasonable efforts."
- This falls into the confused category for me. I'm not sure how you have a 40% p(doom) from something other than unaligned AGI. Could you spell out for me what could make such a large number?
Hi Richard! Thanks for the comment. It seems to me that might apply to < 5% of people in capabilities?
Thanks for your comment Thomas! I appreciate the effort. I have some questions:
- by working on capabilities, you free up others for alignment work who were previously doing capabilities but would prefer alignment
I am a little confused by this, would you mind spelling it out for me? Imagine "Steve" took a job at "FakeLab" in capabilities. Are you saying Steve making this decision creates a Safety job for "Jane" at "FakeLab", that otherwise wouldn't have existed?
- more competition on product decreases aggregate profits of scaling labs
Again I am a bit confused. You're suggesting that if, for e.g., General Motors announced tomorrow they were investing $20 billion to start an AGI lab, that would be a good thing?
A judgement I'm attached to is that a person is either extremely confused or callous if they work in capabilities at a big lab. Is there some nuance I'm missing here?
Prediction: In 6-12 months people are going to start leaving Deepmind and Anthropic for similar sounding reasons to those currently leaving OpenAI (50% likelihood).
> Surface level read of what is happening at OpenAI; employees are uncomfortable with specific safety policies.
> Deeper, more transferable, harder to solve problem; no person that is sufficiently well-meaning and close enough to the coal face at Big Labs can ever be reassured they're doing the right thing continuing to work for a company whose mission is to build AGI.
Basically, this is less about "OpenAI is bad" and more "Making AGI is bad".
I went to buy a ceiling fan with a light in it recently. There was one on sale that happened to also tick all my boxes, joy! But the salesperson warned me "the light in this fan can't be replaced and only has 10,000 hours in it. After that you'll need a new fan. So you might not want to buy this one." I chuckled internally and bought two of them, one for each room.
I've decided to post something very weird because it might (in some small way) help shift the Overton Window on a topic: as long as the world doesn't go completely nuts due to AI, I think there is a 5%-20% chance I will reach something close to full awakening / enlightenment in about 10 years. Something close to this:
Very quick thoughts on setting time aside for strategy, planning and implementation, since I'm into my 4th week of strategy development and experiencing intrusive thoughts about needing to hurry up on implementation;
- I have a 52 week LTFF grant to do movement building in Australia (AI Safety)
- I have set aside 4.5 weeks for research (interviews + landscape review + maybe survey) and strategy development (segmentation, targeting, positioning),
- Then 1.5 weeks for planning (content, events, educational programs), during which I will get feedback from others on the plan and then iterate it.
- This leaves me with 46/52 weeks to implement ruthlessly.
In conclusion, 6 weeks on strategy and planning seems about right. 2 weeks would have been too short, 10 weeks would have been too long, this porridge is juuuussttt rightttt.
keen for feedback from people in similar positions.
Yeah it is a private purchase, unlike eating, so less likely to create some social effect by abstaining (i.e. vegan). I will say though, I've been vegan for about 7 years and I don't think I've nudged anyone :|
I have an intuition that if you tell a bunch of people you're extremely happy almost all the time (e.g. walking around at 10/10) then many won't believe you, but if you tell them that you're extremely depressed almost all the time (e.g. walking around at 1/10) then many more would believe you. Do others have this intuition? Keen on feedback.
Two jobs in AI Safety Advocacy that AFAICT don't exist, but should and probably will very soon. Will EAs be the first to create them though? There is a strong first mover advantage waiting for someone -
1. Volunteer Coordinator - there will soon be a groundswell from the general population wanting to have a positive impact in AI. Most won't know how to. A volunteer manager will help capture and direct their efforts positively, for example, by having them write emails to politicians
2. Partnerships Manager - the President of the Voice Actors guild reached out to me recently. We had a very surprising number of cross over in concerns and potential solutions. Voice Actors are the canary in the coal mine. More unions (etc) will follow very shortly. I imagine within 1 year there will be a formalised group of these different orgs advocating together.
Please help me find research on aspiring AI Safety folk!
I am two weeks into the strategy development phase of my movement building and almost ready to start ideating some programs for the year.
But I want these programs to be solving the biggest pain points people experience when trying to have a positive impact in AI Safety .
Has anyone seen any research that looks at this in depth? For example, through an interview process and then survey to quantify how painful the pain points are?
Some examples of pain points I've observed so far through my interviews with Technical folk:
- I often felt overwhelmed by the vast amount of material to learn.
- I felt there wasn’t a clear way to navigate learning the required information
- I lacked an understanding of my strengths and weaknesses in relation to different AI Safety areas (i.e. personal fit / comparative advantage) .
- I lacked an understanding of my progress after I get started (e.g. am I doing well? Poorly? Fast enough?)
- I regularly experienced fear of failure
- I regularly experienced fear of wasted efforts / sunk cost
- Fear of admitting mistakes or starting over might prevent people from making necessary adjustments.
- I found it difficult to identify my desired role / job (i.e. the end goal)
- When I did think I knew my desired role, identifying the specific skills and knowledge required for a desired role was difficult
- There is no clear career pipeline: Do X and then Y and then Z and then you have an A% chance of getting B% role
- Finding time to get upskilled while working is difficult
- I found the funding ecosystem opaque
- A lot of discipline and motivation over potentially long periods was required to upskill
- I felt like nobody gave me realistic expectations as to what the journey would be like
Thanks :) Uh, good question. Making some good links? Have you done much nondual practice? I highly recommend Loch Kelly :)
Hi Jonas! Would you mind saying about more about TMI + Seeing That Frees? Thanks!
Yesterday Greg Sadler and I met with the President of the Australian Association of Voice Actors. Like us, they've been lobbying for more and better AI regulation from government. I was surprised how much overlap we had in concerns and potential solutions:
1. Transparency and explainability of AI model data use (concern)
2. Importance of interpretability (solution)
3. Mis/dis information from deepfakes (concern)
4. Lack of liability for the creators of AI if any harms eventuate (concern + solution)
5. Unemployment without safety nets for Australians (concern)
6. Rate of capabilities development (concern)
They may even support the creation of an AI Safety Institute in Australia. Don't underestimate who could be allies moving forward!
Ilya Sutskever has left OpenAI https://twitter.com/ilyasut/status/1790517455628198322
Thanks for letting me know!
More people are going to quit labs / OpenAI. Will EA refill the leaky funnel?
[PHOTO] I sent 19 emails to politicians, had 4 meetings, and now I get emails like this. There is SO MUCH low hanging fruit in just doing this for 30 minutes a day (I would do it but my LTFF funding does not cover this). Someone should do this!
I expect (~ 75%) that the decision to "funnel" EAs into jobs at AI labs will become a contentious community issue in the next year. I think that over time more people will think it is a bad idea. This may have PR and funding consequences too.
Help clear something up for me: I am extremely confused (theoretically) how we can simultaneously have:
1. An Artificial Superintelligence
2. It be controlled by humans (therefore creating misuse of concentration of power issues)
My intuition is that once it reaches a particular level of power it will be uncontrollable. Unless people are saying that we can have models 100x more powerful than GPT4 without it having any agency??
Something I'm confused about: what is the threshold that needs meeting for the majority of people in the EA community to say something like "it would be better if EAs didn't work at OpenAI"?
Imagining the following hypothetical scenarios over 2024/25, I can't predict confidently whether they'd individually cause that response within EA?
- Ten-fifteen more OpenAI staff quit for varied and unclear reasons. No public info is gained outside of rumours
- There is another board shakeup because senior leaders seem worried about Altman. Altman stays on
- Superalignment team is disbanded
- OpenAI doesn't let UK or US AISI's safety test GPT5/6 before release
- There are strong rumours they've achieved weakly general AGI internally at end of 2025
"alignment researchers are found to score significantly higher in liberty (U=16035, p≈0)" This partly explains why so much of the alignment community doesn't support PauseAI!
"Liberty: Prioritizes individual freedom and autonomy, resisting excessive governmental control and supporting the right to personal wealth. Lower scores may be more accepting of government intervention, while higher scores champion personal freedom and autonomy..."
https://forum.effectivealtruism.org/posts/eToqPAyB4GxDBrrrf/key-takeaways-from-our-ea-and-alignment-research-surveys#comments
Hi Tomás! is there a prediction market for this that you know of?
I think it is unrealistic to ask people to internalise that level of ambiguity. This is how EA's turn themselves into mental pretzels.
Something someone technical and interested in forecasting should look into: can LLMs reliably convert peoples claims into a % of confidence through sentiment analysis? This would be useful for Forecasters I believe (and rationality in general)
That seems fair enough!
Hi Johannes! Thanks for the suggestion :) I'm not sure i'd want it in the middle of a video call, but maybe in a forum context like this could be cool?
Putting my EA Forum comment here:
I'd like to make clear to anyone reading that you can support the PauseAI movement right now, only because you think it is useful right now. And then in the future, when conditions change, you can choose to stop supporting the PauseAI movement.
AI is changing extremely fast (e.g. technical work was probably our best bet a year ago, I'm less sure now). Supporting a particular tactic/intervention does not commit you to an ideology or team forever!
There have been multiple occasions where I've copy and pasted email threads into an LLM and asked it things like:
- What is X person saying
- What are the cruxes in this conversation?
- Summarise this conversation
- What are the key takeaways
- What views are being missed from this conversation
I really want an email plugin that basically brute forces rationality INTO email conversations.
If you're into podcasts, the Very Bad Wizards guys did an ep on this essay, which I enjoyed: https://verybadwizards.com/episode/episode-227-a-terrible-master-david-foster-wallaces-this-is-water
Alcoholics are encouraged not to talk passed Liquor Stores. Basically, physical availability is the biggest lever - keep your phone / laptop in a different room when you don't absolutely need them!
If GPT5 actually comes with competent agents then I expect this to be a "Holy Shit" moment at least as big as ChatGPT's release. So if ChatGPT has been used by 200 million people, then I'd expect that to at least double within 6 months of GPT5 (agent's) release. Maybe triple. So that "Holy Shit" moment means a greater share of the general public learning about the power of frontier models. With that will come another shift in the Overton Window. Good luck to us all.