Posts
Comments
Computing the exact layer-truncated residual streams on GPT-2 Small, it seems that the effective layer horizon is quite large:
I'm mean ablating every edge with a source node more than n layers back and calculating the loss on 100 samples from The Pile.
Source code: https://gist.github.com/UFO-101/7b5e27291424029d092d8798ee1a1161
I believe the horizon may be large because, even if the approximation is fairly good at any particular layer, the errors compound as you go through the layers. If we just apply the horizon at the final output the horizon is smaller.
However, if we apply at just the middle layer (6), the horizon is surprisingly small, so we would expect relatively little error propagated.
But this appears to be an outlier. Compare to 5 and 7.
Source: https://gist.github.com/UFO-101/5ba35d88428beb1dab0a254dec07c33b
In this piece, we want to paint a picture of the possible benefits of AI, without ignoring the risks or shying away from radical visions.
Thanks for this piece! In my opinion you are still shying away from discussing radical (although quite plausible) visions. I expect the median good outcome from superintelligence involves everyone being mind uploaded / living in simulations experiencing things that are hard to imagine currently.
Even short of that, in the first year after a singularity, I would want to:
- Use brain computer interfaces to play videogames / simulations that feel 100% real to all senses, but which are not constrained by physics.
- Go to Hogwarts (in a 100% realistic simulation) and learn magic and make real (AI) friends with Ron and Hermione.
- Visit ancient Greece or view all the most important events of history based on superhuman AI archeology and historical reconstruction.
- Take medication that makes you always feel wide awake, focused etc. with no side effects.
- Engineer your body / use cybernetics to make yourself never have to eat, sleep, wash, etc. and be able to jump very high, run very fast, climb up walls, etc.
- Use AI as the best teacher ever to learn maths, physics and every subject and language and musical instruments to super-expert level.
- Visit other planets. Geoengineer them to have crazy landscapes and climates.
- Play God and oversee the evolution of life on other planets.
- Design buildings in new architectural styles and have AI build them.
- Genetically modify cats to play catch.
- Listen to new types of music, perfectly designed to sound good to you.
- Design the biggest roller coaster ever and have AI build it.
- Modify your brain to have better short term memory, eidetic memory, be able to calculate any arithmetic super fast, be super charismatic.
- Bring back Dinosaurs and create new creatures.
- Ask AI for way better ideas for this list.
I expect UBI, curing aging etc. to be solved within a few days of a friendly intelligence explosion.
Although I think we also plausibly will see a new type of scarcity. There is limited amount of compute you can create using the materials / energy in the universe. And if in fact most humans are mind-uploaded / brains in vats living in simulations, we will have to divide this among ourselves in order to run the simulations. If you have twice as much compute, you can simulate your brain twice as fast (or run two of you in parallel?), and thus experience twice as much subjective time - and so live twice as long until the heat death of the universe.
Note that the group I was in only played on the app. I expect this makes it significantly harder to understand what's going on.
Yes that's correct, this wording was imprecise.
Would you ever really want mean ablation except as a cheaper approximation to resample ablation?
Resample ablation is not more expensive than mean (they both are just replacing activations with different values). But to answer the question, I think you would - resample ablation biases the model toward some particular corrupt output.
It seems to me that if you ask the question clearly enough, there's a correct kind of ablation. For example, if the question is "how do we reproduce this behavior from scratch", you want zero ablation.
Yes I agree. That's the point we were trying to communicate with "the ablation determines the task."
- direct effect vs indirect effect corresponds to whether you ablate the complement of the circuit (direct effect) vs restoring the circuit itself (indirect effect, mediated by the rest of the model)
- necessity vs sufficiency corresponds to whether you ablate the circuit (direct effect necessary) / restore the complement of the circuit (indirect effect necessary) vs restoring the circuit (indirect effect sufficient) / ablating the complement of the circuit (direct effect sufficient)
Thanks! That's great perspective. We probably should have done more to connect ablations back to the causality literature.
- "all tokens vs specific tokens" should be absorbed into the more general category of "what's the reference dataset distribution under consideration" / "what's the null hypothesis over",
- mean ablation is an approximation to resample ablation which itself is an approximation to computing the expected/typical behavior over some distribution
These don't seem correct to me, could you explain further? "Specific tokens" means "we specify the token positions at which each edge in the circuit exists".
I think so. Mostly we learned about trading and the price discovery mechanism that is a core mechanic of the game. We started with minimal explanation of the rules, so I expect these things can be grokked faster by just saying them when introducing the game.
We just played Figgie at MATS 6.0, most players playing for the first time. I think we made lots of clearly bad decisions for the first 6 or 7 games. And reached a barely acceptable standard by about 10-15 games (but I say this as someone who was also playing for the first time).
(crossposted to the EA Forum)
Nonetheless, the piece exhibited some patterns that gave me a pretty strong allergic reaction. It made or implied claims like:
* a small circle of the smartest people believe this
* i will give you a view into this small elite group who are the only who are situationally aware
* the inner circle longed tsmc way before you
* if you believe me; you can get 100x richer -- there's still alpha, you can still be early
* This geopolitical outcome is "inevitable" (sic!)
* in the future the coolest and most elite group will work on The Project. "see you in the desert" (sic)
* Etc.
These are not just vibes - they are all empirical claims (except the last maybe). If you think they are wrong, you should say so and explain why. It's not epistemically poor to say these things if they're actually true.
If this were the case, wouldn't you expect the mean of the code steering vectors to also be a good code steering vector? But in fact, Jacob says that this is not case. Edit: Actually it does work when scaled - see nostalgebraist's comment.
Thanks. So will the building foundations be going through several meters of foam glass to the ice below?
Lightweight aggregates like expanded/foamed glass will form that layer, though likely with a honeycomb of basalt fiber-reinforced concrete and a final reinforced concrete topping layer.
Ok thanks. So will the top layer of concrete on foamed glass be floating of a layer of melted ice? Won't it gradually sink as more ice melts into denser water?
It's unclear to me how thick this layer is supposed to be. Will building foundations go though it and be anchored in the pykrete below? Presumably it's not possible to build building foundations in foamed glass?
This is a very interesting idea.
I'm still unclear if it's actually feasible to build on the ice. Even if most of the mass remains frozen for hundreds of years won't the surface of the ice be constantly melting such that creating building foundations is almost impossible?
you may be able to construct huge stacked layers about 50-100 meters tall made of reinforced pykrete with a whole city or park per layer and very convenient vertical transport between them. These layers will need a very slight amount of active cooling and a sensor network to monitor temperature
Doesn't this prove too much? If pykrete is such a cheap strong material, why don't we use it for regular buildings?
Is it possible that the canary string itself has been learned but not any documents that used the canary string in order to be removed from the dataset?
It seems we cannot allow all behavior-preserving optimizations, because that might lead to a kind of LLM that dutifully says "I'm conscious" without actually being so.
Surely 'you' are the algorithm, not the implementation. If I get refactored into a giant lookup table, I don't think that makes the algorithm any less 'me'.
I agree that it does not have something it mind but it could in principle have something in mind in the sense that it could represent some object in the residual stream in the tokens where it says "I have something in mind". And then future token positions could read this "memory".
Could the prediction market for each post be integrated more elegantly into the UI, rather than posted as a comment?
Yeah that's the crux I think. Or maybe we agree but are just using "substantial"/"most" differently.
It mostly comes down to intuitions so I think there probably isn't a way to resolve the disagreement.
Yes that's accurate.
Notably, as described this is not specifically a downside of anything I'm arguing for in my comment or a downside of actually being a contractor.
In your comment you say
- For some safety research, it’s helpful to have model access in ways that labs don’t provide externally. Giving employee level access to researchers working at external organizations can allow these researchers to avoid potential conflicts of interest and undue influence from the lab. This might be particularly important for researchers working on RSPs, safety cases, and similar, because these researchers might naturally evolve into third-party evaluators.
- Related to undue influence concerns, an unfortunate downside of doing safety research at a lab is that you give the lab the opportunity to control the narrative around the research and use it for their own purposes. This concern seems substantially addressed by getting model access through a lab as an external researcher.
I'm essentially disagreeing with this point. I expect that most of the conflict of interest concerns remain when a big lab is giving access to a smaller org / individual.
(Unless you think me being a contractor will make me more likely to want model access for whatever reason.)
From my perspective the main takeaway from your comment was "Anthropic gives internal model access to external safety researchers." I agree that once you have already updated on this information, the additional information "I am currently receiving access to Anthropic's internal models" does not change much. (Although I do expect that establishing the precedent / strengthening the relationships / enjoying the luxury of internal model access, will in fact make you more likely to want model access again in the future).
I'm not sure what the confusion is exactly.
If any of
- you have a fixed length contract and you hope to have another contract again in the future
- you have an indefinite contract and you don't want them to terminate your relationship
- you are some other evals researcher and you hope to gain model access at some point
you may refrain from criticizing Anthropic from now on.
It seems a substantial drawback that it will be more costly for you to criticize Anthropic in the future.
Many of the people / orgs involved in evals research are also important figures in policy debates. With this incentive Anthropic may gain more ability to control the narrative around AI risks.
Oh wow - missed that. Thanks!
Some in the field argued as recently as 2020 that no pure LLM would ever able to correctly complete Three plus five equals.
I think you don't mean this literally as the paper linked does not argue for this actual position. Can you clarify exactly what you mean?
The model has capacity to learn to do this, but can't just leverage existing capabilities in the foundation model, as the performance of that model is limited to that of the best humans it saw in the self-supervised training data. So, we need to do RL for many more time steps.
It's unclear to me if this is true because modelling the humans that generated the training data sufficiently well probably requires the model to be smarter than the humans it is modelling. So I expect the current regime where RLHF just elicits a particular persona from the model rather than teaching any new abilities to be sufficient to reach superhuman capabilities.
Oh man don't say it. Your comment is an infohazard.
4 respondents see a hard takeoff as likely (at varying degrees of hardness), and 1 finds it unlikely
Do people who say that hard takeoff is unlikely mean that they expect rapid recursive self-improvement to happen only after the AI is already very powerful? Presumably most people agree that a sufficiently smart AI will be able to cause an intelligence explosion?
Nice idea. Might try to work it into some of our material.
Unfortunately the sharing function is broken for me.
I am confused by takes like this - it just seems so blatantly wrong to me.
For example, yesterday I showed GPT-4o this image.
I asked it to show why (10) is the solution to (9). It wrote out the derivation in perfect Latex.
I guess this is in some sense a "trivial" problem, but I couldn't immediately think of the solution. It is googleable, but only indirectly, because you have to translate the problem to a more general form first. So I think for you to claim that LLMs are not useful you have to have incredibly high standards for what problems are easy / googleable and not value the convenience of just asking the exact question with the opportunity to ask followups.
BBC Tech News as far as I can tell has not covered any of the recent OpenAI drama about NDAs or employees leaving.
But Scarlett Johansson 'shocked' by AI chatbot imitation is now the main headline.
Thanks, this is really useful.
I am of the opinion that you should use good epistemics when talking to the public or policy makers, rather than using bad epistemics to try to be more persuasive.
Do you have any particular examples as evidence of this? This is something I've been thinking a lot about for AI and I'm quite uncertain. It seems that ~0% of advocacy campaigns have good epistemics, so it's hard to have evidence about this. Emotional appeals are important and often hard to reconcile with intellectual honesty.
Of course there are different standards for good epistemics and it's probably bad to outright lie, or be highly misleading. But by EA standards of "good epistemics" it seems less clear if the benefits are worth the costs.
As one example, the AI Safety movement may want to partner with advocacy groups who care about AI using copyrighted data or unions concerned about jobs. But these groups basically always have terrible epistemics and partnering usually requires some level of endorsement of their positions.
As an even more extreme example, as far as I can tell about 99.9% of people have terrible epistemics by LessWrong standards so to even expand to a decently sized movement you will have to fill the ranks with people who will constantly say and think things that you think are wrong.
I'm not sure if this is intentional but this explanation implies that edge patching can only be done between nodes in adjacent layers, which is not the case.
Yes you're correct that it does not work with LayerNorm between layers. I'm not aware of any models that do this. Are you?
Did you try how this works in practice? I could imagine an SGD-based circuit finder could be pretty efficient (compared to brute-force algorithms like ACDC), I'd love to see that comparison some day!
Yes it does work well! I did a kind of write up here but decided not to publish for various reasons.
Do you have a link to a writeup of Li et al. (2023) beyond the git repo?
https://arxiv.org/abs/2309.05973
I quit YouTube a few years ago and it was probably the single best decision I've ever made.
However I also found that I naturally substitute it with something else. For example, I subsequently became addictived to Reddit. I quit Reddit and substituted for Hackernews and LessWrong. When I quit those I substituted for checking Slack, Email and Discord.
Thankfully being addicted to Slack does seem to be substantially less harmful than YouTube.
I've found the app OneSec very useful for reducing addictions. It's an app blocker that doesn't actually block, it just delays you opening the page, so you're much less likely to delete it in a moment of weakness.
Or is that sentence meant to indicate that an instance running after training might figure out how to hack the computer running it so it can actually change it's own weights?
I was thinking of a scenario where OpenAI deliberately gives it access to its own weights to see if it can self improve.
I agree that it would be more likely to just speed up normal ML research.
While I want people to support PauseAI
the small movement that PauseAI builds now will be the foundation which bootstraps this larger movement in the future
Is one of the main points of my post. If you support PauseAI today you may unleash a force which you cannot control tomorrow.
If you want to be healthier, we know ways you can change your diet that will help: Increase your overall diet “quality”. Eat lots of fruits and vegetables. Avoid processed food. Especially avoid processed meats. Eat food with low caloric density. Avoid added sugar. Avoid alcohol. Avoid processed food.
I'm confused - why are you so confident that we should avoid processed food. Isn't the whole point of your post that we don't know whether processed oil is bad for you? Where's the overwhelming evidence that processed food in general is bad?
Reconstruction loss is the CE loss of the patched model
If this is accurate then I agree that this is not the same as "the KL Divergence between the normal model and the model when you patch in the reconstructed activations". But Fengyuan described reconstruction score as:
measures how replacing activations changes the total loss of the model
which I still claim is equivalent.
I think just showing would be better than reconstruction score metric because is very noisy.
there is a validation metric called reconstruction score that measures how replacing activations change the total loss of the model
That's equivalent to the KL metric. Would be good to include as I think it's the most important metric of performance.
Patch loss is different to L2. It's the KL Divergence between the normal model and the model when you patch in the reconstructed activations at some layer.
It would be good to benchmark the normalized and baseline SAEs using the standard metrics of patch loss and L0.
What is Neuron Activity?
Does anything think this could actually be done in <20 years?
materialism where we haven't discovered all the laws of physics yet — specifically, those that constitute the sought-for materialist explanation of consciousness
It seems unlikely that new laws of physics are required to understand consciousness? My claim is that understanding consciousness just requires us to understand the algorithms in the brain.
Without that real explanation, “atoms!” or “materialism!”, is just a label plastered over our ignorance.
Agreed. I don't think this contradicts what I wrote (not sure if that was the implication).
The Type II error of behaving as if these and future systems are not conscious in a world where they are in fact conscious.
Consciousness does not have a commonly agreed upon definition. The question of whether an AI is conscious cannot be answered until you choose a precise definition of consciousness, at which point the question falls out of the realm of philosophy into standard science.
This might seem like mere pedantry or missing the point, because the whole challenge is to figure out the definition of consciousness, but I think it is actually the central issue. People are grasping for some solution to the "hard problem" of capturing the je ne sais quoi of what it is like to be a thing, but they will not succeed until they deconfuse themselves about the intangible nature of sentience.
You cannot know about something unless it is somehow connected the causal chain that led to the current state of your brain. If we know about a thing called "consciousness" then it is part of this causal chain. Therefore "consciousness", whatever it is, is a part of physics. There is no evidence for, and there cannot ever be evidence for, any kind of dualism or epiphenomenal consciousness. This leaves us to conclude that either panpsychism or materialism is correct. And causally-connected panpsychism is just materialism where we haven't discovered all the laws of physics yet. This is basically the argument for illusionism.
So "consciousness" is the algorithm that causes brains to say "I think therefore I am". Is there some secret sauce that makes this algorithm special and different from all currently known algorithms, such that if we understood it we would suddenly feel enlightened? I doubt it. I expect we will just find a big pile of heuristics and optimization procedures that are fundamentally familiar to computer science. Maybe you disagree, that's fine! But let's just be clear that that is what we're looking for, not some other magisterium.
If consciousness is indeed sufficient for moral patienthood, then the stakes seem remarkably high from a utilitarian perspective.
Agreed. If your utility function is that you like computations similar to the human experience of pleasure and you dislike computations similar to the human experience of pain (mine is!). But again, let's not confuse ourselves by thinking there's some deep secret about the nature of reality to uncover. Your choice of meta-ethical system is of the same type signature as your choice of favorite food.
The subfaction veto only applies to faction level policy. The faction veto is decided by pure democracy within the faction.
I would guess in most scenarios most subfactions would agree when to use the faction veto. Eg. all the Southern states didn't want to end slavery.
Yes, Garrett is referring to this post: https://www.lesswrong.com/posts/yi7shfo6YfhDEYizA/more-people-getting-into-ai-safety-should-do-a-phd
Presumably the factions (eg. Southern states) also have sub factions, so maybe a better system would be described with the recursive acronym DVDF:
Democracy with Veto for each faction, plus DVDF within Factions.