Posts
Comments
Yeah. I think the part of the DNA specifying the brain is comparable to something like the training algorithm + initial weights of an LLM. I don't know how much space those would take if compressed, but probably very little, with the resulting model being much bigger than that. (And the model is in turn much smaller than the set of training data that went into creating it.)
Page 79-80 of the Whole Brain Emulation roadmap gave estimated storage requirements for uploading a human brain. The estimate depends on what we expect to be the scale on which the brain needs to be emulated. Workshop consensus at the time was that the most likely scale would be level 4-6 (see p. 13-14). This would put the storage requirements somewhere between 8000 and 1 million terabytes.
Mostly because water use was the most common criticism I'd happened to run into. The linked article has more metrics.
Glad you found it useful!
In case you hadn't ran into the term yet, the thing about conflating strength, grit and masculinity sounds like an instance of a bucket error.
Yeah, agree. Not sure what to do about that.
Of course, vibes are not infallible. As I mentioned in the post, a bad first vibe can change when you get to know someone more. So if you can collect more information in a low-risk way, it may be worth it. Reading someone's writing is usually (though not always) pretty low-risk.
I think a lot of my learning when I've changed my mind about someone's vibe has been implicit, similar to what Anni's comment is pointing at. Getting a better sense of what flavors of bad feelings are less reliable, which I expect to also update the mechanism that produced those particular kinds of vibes in the first place.
(I read it as humor rather than criticism.)
If we have two kinds of people and two kinds of effects that this advice might have:
- Dogmatics who would use this as an excuse for censorship, with the world getting worse as they hear this message
- Genuinely thoughtful people inclined toward self-doubt who are encouraged to listen to their gut, with the world getting better as it helped them avoid substantial damage
Then I acknowledge that the first effect probably exists, but I expect the second effect to dominate. The kinds of people who would ignore everything underneath the title and were just looking for an excuse for dogmatism would likely find some other excuse anyway, so the size of the effect seems small. While the kinds of people who are thoughtful but heavy on self-doubt seem to me much more substantially affected by public messaging giving them permission to listen to their intuition. (Source: I'm one, and I think I would have benefited from this kind of advice.)
We acknowledge that OpenAI does have access to a large fraction of FrontierMath problems and solutions, with the exception of a unseen-by-OpenAI hold-out set that enables us to independently verify model capabilities.
Can you say exactly how large of a fraction is the set that OpenAI has access to, and how much is the hold-out set?
And here's an excerpt from her book:
In this final chapter, then, I hope to start the conversation about what it means for science and policy to be actively anti-eugenicist, by offering five general principles:
1. Stop wasting time, money, talent, and tools that could be used to improve people’s lives.
2. Use genetic information to improve opportunity, not classify people.
3. Use genetic information for equity, not exclusion.
4. Don’t mistake being lucky for being good.
4. Consider what you would do if you didn’t know who you would be.For each of these principles, I will contrast three positions. First, the eugenic position positions genetic influence as a naturalizer of inequality. If social inequalities have genetic causes, then those inequalities are portrayed as the inevitable manifestations of a “natural” order. Genetic information about people can be used to slot them more effectively into that order. Second, the genome-blind position position sees genetic data as the enemy of social equality and so objects to any use of genetic information in social science and policy. Whenever possible, the genome-blind position seeks not to know: scientists ought not to study genetic differences or how they are linked to social inequalities, and other people in society ought not to use any scientific information that is generated for any practical purposes. These two positions can be contrasted with what I am proposing is an anti-eugenic position that does not discourage genetic knowledge but deliberately aims to use genetic science in ways that reduce inequalities in the distribution of freedoms, resources, and welfare. [...]
Stop Wasting Time, Money, Talent, and Tools
- EUGENIC: Point to the existence of genetic influence to deny the possibility of intervening to improve people’s lives.
- GENOME-BLIND: Ignore genetic differences even if it wastes resources and slows down science.
- ANTI-EUGENIC: Use genetic data to accelerate the search for effective interventions that improve people’s lives and reduce inequality of outcome. [...]
Use Genetic Information to Improve Opportunity, Not Classify People
- EUGENIC: Classify people into social roles or positions based on their genetics.
- GENOME-BLIND: Pretend that all people have an equal likelihood of achieving all social roles or positions after taking into account their environment.
- ANTI-EUGENIC: Use genetic data to maximize the real capabilities of people to achieve social roles and positions. [...]
Let’s go back to a specific example that I told you about in chapter 7, about the relationship between the educational attainment polygenic index and mathematics course-taking in high school. Students who had a higher polygenic index were more likely to be enrolled in geometry (versus algebra 1) in the ninth grade, which put them on track to complete calculus by the end of high school. Students who had a higher polygenic index were also less likely to drop out of math once it became optional. What can and should be done with that information?
The eugenic proposal would be to test students’ DNA and use it to assign them to mathematics tracks, such that students with low polygenic indices are excluded from opportunities to learn advanced mathematics. The gene-blind proposal would be to insist that the research connecting genetics and mathematics course taking shouldn’t have been done in the first place. The anti-eugenic proposal is to apply that genetic knowledge toward (a) understanding how teachers and schools can maximize the mathematics learning of their students, and (b) spotlighting how academic tracking entrenches inequalities between students.
Regarding the first goal, consider that one of the greatest challenges to understanding which teachers and schools are best serving the needs of students is that students with different learning needs are not randomly distributed across teachers and schools. A trenchant criticism of using standardized test scores as a metric for teacher and school “accountability”—that is, for identifying poorly performing teachers and schools—is that student test scores are highly correlated with student characteristics, such as family socioeconomic status, that precede the child’s entry to school and that are non-randomly clustered across schools. “Good” schools, defined as schools with high average test scores, are, in actuality, often better described as rich schools with high concentrations of affluent students. (A similar problem besieges identifying the best doctors and hospitals: the best doctor is not the one who avoids treating the sickest patients.)
Researchers have long recognized that estimating school effects on student academic outcomes is a tricky problem, and one can begin to make fair, “apples-to-apples” comparisons among schools only if one incorporates measures of student characteristics such as family background, previous levels of academic knowledge, etc. The appropriate question is not “How do students in school X fare differently than students in school Y?” because the students in school X could be already different from the students in school Y in ways other than the school they attend. The appropriate question is, “How would a particular student have fared differently if he had attended school X rather than school Y?” (Again, we see the importance of counterfactual reasoning for causal inference, as I explained in chapter 5).
In attempting to identify school effects, it is commonplace for researchers, educators, and policymakers to consider information about one accident of birth: a student’s socioeconomic status. But I and others have observed in our research that information from a student’s DNA, in the form of a polygenic index, also predicts academic outcomes, above and beyond information on family socioeconomic status. As I described above, this does not mean that we should use polygenic indices to classify students and restrict their opportunities to learn. It does mean, however, that we can evaluate how students who have equivalent polygenic indices fare differently in their outcomes when they attend different schools.
In one study of US high school students, we found that students with low education-related polygenic indices were, on average, less likely to continue in their mathematics education in high school. But their dropout rates differ substantially across school contexts. In schools that primarily serve students whose parents have high school diplomas, even students with low polygenic indices take a few years of math after the ninth grade. In fact, students with low polygenic indices in high-status schools fare about as well, in terms of their persistence in math, as students with average polygenic indices who attend low-status schools.
This finding is just barely scratching the surface. What, specifically, is happening in higher-status schools that keeps even students who are statistically likely to drop out of math from actually dropping out? How do you make the practices of such schools more widely available to all students? The path from basic research like this study to educational policy reform is long and tortuous.
But even though it is just a first step, this study is revealing a basic and important truth: given a certain fixed starting point in life—inheriting a certain combination of DNA variants—some people get much further in developing their capability to solve mathematical problems. These mathematical skills have lifelong benefits for an individual in terms of future education, participation in the labor force, and ease with navigating problems of everyday living. In fact, math literacy is so important for a student’s future that the opportunity to learn math has been called a civil right. Genetic data has thus revealed an inequality of environmental opportunity, one that calls out for redress.
Other environmental inequalities could be similarly diagnosed using genetic data. Which health interventions reach people who are currently most genetically at risk for poor outcomes? Which schools have the lowest rates of disciplinary problems among youth who are currently at most genetic risk for aggression, delinquency, or substance use problems? Which areas of the country are “opportunity zones,” where opportunity is defined not solely in terms of how children from low-income families fare, but also in terms of how children who are genetically at risk for school problems or mental health problems fare? If researchers embrace principle #1, and start embracing the possibilities of genetic data, we will have a wealth of new information to address these questions.
Use Genetic Information for Equity, Not Exclusion
- EUGENIC: Use genetic information to exclude people from health care systems, insurance markets, etc.
- GENOME-BLIND: Prohibit the use of genetic information per se but otherwise keep markets and systems the same.
- ANTI-EUGENIC: Create health care, educational, housing, lending, and insurance systems where everyone is included, regardless of the outcome of the genetic lottery. [...]
Don’t Mistake Being Lucky for Being Good
- EUGENIC: Point to genetic effects on intelligence as proof that some people naturally have more merit than others.
- GENOME-BLIND: Accept the logic of meritocracy while ignoring the role of genetic luck in developing skills and behaviors that are perceived as meritorious.
- ANTI-EUGENIC: Recognize genetics as a type of luck in life outcomes, undermining the meritocratic logic that people deserve their successes and failures on the basis of succeeding in school. [...]
Consider What You Would Do, If You Didn’t Know Who You Would Be
- EUGENIC: The biologically superior are entitled to greater freedoms and resources.
- GENOME-BLIND: Society should be structured as if everyone is exactly the same in their biology.
- ANTI-EUGENIC: Society should be structured to work to the advantage of people who were least advantaged in the genetic lottery.
But interestingly, Turkheimer also says some nice things about the book The Genetic Lottery: Why DNA Matters for Social Equality, by his former student Kathryn Paige Harden. Unlike Turkheimer, Harden is much more willing to ascribe a straightforward causal role of genetics in intelligence and other traits. But Harden reaches essentially the same sociopolitical conclusions as Turkheimer (according to Turkheimer). I didn’t read Harden’s book, but I presume that those conclusions are things like “racism is bad”, “we should help the downtrodden”, “don’t be an asshole”, and so on—conclusions that I myself enthusiastically endorse as well.
This New Yorker profile about Harden gets a bit into her philosophy. She wants society to use genetic data to design more effective social interventions for making people better off, and for an improved understanding of the effect of genetics to make people more receptive to programs designed to increase equality of outcome:
The first thing that social-science genomics can do is help researchers control for confounding genetic variables that are almost universally overlooked. As Harden puts it in her book, “Genetic data gets one source of human differences out of the way, so that the environment is easier to see.” For example, beginning in 2002, the federal government spent almost a billion dollars on something called the Healthy Marriage Initiative, which sought to reduce marital conflict as a way of combatting poverty and juvenile crime. Harden was not surprised to hear that the policy had no discernible effect. Her own research showed that, when identical-twin sisters have marriages with different levels of conflict, their children have equal risk for delinquency. The point was not to estimate the effects of DNA per se, but to provide an additional counterfactual for analysis: would an observed result continue to hold up if the people involved had different genes? Harden can identify studies on a vast array of topics—Will coaching underresourced parents to speak more to their children reduce educational gaps? Does having dinner earlier improve familial relationships?—whose conclusions she considers dubious because the researchers controlled for everything except the fact that parents pass along to their children both a home environment and a genome.
She acknowledged that gwas techniques are too new, and the anxieties about behavior genetics too deeply entrenched, to have produced many immediately instrumental examples so far. But she pointed to a study from last year as proof of concept. A team of researchers led by Jasmin Wertz, at Duke, used GWAS results to examine four different “aspects of parenting that have previously been shown to predict children’s educational attainment: cognitive stimulation; warmth and sensitivity; household chaos (reverse-coded to indicate low household chaos); and the safety and tidiness of the family home.” They found that one of them—cognitive stimulation—was linked to children’s academic achievement and their mothers’ genes, even when the children did not inherit the relevant variants. Parental choices to read books, do puzzles, and visit museums might be conditioned by their own genes, but they nevertheless produced significant environmental effects.
Even the discovery that a particular outcome is largely genetic doesn’t mean that its effects will invariably persist. In 1972, the U.K. government raised the age at which students could leave school, from fifteen to sixteen. In 2018, a research group studied the effects of the extra year on the students as adults, and found that their health outcomes for measures like body-mass index, for whatever reason, improved slightly on average. But those with a high genetic propensity for obesity benefitted dramatically—a differential impact that might easily have gone unnoticed.
Some of Harden’s most recent research has looked at curricular tracking for mathematics, an intuitive instance of how gene-environment interactions can create feedback loops. Poor schools, Harden has found, tend to let down all their students: those with innate math ability are rarely encouraged to pursue advanced classes, and those who struggle are allowed to drop the subject entirely—a situation that often forecloses the possibility of college. The most well-off schools are able to initiate virtuous cycles in the most gifted math students, and break vicious cycles in the less gifted, raising the ceiling and the floor for achievement. [...]
Harden is not alone in her drive to fulfill Turkheimer’s dream of a “psychometric left.” Dalton Conley and Jason Fletcher’s book, “The Genome Factor,” from 2017, outlines similar arguments, as does the sociologist Jeremy Freese. Last year, Fredrik deBoer published “The Cult of Smart,” which argues that the education-reform movement has been trammelled by its willful ignorance of genetic variation. Views associated with the “hereditarian left” have also been articulated by the psychiatrist and essayist Scott Alexander and the philosopher Peter Singer. Singer told me, of Harden, “Her ethical arguments are ones that I have held for quite a long time. If you ignore these things that contribute to inequality, or pretend they don’t exist, you make it more difficult to achieve the kind of society that you value.” He added, “There’s a politically correct left that’s still not open to these things.” [...]
The ultimate claim of “The Genetic Lottery” is an extraordinarily ambitious act of moral entrepreneurialism. Harden argues that an appreciation of the role of simple genetic luck—alongside all the other arbitrary lotteries of birth—will make us, as a society, more inclined to ensure that everyone has the opportunity to enjoy lives of dignity and comfort. She writes, “I think we must dismantle the false distinction between ‘inequalities that society is responsible for addressing’ and ‘inequalities that are caused by differences in biology.’ ” She cites research showing that most people are much more willing to support redistributive policies if differences in opportunity are seen as arbitrarily unfair—and deeply pervasive.
As she put it to me in an e-mail, “Even if we eliminated all inequalities in educational outcomes between sexes, all inequalities by family socioeconomic status, all inequalities between different schools (which as you know are very confounded with inequalities by race), we’ve only eliminated a bit more than a quarter of the inequalities in educational outcomes.” She directed me to a comprehensive World Bank data set, released in 2020, which showed that seventy-two per cent of inequality at the primary-school level in the U.S. is within demographic groups rather than between them. “Common intuitions about the scale of inequality in our society, and our imaginations about how much progress we would make if we eliminated the visible inequalities by race and class, are profoundly wrong,” she wrote. “The science confronts us with a form of inequality that would otherwise be easy to ignore.”
The perspective of “gene blindness,” she believes, “perpetuates the myth that those of us who have ‘succeeded’ in twenty-first century capitalism have done so primarily because of our own hard work and effort, and not because we happened to be the beneficiaries of accidents of birth—both environmental and genetic.” She invokes the writing of the philosophers John Rawls and Elizabeth Anderson to argue that we need to reject “the idea that America is or could ever be the sort of ‘meritocracy’ where social goods are divided up according to what people deserve.” Her rhetoric is grand, though the practical implications, insofar as she discusses them, are not far removed from the mid-century social-democratic consensus—the priorities of, say, Hubert Humphrey. If genes play a significant role in educational attainment, then perhaps we ought to design our society such that you don’t need a college degree to secure health care.
So I think we have two separate questions here:
- Do psychological issues involve reactivation of an earlier memory such that the reactivation plays a causal role in the issue?
- Can you address an issue without explicitly working with an earlier memory?
For the first question, I'd say "it depends". On one end, we have something like PTSD flashbacks - here a reactivation of a memory is clearly in a causal role, since "a memory getting reactivated to such an extent that the person experiences themselves as literally reliving it" is what a flashback is.
Slightly less strong but still strongly suggestive of a causal role is something where a person imagines themselves doing something, but then - maybe just at the back of their mind - recalls a painful memory and flinches away from doing that. E.g. they consider speaking up, but then a flicker of a memory comes up about a time when they spoke up and somebody ridiculed them, and they quickly close their mouth. Here there seems to at least be a causal path from the memory to the issue, in that the memory is charged with negative affect and that the memory coming up causes the person to reorient to something that makes the memory recede in intensity.
Then we have cases where there's no obvious memory at first, but directing attention to the issue and asking questions about it brings up a memory, even though none of the questions ever ask about memories directly. For example, someone might feel like they have to act in a certain way in a particular social situation despite finding it unpleasant. Now a therapist might ask them something like "what would be bad about acting differently" and have them focus on what feels emotionally or intuitively bad about it (rather than what logical justifications their mind would offer). Then there might be a line of questioning that went something like:
- "I have difficulty getting a turn to speak because I tend to wait extra long to make sure others have finished speaking before I speak up. And then I wait so long that someone else always starts talking before I can."
- "Okay, so what would be bad about speaking up before you're certain that others have finished speaking?"
- "Then I might interrupt them before they're finished."
- "Okay, what would be bad about interrupting them before they're finished?"
- "That'd feel unfair toward them."
- "In what way does it feel unfair?"
- "Hmm, I'm getting a memory of a time when I was trying to speak up but my father interrupted me, and then I tried talking anyway and then he acted like I had interrupted him and that I should let him talk first. That felt really bad and unfair. I guess I want to make sure to act better than he did, and make sure that I never interrupt someone else when it's their turn to speak."
Is the memory in a causal role here? Probably depends on how exactly we define "causal". But at least it seems like there is some kind of a model about how the person wants to act or not act ("interrupting other people is unfair toward them, and should be avoided") that was formed due to an earlier experience. When one tries to elicit details about how exactly the model works, the model seems to structurally incorporate the original experience as a reference point for what exactly the core bad thing is. And working with the memory often seems to help with one's issues.
Given that this kind of a memory seems to have a similar character as the PTSD and the "I can hear the people mocking me" memories, just buried slightly deeper, to me the simplest and most plausible explanation would be that it has a causal role in the same way as the less-buried ones do.
Then on the other hand, it's not always the case that this kind of questioning leads to any clear-cut memory. Sometimes what comes up feels more like a general model that has been formed out of multiple different life experiences, with none of original instances having been stored. Or there might be an issue that seems to go back to an age young enough that the person doesn't have any explicit memories of it, and it has only left a general emotional imprint. In those cases the memory doesn't seem to have any causal role, because there doesn't seem to be any memory around to begin with.
Or at least not one that would be easily accessible. I've heard of claims from people who got into states of deep meditation or strong doses of psychedelics that they managed to access very early painful memories that wouldn't have been available in a normal therapeutic context, and then got independent confirmation for the truthfulness of those memories afterward. I've not looked into these in detail but I'm inclined to suspect they're true. In part due to my personal experiences of old memories spontaneously coming up in altered states of consciousness (and this sometimes shifting behaviors), in part because "all behaviors involve an original memory trace being stored somewhere and that trace then driving behavior, with some of those traces just being buried deep or in not normally forward-compatible storage formats" would again seem like the most parsimonious model.
As for the second question, I'd again say it depends. If someone is suffering from a PTSD flashback, it's going to be hard to do anything about that without working with the traumatic memory in some way! But for the ones where the problem isn't so directly driven by an explicit memory reactivation, there are definitely a lot of approaches that work by changing other parts of the model. E.g. if the model makes a particular prediction about the world in general ("people will always find it unfair if I speak up before being absolutely certain that they're finished"), then it's often possible to disprove that prediction without going into the details of the original memory. And while some therapies focus on the episodic memory component of the learned model, others work on different components.
Cool, thanks!
(B’) “…Gee, I guess this reprocessing must have been a kind of ‘training / practice / exercise’ during which I could forge new better subconscious habits and associations related to ‘the feeling of anxiety’ in general. And these new subconscious habits and associations are now serving me well in a wide variety of adult contexts.”
I think that there are definitely techniques that work on one's reaction to the feeling of anxiety in general, but the specific ones that I had in mind don't feel like they'd be doing that. Rather they seem much more localized, in that they eliminate some particular anxiety trigger or specific kind of anxiety from getting triggered in the first place. But then if something else happens to trigger the same or a similar anxiety, the person isn't necessarily any better at dealing with that.
So if someone feels the same kind of anxious around both spiders and snakes, then this kind of an intervention might eliminate the fear of spiders entirely, while leaving the reaction to snakes entirely unaffected (or vice versa).
Also, AFAICT, people achieve great therapeutic success by methods that involve bringing up childhood memories, but other people also achieve great therapeutic success by methods that don’t. :)
Oh yeah definitely, didn't mean to imply that working with memories would be the only approach that worked.
I think that getting too fixated on status games is usually due to some kind of insecurity, e.g. feeling that you need to accumulate status in order to be accepted, respected or something like that. (One can certainly play status games for the fun of it without having such an insecurity, but that seems unlikely to lead to the level of fixation where status game would become the primary activity that humans engage in.)
If every human can close friends or even lovers with AI systems as you suggest, then I would expect that to provide the kind of deep unconditional feeling of security that makes the need for playing status games fall away. If you feel deeply safe, loved, respected, etc., then status can certainly still feel nice to have, but it's unlikely to feel that important for most people. In the same way that e.g. a loving parent focused on taking care of and spending time with their family may find themselves becoming much less interested in spending their time playing status games.
Fantastically detailed post, thank you for taking the time to write up all this!
I'd be curious to hear your thoughts about the following. There seems to be an obvious conflict between:
- Behavioral genetics, which has all these findings about the childhood environment only having a limited effect
- Therapy, where memory reconsolidation can achieve significant chances in people's feelings and behavior by changing subconscious beliefs, and many of those beliefs seem to be related to childhood events and experiences
One hypothesis I've had for reconciling those is based on what evolutionary psychologists have learned about fear of snakes. At one point, it was thought that humans might be hardwired with a fear of spiders and snakes in particular. But later work then suggested that this is wrong - instead, humans are evolutionarily biased towards paying extra attention to things like spiders and snakes.
Then because we pay more attention to things that look like that, it's more likely that we notice something scary about them. Or if we've been told that they're dangerous, then just repeatedly noticing them increases the chance that we develop a mild phobia around them (as it's increasing the prior of "this dangerous thing might be around and you should notice it"). And that seems to explain why things like spider and snake phobias are much more common than things like electricity phobias:
Seligman’s account suggested that specialised, central mechanisms of fear learning more readily connect aversive events, such as electric shock, with fear-relevant stimuli, such as snakes – which presented genuine threats to our evolutionary ancestors – than with ‘fear-irrelevant’ stimuli such as geometric shapes or flowers. This account predicts that fear of fear-relevant objects should be learned faster, and be extinguished more slowly when shock no longer occurs, as well as being resistant to topdown modification, for example, by instructions indicating that shocks will not occur.
The results of early experiments were consistent with some of these predictions (e.g., [50,51]), but none has withstood extended experimental investigation. Faster or better conditioning with fear-relevant stimuli has rarely been observed, and there is ample evidence that, like most associative learning (e.g., [52]), it can be modified by instruction (reviewed in [53,54]). Initially it seemed that responses to fear-relevant stimuli might extinguish more slowly. However, a recent systematic review [55] found that most positive findings came from a single laboratory, and a large majority of the full set of studies had failed to find differences between fear-relevant and fear-irrelevant stimuli in the rate of extinction.
These results suggest that fear of snakes and other fear-relevant stimuli is learned via the same central mechanisms as fear of arbitrary stimuli. Nevertheless, if that is correct, why do phobias so often relate to objects encountered by our ancestors, such as snakes and spiders, rather than to objects such as guns and electrical sockets that are dangerous now [10]? Because peripheral, attentional mechanisms are tuned to fear-relevant stimuli, all threat stimuli attract attention, but fear-relevant stimuli do so without learning (e.g., [56]). This answer is supported by evidence from conditioning experiments demonstrating enhanced attention to fear-relevant stimuli regardless of learning (Box 2), studies of visual search [57–59], and developmental psychology [60,61]. For example, infants aged 6–9 months show a greater drop in heart rate – indicative of heightened attention rather than fear – when they watch snakes than when they watch elephants [62].
Now suppose that some people carried genes that made them pay extra attention to snakes and/or spiders, and other people didn't. In that situation, you might observe both that:
- The probability of having a snake or spider phobia was strongly heritable - people with those genes were likely to develop that phobia, with parenting style having little effect
- Therapists employing memory reconsolidation-based methods for treating the phobia could often find some specific childhood experiences that had to do with spiders, that seemed to be at the root of the phobia (and doing reconsolidation on these experiences reliably helped with the phobia)
We could then suppose that a lot of other psychological traits are similar: if you have a certain set of genes, it will make you much more likely to have a particular kind of psychological reaction in response to external events. While a person with another set of genes would react differently. And while people in different kinds of environments will differ in exactly what kinds of events they are exposed, then assuming that they belong to a roughly similar social class within the same country, they will probably still have some experiences that are roughly similar.
For instance, when I was little, some older children in our neighborhood made up a story about a man going around the neighborhood and kidnapping children. I expect that a lot of kids who weren't particularly inclined toward high neuroticism soon forgot about the whole thing. Meanwhile I got really scared about it and asked my parents if we couldn't move somewhere else, and then much later as an adult found myself having minor anxiety that seemed to have its roots in this childhood experience.
Now if we hadn't lived in that particular middle-class neighborhood, I wouldn't have encountered that particular rumor and it wouldn't have left a mark on me. But given that I had high-neuroticism genes that made me seriously freaked out by some older children deciding to scare the younger ones a bit, probably something else would have happened in any other middle-class neighborhood that would have felt equally scary and made me somewhat more inclined to anxiety in the future.
That would again lead to the pattern where most major differences seem to come from genetic differences, and at the same time many people with psychological problems can consistently trace the source of their problems to childhood experiences.
Worth keeping in mind that OpenAI is burning through crazy amounts of money and is constantly in need of more:
OpenAI raised $6.6 billion [in October], the largest venture capital round in history. But it plans to lose $5 billion [2024] alone. And by 2026, it could be losing $14 billion per year. That’s head exploding territory. If OpenAI keeps burning money at this rate, it will have to raise another round soon. Perhaps as early as 2025.
As a result, Altman has a significant financial incentive to believe/say that OpenAI is on the verge of a breakthrough and that it's worth it for their investors to continue giving them money.
I think even "a detailed picture of the topic of the lesson" can be too high of an expectation for many topics early on. (Ideally it wouldn't be, if things were taught well, but they often aren't.) A better goal would be to have just something you understand well enough that you can grab on to, that you can start building out from.
Like if the topic was a puzzle, it's fine if you don't have a rough sense of where every puzzle piece goes right away. It can be enough that you have a few corner pieces in place, that you then start building out from.
I recall it heard claimed that a reason why financial crimes sometimes seem to have disproportionately harsh punishments relative to violent crimes is that financial crimes are more likely to actually be the result of a cost-benefit analysis.
Fantastic post. This has been frequently on my mind after reading it, and especially the surface/character layer split feels very distinct now that I have an explicit concept for it. And then at one point I asked it to profile me based on some fiction I co-wrote with it and it managed to guess that I was Finnish from something I didn't think had any clues in that direction, which gave me a novel feeling of getting a glimpse into that vast alien ground layer.
The analogy to the character and player distinction in humans also feels very apt.
Don't most browsers come with spellcheck built in? At least Chrome automatically flags my typos.
Thanks. Still not convinced, but it will take me a full post to explain why exactly. :)
Though possibly some of this is due to a difference in definitions. When you say this:
what I consider AGI - which importantly is fully general in that it can learn new things, but will not meet the bar of doing 95% of remote jobs because it's not likely to be human-level in all areas right away
Do you have a sense of how long you expect it will take for it to go from "can learn new things" to "doing 95% of remote jobs"? If you e.g. expect that it might still take several years for AGI to master most jobs once it has been created, then that might be more compatible with my model.
Hmm, some years back I was hearing the claim that self-driving cars work badly in winter conditions, so are currently limited to the kinds of warmer climates where Waymo is operating. I haven't checked whether that's still entirely accurate, but at least I haven't heard any news of this having made progress.
Thanks, this is the kind of comment that tries to break down things by missing capabilities that I was hoping to see.
Episodic memory is less trivial, but still relatively easy to improve from current near-zero-effort systems
I agree that it's likely to be relatively easy to improve from current systems, but just improving it is a much lower bar than getting episodic memory to actually be practically useful. So I'm not sure why this alone would imply a very short timeline. Getting things from "there are papers about this in the literature" to "actually sufficient for real-world problems" often takes a significant time, e.g.:
- I believe that chain-of-thought prompting was introduced in a NeurIPS 2022 paper. Going from there to a model that systematically and rigorously made use of it (o1) took about two years, even though the idea was quite straightforward in principle.
- After the 2007 DARPA Grand Challenge there was a lot of hype about how self-driving cars were just around the corner, but almost two decades later, they’re basically still in beta.
My general prior is that this kind of work - from conceptual prototype to robust real-world application - can in general easily take between years to decades, especially once we move out of domains like games/math/programming and into ones that are significantly harder to formalize and test. Also, the more interacting components you have, the trickier it gets to test and train.
Thanks. I think this argument assumes that the main bottleneck to AI progress is something like research engineering speed, such that accelerating research engineering speed would drastically increase AI progress?
I think that that makes sense as long as we are talking about domains like games / math / programming where you can automatically verify the results, but that something like speed of real-world interaction becomes the bottleneck once shifting to more open domains.
Consider an AI being trained on a task such as “acting as the CEO for a startup”. There may not be a way to do this training other than to have it actually run a real startup, and then wait for several years to see how the results turn out. Even after several years, it will be hard to say exactly which parts of the decision process contributed, and how much of the startup’s success or failure was due to random factors. Furthermore, during this process the AI will need to be closely monitored in order to make sure that it does not do anything illegal or grossly immoral, slowing down its decision process and thus whole the training. And I haven’t even mentioned the expense of a training run where running just a single trial requires a startup-level investment (assuming that the startup won’t pay back its investment, of course).
Of course, humans do not learn to be CEOs by running a million companies and then getting a reward signal at the end. Human CEOs come in with a number of skills that they have already learned from somewhere else that they then apply to the context of running a company, shifting between their existing skills and applying them as needed. However, the question of what kind of approach and skill to apply in what situation, and how to prioritize between different approaches, is by itself a skillset that needs to be learned... quite possibly through a lot of real-world feedback.
I think their relationship depends on whether crossing the gap requires grind or insight. If it's mostly about grind then a good expert will be able to estimate it, but insight tends to unpredictable by nature.
Another way of looking at my comment above would be that timelines of less than 5 years would imply the remaining steps mostly requiring grind, and timelines of 20+ years would imply that some amount of insight is needed.
we're within the human range of most skill types already
That would imply that most professions would be getting automated or having their productivity very significantly increased. My impression from following the news and seeing some studies is that this is happening within copywriting, translation, programming, and illustration. [EDIT: and transcription] Also people are turning to chatbots for some types of therapy, though many people will still intrinsically prefer a human for that and it's not affecting the employment of human therapists yet. With o3, math (and maybe physics) research is starting to be affected, though it mostly hasn't been yet.
I might be forgetting some, but the amount of professions left out of that list suggests that there are quite a few skill types that are still untouched. (There are of course a lot of other professions for which there have been moderate productivity boosts, but AFAIK mostly not to the point that it would affect employment.)
Doesn't that discrepancy (how much answers vary between different ways of asking the question) tell you that the median AI researcher who published at these conferences hasn't thought about this question sufficiently and/or sanely?
We know that AI expertise and AI forecasting are separate skills and that we shouldn't expect AI researchers to be skilled at the latter. So even if researchers have thought sufficiently and sanely about the question of "what kinds of capabilities are we still missing that would be required for AGI", they would still be lacking the additional skill of "how to translate those missing pieces into a timeline estimate".
Suppose that a researcher's conception of current missing pieces is a mental object M, their timeline estimate is a probability function P, and their forecasting expertise F is a function that maps M to P. In this model, F can be pretty crazy, creating vast differences in P depending how you ask, while M is still solid.
I think the implication is that these kinds of surveys cannot tell us anything very precise such as "is 15 years more likely than 23", but we can use what we know about the nature of F in untrained individuals to try to get a sense of what M might be like. My sense is that answers like "20-93 years" often translate to "I think there are major pieces missing and I have no idea of how to even start approaching them, but if I say something that feels like a long time, maybe someone will figure it out in that time", "0-5 years" means "we have all the major components and only relatively straightforward engineering work is needed for them", and numbers in between correspond to Ms that are, well, somewhere in between those.
Yeah I'm not sure of the exact date but it was definitely before LLMs were a thing.
Relative to 10 (or whatever) years ago? Sure I've made quite a few of those already. By this point it'd be hard to remember my past beliefs well enough to make a list of differences.
Due to o3 specifically? I'm not sure, I have difficulty telling how significant things like ARC-AGI are in practice, but the general result of "improvements in programming and math continue" doesn't seem like a huge surprise by itself. It's certainly an update in favor of the current paradigm continuing to scale and pay back the funding put into it, though.
Am I understanding right that inference compute scaling time is useful for coding, math, and other things that are machine-checkable, but not for writing, basic science, and other things that aren't machine-checkable?
I think it would be very surprising if it wasn't useful at all - a human who spends time rewriting and revising their essay is making it better by spending more compute. When I do creative writing with LLMs, their outputs seem to be improved if we spend some time brainstorming the details of the content beforehand, with them then being able to tap into the details we've been thinking about.
It's certainly going to be harder to train without machine-checkable criteria. But I'd be surprised if it was impossible - you can always do things like training a model to predict how much a human rater would like literary outputs, and gradually improve the rater models. Probably people are focusing on things like programming first both because it's easier and also because there's money in it.
I doubt that anyone even remembers this, but I feel compelled to say it: there was some conversation about AI maybe 10 years ago, possibly on LessWrong, where I offered the view that abstract math might take AI a particularly long time to master compared to other things.
I don't think I ever had a particularly good reason for that belief other than a vague sense of "math is hard for humans so maybe it's hard for machines too". But formally considering that prediction falsified now.
Fixed, thanks
Mostly just personal experience with burnout and things that I recall hearing from others; I don't have any formal papers to point at. Could be wrong.
I think it's fine if the users are clearly informed about this happening, e.g. the DM interface showing a small message that explains how metadata is used. (But I think it shouldn't be any kind of one-time consent box that's easy to forget about.)
That makes sense in general, though in this particular case I do think it makes sense to divide the space into either "things that have basically zero charge" or "things that have non-zero charge".
I feel like activists are generally seen like this when one disagrees with their cause, and seen as brave people doing an important thing when one agrees with their cause. If one doesn't have an opinion, could go either way, depending on how much they seem to violate generally-accepted norms and how strongly the person in question feels about those norms.
Okay, going through the messages in detail, the best account of what I can reconstruct of what actually happened is:
- The mechanics in this particular game involved 1) a choice of what kind of an action to play 2) once the action had been chosen, a choice of where exactly to play it. Person A had previously agreed to make certain plays.
- For one of the plays (call this "action 1"), communication had been ambiguous. A had ended up thinking that we'd agreed on the action to play but left the choice of where to play it up to him, whereas person B had ended up with the belief that we'd decided both on the action and the location.
- We had also agreed on A doing another thing, call it "action 2a".
- When the time came, A noticed that if he played action 1 in a particular location, he could do another thing ("action 2b") that would still lead to the same outcome that 2a would have led to.
- Person A now said that it didn't matter whether he played action 2a or action 2b, since by that point either one would lead to the same outcome.
- However person B objected that A's claim of "both 2a and 2b lead to the same result" was only true given that A had already decided to play that action in a different location than had already been decided, while B held that the choice of where to play it was part of what needed to be decided together.
And the specific thing that B found triggering was that (in her view) A didn't even acknowledge that he was deviating from something that had already been agreed upon (the choice of where to play action 1), and instead that gave (what seemed to B like an) excuse for why it was okay to unilaterally change action 2.
That seems complex enough that I'm not sure how to rewrite the post to take it into account while also keeping it clear.
I don't think alternative stories have negligible probability
Okay! Good clarification.
I think it's good to discuss norms about how appropriate it is to bring up cynical hypotheses about someone during a discussion in which they're present.
To clarify, my comment wasn't specific to the case where the person is present. There are obvious reasons why the consideration should get extra weight when the person is present, but there's also a reason to give it extra weight if none of the people discussed are present - namely that they won't be able to correct any incorrect claims if they're not around.
so I think it went fine
Agree.
(As I mentioned in the original comment, the point I made was not specific to the details of this case, but noted as a general policy. But yes, in this specific case it went fine.)
I think it's both true what you say, that removing blocks can give you instant improvements that no amount of practice ever would, and also that one can make progress with practice in the right conditions.
Oh. This discussion got me to go back and review some messages written in the aftermath of this, when I was trying to explain things to A... and I noticed a key thing I'd misremembered. (I should have reviewed those messages before posting this, but I thought that they only contained the same things that I already covered here.)
It wasn't that A was making a different play that was getting the game into a better state; it was that he was doing a slightly different sequence of moves that nevertheless brought the game into exactly the same state as the originally agreed upon moves would have. That was what the "it doesn't matter" was referring to.
Well that explains much better why this felt so confusing for the rest of us. I'll rewrite this to make it more accurate shortly. Thanks for the comments on this version for making me look that up!
but there doesn't have to be any past betrayal to object to betrayal in the present; people don't need to have ever been betrayed in the past to be against it as a matter of principle.
True, but that is assuming that everyone was perceiving this as a betrayal. A relevant question is also, what made A experience this as a betrayal, when there were four people present and none of the other three did? (It wasn't even B's own plan that was being affected by the changed move, it was my plan - but I was totally fine with that, and certainly didn't experience that as a betrayal.)
Betrayal usually means "violating an agreement in a way that hurts one person so that another person can benefit" - it doesn't usually mean "doing something differently than agreed in order to get a result that's better for everyone involved". In fact, there are plenty of situations where I would prefer someone to not do something that we agreed upon, if the circumstances suddenly change or there is new information that we weren't aware of before.
Suppose that I'm a vegetarian and strongly opposed to buying meat. I ask my friend to bring me a particular food from the store, mistakenly thinking it's vegetarian. At the store, my friend realizes that the food contains meat and that I would be unhappy if they followed my earlier request. They bring me something else, despite having previously agreed to bring the food that I requested. I do not perceive this as a betrayal, I perceive this as following my wishes. While my friend may not be following our literal agreement, they are following my actual goals that gave rise to that agreement, and that's the most important thing.
In the board game, three of us (A, me, and a fourth person who I haven't mentioned) were perceiving the situation in those terms: that yes, A was doing something differently than we'd agreed originally. But that was because he had noticed something that actually got the game into a better state, and "getting the game into as good of a state as possible" was the purpose of the agreement.
Besides, once B objected, A was entirely willing to go back to the original plan. Someone saying "I'm going to do things differently" but then agreeing to do things the way that were originally agreed upon as soon as the other person objects isn't usually what people mean by betrayal, either.
And yet B was experiencing this as a betrayal. Why was that?
I would strongly caution against assuming mindreading is correct.
I definitely agree! At the same time, I don't think one should take this far as never having hypotheses about the behavior of other people. If a person is acting differently than everyone else in the situation is, and thing X about them would explain that difference, then it seem irrational not to at least consider that hypothesis.
But of course one shouldn't just assume themselves to be correct without checking. Which I did do, by (tentatively) suggesting that hypothesis out loud and letting B confirm or disconfirm it. And it seemed to me that this was actually a good thing, in that a significant chunk of B's experience of being understood came from me having correctly intuited that. Afterward she explicitly and profusely thanked me for having spoken up and figured it out.
Also, as I mentioned, this is a slightly fictionalized account that I wrote based on my recollection of the essence of what happened. But the exact details of what was actually said were messier than this, and the logic of exactly what was going on didn't seem as clear as it does in this narrative. Regenerating the events based on my memory of the essence of the issue makes things seem clearer than they actually were, because that generator doesn't contain any of the details that made the essence of the issue harder to see at the time.
So if this conversation had actually taken place literally as I described it, then the hypothesis that you object to would have been more redundant. In the actual conversation that happened, things were less clear, and quite possibly the core of the issue may actually have been slightly different from what seems to make sense to me in retrospect when I try to recall it.
My read was that one might certainly just object to the thing on those grounds alone, but that the intensity of B's objection was such that it seemed unlikely without some painful experience behind it. B also seemed to become especially agitated by some phrases ("it doesn't matter") in particular, in a way that looked to me like she was being reminded of some earlier experience where similar words had been used.
And then when I tried to explain things to A and suggested that there was about something like that going on, B confirmed this.
(I read
I think many well-intentioned people will say something like this, and that this is probably because of two reasons
as implying that the list of reasons is considered to exhaustive, such that any reasons besides those two have negligible probability.)
The truth of that literal statement depends on exactly how much trust someone would need in somebody else before having sex with them - e.g. to my knowledge, studies tend to find that most single men but very few if any women would be willing to have sex with a total stranger. Though I've certainly also known women who have had a relatively low bar of getting into bed with someone, even if they wouldn't quite do it with a total stranger.
But more relevantly, even if that statement was correct, I don't think it'd be a particularly good analogy to Circling. It seems to involve the "obligatory openness" fallacy that I mentioned before. I'm not sure why some people with Circling experience seemed to endorse it, but I'm guessing it has to do with some Circling groups being more into intimacy than others. (At the time of that discussion, I had only Circled once or twice, so probably didn't feel like I had enough experience to dispute claims by more experienced people.)
My own experience with Circling is that it's more like meeting a stranger for coffee. If both (all) of you feel like you want to take it all the way to having sex, you certainly can. But if you want to keep it to relatively shallow and guarded conversation because you don't feel like you trust the other person enough for anything else, you can do that too. Or you can go back and forth in the level of intimacy, depending on how the conversation feels to you and what topics it touches on. In my experience of Circling, I definitely wouldn't say that it feeling anywhere near as intimate as sex would be the norm.
You can also build up that trust over time. I think Circling is best when done with people who you already have some pre-existing reason to trust, or in a long-term group where you can get to know the people involved. That way, even if you start at a relatively shallow level, you can go deeper over time if (and only if) that feels right.
I don't know the details. The official explanation is this:
When individuals with little training attempt to facilitate Circling, or teach/train others their arbitrarily altered versions and still call it Circling, then consumers and students – at best – receive a sub-standard experience and the reputation of Circling suffers greatly, along with its impact in the world.
Between the three schools there are hundreds of accounts of:
- People taking one or two 3-hour workshops, or merely experiencing Circling at a drop in event or festival, and then advertising that they are leading their own Circling workshops
- People coming to a few drop in events & turning around and offer “Circling” to large corporations for corporate culture training.
- People claiming they were emotionally abused by facilitators at an event that advertised itself as “Circling” but had no ties to any of the 3 Certified Circling Schools
In order to protect the public consumer and the legacy of Circling, we need to use the term “Circling” consistently and limit the use of the term to those who are actually using and teaching the authentic communication and relating tools taught by the Certified Circling Schools.
... but then I also heard it claimed that Circling Europe, previously one of the main Circling schools in very good standing, ended up not having a permission to use the trademark because the licensing fees for it would have been so exorbitant that CE found it better to use a different name than to pay them. So maybe it was more of a cash grab? (Or just a combination of several different motives.)
What's the long version of the professional's standard advice?
Historically there were plenty of rationalizations for slavery, including ones holding that slaves weren't really people and were on par with animals. Such an argument would be much easier for a mind running on a computer and with no physical body - "oh it just copies the appearance of suffering but it doesn't really suffer".
I think many people have learned to believe the reasoning step "If people believe bad things about my team I think are mistaken with the information I've given them, then I am responsible for not misinforming people, so I should take the information away, because it is irresponsible to cause people to have false beliefs". I think many well-intentioned people will say something like this, and that this is probably because of two reasons (borrowing from The Gervais Principle):
(Comment not specific to the particulars of this issue but noted as a general policy:) I think that as a general rule, if you are hypothesizing reasons for why somebody might say a thing, you should always also include the hypothesis that "people say a thing because they actually believe in it". This is especially so if you are hypothesizing bad reasons for why people might say it.
It's very annoying when someone hypothesizes various psychological reasons for your behavior and beliefs but never even considers as a possibility the idea that maybe you might have good reasons to believe in it. Compare e.g. "rationalists seem to believe that superintelligence is imminent; I think this is probably because that lets them avoid taking responsibility about their current problems if AI will make those irrelevant anyway, or possibly because they come from religious backgrounds and can't get over their subconscious longing for a god-like figure".
We can also learn something about how o1 was trained from the capabilities it exhibits. Any proposed training procedure must be compatible with the following capabilities:
- Error Correction: "[o1] learns to recognize and correct its mistakes."
- Factoring: "[o1] learns to break down tricky steps into simpler ones."
- Backtracking: "[o1] learns to try a different approach when the current one isn't working."
I would be cautious of drawing particularly strong conclusions from isolated sentences in an announcement post. The purpose of the post is marketing, not technical accuracy. It wouldn't be unusual for engineers at a company to object to technical inaccuracies in marketing material and have their complaints ignored.
There probably aren't going to be any blatant lies in the post, but something like "It'd sound cool if we said that the system learns to recognize and correct its mistakes, would there be a way of interpreting the results like that if you squinted the right way? You're saying that in principle yes, but yes in a way that would also apply to every LLM since GPT-2? Good enough, let's throw that in" seems very plausible.