Posts
Comments
Sutskever's response to Dwarkesh in their interview was a convincing refutation of this argument for me:
Dwarkesh Patel
So you could argue that next-token prediction can only help us match human performance and maybe not surpass it? What would it take to surpass human performance?
Ilya Sutskever
I challenge the claim that next-token prediction cannot surpass human performance. On the surface, it looks like it cannot. It looks like if you just learn to imitate, to predict what people do, it means that you can only copy people. But here is a counter argument for why it might not be quite so. If your base neural net is smart enough, you just ask it — What would a person with great insight, wisdom, and capability do? Maybe such a person doesn't exist, but there's a pretty good chance that the neural net will be able to extrapolate how such a person would behave. Do you see what I mean?
Dwarkesh Patel
Yes, although where would it get that sort of insight about what that person would do? If not from…
Ilya Sutskever
From the data of regular people. Because if you think about it, what does it mean to predict the next token well enough? It's actually a much deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It's not statistics. Like it is statistics but what is statistics? In order to understand those statistics to compress them, you need to understand what is it about the world that creates this set of statistics? And so then you say — Well, I have all those people. What is it about people that creates their behaviors? Well they have thoughts and their feelings, and they have ideas, and they do things in certain ways. All of those could be deduced from next-token prediction. And I'd argue that this should make it possible, not indefinitely but to a pretty decent degree to say — Well, can you guess what you'd do if you took a person with this characteristic and that characteristic? Like such a person doesn't exist but because you're so good at predicting the next token, you should still be able to guess what that person who would do. This hypothetical, imaginary person with far greater mental ability than the rest of us
"Dwarkesh chose excellent questions throughout, displaying an excellent sense of when to follow up and how, and when to pivot."
This is the basic essence of why Dwarkesh does such good interviews. He does the groundwork to be able to ask relevant and interesting questions, ie. actually reading their books/works, and seems to consistently have put actual thought into analysing the worldview of his subjects.
The unambitiousness of modern geoengineering in general is dismaying.
For my Australian perspective: in the early 1900s there were people discussing how make use of the massive tracts of desert wasteland than make up most of the outback (ie: https://en.wikipedia.org/wiki/Bradfield_Scheme). None of this stuff could be considered today - one tree getting chopped down is liable to make the news: https://www.bbc.com/news/world-australia-54700074
Hard to escape the conclusion that we should all go lie in a ditch so as to guarantee that no impact to anything occurs ever.
This seems about right. Sam is a bit of a cowboy and probably doesn't bother involving the board more than he absolutely has to.
Stefánsson's "The Fat of the Land" is not really worth reading for any scientific insight today, but it's entertaining early 1900s anthropology.
I don't have much of an opinion on any specific diet approach, but I can tell you my own experience with weight loss: I've always been between 15-22% bodyfat, but I have always tended to slowly gain weight if not actively dieting. My routine for about 10 years now has been to diet to 15%, and then at some point notice that I've been getting fatter and diet back down to 15% by counting calories and CICO logic. I find dieting annoying but consistent, predictable, and doable.
This routine isn't ideal, so I too am a 'victim' of the weight gain phenomenon. I can't say that I've established a truly sustainable diet for myself - but it works well enough.
I have no satisfying answers for "why are we getting fatter" or "what makes caloric deficits so hard to maintain". I appreciate the diet blogging community that tries to tackle these questions with citizen science.
I assume you're familiar with Vilhjálmur Stefánsson's work if you are interested in low protein carnivore diets, but I was really was surprised to see how similar the 'ex150' sounds compared to the classic ~80:20 fat:protein experiments. These aren't really new ideas - although I'm sure there's a lot more information available on the details.
Anyway, dieting seems like something where while people on average fail, you do see some individual successes, so it's worth poking around the edges and giving things a go. It's always nice to see results from the coalface.
Ultimately the new GLP-1 agonist weightloss drugs seem to be awesome by both data and anecdata. So the food composition experimentation might fade away a bit over the next few years for the express purpose of weight loss.
Good post.
For Westerners looking to get a palatable foothold on the priorities and viewpoints of the CCP (and Xi), I endorse "The Avoidable War" written last year by Kevin Rudd (former Prime Minister of Australia, speaks mandarin, has worked in China, loads of diplomatic experience - imo about as good of a person that exists to interpret Chinese grand strategy and explain it from a Western viewpoint). The book is (imo, for a politician), impressively objective in its analysis.
Some good stuff in there explaining the nature of Chinese cynicism about foreign motivations that echoes some of what is written in this post, but with a bit more historical background and strategic context.
Yeah - it's odd, but TC is a self-professed contrarian after all.
I think the question here is: why doesn't he actually address the fundamentals of the AGI doom case? The "it's unlikely / unknown" position is really quite a weak argument which I doubt he would make if he actually understood EY's position.
Seeing the state of the discourse on AGI risk just makes it more and more clear that the AGI risk awareness movement has failed to express its arguments in terms that non-rationalists can understand.
People like TC should the first type of public intellectual to grok it, because EY's doom case is is highly analogous to market dynamics. And yet.
For example, a major point of disagreement between me and Eliezer is that Eliezer often dismisses plans as “too complicated to work in practice,” but that dismissal seems divorced from experience with getting things to work in practice (e.g. some of the ideas that Eliezer dismisses are not much more complex than RLHF with AI assistants helping human raters). In fact I think that you can implement complex things by taking small steps—almost all of these implementation difficulties do improve with empirical feedback.
EY's counter to this?
@Gerald Monroe On the question of Japan's unique lack of variation, I think it's unlikely to be decisive here. The 'monoculture' argument may have some merit, but even a genetically 'homogenous' population has plenty of variation - especially one 125m strong like the Japanese.
Fertility related traits are just so fundamental to genetic fitness that selection is guaranteed to wring out the higher fertility alleles where the environment allows.
Imo Japan is one of the more illuminating examples on this topic:
- Japan had a TFR of 5 in the 1930s. It's been only 3 generations since Japan's TFR began to fall, and France took 5 to stablise around the current level (1830s-1980s). I agree that the trend since 2005 is too short term to be sure, but it's interesting to note! The above modelling suggests that a faster fertility transition should result in a faster bounceback - the lower the TFR the more adaptive high-TFR genes + cultures will be relatively.
- The fertility transition hit East Asia harder and faster than it did Europe. There's merit to the theory that it's because Europe had a slower transition to today's mainstream fertility-surpressing universal culture (technological advancement, enlightenment values. women's lib etc), since much of these cultural changes were developed in the West (consider the analogy to megafauna in Africa).
It's extremely difficult to quantify this sort of thing but it does support a model where both genes and culture are load-bearing inputs to TFR. In countries where culture propped up fertility one way or another there could be said to be a cultural fertility overhang, and when these forces were removed TFR naturally cratered in the short term. Where countries had less cultural overhang, or a slower transition from high-TFR culture to low-TFR culture, the transition was less dramatic because there was time for cultural counter-developments or genetic selection to act.
The example of Sth Korea (TFR >5 until the 60s) supports some of these theories. The timing is especially interesting - the 60s were a major leap forward in progressive cultural hegemony, and Sth Korea (an extremely poor society prior) copped that right in the face after the Korean War. The idea is that the speed of TFR-decline is related to the severity of cultural change - makes sense to me.
An optimistic Sth Korean pro-natalist could interpret this current ultra-low TFR period as evidence of an extremely effective 'weeding' process which is sure to be followed by a period of high fertility preferences as the most enthusiastic 'breeders' will be all that remain!
Note: When I refer to culture above I take the expansive definition which includes technology, wealth, social changes - ie. anything that isn't genetic.
There's good reason to believe the fertility transition as a general phenomenon is subject to negative feedback, thus almost certain to be self correcting. Selection self-evidently favours high fertility culture and genes.
See this study for an attempt to model the effects:
Correlations in fertility across generations: can low fertility persist?
Martin Kolk, Daniel Cownden and Magnus Enquist
Published:22 March 2014
https://royalsocietypublishing.org/doi/10.1098/rspb.2013.2561#d3e788,
"... Our models suggest a mechanism in which the recent fertility decline may be reversed in the long run. Intergenerational fertility correlations create cultural and genetic selection processes that favour lifestyles with higher fertility. Only through continuous cultural change, introducing novel lifestyles associated with reduced child-bearing, can low fertility persist."
An example in favour of this model is France. France is generally regarded as a low-fertility pioneer, but today has the highest TFR in Western Europe.
It's still worth paying attention to policy, but worth noting the strong undercurrents at play which are likely somewhat independent of govt tweaking. I think genetics are likely a more relevant factor than has been mentioned in this thread so far.
Agree - Gell-Mann amnesia sums up my experience with trying to get ChatGPT to be useful for a professional context so far. It is weak on accuracy and details.
My questions:
- Is this something that can be overcome through skilful prompting?
- Is there something about LLMs that makes these issues difficult to overcome?
- ChatGPT is very conservative with providing factual information even when it's possible to tease out the relevant information (i.e. topics like law or commercial activities), is this purposeful throttling?
The speed issue is the #1 factor stopping me from trying audiobooks. A book might take me 4-8 hours to read but the internet tells me audio is 2-3x slower. I have a lot of other prejudices against audiobooks (flipping / skimming less easy, less focus on the task etc) but that's the main one.
Seconded - I'd like to see more of this angle of analysis too. I assume the reason why the 'soft take-off' is underdiscussed is that tech people a) pay more attention to the news on AI, and b) see the future of this stuff viscerally and the endgame is what looms. I think that's not wrong, because the endgame is indeed transformative. But how we get there and how quickly it happens is a completely open question.
I work in the AEC industry (Architecture, Engineering, Construction) - 90%+ of people have zero idea about recent advances in AI. But on the other hand, should they be personally worried about their employment prospects in the next decade? I feel like lots of LW-type people would say "Yes!" - I can only speak personally, but it's really hard for me to see it happening. If only for the fact that doing anything in meatspace takes a long time. There are plenty of great 'digital' solutions to problems in this industry that have been around for 10+ years and have still made no headway. I know AGI is different, but it's worth mentioning how slow things can be, and how much of a grinding bureaucracy many industries are.
The other way I think about it is that ultimately human concerns (politics, agency etc) underpin all economic activity, and there will be massive political and bureaucratic opposition to extreme levels of economic 'disruption' (in the negative sense of mass unemployment etc). I foresee active responses from the populace and governments shaping the path this takes to a significant degree (eg. increases in labour force protectionism). Not so much that capitalism just takes to AI like a duck to water, governments let it happen, and 99% of people end up in a terrafoam box in a few short years.
I can anecdotally report that when I started consistently getting up immediately upon my alarm going off the subjective feeling of the first 5-10 minutes was far superior and I didn't feel much tiredness even with a relatively short night of sleep. I started doing this through setting my alarm to the maximum latest time possible and still allow me to get to work on time, and then noticed how much better I felt while doing this (previously I was a chronic snooze-button masher and felt pretty groggy waking up).
I have noticed in the WFH/office phases of the past 2 years the effect has persisted - a worse subjective waking up experience when WFH. I think it's because without the pressure of needing to be at work, I don't develop an automatic routine of simply getting up unthinkingly when my alarm goes off.
I think part of this is related to how good your internal biological clock is. I'm the type of person that will wake up pretty consistently a few minutes before an alarm goes off anyway (sometimes even when the alarm is not my usual time, such as for a flight). Maybe the 'get up immediately' approach wouldn't work as well if this isn't you.
So I looked into this, and the 'refine win condition' seems to be an actual technique that some of the best 3pt shooters do employ! I looked up some interviews with some of the best basketball shooters (Steph Curry and Ray Allen) and they mention playing little games with themselves while practice. They will do things such a define win condition as a swish instead of just making the shot, or employing some particular type of rhythm or footwork, or slightly altering the angle of their shot.
But on the other hand, they also mention plenty of repetitive drilling, and watching the available footage of them practicing, it is hard to see a 'stop after win' approach in action. There are videos of Steph Curry practicing with a coach who passes him the ball - it looks pretty repetitive to me (although maybe he's playing some mental games with himself that are hard to pick up?).
Thanks for the post - skill acquisition does seem like an area lacking attention from the rationalist community in general.
I wonder if these learnings would apply to sports. "Stop after win" doesn't seem like a productive idea if you are trying to get better at, say, shooting a 3pt shot in basketball - the traditional approach is "reps reps reps". Thoughts?
Ctrl+F "CFR" & "IFR" = 0 results. Apologies if you used different terminology here but I feel it is worth pointing out that CFR has cratered in many nations, even those with high case counts and high testing rates. The old assumption of ~.5-1% IFR seems to be thoroughly passed its use-by date. For highly vaxed populations current IFR is probably sub 0.1% overall, with the same qualifiers about age/morbidity profile etc.
Particularly of interest is Australia, which could be said to have had a lot of 'dry tinder' given the lack of pre-omicron deaths overall, and experienced a fairly high case peak recently - but Australia's rolling CFR is .06% as of today.
New variants notwithstanding, surely it is fair to say 'it's over' (actually for real this time).
Apparently the issue was subject to a quite a bit of activist opposition anyway - but the timing is suspicious, especially since the Serbian PM was personally involving herself in the issue.
I don't like listening to Rogan, but I do like hearing him translated into rationalist-speak.
Four, he has uniquely powerful ideas about how to do project management well...
I am interested in this. Any suggestions for posts that focus on project management specifically?
Other examples of drugs that also seem to work for weight loss are GLP-1 agonists, which are relatively new and potentially promising.
I keep seeing references to 'long covid' by smart people, usually in the context of "we need to confirm what the effects are" but have never seen any real evidence from obscure or mainstream sources that it's something worth thinking about much. Surely if it were a real and important thing it would be identified and proven as such by now?
At this moment it strikes me as something that's probably either bullshit or a minor consideration until further evidence emerges to the contrary, but smart people continue to mention it. What am I missing?
I think it can be argued that in the West that practical achievement and directed pursuit probably is like this at a core level - humans are capable extraordinary effort if the conditions (and pressures) are right. Parenthood is one of the things that is still like everything used to be - pushed to the edge of human capability because there isn't another option. I think for many other modern pursuits a common failure mode is indulging in the eases and luxuries of modern existence - no such option with parenthood or language immersion.
Related thought: slogans like 'Just Do It' are effective because people grok that the human mind will trick you into taking easy way out if the option is there. 'No excuses' is an attractive motivating strategy and does usually work in the short term, but long term environment design is more imporant.
AFAIK the main effect from the PM's policy change seems to be around relaxing indemnity rules for GPs so that they could hand out AZ if they wanted to without getting sued by people who develop the blood clot disorder. Previously this was an issue due to the current ATAGI advice recommending against it.
I thought the PM's statement on this wasn't too crazy - the blood clot risks are objectively still very low and the ATAGI report contemplates the then near-zero covid in Aus as you note. I assume somebody in govt realised that at current and projected vaccine rates it'll be a long time before the country opens up / stops having to lockdown extremely hard every time covid leaks into the community - and then the recent NSW outbreak brought the issue to a head.
Vaccination supply has not been that reliable or consistent so far, and AZ is the only vax currently made locally so I think vaccination regimes that don't involve AZ do risk a longer 'fortress Australia' period.
Thanks for the article - very informative and exactly the kind of content I enjoy!
In 2017 Australia held a public survey on whether same-sex marriage should be allowed, the results of which were pledged informally to be enacted by the government. Public votes on specific issues are relatively rare in Australia, so the debate around the procedural merits of voting on this particular issue were quite active.
I recall the main arguments against conducting a survey were mostly procedural criticisms, that it is wastefully expensive to hold a postal survey when public polling had revealed consistent majority support for same sex marriage for a number of years already. Wikipedia tells me the survey cost $80m AUD, so I wonder how much this Swiss system costs over the long haul?
The arguments in favour were mostly that it would break political and procedural gridlock over the issue and settle things once and for all with the legitimising stamp of direct democracy.
In the aftermath I found myself thinking 'we should do this more often' - so it's nice to see that somewhere in fact does do it more often!
PS: It's interesting to see the high rejection rate for referendums in Switzerland. The same-sex marriage survey was in fact suggested by the centre-right party who were historically opposed to same-sex marriage, and it's generally accepted that they viewed the direct (voluntary) vote as the best chance to get a 'no' or ambiguous result on this matter and introduce a long-term mandate against legalising same-sex marriage.