Posts
Comments
Wikipedia says it's a SaaS company "specializing in AI-powered document processing and automation, data capture, process mining and OCR": https://en.wikipedia.org/wiki/ABBYY
To be clear, GiveWell won’t be shocked by anything I’ve said so far. They’ve commissioned work and published reports on this. But as you might expect, these quality of life adjustments wouldnt feature in GiveWell’s calculations anyway, since the pitch to donors is about the price paid for a life, or a DALY.
Can you clarify what you mean by these quality of life adjustments not featuring in GiveWell's calculations?
To be more concrete, let's take their CEA of HKI's vitamin A supplementation (VAS) program in Burkina Faso. They estimate that a $1M grant would avert 553 under-5 deaths (~80% of total program benefit) and incrementally increase future income for the ~560,000 additional children receiving VAS (~20%) (these figures vary considerably by location by the way, from 60 deaths averted in Anambra, Nigeria to 1,475 deaths averted in Niger) then they convert this to 81,811 income-doubling equivalents (their altruistic common denominator — they don't use DALYs in any of their CEAs, so I'm always befuddled when people claim they do), make a lot of leverage- and funging-related adjustments which reduces this to 75,272 income doublings, then compare it with the 3,355 income doublings they estimate would be generated by donating that $1M to GiveDirectly to get their 22.4x cash multiplier for HKI VAS in Burkina Faso.
So: are you saying that GiveWell should add a "QoL discount" when converting lives saved and income increase, like what Happier Lives Institute suggests for non-Epicurean accounts of badness of death?
I agree; Eichmann in Jerusalem and immoral mazes come to mind.
the obvious thing to happen is that nvidia realizes it can just build AI itself. if Taiwan is Dune, GPUs are the spice, then nvidia is house Atreides
You mention in another comment that your kid reads the encyclopaedia for fun, in which case I don't think The Martian would be too complex, no?
I'm also reminded of how I started perusing the encyclopaedia for fun at age 7. At first I understood basically nothing (English isn't my native language), but I really liked certain pictures and diagrams and keep going back to them wanting to learn more, realising that I'd comprehend say 20% more each time, which taught me to chase exponential growth in comprehension. Might be worth teaching that habit.
That's fair.
Society seems to think pretty highly of arithmetic. It’s one of the first things we learn as children. So I think it’s weird that only a tiny percentage of people seem to know how to actually use arithmetic. Or maybe even understand what arithmetic is for.
I was a bit thrown off by the seeming mismatch between the title ("underrated") and this introduction ("rated highly, but not used or understood as well as dynomight prefers").
The explanation seems straightforward: arithmetic at the fluency you display in the post is not easy, even with training. If you only spend time with STEM-y folks you might not notice, because they're a very numerate bunch. I'd guess I'm about average w.r.t. STEM-y folks and worse than you are, but I do quite a bit of spreadsheet-modeling for work, and I have plenty of bright hardworking colleagues who can't quite do the same at my level even though they want to, which suggests not underratedness but difficulty.
(To be clear I enjoy the post, and am a fan of your blog. :) )
This is great, thank you Sable. Some thoughts:
- Percentage seems logarithmic, like the Richter scale
- I wish there was a tabulated / spreadsheet version with the columns 'percentage', 'description', 'emotionally', 'cognitively', 'personally' or something (might create this myself based on your main text)
- Your "when depressed, I am not (in reality) the person I see myself as. I have a self-image, a conception of what I’m supposed to be like, and my depressed self isn’t it" resonates with how I felt when I was depressed. That said in my case my self-image was quite grounded in reality (the conception was of "where I was supposed to be in life" with externally observable correlates others could verify for themselves), improving my life condition over the years was key to me reducing my depression, and this matches some (but to my chagrin not all) of the depressed people I have tried to help
Misses my point, but never mind.
Counterpoint: the likes of Kelsey Piper and Dylan Matthews are great. Their business does seem to be to help us understand and better the world.
Setting low prices might mean the few gallons of gas, bottles of water, or flights that are available are allocated to people who get to them first, or who can wait in line the longest, rather than based on who is willing to pay, but it’s not clear that these allocations are more egalitarian.
If that last sentence can be rephrased as "the counterfactual benefit is unclear", then given that the counterfactual cost is clear (beneficiaries need to pay more, potentially a lot more) doesn't this show that price gouging is probably net negative i.e. aren't you arguing against, not for, it here?
(Genuine question, not a gotcha. I'm admittedly biased because I dislike the idea of profiting off others' misery, suspect that purchasing power is moderately anticorrelated with need via wealth/income i.e. fewer DALYs are averted per person helped this way, and find the bounty reframing uncompelling, but I'm willing to change my mind. Another personal bias: I work in the market-shaping for global health space, which is about redressing market failures due to e.g. mismatch between need and ability to pay for it, and I suspect a similar dynamic is at play after hurricanes have devastated an area.)
(Unrelated to the post: I'm glad you're back, ADS. I wasn't the only one who wondered)
Seems way underestimated. While I don't think he's at "the largest supergeniuses" level either, even +3 SD implies just top 1 in ~700 i.e. millions of Eliezer-level people worldwide. I've been part of more quantitatively-selected groups talent-wise (e.g. for national scholarships awarded on academic merit) and I've never met anyone like him.
Similar points apply to adding too many unnecessary links. Specifically links where it isn't clear where they lead and what point is made in the link target, as in the previous sentence.
This used to be a recurring failure mode of my own writing, which I've since partially mitigated. Reflecting on why, I think I wanted to do some combination of
- justifying contentious or surprising claims
- preventing being pattern-matched to straw versions of ideas / arguments commonly referenced and adjacent in concept-space
- finding excuses to share cool reads
- making sense of links I'd read by relating them, using publication as a focusing mechanism
I didn't notice the cost of overdoing it until I saw writers who did it worse, and became horrified at the thought that I was slowly becoming them.
(Gwern links a lot but it doesn't feel "worse", on the contrary I enjoy his writing, so "worseness" is as much about adding more value to the reader than the cost of disrupting their flow as it is about volume. His approach is also far more thought-out of course.)
Thanks for writing this. I only wish it was longer.
Ha, that's awesome. Thanks for including the screenshot in yours :) Scott's "invisible fence" argument was the main one I thought of actually.
You might be interested in Scott Aaronson's thoughts on this in section 4: Why Is Proving P != NP Difficult?, which is only 2 pages.
Yeah pretty much. In more detail:
Bezos explained why he chose to only sell books on his website — at least, at first — in a “lost” video interview recorded at a Special Libraries Association conference in June 1997, which resurfaced in 2019 when it was posted online by entrepreneur Brian Roemmele.
Out of all the different products you might be able to sell online, books offered an “incredibly unusual benefit” that set them apart, Bezos said.
“There are more items in the book category than there are items in any other category, by far,” said Bezos. “Music is No. 2 — there are about 200,000 active music CDs at any given time. But in the book space, there are over 3 million different books worldwide active in print at any given time across all languages, [and] more than 1.5 million in English alone.”
When Bezos launched Amazon in 1994, the internet and e-commerce industry were still in their earliest stages. He knew it would take some time before online shopping became ubiquitous, he said, so he wanted to start with a concept that couldn’t be replicated by a seller with only physical locations.
“When you have that many items, you can literally build a store online that couldn’t exist any other way,” he explained. “That’s important right now, because the web is still an infant technology. Basically, right now, if you can do things using a more traditional method, you probably should do them using the traditional method.”
Still, Bezos hinted at the company’s potential for expansion, noting that “we’re moving forward in so many different areas.”
“This is Day 1,” he added. “This is the very beginning. This is the Kittyhawk stage of electronic commerce.”
By this assessment, who in real life do you think has proven above-average intelligence?
you presumably also think that teleportation would only create copies while destroying the originals. You might then be hesitant to use teleportation.
As an aside, Holden's view of identity makes him unconcerned about this question, and I've gradually gotten round to it as well.
Is 3.1 points small? Well, a 100 IQ is higher than that of 50% of the population, while a 103.1 IQ is higher than 58%. Adding 3.1 IQ points to a kid ranked 13th in a 25-person class would push them up to around 11th. And, personally, if you were going to drop my IQ by 3.1 points, I would not be super stoked about it.
And remember, 3.1 points is still just the impact of a modest increase in breastfeeding intensity. If you ran a trial that compared no breastfeeding to exclusive breastfeeding for 12 months, the impact would surely have been much larger.
For context, in high-income countries lead poisoning is estimated to have lowered IQ by a comparable amount (the paper doesn't explicitly state the IQ drop, but does say that the mean blood lead level in HICs is 1.3 μg/dL and provides the chart below), and lead poisoning is taken pretty seriously.
00166-3/asset/ebfd713b-2f9e-4017-8506-aca6d1b78662/main.assets/gr1.jpg)
ARIA feels like it has the same vibe (I might be wrong); I found out about it from davidad's bio (he's a PD).
I thought this review was fine: https://recordcrash.substack.com/p/mad-investor-chaos-woman-asmodeus
How would you falsify this model?
I have a similar experience. Do you know of any LLMs that aren't as agreeable in a useful way?
Ben Pace
Can you say slightly more detail about how you think the preference synthesizer thing is suposed to work?
zhukeepa
Well, yeah. An idealized version would be like a magic box that's able to take in a bunch of people with conflicting preferences about how they ought to coordinate (for example, how they should govern their society), figure out a synthesis of their preferences, and communicate this synthesis to each person in a way that's agreeable to them.
...
Ben Pace
Okay. So, you want a preference synthesizer, or like a policy-outputter that everyone's down for?
zhukeepa
Yes, with a few caveats, one being that I think preference synthesis is going to be a process that unfolds over time, just like truth-seeking dialogue that bridges different worldviews.
...
zhukeepa
Yeah. I think the thing I'm wanting to say right now is a potentially very relevant detail in my conception of the preference synthesis process, which is that to the extent that individual people in there have deep blind spots that lead them to pursue things that are at odds with the common good, this process would reveal those blind spots while also offering the chance to forgive them if you're willing to accept it and change.
I may be totally off, but whenever I read you (zhukeepa) elaborating on the preference synthesizer idea I kept thinking of democratic fine-tuning (paper: What are human values, and how do we align AI to them?), which felt like it had the same vibe. It's late night here so I'll butcher their idea if I try to explain them, so instead I'll just dump a long quote and a bunch of pics and hope you find it at least tangentially relevant:
We report on the first run of “Democratic Fine-Tuning” (DFT), funded by OpenAI. DFT is a democratic process that surfaces the “wisest” moral intuitions of a large population, compiled into a structure we call the “moral graph”, which can be used for LLM alignment.
- We show bridging effects of our new democratic process. 500 participants were sampled to represent the US population. We focused on divisive topics, like how and if an LLM chatbot should respond in situations like when a user requests abortion advice. We found that Republicans and Democrats come to agreement on values it should use to respond, despite having different views about abortion itself.
- We present the first moral graph, generated by this sample of Americans, capturing agreement on LLM values despite diverse backgrounds.
- We present good news about their experience: 71% of participants said the process clarified their thinking, and 75% gained substantial respect for those across the political divide.
- Finally, we’ll say why moral graphs are better targets for alignment than constitutions or simple rules like HHH. We’ll suggest advantages of moral graphs in safety, scalability, oversight, interpretability, moral depth, and robustness to conflict and manipulation.
In addition to this report, we're releasing a visual explorer for the moral graph, and open data about our participants, their experience, and their contributions.
...
Our goal with DFT is to make one fine-tuned model that works for Republicans, for Democrats, and in general across ideological groups and across cultures; one model that people all around the world can all consider “wise”, because it's tuned by values we have broad consensus on. We hope this can help avoid a proliferation of models with different tunings and without morality, fighting to race to the bottom in marketing, politics, etc. For more on these motivations, read our introduction post.
To achieve this goal, we use two novel techniques: First, we align towards values rather than preferences, by using a chatbot to elicit what values the model should use when it responds, gathering these values from a large, diverse population. Second, we then combine these values into a “moral graph” to find which values are most broadly considered wise.
Example moral graph, which "charts out how much agreement there is that any one value is wiser than another":
Also, "people endorse the generated cards as representing their values—in fact, as representing what they care about even more than their prior responses. We paid for a representative sample of the US (age, sex, political affiliation) to go through the process, using Prolific. In this sample, we see a lot of convergence. As we report further down, people overwhelmingly felt well-represented with the cards, and say the process helped them clarify their thinking", which is why I paid attention to DFT at all:
Not a substantive response, just wanted to say that I really really like your comment for having so many detailed real-world examples.
Just to check, you're referring to these?
They have in fact been published (it's in your link), at least the ones authors agreed to make publicly available: these are all the case studies, and Moritz von Knebel's write-ups are
Nisan's comment upthread links to one such double-blind test, text reproduced here to save people the effort of clicking:
In order to assess the applicability and quality of the entropy-based tuning scheme, a detailed test was carried out in spring 2015 at the University of Music Würzburg in cooperation with Prof. Andreas C. Lehmann, Master piano builder Burkard Olbrich and Michael Kohl. To this end two structurally identical Steinway-C grands were compared, one of them professionally tuned by ear and the other one according to the EPT (see figure). 28 pianists played and compared the two instruments in a double-blind test, evaluating them in a questionnaire.
The participants can be divided roughly into two groups. The first group of 20 participants represents the semi-professional sector, including piano students at the University of Music and serious amateurs. The second group of eight people consists of professional pianists with many years of experience, including e.g. professors for piano playing at the University of Music.
Because of the small number of participants statistical statements are limited. Nevertheless, the test has led to a clear overall picture that can be summarized as follows:
Pianists belonging to the group of semi-professional musicians do not show a clear preference for one of the two grands.
Pianists with a long professional experience show a statistically significant preference for the aurally tuned grand. Moreover, this grand is perceived as being more harmonic and in a better tune, exhibiting less beats than the electronically tuned instrument.
In conclusion, it seems that the current version of the EPT generates tunings which can be considered as acceptable in a semi-professional context. On the other hand, the EPT cannot compete with high-quality aural tunings on a professional scale. However, given the fact that the EPT tunes randomly according to a very simple one-line formula this is not too surprising. What is surprising, though, is that the entropy-based method seems to produce acceptable results on a semi-professional scale.
I wish this linked to a more substantive writeup of the test.
I would bet a lot that this system will fail on almost any combinatorics problem at the moment...
I don't know if you guessed this prior to reading the post, but if so good guess:
AlphaProof solved two algebra problems and one number theory problem by determining the answer and proving it was correct. This included the hardest problem in the competition, solved by only five contestants at this year’s IMO. AlphaGeometry 2 proved the geometry problem, while the two combinatorics problems remained unsolved.
I've been wondering about this ever since I saw that sentence, so now I'm curious to see your post explaining your reasoning.
ETA: I just saw sunwillrise's great comment on this and am now wondering how your reasoning compares
There was a time when I was working on n-dimensional data structures that I could cleanly think in 4 or 5 dimensional "images". They weren't quite visual, since vision is so 2-d, but I could independently manipulate features of the various dimensions separately.
This remark is really interesting. It seems related to the brain rewiring that happens after, say, a subject has been blindfolded for a week, in that their hearing and tactile discrimination improves a lot to compensate.
If we dug deep into the sub-components of Carlsen’s IQ (or perhaps the sub-components of the sub-components), we’d probably find some sub-component where he measured off the charts.
It's still probably a combination of sub-components + memory (often underrated in IQ discussions) + physical fitness (relevant in long tournaments), not any single one. Quotes on his childhood:
At two years, he could solve 500-piece jigsaw puzzles; at four, he enjoyed assembling Lego sets with instructions intended for children aged 10–14.
Simen Agdestein emphasises Carlsen's exceptional memory, stating that he was able to recall the locations, populations, flags and capitals of all the countries in the world by age five. Later, he memorised the locations, populations, coats-of-arms and administrative centres of "virtually all" 356 Norwegian municipalities.
and on his distinctive style:
Kasparov expressed similar sentiments: "[Carlsen] has the ability to correctly evaluate any position, which only Karpov could boast of before him.
In a 2012 interview, Vladimir Kramnik stated that Carlsen's "excellent physical shape" was a contributing factor to his success against other top players as it prevents "psychological lapses", which enables him to maintain a high standard of play over long games and at the end of tournaments, when the energy levels of others have dropped.
Levon Aronian said in 2015: "Magnus' main secret is his composure and the absence of any soul-searching after mistakes during a game." (i.e. chess tilt)
Tyler Cowen gave a point of view on Carlsen's playing style: "Carlsen is demonstrating one of his most feared qualities, namely his 'nettlesomeness,'... Using computer analysis, you can measure which players do the most to cause their opponents to make mistakes. Carlsen has the highest nettlesomeness score by this metric, because his creative moves pressure the other player and open up a lot of room for mistakes. In contrast, a player such as Kramnik plays a high percentage of very accurate moves, and of course he is very strong, but those moves are in some way calmer and they are less likely to induce mistakes in response."
Ben Todd at 80K published an old (2014) exploratory career profile on being a program manager in international orgs "like the World Bank, World Health Organisation, International Monetary Fund and United Nations", and notes that
We recommend this career if it is a better fit for you than our other recommended careers.
These positions may offer the opportunity to influence substantial budgets, since these organisations govern huge pools of aid money and international regulation. Typically in these organisations the average budget spent on programs per employee is on the order of US$1-10 million.1 Since it’s difficult to give workers the right incentives and the work is difficult, we think it’s likely that additional intelligent, rational and altruistic people can have a substantial impact through improving the efficiency of how these funds are spent. However, we’re highly uncertain about the expected size of the influence.
These organisations are highly influential over important global challenges, so you’ll be working with highly influential people, which increases our assessment of advocacy potential and career capital. The high prestige of these positions also contributes to our higher rating of their career capital.
On the other hand, reading Backstabbing for Beginners by Michael Soussan a few years back, detailing his experience as program coordinator for the Oil-for-Food programme, greatly soured my then cautiously net-positive view of the UN.
I wasn't aware of these options, thank you.
There's also Byrne Hobart's article Understanding Jane Street (5.5k words), although pitched at an even lower level maybe. He recommends The Laws of Trading as further reading, alongside:
This interview with Yaron Minsky is a great look at their decision to use Ocaml. If you're interested in more about the mechanics of exchanges and trading, the Hide not Slide Substack is good. As a starting point, here's their writeup on Jane Street. Max Dama on Automated Trading is old, but a very helpful overview of the industry for technical people. If you want to learn Ocaml, Jane Street's Yaron Minsky has coauthored a good book on it.
Your topline answers to the questions you assume xlr8harder cares about more seem similar to Holden Karnofsky's, and I haven't seen his essay on this mentioned so in this thread so I thought it'd be useful to link it here: What counts as death? An unconventional but simple take on personal identity, that dissolves most paradoxes
My philosophy on "what counts as death" is simple, though unconventional, and it seems to resolve most otherwise mind-bending paradoxical thought experiments about personal identity. It is the same basic idea as the one advanced by Derek Parfit in Reasons and Persons;1 Parfit also claims it is similar to Buddha's view2 (so it's got that going for it).
I haven't been able to find a simple, compact statement of this philosophy, and I think I can lay it out in about a page. So here it is, presented simply and without much in the way of caveats (this is "how things feel to me" rather than "something I'm confident in regardless of others' opinions"):
Constant replacement. In an important sense, I stop existing and am replaced by a new person each moment (second or minute or whatever).
The sense in which it feels like I "continue to exist, as one unified thread through time" is just an illusion, created by the fact that I have memories of my past. The only thing that is truly "me" is this moment; next moment, it will be someone else.
Kinship with past and future selves. My future self is a different person from me, but he has an awful lot in common with me: personality, relationships, ongoing projects, and more. Things like my relationships and projects are most of what give my current moment meaning, so it's very important to me whether my future selves are around to continue them.
So although my future self is a different person, I care about him a lot, for the same sorts of reasons I care about friends and loved ones (and their future selves).3
If I were to "die" in the common-usage (e.g., medical) sense, that would be bad for all those future selves that I care about a lot.4
(I do of course refer to past and future Holdens in the first person. When I refer to someone as "me," that means that they are a past or future self, which generally means that they have an awful lot in common with me. But in a deeper philosophical sense, my past and future selves are other people.)
And that's all. I'm constantly being replaced by other Holdens, and I care about the other Holdens, and that's all that's going on.
- I don't care how quickly the cells in my body die and get replaced (if it were once per second, that wouldn't bother me). My self is already getting replaced all the time, and replacing my cells wouldn't add anything to that.
- I don't care about "continuity of consciousness" (if I were constantly losing consciousness while all my cells got replaced, that wouldn't bother me).
- If you vaporized me and created a copy of me somewhere else, that would just be totally fine. I would think of it as teleporting. It'd be chill.
- If you made a bunch of copies of me, I would be all of them in one sense (I care about them a lot, in the same way that I normally care about future selves) and none of them in another sense (just as I am not my future selves).
- If you did something really weird like splitting my brain in half and combining each half with someone else's brain, that would create two people that I care about more than a stranger and less than "Holden an hour from now."
I don't really find any thought experiments on this topic trippy or mind bending. They're all just cases where I get replaced with some other people who have some things in common with me, and that's already happening all the time.
Footnotes
For key quotes from Reasons and Persons, see pages 223-224; 251; 279-282; 284-285; 292; 340-341. For explanations of "psychological continuity" and "psychological connectedness" (which Parfit frequently uses in discussing what matters for what counts as death), see page 206.
"Psychological connectedness" is a fairly general idea that seems consistent with what I say here; "psychological continuity" is a more specific idea that is less important on my view (though also see pages 288-289, where Parfit appears to equivocate on how much, and how, psychological continuity matters). ↩
- "As Appendix J shows, Buddha would have agreed. The Reductionist View [the view Parfit defends] is not merely part of one cultural tradition. It may be, as I have claimed, the true view about all people at all times." Reasons and Persons page 273. Emphasis in original. ↩
- There's the additional matter that he's held responsible for my actions, which makes sense if only because my actions are predictive of his actions. ↩
- I don't personally care all that much about these future selves' getting to "exist," as an end in itself. I care more about the fact that their disappearance would mean the end of the stories, projects, relationships, etc. that I'm in. But you could easily take my view of personal identity while caring a lot intrinsically about whether your future selves get to exist. ↩
This is great, thank you for making it.
So instead, we tell the lawyers to go nuts. Be as biased as possible, and, as long as they're equally skilled and there aren't background factors that favor one position over the other, this ensures that each presented position is equally far from the truth. The jury now has a fair overview of both sides of the case, without a malicious lawyer being able to advantage one over the other.
This reminds me of Peter Watts' classic post about (among others) how science works:
Science doesn’t work despite scientists being asses. Science works, to at least some extent, because scientists are asses. Bickering and backstabbing are essential elements of the process. Haven’t any of these guys ever heard of “peer review”?
There’s this myth in wide circulation: rational, emotionless Vulcans in white coats, plumbing the secrets of the universe, their Scientific Methods unsullied by bias or emotionalism. Most people know it’s a myth, of course; they subscribe to a more nuanced view in which scientists are as petty and vain and human as anyone (and as egotistical as any therapist or financier), people who use scientific methodology to tamp down their human imperfections and manage some approximation of objectivity.
But that’s a myth too. The fact is, we are all humans; and humans come with dogma as standard equipment. We can no more shake off our biases than Liz Cheney could pay a compliment to Barack Obama. The best we can do— the best science can do— is make sure that at least, we get to choose among competing biases.
That’s how science works. It’s not a hippie love-in; it’s rugby. Every time you put out a paper, the guy you pissed off at last year’s Houston conference is gonna be laying in wait. Every time you think you’ve made a breakthrough, that asshole supervisor who told you you needed more data will be standing ready to shoot it down. You want to know how the Human Genome Project finished so far ahead of schedule? Because it was the Human Genome projects, two competing teams locked in bitter rivalry, one led by J. Craig Venter, one by Francis Collins — and from what I hear, those guys did not like each other at all.
This is how it works: you put your model out there in the coliseum, and a bunch of guys in white coats kick the shit out of it. If it’s still alive when the dust clears, your brainchild receives conditional acceptance. It does not get rejected. This time. ...
Science is so powerful that it drags us kicking and screaming towards the truth despite our best efforts to avoid it. And it does that at least partly fueled by our pettiness and our rivalries. Science is alchemy: it turns shit into gold.
It took me about 5 years. Again, I don't think it's a useful approach if you don't like exercising in the first place; for me 5 years of resistance training has felt less like a weight-loss strategy and more like an excuse to have fun chasing goals and make like-minded friends along the way.
Lots of people believe they can eat more if they just exercise more. Unfortunately our bodies are highly efficient relative to the density of modern food, so “exercising it away” is not a realistic plan.
'Exercising it away' seems misguided given our bodies' energetic efficiency, as you said. What's instead worked for me is raising my basal metabolic rate substantially by adding muscle, which is very energetically expensive, via ~3 resistance training sessions a week.
Admittedly I don't know of a way to maintain the required muscle mass for this strategy to work long-term without enjoying physical activity, which I seem to enjoy the way most people enjoy good food, which probably makes this useless as general advice.
Thought it would be useful to share this 2017 HN thread.
The Myths of Creativity by David Burkus has this passage on class 1 vs class 2 disagreement:
In the 1970s at Xerox PARC, regularly scheduled arguments were routine. The company that gave birth to the personal computer staged formal discussions designed to train their people on how to fight properly over ideas and not egos. PARC held weekly meetings they called "Dealer" (from a popular book of the time titled Beat the Dealer). Before each meeting, one person, known as "the dealer," was selected as the speaker. The speaker would present his idea and then try to defend it against a room of engineers and scientists determined to prove him wrong. Such debates helped improve products under development and sometimes resulted in wholly new ideas for future pursuit. The facilitators of the Dealer meetings were careful to make sure that only intellectual criticism of the merit of an idea received attention and consideration. Those in the audience or at the podium were never allowed to personally criticize their colleagues or bring their colleagues' character or personality into play.
Bob Taylor, a former manager at PARC, said of their meetings, "If someone tried to push their personality rather than their argument, they'd find that it wouldn't work." Inside these debates, Taylor taught his people the difference between what he called Class 1 disagreements, in which neither party understood the other party's true position, and Class 2 disagreements, in which each side could articulate the other's stance. Class 1 disagreements were always discouraged, but Class 2 disagreements were allowed, as they often resulted in a higher quality of ideas. Taylor's model removed the personal friction from debates and taught individuals to use conflict as a means to find common, often higher, ground.
Alan Kay responded to the above with
This is one of those stories that has distorted over time. "Dealer" was a weekly meeting for many purposes, the main one was to provide a vehicle for coordination, planning, communication without having to set up a management structure for brilliant researchers who had some "lone wolves" tendencies.
Part of these meetings were presentations by PARC researchers. However, it was not a gantlet to be run, and it was not to train people to argue in a constructive way (most of the computer researchers at PARC were from ARPA community research centers, and learning how to argue reasonably was already part of that culture).
Visitors from Xerox frequently were horrified by the level of argument and the idea that no personal attacks were allowed had to be explained, along with the idea that the aim was not to win an argument but to illuminate. Almost never did the participants have to be reminded about "Class 1" and "Class 2", etc. The audience was -not- determined to prove the speaker wrong. That is not the way things were done.
which I suppose suggests the answer to this comment's question is "probably not".
re: The Fractal Prince (really the whole Quantum Thief trilogy), I may be biased, but when I first read it I had 2 reactions: (1) this is the most targeted-at-my-ingroup novel I have ever read (2) nobody outside of my ingroup will get the kajillion references flying around, since Hannu Rajaniemi never bothered footnoting / defending any of them (unlike say what Peter Watts did with Blindsight), so people will think he's just making up technobabble when he's not, which means he'll be generally underappreciated despite the effusive praise (which will be of the generic "he's so smart" variety), which made my heart sink.
But Gwern not only got it (unsurprisingly), he articulated it better than I ever could, so thanks Gwern:
Hannu makes no concessions to the casual reader, as he mainlines straight into his veins the pre-deep-learning 2010-era transhumanist zeitgeist via Silicon Valley—if it was ever discussed in a late-night bull session after a Singularity University conference, it might pop up here. Hannu stuffs the novels with blink-and-you’ll-miss-it ideas on the level of Olaf Stapeldon. A conventional Verne gun is too easy a way of getting to space—how about beating Project Orion by instead using a nuclear space gun (since emulated brains don’t care about high g acceleration)? Or for example, the All-Defector reveals that, since other universes could be rewriting their rules to expand at maximum speed, erasing other universes before they know it, he plans to rewrite our universe’s rule to do so first (ie. he will defect at the multiversal level against all other universes); whereas beginner-level SF like The Three Body Problem would dilate on this for half a book, Hannu’s grand reveal gets all of 2 paragraphs before crashing into the eucatastrophic ending.
For world-building, he drops neologisms left and right, and hard ones at that—few enough American readers will be familiar with the starting premise of “Arsène Lupin in spaaaace!” (probably more are familiar with the anime Lupin The Third these days), but his expectations go far beyond that: the ideal reader of the trilogy is not merely one familiar with the Prisoner’s Dilemma but also with the bizarre zero-determinant PD strategies discovered ~2008, and not just with such basic physics as quantum entanglement or applications like quantum dots, but exotic applications to quantum auctions & game theory (including Prisoner’s Dilemma) & pseudo-telepathy (yes, those are things), and it would definitely be helpful if that reader happened to also be familiar with Eliezer Yudkowsky’s c. 2000s writings on “Coherent Extrapolated Volition”, with a dash of Nikolai Fyodorovich Fyodorov’s Russian Cosmism for seasoning (although only a dash2).
This leads to an irony: I noted while reading Masamune Shirow’s Ghost in the Shell cyberpunk manga that almost everything technical in the GitS manga turned out to be nonsense despite Shirow’s pretensions to in-depth research & meticulous attention to detail; while in QT, most technical things sound like cyberpunk nonsense and Hannu doesn’t insert any editorial notes like Shirow does to defend them, but are actually real and just so arcane you haven’t heard of them.
For example, some readers accuse Hannu of relying on FTL communication via quantum entanglement, which is bad physics; but Hannu does not! If they had read more closely (similar to the standard reader failure to understand the physics of “Story of Your Life”), they would have noticed that at no point is there communication faster-than-light, only coordination faster-than-light—‘spooky action at a distance’. He is instead employing advanced forms of quantum entanglement which enable things like secret auctions or for coordinated strategies of game-playing (quantum coordination, like treating the particle measurements as flipping a coin and one person does the ‘Heads’ strategy and the other person does the ‘Tails’ strategy does not require communication, obviously, but surprisingly, quantum coordination can be superior to all apparently-equivalent communication-free classical strategies). He explains briefly that the zoku use quantum entanglement in these ways, but a reader could easily miss that, given all the other things they are trying to understand and how common ‘quantum woo’ is.3
Rajaniemi confirmed that Gwern "got it" like nobody else did:
As a longtime fan of gwern 's work -- gwern.net is the best rabbit hole on the Internet -- it's a treat to see this incredibly thoughtful (and slightly spoilery) review of the Quantum Thief trilogy. gwern.net/review/book#quantu… Gwern perfectly nails the emotional core of the trilogy and, true to form, spots a number of easter eggs I thought no one would ever find. This may be my favorite review of all time.
I admire Rajaniemi for pulling it off as you said, but I'm somehow not that surprised. He's bright (mathematical physics PhD) and has been working at writing-as-craft for a while:
But talking to him about his rapid career, it’s quickly apparent he’s no stranger to being compared to other sci-fi rising stars, having first seriously begun writing in 2002 while studying his PhD as part of writing group called Writers Bloc – which includes authors Charles Stross and Alan Campbell. “It is, and always has been a place with quite a harsh level of criticism,” he says. “But in a healthy and professional way, of course, so it was a good group of people and environment in which to develop.”
I personally got the Quantum Thief trilogy because I'd been blown away by Stross' Accelerando, wanted more, and saw Stross say of Rajaniemi: "Hard to admit, but I think he’s better at this stuff than I am. The best first SF novel I’ve read in many years."
I see. You may be interested in a contrary(?) take from the Welfare Footprint Project's researchers; in their FAQ they write
4. Why don't you use the term 'suffering', instead of 'pain'?
We prefer not to use the term suffering for various reasons. First, our analyses are concerned with “any” negative affective state (including mild ones), whereas the term suffering is often used to denote more severe states that are accompanied by concurrent negative feelings such as the perception of lack of control, fear, anxiety, the impossibility to enjoy pleasant activities or even a threat to one’s sense of self. Additionally, it is not yet possible to determine objectively when an unpleasant state becomes suffering. This is so far a value judgement, which we leave open to users of our estimates. The term ‘pain’ (both physical and psychological), in turn, is associated with negative affective experiences of a wide range of intensities.
They define their terms further here. To be fair, they focus on non-human animal welfare; I suppose your suffering vs joy distinction is more currently actionable in human-focused contexts e.g. CBT interventions.
The motivation of this post is to address the persistent conflation between suffering and pain I have observed from members of the EA community, even amongst those who purport to be “suffering-focused” in their ethical motivations.
I'm pretty suffering-focused in practice for EA-related actions (mostly donations), so I was hoping you'd say more. So:
Having this distinction in mind is critical in order to develop ethical policies and effective interventions. For instance, as previously mentioned, CBT and mindfulness practices have been shown to reduce suffering by altering the mental response to pain rather than addressing the pain itself. If (the alleviation of) suffering is what we care about, this distinction guides us to focus on the root causes of suffering in our ethical considerations, rather than merely alleviating pain. Recognizing that suffering often lies in an aversive mental reaction to pain rather than the pain itself enables more precise scientific research and more effective strategies for reducing overall suffering.
This was probably the first intervention that came to mind for me as well when seeing your claim that distinguishing pain and suffering matters in the EA-action-guiding sense; unfortunately it's already a thing, e.g. HLI recommending StrongMinds. I'd be interested if you have any other ideas for underexplored / underappreciated cause areas / intervention groups that might be worth further investigation when reevaluated via this pain vs suffering distinction? (This is my attempt to make this distinction pay rent, albeit in actions instead of anticipated experiences.)
That's not the sense I get from skimming his second most recent post, but I don't understand what he's getting at well enough to speak in his place.
Not an answer to your question, just an extended quote from the late Fields medalist Bill Thurston from his classic essay On proof and progress which seemed relevant:
Mathematicians have developed habits of communication that are often dysfunctional. Organizers of colloquium talks everywhere exhort speakers to explain things in elementary terms. Nonetheless, most of the audience at an average colloquium talk gets little of value from it. Perhaps they are lost within the first 5 minutes, yet sit silently through the remaining 55 minutes. Or perhaps they quickly lose interest because the speaker plunges into technical details without presenting any reason to investigate them. At the end of the talk, the few mathematicians who are close to the field of the speaker ask a question or two to avoid embarrassment.
... Outsiders are amazed at this phenomenon, but within the mathematical community, we dismiss it with shrugs. ...
Mathematical knowledge can be transmitted amazingly fast within a subfield. When a significant theorem is proved, it often (but not always) happens that the solution can be communicated in a matter of minutes from one person to another within the subfield. The same proof would be communicated and generally understood in an hour talk to members of the subfield. It would be the subject of a 15- or 20-page paper, which could be read and understood in a few hours or perhaps days by members of the subfield.
Why is there such a big expansion from the informal discussion to the talk to the paper? One-on-one, people use wide channels of communication that go far beyond formal mathematical language. They use gestures, they draw pictures and diagrams, they make sound effects and use body language. Communication is more likely to be two-way, so that people can concentrate on what needs the most attention. With these channels of communication, they are in a much better position to convey what’s going on, not just in their logical and linguistic facilities, but in their other mental facilities as well.
In talks, people are more inhibited and more formal. Mathematical audiences are often not very good at asking the questions that are on most people’s minds, and speakers often have an unrealistic preset outline that inhibits them from addressing questions even when they are asked. In papers, people are still more formal. Writers translate their ideas into symbols and logic, and readers try to translate back.
Why is there such a discrepancy between communication within a subfield and communication outside of subfields, not to mention communication outside mathematics?
Mathematics in some sense has a common language: a language of symbols, technical definitions, computations, and logic. This language efficiently conveys some, but not all, modes of mathematical thinking. Mathematicians learn to translate certain things almost unconsciously from one mental mode to the other, so that some statements quickly become clear. Different mathematicians study papers in different ways, but when I read a mathematical paper in a field in which I’m conversant, I concentrate on the thoughts that are between the lines. I might look over several paragraphs or strings of equations and think to myself “Oh yeah, they’re putting in enough rigamarole to carry such-and-such idea.” When the idea is clear, the formal setup is usually unnecessary and redundant—I often feel that I could write it out myself more easily than figuring out what the authors actually wrote. It’s like a new toaster that comes with a 16-page manual. If you already understand toasters and if the toaster looks like previous toasters you’ve encountered, you might just plug it in and see if it works, rather than first reading all the details in the manual.
People familiar with ways of doing things in a subfield recognize various patterns of statements or formulas as idioms or circumlocution for certain concepts or mental images. But to people not already familiar with what’s going on the same patterns are not very illuminating; they are often even misleading. The language is not alive except to those who use it.
Okay, I liked that passage but maybe it wasn't very useful. Ravi Vakil's advice to potential PhD students attending talks seems more useful, especially the last bullet:
- At the end of the talk, you should try to answer the questions: What question(s) is the speaker trying to answer? Why should we care about them? What flavor of results has the speaker proved? Do I have a small example of the phenonenon under discussion? You can even scribble down these questions at the start of the talk, and jot down answers to them during the talk.
- Try to extract three words from the talk (no matter how tangentially related to the subject at hand) that you want to know the definition of. Then after the talk, ask me what they mean. ...
- New version of the previous jot: try the "three things" exercise.
- See if you can get one lesson from the talk (broadly interpreted).
- Try to ask one question at as many seminars as possible, either during the talk, or privately afterwards. The act of trying to formulating an interesting question (for you, not the speaker!) is a worthwhile exercise, and can focus the mind.
I'm guessing Rob is referring to footnote 54 in What do XPT forecasts tell us about AI risk?:
And while capabilities have been increasing very rapidly, research into AI safety, does not seem to be keeping pace, even if it has perhaps sped-up in the last two years. An isolated, but illustrative, data point of this can be seen in the results of the 2022 section of a Hypermind forecasting tournament: on most benchmarks, forecasters underpredicted progress, but they overpredicted progress on the single benchmark somewhat related to AI safety.
That last link is to Jacob Steinhardt's tweet linking to his 2022 post AI Forecasting: One Year In, on the results of their 2021 forecasting contest. Quote:
Progress on a robustness benchmark was slower than expected, and was the only benchmark to fall short of forecaster predictions. This is somewhat worrying, as it suggests that machine learning capabilities are progressing quickly, while safety properties are progressing slowly. ...
As a reminder, the four benchmarks were:
- MATH, a mathematics problem-solving dataset;
- MMLU, a test of specialized subject knowledge using high school, college, and professional multiple choice exams;
- Something Something v2, a video recognition dataset; and
- CIFAR-10 robust accuracy, a measure of adversarially robust vision performance.
...
Here are the actual results, as of today:
- MATH: 50.3% (vs. 12.7% predicted)
- MMLU: 67.5% (vs. 57.1% predicted)
- Adversarial CIFAR-10: 66.6% (vs. 70.4% predicted)
- Something Something v2: 75.3% (vs. 73.0% predicted)
That's all I got, no other predictions.
Great post, especially the companion piece :)
I'm tangentially reminded of professional modeler & health economist froolow's refactoring of GiveWell's cost-effectiveness models in his A critical review of GiveWell's 2022 cost-effectiveness model (sections 3 and 4), which I think of as complementary to your post in that it teaches-via-case-study how to level up your spreadsheet modeling.
Here's GiveWell's model architecture:
And here's froolow's refactoring:
The difference in micro-level architecture is also quite large:
As someone who's spent a lot of his (short) career building dashboards and models in Google Sheets, and having seen GiveWell's CEAs, I empathized with froolow's remarks here:
After the issue of uncertainty analysis, I’d say the model architecture is the second biggest issue I have with the GiveWell model, and really the closest thing to a genuine ‘error’ rather than a conceptual step which could be improved. Model architecture is how different elements of your model interact with each other, and how they are laid out to a user.
It is fairly clear that the GiveWell team are not professional modellers, in the same way it would be obvious to a professional programmer that I am not a coder (this will be obvious as soon as you check the code in my Refactored model!). That is to say, there’s a lot of wasted effort in the GiveWell model which is typical when intelligent people are concentrating on making something functional rather than using slick technique. A very common manifestation of the ‘intelligent people thinking very hard about things’ school of model design is extremely cramped and confusing model architecture. This is because you have to be a straight up genius to try and design a model as complex as the GiveWell model without using modern model planning methods, and people at that level of genius don’t need crutches the rest of us rely on like clear and straightforward model layout. However, bad architecture is technical debt that you are eventually going to have to service on your model; when you hand it over to a new member of staff it takes longer to get that member of staff up to speed and increases the probability of someone making an error when they update the model.