LessWrong 2.0 Reader

Yes, though I'm not confident.

I don’t see how this is any evidence against John’s point.

Presumably the reason you need such crushingly obvious results which can be seen regardless of the validity of your statistical tool before the field can move on is because you need to convince the median researchers.

The sharp researchers have predictions about where the field is going based on statistical evidence and mathematical reasoning, and presumably can be convinced of the ultimate state far before the median, and work toward proving or disproving their hypotheses, and then once its clear to them, making the case stupidly obvious for the lowest common denominator in the room. And I expect this is where most of the real conceptual progress lies.

Even in the word where as you claim this is a marginal effect, if we could speed up any given advance in academic biomedicine by a year, that is an incredible achievement! Many people may die in that year who could’ve been saved had the median not wasted time (assuming the year saved carries over to clinical medicine).

Hmm, I still might not be following, but I’ll write something anyway. :)

Take some “concept” in your world-model, operationalized as a particular cluster C of neurons in some part of your cortex that tend to activate together.

How might we figure out what what C “means”?

One part of the answer is entirely within the cortex world-model: C has particular relationships to other things in the cortex world-model, which in term have relationships to still other things etc. Clusters of neurons related to “bird” have some connection to clusters of neurons related to “flying”. That by itself might already be enough to pin down the “meanings” of different things, just because there’s so much structure there, and we can try to match it up with structures in the world, by analogy with unsupervised machine translation. But if not…

The other part of the answer is about how the cortex world-model relates to the real world. Maybe C directly predicts some particular pattern in low-level sensory inputs. Maybe C directly activates some particular pattern in motor output. Or maybe the connection is less direct—a certain abstract pattern in the space of abstract patterns in the space of abstract patterns in the space of low-level sensory inputs, or whatever. If we look at naturalistic visual inputs that directly or indirectly trigger C, and they’re disproportionately pictures of clocks, then that’s some evidence that C “means” clock.

So, how about “cold”? Our body has a couple relevant sensors: peripheral nerves that express TRPM8 (“cold and menthol receptor 1”), hypothalamus neurons that detect blood temperature via TRPV1, etc. (I’m not an expert on the details.) As usual [LW · GW], these sensory signals are processed in two areas in parallel. In the hypothalamus & brainstem (“Steering Subsystem”), they trigger innate reactions like shivering, unpleasant feelings / desire to warm up, and so on. And in the cortex, they’re treated as just so many more channels of unlabeled input data that the world-model needs to predict.

In the course of predicting them well, the world-model invents some slightly-higher-level concept (or family of closely-interlinked concepts) that we call “cold”. And it notices and memorizes predictively-useful relationships between this new “cold” concept and other things in the world-model, e.g. shivering and ice.

I don’t think there’s more to the concept “cold” than the sum total of its associations with every other concept, with sensory input, and with motor output. And we can explain those latter associations via the structure of the world and body in conjunction with a learning algorithm running throughout your life experience.

You can sorta write code for a relevant part of what's happening in the mind when e.g. the freezing emotion/sensation is triggered.

I like to draw the distinction between understanding learning algorithms and understanding trained models. The former is kinda like what you learn in an ML course (gradient descent, training data, etc.) , the latter is kinda like what you learn in a mechanistic interpretability paper. I don’t think it’s realistic to “write code” for the “cold” concept, because I think it (like all concepts) emerges at the trained model level. It emerges from a learning algorithm, training environment, loss function, etc.

Of course, we can chat about the trained model level to some extent. Why is “cold” associated with shivering? Because in the training environment of life experience, those two things have tended to go together, such that each provides nonzero Bayesian evidence that the other should be active, or will be soon. Ditto with the connection between cold and ice cream, and everything else. So we can chat about it, but it would take forever to directly write code for all those things. Hence the learning algorithm. Does that help?

Here's my solution to your problem. Small major donors should collectively organize together and make decisions democratically.

I would therefore expand the donor lottery into a democratic committee. Instead of selecting only 1 participant, select ~10 participants, similar to jury duty. With more participants, we enjoy more diverse opinion and a better representative sample (Yes 10 is a terrible sample, but it's way better than 1. If the number of members in the pool increase, the sample size should be increased). More people also facilitate better deliberative discussion and information sharing. 

The rationale of a lottocratically selected committee is also different from a donor lottery. Lottocratic committees have democratic legitimacy (often called "sortition"). They are created similarly to how jury pools are created, with similar democratic credentials. 

As the sample size of the committee increases, it becomes more and more legitimate as a representative statistical sample of the donor membership. 

The tradeoff is cost. A lottocratic body of 10 is 10 times more costly than a body of 1. But it's also much more efficient than individual action. Imagine 50 people are in your pool. A lottocratic body of 10 reduces cognitive load by 5x. In my opinion, the body of 10 will also make better decisions than a single temporary dictator. 

A variety of reasons why collective decisions are often better include:

• The practice of deliberative democracy - Deliberation can produce better informed results.
• Division of labor - Effective committees can organize research tasks and divide up cognitive labor to enhance decision making.
• Condorcet's Jury Theorem - Greater number of participants increases decision accuracy.
• Median voter theorem - Greater number of participants activate the possibility of tending towards the median preferences of the pool.

A single winner in contrast invites chaos to charitable selections, and is unrepresentative of the whole. 

I disagree with “He seems to have no inside information.” He presented himself as having no inside information, but that’s presumably how he would have presented himself regardless of whether he had inside information or not. It’s not like he needed to convince others that he knows what he’s doing, like how in the stock market you want to buy then pump then sell. This is different—it’s a market that’s about to resolve. The smart play from his perspective would be to aggressively trash-talk his own competence, to lower the price in case he wants to buy more.

My quick reactions:

1. Interp: no inside view; my impression is that my friends are pessimistic about such safety cases in the next few years
2. Control: hooray for Anthropic finally writing something about control; some details seem good; some details seem odd ("contexts that were specifically whitelisted"); I think the naive pre-deployment honeypots path doesn't work because a scheming model wins with the defect rarely strategy (related: https://www.lesswrong.com/posts/vYWDEtJ2b8tkbG7Rv/behavioral-red-teaming-is-unlikely-to-produce-clear-strong [LW · GW]) [very sophisticated or expensive honeypotting setups could work but the post doesn't seem to be imagining that or noticing the issue]; that's important and the current language in the post is concerning
3. Incentives: I don't really understand, or I'm dubious, or I expect that A3.1 is very hard to demonstrate

Mindmeld

In theory AIs can transmit information far faster and more directly than humans. They can directly send weight/activation vectors to one another. The most important variable on whether entities (cells, organisms, polities, companies, ideologies, empire etc) stay individuals or amalgate into a superorganism is communication bandwith & copy fidelity. 
Both of these differ many order of magnitude for humans versus AIs. At some point, mere communication becomes a literal melding of minds. It seems quite plausibly then that AIs will tend to mindmeld if left alone. 

The information rate of human speech is around 39 bits per second, regardless of the language being spoken or how fast or slow it is spoken. This is roughly twice the speed of Morse code. 

Some say that the rate of 39 bits per second is the optimal rate for people to convey information. Others suggest that the rate is limited by how quickly the brain can process or produce information. For example, one study found that people can generally understand audio recordings that are sped up to 120%. 

While the rate of speech is relatively constant, the information density and speaking rate can vary. For example, the information density of Basque is 5 bits per syllable, while Vietnamese is 8 bits per syllable. 

Current state of the art fibre optic cables can transmit up to 10 terabits a second. 

That's probably a wild overestimate for AI communication though. More relevant bottlenecks are limits on processing informations [plausibly more in the megabits range], limits on transferability of activation vectors (but training could improve this). 

>>  'a massive transfer of wealth from "sharps" '. 

no. That's exactly the point. 

1. there might no be any real sharps (=traders having access to real private arbitragiable information that are consistently taking risk-neutral bets on them) in this market at all.

This is because a) this might simple be a noisy, high entropy source that is inherently difficult to predict, hence there is little arbitragiable information and/or b) sharps have not been sufficiently incenticiz

2. The transfer of wealth is actually disappointing because Theo the French Whale moved the price so much. 

For an understanding of what the trading decisions of a verifiable sharp looks like one should take a look at Jim Simons' Medaillon fund. They do enormous hidden information collection, ?myssterious computer models, but at the end of the day take a large amount of very hedged tiny edge positions. 

***************************************************

You are misunderstanding my argument (and most of the LW commentariat with you). I might note that I made my statement before the election result and clearly said 'win or lose' but it seems that even on LW people think winning on a noisy N=1 sample is proof of rationality. 

mildly disapprove of words like "a widely-used strategy"

The text says “A widely-used strategy for arguing for norms of rationality involves avoiding dominated strategies”, which is true* and something we thought would be familiar to everyone who is interested in these topics. For example, see the discussion of Dutch book arguments in the SEP entry on Bayesianism and all of the LessWrong discussion on money pump/dominance/sure loss arguments (e.g., see all of the references in and comments on this post [LW · GW]). But fair enough, it would have been better to include citations.

"we often encounter claims"

We did include (potential) examples in this case. Also, similarly to the above, I would think that encountering claims like “we ought to use some heuristic because it has worked well in the past” is commonplace among readers so didn’t see to provide extensive evidence.

*Granted, we are using “dominated strategy” in the wide sense of “strategy that you are certain is worse than something else”, which glosses over technical points like the distinction between dominated strategy and sure loss.

links 11/6/2024: https://roamresearch.com/#/app/srcpublic/page/11-06-2024

• https://angrystaffofficer.com/2018/09/19/if-the-hoth-crash-was-an-air-force-investigation/
  • this taught me the phrase "mishap pilot"
• https://pmc.ncbi.nlm.nih.gov/articles/PMC5656536/ this is measles virus used against relapsed multiple myeloma; one complete response out of 32 patients.
  • https://www.nature.com/articles/s41375-020-0828-7.pdf the one patient with the CR had strong T-cell responses to measles virus proteins. suggests that when this works it's via immune response.
  • https://ajronline.org/doi/pdf/10.2214/AJR.09.3672 it works on mouse pancreatic cancer
  • https://ascopubs.org/doi/abs/10.1200/JCO.2022.40.6_suppl.509 seems to be able to treat bladder cancer?
  • https://pmc.ncbi.nlm.nih.gov/articles/PMC3018921/ blocks medulloblastoma growth in mice
  • https://en.wikipedia.org/wiki/CD46 the receptor for measles virus is also frequently expressed by cancer cells
  •  
• https://www.frontiersin.org/journals/oncology/articles/10.3389/fonc.2023.1095219/full targeting CDKs in sarcomas -- there are some clinical trials happening
  • https://ascopubs.org/doi/abs/10.1200/PO.24.00219 palcociclib: one partial response out of 42 sarcoma patients
  • https://aacrjournals.org/clincancerres/article/29/17/3484/728559 suggestive in-vitro/animal evidence
• targeting FGFRs in advanced solid tumors with FGFR mutations/overexpression:  https://dial.uclouvain.be/pr/boreal/object/boreal%3A285422/datastream/PDF_01/view
  • 3% complete response, 25% partial response with erdafitinib
  • https://pmc.ncbi.nlm.nih.gov/articles/PMC8231807/ FGFR inhibitors are typically toxic
• MPNSTs cluster into two distinct types of genomic alteration with different drug vulnerabilities https://www.nature.com/articles/s41467-023-38432-6.pdf
• targeting MDM2 in advanced solid tumors: there's a trial.  https://clinicaltrials.gov/study/NCT03611868
  • https://ascopubs.org/doi/10.1200/JCO.2022.40.16_suppl.9517 2 complete responses in melanoma, 1 PR each in liposarcoma, urothelial, and NSCLC, but none in MPNST.
  • http://www.annclinlabsci.org/content/46/6/627.full it's being explored as a target in cancer
• https://www.sciencedirect.com/science/article/abs/pii/S0959804920313228 21% partial response in soft tissue sarcoma to a XPO1 inhibitor + chemo
  • review article on XPO1 inhibition https://www.nature.com/articles/s41571-020-00442-4
• https://www.centauri-dreams.org/2024/11/05/vegas-puzzling-disk/ the star Vega looks like it has a disc but no planets
• https://www.nature.com/articles/s41598-024-58899-7 CD74 in cancer is an indicator for M1 macrophage infiltration, across cancer types
• https://bibliome.ai/  is a resource for looking up specific genome variants and their references in the literature and open-access databases.
  • when i click through to references they're often inaccurate (they are claimed to reference a variant that they do not, in fact, contain) but tbh this is also true of Google Search and Google Scholar when it comes to rare variants.
•