LessWrong 2.0 Reader

hey, welcome! Congrats on de-lurking, I think? I fondly remember my own teenage years of lurking online -- one certainly learns a lot about the human condition.

If I was sending my 14-year-old self a time capsule of LW, it'd start with the sequences, and beyond that I'd emphasize the writings of adults examining how their own cognition works. Two reasons -- first, being aware that one is living in a brain as it finishes wiring itself together is super entertaining if you're into that kind of thing, and even more fun when you have better data to guess how it's going to end up. (I got the gist of that from having well-educated and openminded parents, who explained that it's prudent to hold off on recreational drug use until one's brain is entirely done with being a kid, because most recreational substances make one's brain temporarily more childlike in some way and the real thing is better. Now I'm in my 30s and can confirm that's how such things, including alcohol, have worked for me)

Second, my 20s would have been much better if someone had taken kid-me aside and explained some neurodiversity stuff to her: "here's the range of normal, here's the degree of suffering that's not expected nor normal and is worth consulting a professional for even if you're managing through great effort to keep it together", etc.

If you'd like to capitalize on your age for some free internet karma, I would personally enjoy reading your thoughts on what your peers think of technology, how they get their information, and how you're all updating the language at the moment.

I also wish that my 14-year-old self had paid more attention to the musical trends and attempted to guess which music that was popular while I was of highschool age would stand the test of time and remain on the radio over the subsequent decades. In retrospect, I'm pretty sure I could probably have taken some decent guesses, but I didn't so now I'll never know whether I would have guessed right :)

It is not that high here, but this is something I will look into if we can get to Seattle. But does this not require a license?

Why do you guys think this is happening? It sounds to me like one possibility is that maybe the model might have some amount of ensembling (thinking back to The Clock and The Pizza where in a toy setting ensembling happened). W.r.t. "across all steering vectors" that's pretty mysterious, but at least in the specific examples in the post even 9 was semi-fantasy.

Also what are ya'lls intuitions on picking layers for this stuff. I understand that you describe in the post that you control early layers because we suppose that they might be acting something like switches to different classes of functionality. However, implicit in layer 10 it seems like you probably don't want to go too early because maybe in the very early layers it's unembedding and learning basic concepts like whether a word is a noun or whatever. Do you choose layers based on experience tinkering in jupyter notebooks and the like, or have you run some sort of grid to get a notion of what the effects elsewhere are. If the latter, it would be nice to know to aid in hypothesis formation and the like.

That's it! Thank you.

Pathogens, whether natural or artificial, have a fairly well-defined attack surface; the hosts’ bodies. Human bodies are pretty much static targets, are the subject of massive research effort, have undergone eons of adaptation to be more or less defensible, and our ability to fight pathogens is increasingly well understood.

It's certainly not true. Pathogen can target agriculture or ecosystems.

I hope you market it under the name Soylent.

I was envisioning that you can organize a festival incrementally, investing more time and money into it as you receive more and more validation, and that taking this approach would de-risk it to the point where overall, it's "not that risky".

For example, to start off you can email or message a handful of potential attendees. If they aren't excited by the idea you can stop there, but if they are then you can proceed to start looking into things like cost and logistics. I'm not sure how pragmatic this iterative approach actually is though. What do you think?

Also, it seems to me that you wouldn't have to actually risk losing any of your own money. I'd imagine that you'd 1) talk to the hostel, agree on a price, have them "hold the spot" for you, 2) get sign ups, 3) pay using the money you get from attendees.

Although now that I think about it I'm realizing that it probably isn't that simple. For example, the hostel cost ~$5k and maybe the money from the attendees would have covered it all but maybe less attendees signed up than you were expecting and the organizers ended up having to pay out of pocket.

On the other hand, maybe there is funding available for situations like these.

Thanks for all these additional datapoints! I'll try to respond all of your questions in turn:

Did you find that your AIS survey respondents with more AIS experience were significantly more male than newer entrants to the field?

Overall, there don't appear to be major differences when filtering for amount of alignment experience. When filtering for greater than vs. less than 6 months of experience, it does appear that the ratio looks more like ~5 M:F; at greater than vs. less than 1 year of experience, it looks like ~8 M:F; the others still look like ~9 M:F. Perhaps the changes you see over the past two years at MATS are too recent to be reflected fully in this data, but it does seem like a generally positive signal that you see this ratio changing (given what we discuss [LW · GW] in the post).

Has AE Studio considered sponsoring significant bounties or impact markets for scoping promising new AIS research directions?

We definitely want to do everything we can to support increased exploration of neglected approaches—if you have specific ideas here, we'd love to hear them and discuss more! Maybe we can follow up offline on this.

Did survey respondents mention how they proposed making AIS more multidisciplinary? Which established research fields are more needed in the AIS community?

We don't appear to have gotten many practical proposals for how to make AIS more multidisciplinary, but there were a number of specific disciplines mentioned in the free responses, including cognitive psychology, neuroscience, game theory, behavioral science, ethics/law/sociology, and philosophy (epistemology was specifically brought up across multiple respondents). One respondent wrote, "AI alignment is dominated by computer scientists who don't know much about human nature, and could benefit from more behavioral science expertise and game theory," which I think captures the sentiment of many of the related responses most succinctly (however accurate this statement actually is!). Ultimately, encouraging and funding research at the intersection of these underexplored areas and alignment is likely the only thing that will actually lead to a more multidisciplinary research environment.

Did EAs consider AIS exclusively a longtermist cause area, or did they anticipate near-term catastrophic risk from AGI?

Unfortunately, I don't think we asked the EA sample about AIS in a way that would allow us to answer this question using the data we have. This would be a really interesting follow-up direction. I will paste in below the ground truth distribution of EAs' views on the relative promise of these approaches as additional context (eg, we see that the 'AI risk' and 'Existential risk (general)' distributions have very similar shapes), but I don't think we can confidently say much about whether these risks were being conceptualized as short- or long-term.

It's also important to highlight that in the alignment sample (from the other survey), researchers generally indicate that they do not think we're going to get AGI in the next five years. Again, this doesn't clarify if they think there are x-risks that could emerge in the nearer term from less-general-but-still-very-advanced AI, but it does provide an additional datapoint that if we are considering AI x-risks to be largely mediated by the advent of AGI, alignment researchers don't seem to expect this as a whole in the very short term:

The cancer must develop neoantigens that are sufficiently distinct from human surface proteins and consistent across the cancer.

This is not necessarily true. There are some proteins that get produced by embryos and not by adult humans. Sometimes cancer mutates in a way that those proteins get produced by cancer cells.

While the vaccines that target a single of those embryo proteins did not do enough in clinical trials, I don't see a reason why we should completely ignore those proteins. 

Cancer cells must be isolated and have their surface proteins characterized.

Given that cancer cells engage in necrososis much more than regular cells you don't need to isolate cancer cells to get the DNA of cancer. ctDNA can in theory be used to sequence the cancer. Using ctDNA might even be better because it gives you a better idea of whether a mutation is present in most of the cancer cells or only the section from which you removed cells. 

Individualized cancer vaccines are not yet practical

Moderna has just put an individualized cancer vaccine into a phase III trial after positive results from a phase 2b trial:

Hirawat was referring to Moderna and Merck’s December announcement of positive results from a 157-patient phase 2b trial dubbed KEYNOTE-942. The pair said a combination of their personalized mRNA cancer vaccine, coded mRNA-4157 or V940, and the PD-1 inhibitor Keytruda slashed the risk of tumor recurrence or death by 44% compared with Keytruda alone when used as an adjuvant therapy in stage 3/4 melanoma following complete surgical resection.

Do you think that trial is a bad idea?

Whoa! The meetup to beat