LessWrong 2.0 Reader
View: New · Old · Topnext page (older posts) →
next page (older posts) →
I absolutely agree that it makes more sense to fund the person (or team) rather than the project. I think that it makes sense to evaluate a person's current best idea, or top few ideas when trying to decide whether they are worth funding.
Ideally, yes, I think it'd be great if the funders explicitly gave the person permission to pivot so long as their goal of making aligned AI remained the same.
Maybe a funder would feel better about this if they had the option to reevaluate funding the researcher after a significant pivot?
emrik-1 on Fund me please - I Work so Hard that my Feet start Bleeding and I Need to Infiltrate UniversityHe linked his extensive research log on the project above, and has made LW posts of some of their progress. That said, I don't know of any good legible summary of it. It would be good to have. I don't know if that's one of Johannes' top priorities, however. It's never obvious from the outside what somebody's top priorities ought to be.
gotenosente on "If we go extinct due to misaligned AI, at least nature will continue, right? ... right?"It is not at all clear to me that most of the atoms in a planet could be harnessed for technological structures, or that doing so would be energy efficient. Most of the mass of an earthlike planet is iron, oxygen, silicon and magnesium, and while useful things can be made out of these elements, I would strongly worry that other elements that are needed also in those useful things will run out long before the planet has been disassembled. By historical precedent, I would think that an AI civilization on Earth will ultimately be able to use only a tiny fraction of the material in the planet, similarly to how only a very small fraction of a percent of the carbon in the planet is being used by the biosphere, in spite of biological evolution having optimized organisms for billions of years towards using all resources available for life.
The scenario of a swarm of intelligent drones eating up a galaxy and blotting out its stars I think can empirically be dismissed as very unlikely, because it would be visible over intergalactic distances. Unless we are the only civilization in the observable universe in the present epoch, we would see galaxies with dark spots or very strangely altered spectra somewhere. So this isn't happening anywhere.
There are probably some historical analogs for the scenario of a complete takeover, but they are very far in the past, and have had more complex outcomes than intelligent grey goo scenarios normally portray. One instance I can think of is the Great Oxygenation Event. I imagine an observer back then might have envisioned that the end result of the evolution of cyanobacteria doing oxygenic photosynthesis would be the oceans and lakes and rivers all being filled with green slime, with a toxic oxygen atmosphere killing off all other life. While indeed this prognosis would have been true to a first order approximation - green plants do dominate life on Earth today - the reality of what happened is infinitely more complex than this crude picture suggests. And even anaerobic organisms survive to this day in some niches.
The other historical precedent that comes to mind would be the evolution of organisms that use DNA to encode genetic information using the specific genetic code that is now universal to all life, in whatever pre-DNA world existed at the beginning of life. These seem to have indeed completely erased all other kinds of life (claims of a shadow biosphere of more primitive organisms are all dubious to my knowledge), but also have not resulted in a less complex world.
So am I. So are a lot of would-be researchers. There are many people who think they have a shot at doing this. Most are probably wrong. I'm not saying an org is a good solution for him or me. It would have to be an org willing to encompass and support the things he had in mind. Same with me. I'm not sure such orgs exist for either of us.
With a convincing track-record, one can apply for funding to found or co-found a new org based on your ideas. That's a very high bar to clear though.
The FAR AI org might be an adequate solution? They are an organization for coordinating independent researchers.
algon on Fund me please - I Work so Hard that my Feet start Bleeding and I Need to Infiltrate UniversityOh, I see! That makes a lot more sense. But he should really write up/link to his project then, or his collaborator's project.
charlie-steiner on In the context of AI interp. What is a feature exactly?A different way of stating the usual Anthropic-esque concept of features that I find useful: Features are the things that are getting composed when a neural network is taking advantage of compositionality. This isn't begging the question, you just can't answer this without knowing about the data distribution and the computational strategy of the model after training.
For instance, the reason the neurons aren't always features, even though it's natural to write the activations (which then get "composed" into the inputs to the next layer) in the neuron basis, is because if your data only lies on a manifold in the space of all possible values, the local coordinates of that manifold might rarely line up with the neurons basis.
nathan-helm-burger on Fund me please - I Work so Hard that my Feet start Bleeding and I Need to Infiltrate UniversityYeah, so... I find myself feeling like I have some things in common with the post author's situation. I don't think "work for free at an alignment org" is really an option? I don't know about any alignment orgs offering unpaid internships. An unpaid worker isn't free for an org, you still need to coordinate them, assess their output, etc. The issues with team bloat and how much to try to integrate a volunteer are substantial.
I wish I had someone I could work with on my personal alignment agenda, but it's not necessarily easy to find someone interested enough in the same topic and trustworthy enough to want to commit to working with them.
Which brings up another issue. Research which has potential capabilities side-effects is always going to be a temptation to some degree. How can potential collaborators or grant makers trust that researchers will resist the temptation to cash in on powerful advances and also prevent the ideas from leaking? If the ideas are unsafe to publish then the ideas can't contribute piecemeal to the field of alignment research, they have to be valuable alone. That places a much higher bar for success. Which makes it seem like a riskier bet from the perspective of funders. One way to partially ameliorate this issue of trust is having Orgs/Companies. They can thoroughly investigate a person's competence and trustworthiness. Then the person can potentially contribute to a variety of different projects once onboarded. Management can supervise and ensure that individual contributors are acting in alignment with the company's values and rules. That's a hard thing for a grant-making institution to do. They can't afford that level of initial evaluation, much less the ongoing supervision and technical guidance. So... Yeah. Tougher problem than it seems on first glance.
anders-lindstroem on Ilya Sutskever and Jan Leike resign from OpenAI [updated]Without resorting to exotic conspiracy theories, is it that unlikely to assume that Altman et al. are under tremendous pressure from the military and intelligence agencies to produce results to not let China or anyone else win the race for AGI? I do not for a second believe that Altman et al. are reckless idiots that do not understand what kind of fire they might be playing with, that they would risk wiping out humanity just to beat Google on search. There must be bigger forces at play here, because that is the only thing that makes sense when reading Leike's comment and observing Open AI's behavior.
emrik-1 on Fund me please - I Work so Hard that my Feet start Bleeding and I Need to Infiltrate UniversitySurely you could work for free as an engineer at an AI alignment org or something and then shift into discussions w/ them about alignment?
To be clear: his motivation isn't "I want to contribute to alignment research!" He's aiming to actually solve the problem. If he works as an engineer at an org, he's not pursuing his project, and he'd be approximately 0% as usefwl.
emrik-1 on Fund me please - I Work so Hard that my Feet start Bleeding and I Need to Infiltrate UniversityI strongly endorse Johannes' research approach. I've had 6 meetings with him, and have read/watched a decent chunk of his posts and YT vids. I think the project is very unlikely to work, but that's true of all projects I know of, and this one seems at least better than almost all of them. (Reality doesn't grade on a curve.)
Still, I really hope funders would consider funding the person instead of the project, since I think Johannes' potential will be severely stifled unless he has the opportunity to go "oops! I guess I ought to be doing something else instead" as soon as he discovers some intractable bottleneck wrt his current project. He's literally the person I have the most confidence in when it comes to swiftly changing path to whatever he thinks is optimal, and it would be a real shame if funding gave him an incentive to not notice reasons to pivot. (For more on this, see e.g. Steve's post [LW · GW].)
I realize my endorsement doesn't carry much weight for people who don't know me, and I don't have much general clout here, but if you're curious here's my EA forum profile [EA · GW] and twitter. Some other things which I hope will nudge you to take my endorsement a bit more seriously: