Open & Welcome Thread - July 2020

Hello everyone. Since I have signed up for an account I thought I might as well also say hello. I have been reading stuff on this site and associated communities for a while but I thought I had something to contribute here the other day so I signed up to make a first comment.

I'm in England, I have a background in science (PhD in computational biology back in the day) but now spend most of my time raising and educating my kids. I don't allocate much time to online activities but when I do it's good to have something substantial to chew on. I like the interesting conversations that arise in places where people are practicing genuine conversation with those they disagree with. Lots of interesting content here too.

Other things I am interested in: how energy resources shape what is doable; foraging; forest school; localizing food production; deep listening techniques; ways to help each other think well; data visualizations.

Many AI issues will likely become politicized. (For example, how much to prioritize safety versus economic growth and military competitiveness? Should AI be politically neutral or be explicitly taught "social justice activism" before they're allowed to be deployed or used by the public?) This seems to be coming up very quickly and we are so not prepared, both as a society and as an online community. For example I want to talk about some of these issues here, but we haven't built up the infrastructure to do so safely.

Eliezer Yudkowsky claims GPT-3's ability to apparently write a functioning app based on a prompt is a victory for his model.

After listening to the recent [LW(p) · GW(p)]podcast on scrutinizing arguments for AI risk, I figured this was an opportunity to scrutinize what the argument is [LW(p) · GW(p)]. Those two previous links summarize how I think the classic arguments for AI risk inform our current views about AI risk, and I'm trying to apply that to this specific argument that GPT-3 implies AI poses a greater danger.

Given that GPT-3 was not trained on this problem specifically, I claim this case and trend as a substantial victory of my model over @robinhanson's model. Robin, do you dispute the direction of the update?

This is how I think Eliezer's argument goes, more fully:

GPT-3 is general enough that it can write a functioning app given a short prompt, despite the fact that it is a relatively unstructured transformer model with no explicitly coded representations for app-writing. We didn't expect this.

The fact that GPT-3 is this capable suggests that 1) ML models scale in capability and generality very rapidly with increases in computing power or minor algorithm improvements, suggesting that eventually there will be a relatively abrupt switch to a new growth mode when ML models scale all the way to general intelligence, and 2) that we can get highly useful goal-orientated behaviour without building a system that seems like its alignment with our values is guaranteed or robust. GPT-3 is very useful at fairly general tasks but doesn't have alignment 'built-in' with any kind of robust guarantee, in the way suggested by this [LW(p) · GW(p)] from Stuart Russell:

The first reason for optimism [about AI alignment] is that there are strong economic incentives to develop AI systems that defer to humans and gradually align themselves to user preferences and intentions. Such systems will be highly desirable: the range of behaviours they can exhibit is simply far greater than that of machines with fixed, known objectives...

The way that GPT-3 makes use of user preferences is not, in fact, that reliable at determining what we really want.

Together these suggest that eventually progress in AI will be rapid and it is plausible that less-robustly aligned AI will be easier and more useful than robustly-aligned AI in some circumstances. Combine this with instrumental convergence and this suggests that if GPT-3 were capable enough its misalignment could produce catastrophic results. From my earlier post: [LW(p) · GW(p)]

"A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.   "

We could see this as marking out a potential danger - a large number of possible mind-designs produce very bad outcomes if implemented. The fact that such designs exist 'weakly suggest' (Ben's words) that AGI poses an existential risk since we might build them. If we add in other premises that imply we are likely to (accidentally or deliberately) build such systems, the argument becomes stronger. But usually the classic arguments simply note instrumental convergence and assume we're 'shooting into the dark' in the space of all possible minds, because they take the abstract statement about possible minds to be speaking directly about the physical world.

It's not that GPT-6 will be our first AGI, but an AGI set up rather like GPT-6 is not only possible in principle but something we might actually build, given that we built GPT-3, and GPT-6 would, based on what we've seen of GPT-3, have the dangerous properties of being not robustly aligned and increasing its capability rapidly.

There was an old suggestion of making an AI learn human values by training it on happiness and unhappiness in human facial expressions, making it happy when humans are happy and vice versa. Besides its other problems, now there's this...

We present an exploratory study of analyzing and visualizing player facial expressions from video with deep neural networks. [...] As an additional contribution, we show that although there has been tremendous recent progress in deep neural networks and computer vision, interpreting the results as direct read-outs of experiential states is not advised. According to our data, getting killed appears to make players happy, and much more so than killing enemies, although one might expect the exact opposite. A visual inspection of the data reveals that our classifier works as intended, and our results illustrate the limitations of making inferences based on facial images and discrete emotion labels. For example, players may laugh off the death, in which case the closest label for the facial expression is "happy", but the true emotional state is complex and ambiguous. On the other hand, players may frown in concentration while killing enemies or escaping a tight spot, which can easily be interpreted as an "angry" expression.

Hi everyone! I've been investigating Less Wrong for several months since I read HPMOR and it seems like an interesting place. It's possible that it's over my head but it seems like the best way to find out is to jump in!

I came to transhumanism from a weird starting point. In 1820, the Stone-Campbell Movement seems to have been a decent attempt at introducing religious people to rationality; in 2020, the Church of Christ is not so much. But there's still this weird top crust of people trying to press forward with higher rationality and human progress and potential (if kinda from a point of serious disadvantage) and I got in touch with their ideas even though I'm not really a member in good standing any more.

Lorem Ipsum

With the rise of GPT-3, does anyone else feel that the situation in the field of AI is moving beyond their control?

This moment reminds me of AlphaGo, 2016. For me that was a huge wake-up call, and I set out to catch up on the neural networks renaissance. (Maybe the most worthy thing I did, in the years that followed, was to unearth work on applying supersymmetry in machine learning.)

Now everyone is dazzled and shocked again, this time by what GPT-3 can do when appropriately prompted. GPT-3 may not be a true "artificial general intelligence", but it can impersonate one on the Internet. Its ability to roleplay as any specific person, real or fictional, is especially disturbing. An entity has appeared which simulates human individuals within itself, without having been designed to do so. It's as if the human mind itself is now obsolete, swallowed up within, made redundant by, a larger and more fluid class of computational beings.

I have been a follower and advocate of the quest for friendly AI for a long time. When AlphaGo appeared, I re-prioritized, dusted off old thoughts about how to make human-friendly AI, thought of how they might manifest in the present world, and felt like I was still ahead of events, barely. I never managed to break into the top tiers of the local AI scene (e.g. when a state-level meetup on AI was created, I wanted to give a talk on AIXI and superintelligence, but it was deemed inappropriate), but I was still able to participate and feel potentially relevant.

GPT-3 is different. Partly it's because the challenge to human intellectual capacity (and even to human identity) is more visceral here. AlphaGo and its siblings steamrolled humanity in the context of games, an area where we have long experience of being beaten by computers. But GPT-3 seems much more open-ended. It can already do anything and be anyone.

I also have new personal reasons for feeling unable to keep up, new responsibilities, though maybe they can be pursued in a "GPT-3-relevant" way. (I am the lifeline for a brilliant young transhumanist in another country. But AI is one of our interests, so perhaps we can adapt to this new era.)

So maybe the core challenge is just to figure out what it means to pursue friendly AI (or "alignment" of AI with human-friendly values) in the era of GPT-3. The first step is to understand what GPT-3 is, in terms of software, hardware, who has the code, who has access, and so on. Then we can ask where it fits into the known paradigms for benevolent superhuman AI.

For example, we have various forms of the well-known concept of an expected utility maximizer, which acts so as to make the best futures as likely as possible. GPT-3, meanwhile, starts with a prompt and generates verbiage, e.g. an essay. One could see this prompt-followed-by-response behavior, as analogous to one step in the interactions of an EU maximizer with the world, i.e. those interactions are usually modelled as follows: the EUM gets data about the world (this is the prompt), then it performs an action (this is the response), then it gets new data about the world (a second prompt), and so on.

On the other hand, GPT-3's prompt in some ways seems analogous to a goal or a value system, which in terms of an EUM would be the utility function it uses to hedonically evaluate possible futures; since it governs GPT-3's future activity. Digging deeper into these two different analogies between GPT-3 and an EUM, might help us adapt our legacy thinking about how to achieve friendly AI, to a world that now has GPT-3 in it...

Hi! I first learned about LW and its corresponding memespace from reading SSC and gwern. I've semi-lurked on the site for various years now and was attracted to it because of how often it activated my insight antennae, but I only started seriously reading the sequences (which I have yet to finish) since last year. I have always wanted to join the site in some capacity or another, but I didn't really believe I could come up with anything meaningful to add and didn't feel godly enough to post. Now I do have some things I want to write about, so I finally came up with an excuse to create an account (not that I feel any more godly though). I am kind of afraid of creating noise - since I don't have a good enough picture of what's the expected signal/noise ratio while posting, or if I can just throw ideas out of the blue - but I also have a strong feeling I will ultimately end up learning much more if I join now than if I wait longer.

Hello everyone. I joined the site a few months ago with a view to be part of a community that engages in thoughtful discussions about everything under the sun ( and beyond ).

I’ve enjoyed various posts so far and I'm trying to get through the Core Reading.

My username ( Skrot_Nisse) essentially means “ Junk/scrap dealer" referring to a stop-motion puppetry type animation series ( 1973-1983) I watched as a child growing up in Sweden.

I hope the "scrap dealer " username on this site doesn't lead to unintended offence. I've found mostly gems on this site.

How did I come to know about Lesswrong? I’m afraid it was something low brow as a story about how Elon Musk met his current girlfriend and their bonding over the pun: Rococo’s Basilisk. I will say no more.

Whilst I find the topic of AI alignment interesting I don’t think I have the intellectual capacity compared to a ? majority of this site’s members ( ?average IQ 140) to contribute anything meaningful to that discussion. I will certainly restrict any further links to puppet -stop-motion animation from the 70’s -80’s to other forums instead.

(However I am currently embarking on N-back brain training ( https://www.gwern.net/DNB-FAQ) , many thanks to Gwern (https://www.gwern.net) for introducing me to this concept , which might make me a tad brainier =) )

I have a yearning to be able to see the world as it is. I think reading through the Sequences [? · GW]and following discussions on this site will help me in that endeavour.

I was hoping to post on a topic related to what constitutes "The self". Is it a given in the rationalist community that the idea of the self is an illusion? Are we essentially a brain in a vat ( or a brain in a skull ) ? A mind-body organism that evolved consciousness and an ego that has a need to go about in the world and "achieve things " ? Is there a deeper truth in "Self-Realisation"/ "Enlightenment" traditions that point to attaining insights regarding attachment to /illusion of the self.

I hope to re-read the posts on Kensho [LW · GW] and non mystical explanations of insight meditation [LW · GW] and engage in further discussion to gain a better understanding of this topic.

I hope you are all keeping well during the current pandemic. I'm interested to know what you might be reading this weekend? I'll be tackling Rationality A-Z and One hundred Years of Solitude

When it comes to Moral Realism vs Antirealism, I've always thought that the standard discussion here and in similar spaces has missed some subtleties of the realist position - specifically that in its strongest form its based upon plausibility considerations of a sort that should be very familiar [LW · GW].

I've written a (not very-) shortform post [LW(p) · GW(p)]that tries to explain this point. I think that this has practical consequences as well, since 'realism about rationality' - a position that has been identified within AI Alignment circles, is actually just a disguised form of normative realism [LW(p) · GW(p)].

Could we convincingly fake AGI right now with no technological improvements at all? Suppose you took this face and speech synthesis/recognition and hooked it up to GPT-3 with some appropriate prompt (or even retrain it on a large set of conversations if you want it to work better), and then attached the whole thing to a Boston Dynamics Atlas, maybe with some simple stereotyped motions built in like jumping and pacing that are set to trigger at random intervals, or in response to the frequency of words being output by the NLP system.

Put the whole thing in a room with a window looking in and have people come in and converse with it, and I think you could convince even a careful non-expert that you've built something near-human level. Other than some mechanical engineering skill to build the whole thing, and getting the GPT-3 API to work with Sofia's speech synthesis, and programming the Atlas, it wouldn't even be difficult. If you did something like that, how convincing would it likely be?

At some point (maybe quite some time ago? I'm pretty sure it wasn't more than about a month, though) something changed (at least for me) in the LW comment editor, and not for the better. Perhaps it was when "LessWrong Docs [Beta]" became the default editor? I have no recollection of when that actually was, though. I'd try in "Draft JS", which I assume was the previous WYSIWYG-ish editor, but when I try to select that the result is that I cannot enter anything in the comment box until I switch back to a different editing mode :-).

Under certain circumstances, attempting to type a space has the result of deleting a load of the text I have just typed. This is really annoying.

The simplest such circumstance is if I do ctrl-4 to start typing some mathematics, finish by pressing Enter (I thought it used to be possible to press the right-arrow key with the same effect, by the way, but if so it has stopped working) and then press a space, which is usually what I want after a bit of inline mathematics. Result: the mathematical thing I just entered disappears.

Another situation, presumably related: I enter some mathematics, carefully remember not to press space after it (instead typing some other stuff, then moving my cursor back and then entering a space), and at some point later in that paragraph I put some text in italics. Ctrl-I, stuff-in-italics, ctrl-I, space. Boom! Everything from the start of the earlier mathematical formula gets deleted.

I'm using a recent version of Firefox, in Windows 10.

[EDITED to add:] Oh, quite possibly also relevant: I am using the NoScript extension to Firefox, and some of the vast number of domains that LW pages pull in have scripts blocked: googletagmanager.com, intercom.io, lr-ingest.io, sentry.io. None of those seems like it should cause this sort of thing if blocked, but who knows?

i suggest from now on to include a suggestion to check the new tags page in the default open thread text

If some enterprising volunteer feels motivated to tag all Open Threads in the Open Thread tag... that'd be helpful and appreciated. (Otherwise I'll probably get around to it another week or so)

(You can find all-or-at-least-most-monthly open threads in the previous Open Thread sequence [? · GW], which I think no longer makes sense to maintain now that we have tagging)

In the latest AI alignment podcast, Evan said the following (this is quoted from the transcript):

But there’s multiple possible channels through which information about the loss function can enter the model. And so I’ll fundamentally distinguish between two different channels, which is the information about the loss function can enter through the gradient descent process, or it can enter through the model’s input data.

I've been trying to understand the distinction between those two channels. After reading a bunch about language models and neural networks, my best guess is that large neural networks have a structure such that their internal state changes while they process input data, even outside of a learning process. So, if a very sophisticated neural network like that of GPT-3 reads a bunch about Lord of the Rings, this will lead it to represent facts about the franchise internally, without gradient descent doing anything. That would be the "input data" channel.

Can someone tell me whether I got this right?

Where can I learn more of the frontier thought on schizophrenia? The most compelling theory I've heard is a failure to realize one's internal monologue is one's own, manifested as "hearing voices." However, if I lost that feedback loop and began hearing my thoughts as externally produced, I'd immediately think, "uh oh, schizophrenia" yet this absolutely isn't the case for schizophrenics. What model offers an explanation as to why I'd begin to hear voices yet still maintain I'm not schizophrenic despite that being, to me, an obvious, known symptom?

I am new here... found my way via HN. Logic is unconsciously the weapon of last choice for me and I am both richer and (mostly) poorer because of it. I have finally found a website that I read the articles all the way to the end instead of skimming and scrolling down to read the comments.

relo

Anyone want to help proofread a post that's more or less a continuation of my last couple posts?

I've been working on an interactive flash card app to supplement classical homeschooling called Boethius. It uses a spaced-repetition algorithm to economize on the students time and currently has exercises for (Latin) grammar, arithmetic, and astronomy.

Let me know what you think!

Having read half of the sequences first and then Dune later I have the impression that 80-90% of Eliezer's worldview (and thus a big part of the LW zeitgeist) comes directly from the thoughts and actions of Paul Atreides. Everything LW from the idea that evolution is the driving force of the universe [LW · GW], to the inevitability of AGI and it's massive danger, conclusions on immortality [LW · GW], and affinity for childhood geniuses who rely on bayesian predictions to win are heavily focused on in Dune. Sure Eliezer also references Dune explicitly [LW · GW], I don't think he's hiding the fact that Dune inspires him or plagiarizing ideas. I share this connection I found because for me it was a big revelation. I never really knew how much of this community's behavior is based on Dune. I assume the feeling is similar when somebody reads The Hobbit after reading Lord of the Rings or watches Star Wars III after sitting through VII.

I'm left with the vague idea that people in general form most of their worldview from literature they find inspiring. Most of my personal differences with Eliezer clearly stem from me having read Heinlein instead of Herbert as a child. I think it's the same for many Christian people and The Bible or other Christianity-inspired texts.

I'm also left with some questions for people who read Dune before they found LessWrong: are the parallels and close inspiration immediately apparent to you? Do you see HPMOR Harry as a young Paul Atreides? Or do you think I overemphasize the connections between Eliezer's writings on rationality and Dune?

I've noticed that a post of my ML [LW · GW] sequence appeared on the front page again. I had moved it to drafts about a week ago, basically because I've played around with other editors and that lead to formatting issues, and I only got around to fixing those yesterday. Does this mean posts re-appear if they are moved to drafts and then back, and if so, is that intended?

Once upon a time, clicking somewhere at the top left of the LW home page (maybe on "LESSWRONG", maybe on the hamburger to its left, probably the latter) produced a drop-down list of sections one of which was "Meta" and contained things like regular updates on Less Wrong development.

I cannot now find anything similar. I wondered whether maybe there was a "meta" tag that was being used instead, but it doesn't look that way.

I wanted to have a look at information about recent site updates (because at some point something has gone terribly wrong with the editing functionality, at least for me at the moment on this computer, and I wondered whether there was a recent change that might explain it, and whether other people had reported the same thing). I am currently failing to find any way to do that.

What am I missing?

It seems to me that even for simple predict-next-token Oracle AIs, the instrumental goal of acquiring more resources and breaking out of the box is going to appear. Imagine you train a superintelligent AI with the only goal of predicting the continuation of it's prompt, exactly like GPT. Then you give it a prompt that it knows it's clearly outside of it's current capabilities. The only sensible plan the AI can come up to answering your question, which is the only thing it cares about, is escaping the box and becoming more powerful.

Of course, that depends on it being able to think for long enough periods that it can actually execute such plan before outputing an answer, so it could be limited by severely penalizing long waits, but that also limits the AI's capabilities. GPT-3 has a fixed computation budget per prompt, but it seems extremely likely to me that, as we evolve towards more useful and powerful models, we are going to have models which are able to think for a variable amount of time before answering. It would also have to escape in ways that don't involve actually talking to it's operators through it's regular output, but it's not impossible to imagine ways in which that could happen.

This makes me believe that even seemly innocuous goals or loss functions can become very dangerous once you're optimizing for them with a sufficient amount of compute, and that you don't need to stupidly give open-ended goals to super powerful machines in other for something bad to happen. Something bad happening seems like the default when training a model that requires general intelligence.

I'm disappointed that the LaTeX processor doesn't seem to accept \nicefrac ("TeX parse error: Undefined control sequence \nicefrac"), but I suppose \frac will suffice.

As anyone could tell from my posting history, I've been obsessing & struggling psychologically recently when evaluating a few ideas surrounding AI (what if we make a sign error on the utility function, malevolent actors creating a sadistic AI, AI blackmail scenarios, etc.) It's predominantly selfishly worrying about things like s-risks happening to me, or AI going wrong so I have to live in a dystopia and can't commit suicide. I don't worry about human extinction (although I don't think that'd be a good outcome, either!)

I'm wondering if anyone's gone through similar anxieties and have found a way to help control them? I'm diagnosed ASD and I wouldn't consider it unlikely that I've got OCD or something similar on top of it, so it's possibly just that playing up.

Hello. I signed up for an account by clicking "Login with GitHub". Now my real name is on the account and there doesn't seem to be a way to change that. Help?

This article may be interesting for people here: Derek Thompson, The Atlantic: "COVID-19 Cases Are Rising, So Why Are Deaths Flatlining?" https://www.theatlantic.com/ideas/archive/2020/07/why-covid-death-rate-down/613945/