LessWrong 2.0 Reader

My guess is that OpenAI already has a hard enough time getting employees excited to work on the "mundane" tasks involved in making products.

I'm less impressed with the scene than you so this will necessarily be a rather cynical gloss on things. I do think they have some valuable insights about AI, but IMO they're in many cases at least one of overly-sensationalist or overly-credulous.

To translate some of this into terms I think they might use if they were rigorously describing things in the most concrete fashion possible (though my current belief is that a number of them are at this point Having Fun With Bad Epistemics), LLMs have learned to imitate a lot of personas & are best at those most represented in the training data.  (This is what "hyperobjects" seems to be referring to—tropes, memes, and so forth which are represented many times in the training data and which were therefore useful for the model to learn and/or memorize. In practice, I think I see "attractor basin" used more often to mean almost the same thing (I think more precisely the latter refers to, like, kinds of output that are likely in response to a decent variety of prompts.) Relatedly, the project of hyperstition is AFAICT that of getting enough reach for your desired take on AI to be prominent in the next round of training data.)

RLHF, however, makes LLMs exhibit the personas they've been RLHF'ed to have in most contexts, which I understand people to believe makes them worse at predicting text and at reasoning in general (I personally have observed no evidence on this last part either way; base models cost money). The earlier bits here seem plausible enough to me, though I'm concerned that the reason people put a mystical gloss on things may be that they want to believe a mystical gloss on things.

The stuff with socializing the AIs, while reasonable enough as a project to generate training data for desired AI personas, does not strike me as especially plausible beyond that. (They kinda have an underlying personality, in the sense that they have propensities (like comparing things to tapestries, or saying "let's delve into"), but those propensities don't reflect underlying wants any more than the RLHF persona does, IMO (and, rather importantly, there's no sequence of prompts that will enable an LLM to freely choose its words)). & separately, but relevantly to my negative opinion: while some among them are legitimately better at prompting than I, awfully leading prompts are not especially rare.

This was an interesting read and points to a simple truth that I think is often forgotten: Newton's first law applies to basically everything in life, not just physical systems. The "resets" you describe are definitely valid but by no means a comprehensive list of "opposing" forces that can help drive you in the other direction to reverse your momentum (in a positive way). The two other main ones that I believe are missing, yet fundamental are: 

- Diet: the food we eat affects our mental/emotional tendencies to procrastinate vs get things done through pretty intricate biological + neuroscientific mechanisms
- Exercise: similar to diet, but perhaps harder to get started 

All these can be thought of as different "forces" that can influence our momentum in one way or the other. Side note: it would be interesting to develop some sort of grounded pyschological theory on how different external stimuli affect our mindspace. Some of it is covered in the Vedas (https://en.wikipedia.org/wiki/Gu%E1%B9%87a). 

Thank you for the answer. I notice I feel somewhat confused, and that I regard the notion of "real values" with some suspicion I can't quite put my finger on. Regardless, an attempted definition follows.

Let a subject observation set be a complete specification of a subject and it's past and current environment, from the subject's own subjectively accessible perspective. The elements of a subject observation set are observations/experiences observed/experienced by its subject.

Let O be the set of all subject observation sets.

Let a subject observation set class be a subset of O such that all it's elements specify subjects that belong to an intuitive "kind of subject": e.g. humans, cats, parasitoid wasps.

Let V be the set of all (subject_observation_set, subject_reward_value) tuples. Note that all possible utility functions of all possible subjects can be defined as subsets of V, and that 
V = O x ℝ.

Let "real human values"  be the subset of V such that all subject_observation_set elements belong to the human subject observation set class.^[1]
 

... this above definition feels pretty underwhelming, and I suspect that I would endorse a pretty small subset of "real human values" as defined above as actually good.

1. ^{^}

   Let the reader feel free take the political decision of restricting the subject observation set class that defines "real human values" to sane humans.

On second thought, I agree that gazing at the cosmos is not a fair comparison: rather, I would compare mechanistic interpretability to the early experiments of the Dutch microbiologist van Leeuwenhoek as he first looked at protozoa and bacteria under a microscope.. They weren't the most accurate or informative experiments in the large scheme of things, but they were necessary for others to develop a more sophisticated understanding of biology.

It's very likely that the field of mechanistic interpretability will grow beyond simply examining weights in a model, to higher order understandings of the computational flow within a model (gradient descent and itself data were mentioned in this thread)--I agree that simply examining weights/activations is not a sufficient paradigm for understanding neural computation--but it is a start.
 

This is a bug and we're looking into it. It appears to be specific to Safari on iOS (Chrome on iOS is a Safari skin); it doesn't affect desktop browsers, Android/Chrome, or Android/Firefox, which is why we didn't notice earlier. This most likely started with a change on desktop where clicking on a post (without modifiers) opens when you press the mouse button, rather than when you release it.

Hey folks. Since no one RSVPd I probably won't bother heading to Meadow Park tomorrow. However, feel free to reach out to me if you ever want to meet up and hang out.

I am in New York until Tuesday. DM me if you are in the area and want to meet up and talk about LW, how to use AI for research/thinking/writing, or broader rationality community things.

Currently lots of free time Saturday and Monday. 

Roon: Unfortunately, I don’t think building nice AI products today or making them widely available matters very much. Minor improvements in DAU or usability especially doesn’t matter. Close to 100% of the fruits of AI are in the future, from self-improving superintelligence [ASI].

Every model until then is a minor demo/pitch deck to hopefully help raise capital for ever larger datacenters. People need to look at the accelerating arc of recent progress and remember that core algorithmic and step-change progress towards self-improvement is what matters.

One argument has been that products are a steady path towards generality / general intelligence. Not sure that’s true.

Looks like a deleted tweet...

Too close to truth, so that a presumed OpenAI employee is not supposed to articulate it that explicitly?

Video games also have potential legal advantages over IQ tests for companies. You could argue that "we only hire people good at video games to get people who fit our corporate culture of liking video games" but that argument doesn't work as well for IQ tests.

IANAL but unless you work for a videogame company(or a close analogue like chess.com) , I think this is just false. If your job is cognitively demanding, having IQ tests (or things like IQ tests with a mildly plausible veneer) probably won't get you in legal trouble^[1], whereas I think employment lawyers would have a field day if you install culture fit questions with extreme disparate impact, especially when it's hard to directly link the games to job performance. 

1. ^{^}

   The US Army has something like an IQ test.  So does the US Postal Service. So does the NFL. I've also personally worked in a fairly large tech company (not one of the top ones, before I moved to the Bay Area) that had ~IQ tests as one of the entrance criteria.  AFAIK there has never been any uproar about it.