LessWrong 2.0 Reader

Yeah sorry to be clear totally agree we (or at least I) don’t know the sizes of models, I was just naming specific models to be concrete. 

But anyway yes I think you got my point: the Jones chart illustrates (what I understood to be) gwern’s view that adding more inference/search does juice your performance to some degree, but then those gains taper off. To get to the next higher sigmoid-like curve in the Jones figure, you need to up your parameter count; and then to climb that new sigmoid, you need more search. What Jones didn’t suggest (but gwern seems to be saying) is that you can use your search-enhanced model to produce better quality synthetic data to train a larger model on. 

I'm confident in saying that the information here is not sufficient to conclude she was making a personal attack.

A wild claim to make about a post that explictly centers around shaming Kat for her posting style predominantly because it's "cringe", by putting her name in  the headline (what I meant by name-calling) and orienting the entire argument around her. If it wasn't meant to tarnish her reputation, why not instead make the post about just her issues with the disagreeable content? 

If she said that was her intent, I'd change my mind. Or if she said something that was unambiguously a personal attack I'd change my mind, but at the moment I see no reason not to read the post as well meaning innocent criticism...I also don't think it's very appropriate for this forum. 

If intent was determined solely by someone's words than I'd agree with you that her caveat made in passing indicates well-meaningness, but the standard you've set for determining intent is as naive as believing when someone says "I like you as a person" before they knock you out with a punch that they do not intend to cause you harm. 

Even by her own words this is not "innocent criticism", because she states that she dislikes the "deluge of posts" and her "futile downvotes accomplish nothing, so instead I am writing this blog post" which indicate she elevated her response to bring about behavior change, not by using the methods native to Reddit, but rather by leaning on those friendly to her position in the group to shame Kat into compliance. 

Before you misquote me as analogizing this post to a punch, let me be clear that I'm merely observing you are clinging so minutely to small signals that are arguably misdirections, that you're missing the extent to which browsing's post is more than "inappropriate" as you acknowledge, because as I said it amounts to bullying someone to get in line on another forum, because it's fun to do. Why don't you hold yourself to a higher standard?

In addition to the object level reasons mentioned by plex, misleading people about the nature of a benchmark is a problem because it is dishonest. Having an agreement to keep this secret indicates that the deception was more likely intentional on OpenAI's part.

If we talk about the quote at the beginning, then its final conclusion seems to me not entirely correct.
What the vast majority of people mean by "emotions" is different from "rational functions of emotions". Yudkowsky in his essay on emotions is playing with words, using terms that are not quite traditional.
Fear is not "I calmly foresee the negative consequences of some actions and therefore I avoid them."
Fear is rather "The thought of the possibility of some negative events makes me tremble, I have useless reflections, I have cognitive distortions that make me unreasonably overestimate (or, conversely, sometimes underestimate) the probability of these negative events, I begin to feel aggression towards sources of information about the possibility of these negative events (and much more in the same spirit)."
Emotions in the human understanding are not at all the same as the rational influence of basic values ​​on behavior in Yudkowsky's interpretation.
Emotions in the human understanding are, first of all, a mad hodgepodge of cognitive distortions.
Therefore, when Yudkowsky says something like "Why do you think that AI will be emotionless? After all, it will have values!", I even see some manipulation here. Well, yes, AI will have values ​​​​influencing behavior. But at the same time, it will not be nervous, freak out, or experience the halo effect. This is absolutely not what a normal ordinary person would call emotions. In fact, here Yudkowsky's imaginary opponents are closer to the truth, depicting AI as dispassionate and emotionless (because the uniform influence of values ​​on behavior without peaks and troughs should look exactly like that).
Does it matter?
It depends. When communicating with ordinary people, we are used to using their and our cognitive distortions. When talking to a person, you know that you can suddenly change the topic of conversation and influence the emotions of the interlocutor. In communication with AI (powerful enough and having managed to modify itself well), all this will not work. It is like trying to outsmart God.
Therefore, it seems to me that a person who tunes himself to thought "I am communicating with an impassive inhuman being" will in some sense be closer to the truth (at least, will have fewer false subconscious hopes) than a person who tunes himself to thought "I am communicating with the same living emotional sympathetic subject that I am." But this is context-dependent.

Years ago, I advocated banning crypto as a means of limiting the damage AI could do, thinking an advanced AI might be able to mine or hack exchanges (eg by guessing passwords of "lost" bitcoins) and accumulate wealth (ie power).

Apparent it could also just make a meme coin, and generate billions from nothing, given a sufficiently edgy coin (for example, AI itself)

https://www.axios.com/2025/01/19/donald-trump-crypto-billionaire

I am once again humbly suggesting that all un-regulated currency, especially distributed ledgers, be banned worldwide as a precautionary measure.

I consider the lattice to be a regulator as well, but, semantics aside, thank you for the example.

What application do you have in mind? If you're trying to reason about formal models without trying to completely rigorously prove things about them, then I think thinking of neural networks as stochastic systems is the way to go. Namely, you view the weights as a random variable solving a stochastic optimization problem to produce a weight-valued random variable, then conditioning it on whatever knowledge about the weights/activations you assume is available. This can be done both in the Bayesian "thermostatic" sense as a model of idealized networks, and in the sense of modeling the NN as SGD-like systems. Both methods are explored explicitly (and give different results) in suitable high width limits by the PDLT and tensor networks paradigms (the latter also looks at "true SGD" with nonnegligible step size). 

Here you should be careful about what you condition on, as conditioning on exact knowledge of too much input-output behavior of course blows stuff up, and you should think of a way of coarse-graining, i.e. "choose a precision scale [LW · GW]" :). Here my first goto would be to assume the tempered Boltzmann distribution on the loss at an appropriate choice of temperature for what you're studying.

If you're trying to do experiments, then I would suspect that a lot of the time you can just blindly throw whatever ML-ish tools you'd use in an underdetermined, "true inference" context and they'll just work (with suitable choices of hyperparameters) 

I think the website just links back to this blog post? Is that intentional?

Edit: I also think the application link requires access before seeing the form?

Just an update.  So far, nothing interesting has happened.  

I've got some more thorough tests I'm working on in my spare time.  
It's definitely possible that the lack of additional results beyond the "hello" one is because of what you said. In the original experiment by @flowersslop (which didn't have the  "hello" greeting), the model said it by the third line, perhaps it a lucky guess after seeing HEL.  Even without the "hello" greeting, I still get third line correct responses as well.

But I haven't had any luck with any less common words yet.  I'm still going to try a bit more experimentation on this front though.  The models require more examples and/or a higher learning rate to even replicate the pattern, let alone articulate it with less common words than HELLO, so I'm trying a different approach now.  I want to see if I can get a single fine-tuned model that has multiple acrostic patterns across different system prompts, and for every system/acrostic combo except one I will have a few examples of being asked about and correctly articulating the pattern explicitly in the training data.  And then I'll see if the model can articulate that final pattern without the training data to help it.  

If there is any emergent meta-awareness (which I've now seen a couple of papers hinting at something similar) happening here, I'm hoping this can coax it out of the model.

Contempt of court penalties for noncompliance with an investigative process is a mainstream example of 1.

Burning Man has some aspects of the second, as do some camping trips, or simply living in a relatively harsh climate. Compare measured levels of corruption in southern vs northern Europe, for instance. When modern democracies fight big wars, the first year involves learning which parts of their warfighting institutions are corrupt and incompetent, & repairing or replacing them.