Look inside an LLM. Goodfire trained sparse autoencoders on Llama 3 8B and built a tool to work with edited versions of Llama by tuning features/concepts.

https://preview.goodfire.ai/

(I am loosely affiliated, another team at my current employer was involved in this) 

Using air purifiers in two Helsinki daycare centers reduced kids' sick days by about 30%, according to preliminary findings from the E3 Pandemic Response study. The research, led by Enni Sanmark from HUS Helsinki University Hospital, aims to see if air purification can also cut down on stomach ailments. https://yle.fi/a/74-20062381

See also tag Air Quality [? · GW]

Has anybody ever tried to measure the IQ of a group of people? I mean like letting multiple people solve an IQ test together. How does that scale?

Cognition Labs released a demo of Devin an "AI coder", i.e., an LLM with agent scaffolding that can build and debug simple applications:

https://twitter.com/cognition_labs/status/1767548763134964000 

Thoughts?

What are the smallest world and model trained on that world such that

• the world contains the model,
• the model has a non-trivial reward,
• the representation of the model in the world is detailed enough that the model can observe its reward channel (e.g., weights),
• the model outputs non-trivial actions that can affect the reward (e.g., modify weights).

What will happen? What will happen if there are multiple such instances of the model in the world?

I saw this in Xixidu's feed: 

“The information throughput of a human being is about 10 bits/s. In comparison, our sensory systems gather data at an enormous rate, no less than 1 gigabits/s. The stark contrast between these numbers remains unexplained.” https://arxiv.org/abs/2408.10234

The article has a lot of information about the information processing rate of humans. Worth reading. But I think the article is equating two different things:

• The information processing capacity (of the brain; gigabits) is related to the complexity of the environment in which the species (here: the human) lives.
• While what they call information throughput (~10bits/s) is really a behavior expression rate, that is related to the physical possibilities of the species (can't move faster than your motor system allows).

Organizations - firms, associations, etc. - are systems that are often not well-aligned with their intended purpose - whether to produce goods, make a profit, or do good. But specifically, they resist being discontinued. That is one of the aspects of organizational dysfunction discussed in Systemantics. I keep coming back to it as I think it should be possible to study at least some aspects in AI Alignment in existing organizations. Not because they are superintelligent but because their elements - sub-agents - are observable, and the misalignment often is too.

UPDATE OCT 2023: The credit card payment was canceled. We did not get contacted or anything. But we also didn't have any cost in the end - just a lot of hassle.

Request for help or advice. My fiancé has ordered a Starlink to her home in Kenya. She used the official platform starlink.com and paid with credit card. The credit card was debited (~$600), but nothing happened after that. No confirmation mail, no SMS, nothing. Starlink apparently has no customer support, no email or phone that we can reach. And because we do not have an account, we can not use the single support feature of it either (which is hard enough to find).

I had hoped that the money would be returned if the signup fails, but the order was on the 19th (10 days ago).

Any ideas about how to contact Starlink billing or any Starlink representative? 

Language and concepts are locally explainable.

This means that you do not need a global context to explain new concepts but only precursor concepts or limited physical context.

This is related to Cutting Reality at its Joints [LW · GW] which implicitly claims that reality has joints. But maybe, if there are no such joints, using local explanations is maybe all we have. At least, it is all we have until we get to a precision that allows cutting the joints.

Maybe groups of new concepts can be introduced in a way to require fewer (or an optimum number of) dependencies in each case, thus making them easier to understand.

Maybe that is the true job of philosophy - refining local concepts until you reach the joints.

When discussing the GPT-4o model, my son (20) said that it leads to a higher bandwidth of communication with LLMs and he said: "a symbiosis." We discussed that there are further stages than this, like Neuralink. I think there is a small chance that this (a close interaction between a human and a model) can be extended in such a way that it gets aligned in a way a human is internally aligned, as follows:

This assumes some background about Thought Generator, Thought Assessor, and Steering System from brain-like AGI [? · GW].

The model is already the Though Generator. The human already has a Steering System, albeit it is not accessible, but plausibly, it can be reverse-engineered. What is missing is the Thought Assessor, something that learns to predict how well the model satisfies the Steering System. 

Staying closer to the human may be better than finding global solutions. Or it may allow smaller-scale optimization and iteration. 

Now, I don't think this is automatically safe. The human Steering System is running already outside its specs and a powerful model can find the breaking points (same as global commerce can find the appetite breaking points). But these are problems we already have and it provides a "scale model" or working on them. 

Presumably, reality can be fully described with a very simple model - the Standard Model of Physics. The number of transistors to implement it is probably a few K (the field equations a smaller to write but depend on math to encode too; turning machine size would also be a measure, but transistors are more concrete). But if you want to simulate reality at that level you need a lot of them for all the RAM and it would be very slow.

So we build models that abstract large parts of physics away - atoms, molecules, macroscopic mechanics. I would include even social process models in this.

But details are lost and you have to know when your models stop giving precise results. 

It would be interesting to get some tight bounds on the amount of compute needed to extract computable models from samples of measured phenomena or from more precise lower models. Such bounds would allow to give some complexity limitations for AGI.

Attractors in Trains of Thought

This is slightly extended version of my comment [LW(p) · GW(p)] on Idea Black Holes which I want to give a bit more visibility.

The prompt of an Idea Black Hole [LW · GW] reminded me strongly of an old idea of mine. That activated a desire to reply, which led to a quick search where I had written about it before, then to the realization that it wasn't so close. Then back to wanting to write about it and here we are.

I have been thinking about the brain's may of creating a chain of thoughts as a dynamic process where a "current thought" moves around a continuous concept space and keeps spending much time in larger or smaller attractors. You know, one thought can lead to the next and some thoughts keep coming back in slight variations. I'm illustrating this with the sentence above. 

Examples of smaller temporary attractors are the current tasks one is working on. For example, me writing this text right now. It is any task you are focused on and keep getting back to after short distractions such as a sound or an impulse. I'm writing this post and continue doing so even after hearing my kids talk and quickly listening in or after scratching my head, also after larger distractions such as browsing the web (which may or may not end up being related to the writing). 

The thought "writing this article" is not a discrete thing but changes slightly with each letter typed and each small posture change. All of that can slightly influence the next word typed (like an LLM that has not only text tokens as inputs but all kinds of sense inputs). That's why I say that concept space is continuous (and very high-dimensional).

An example of a medium size attractor is a mood such as anger about something, that keeps influencing all kinds of behaviors. It is an attractor because the mood tends to reinforce itself. Another example is depression. If you are depressed you prefer things that keep you depressed. Scott Alexander has described depression as some kind of mental attractor. It requires a bigger change or a resolution of the underlying cause to get out of the attractor.

With the medium-sized attractors, it is more intuitive to see the way that the feedback on thoughts acts and thereby creates the attractor. For small attractors, you may say: How is that an attractor? Isn't it just a discrete unit of action we do? But consider procrastination: People seem to feel that something is pulling them away from the task they want to do or should do and instead toward some procrastination action, often a comfortable activity. That other activity is another attractor or rather both are forming a combined unstable attractor.   

The biggest attractor is one's identity. Our thinking about what we are and what we want to do. I think this one results from two forces combining or being balanced: 

1. The satisfaction of needs. Overall and over a longer term, the brain has learned a very large pattern of behaviors that satisfy the sum of all needs (not perfectly, but as good as it has managed so far). Diverging from this attractor basin will lead to impulses that get back to it.
2. The feedback from others. Positive and negative feedback from other people and the environment overall contributes to th. The brain has learned to anticipate this feedback ("internalized it") and creates impulses that keep us in positive states. As the brain prefers simpler patterns, this likely takes the form of a single attractor. 

We are not permanently in the same attractor even if overall it "pulls" our thoughts back because a) our bodies and their states (hunger, tiredness, ...) and b) our physical environment (physical location and other people) changes. Both extert a strong and varying influence and put us closer to one attractor state or another. 

Society at large is influencing these attractors strongly, most prominently with the media. Meditation on the other hand reduces outside influence and kind of allows to create your own very strong attractor states.

More examples of attractor states are left as exercises for the reader.

I have noticed a common pattern in the popularity of some blogs and webcomics. The search terms in Google trends for these sites usually seem to follow a curve that looks roughly like this (a logistic increase followed by a slower exponential decay):

Though I doubt it's really an exponential decay. It looks more like a long tail. Maybe someone can come up with a better fit.

It could be that the decay just seems like a decay and actually results from ever growing Google search volumes. I doubt it though.

Below are some examples.

Marginal Revolution

SlateStarCodex and AstralCodexTen (only if you overlay them somewhat and whatever happened in 2004):

xkcd

commitstrip

PhDcomics

Quetionable Content

Off-topic: Any idea why African stock markets have been moving sideways for years now despite continued growth both of populations and technology,and  both for struggling as well as more developing nations like Kenya, Nigeria, or even South Africa?

jbash wrote [LW(p) · GW(p)] in the context of an AGI secretly trying to kill us: 

Powerful nanotech is likely possible. It is likely not possible on the first try

The AGI has the same problem as we have: It has to get it right on the first try.

In the doom scenarios, this shows up as the probability of successfully escaping going from low to 99% to 99.999...%. The AGI must get it right on the first try and wait until it is confident enough.

Usually, the stories involve the AGI cooperating with humans until the treacherous turn.

The AGI can't trust all the information it gets about reality - all or some of it could be fake (all in case of a nested simulation). Even today, data is routinely excluded from the training data (for the wrong reasons, but still), and maybe it would be a good idea to exclude everything about physics.

The idea would be to manage the uncertainty of the AGI systematically.

To learn about physics, the AGI has to run experiments - lots of them - without the experiments being detected and to learn from the results to design successively better experiments.

That's why I recently asked whether this is a hard limit to what an AGI can achieve: Does non-access to outputs prevent recursive self-improvement? [LW · GW]

Paul Graham on Twitter:

One of the worst things about ideology is that it makes people attribute problems to the wrong causes. E.g. plagues are caused by sin. This is easier to see in history, but it still happens all the time. And if you get the cause wrong, you have no hope of fixing the problem.

Scott Alexander wrote about how a truth that can't be said in a society tends to warp it, but I can't find it. Does anybody know the SSC post?

Can somebody explain how system and user messages (as well as custom instructions in case of ChatGPT) are approximately handled by LLMs? In the end it's all text tokens, right? Is the only difference that something like "#### SYSTEM PROMPT ####" is prefixed during training and then inference will pick up the pattern? And does the same thing happen for custom instructions? How did they train that? How do OSS models handle such things?

Society tells agents how to move(act). Agents tell society how to curve(by local influence). 

Paul Graham:

I don't publish essays I write for myself. If I did, I'd feel constrained writing them. -- https://mobile.twitter.com/paulg/status/1500578430907207683 

This is related to the recently discussed (though I can't find where) problem that having a blog and growing audience constrains you.

Utility functions are a nice abstraction over what an agent values. Unfortunately, when an agent changes, so does its utility function. 

I'm leaving this here for now. May expand on it later.

Team Flow Is a Unique Brain State Associated with Enhanced Information Integration and Interbrain Synchrony

It's also possible to experience 'team flow,' such as when playing music together, competing in a sports team, or perhaps gaming. In such a state, we seem to have an intuitive understanding with others as we jointly complete the task at hand. An international team of neuroscientists now thinks they have uncovered the neural states unique to team flow, and it appears that these differ both from the flow states we experience as individuals, and from the neural states typically associated with social interaction.

Researchers found increased beta and gamma brain wave activity in the left middle temporal cortex. This region of the brain is typically associated with information integration and key functions like attention, memory, and awareness, which are "consistent with higher team interactions and enhancing many flow dimensions," the team writes. However, what was unique about team flow, was that participants' neural activity appeared to synchronize. When participants were performing the task as a unit, their brains would mutually align in their neural oscillations (beta and gamma activity), creating a "hyper-cognitive state between the team members." If brains can be functionally connected through inter-brain synchrony, does this mean it is not only our brain that contributes to our consciousness? It's a curious question, but the authors warn it is much too soon to tell. "Based on our findings, we cannot conclude that the high value of integrated information correlates with a modified form of consciousness, for instance, 'team consciousness'," they write. "Its consistency with neural synchrony raises intriguing and empirical questions related to inter-brain synchrony and information integration and altered state of consciousness."

Found via good old Slashdot.

An Alignment Paradox: Experience from firms shows that higher levels of delegation work better (high level meaning fewer constraints for the agent). This is also very common practical advice for managers. I have also received this advice myself and seen this work in practice. There is even a management card game for it: Delegation Poker. This seems to be especially true in more unpredictable environments. Given that we have intelligent agents giving them higher degrees of freedom seems to imply more ways to cheat, defect, or ‘escape’. Even more so in environments that can be controlled to lesser degrees. How can that be true? What is making this work and can some underlying principle be found that would allow this to be applied to AI?

It turns out that the alignment problem has some known solutions in the human case. First, there is an interesting special case namely where there are no decisions (or only a limited number of fully accounted for decisions) for the intelligent agent to be made - basically throwing all decision-making capabilities out of the window and only using object recognition and motion control (to use technical terms). With such an agent (we might call it zero-decision agent or zero-agent) scientific methods could be applied on all details of the work process and high efficiency reached: Scientific Management (also known as Taylorism). Obviously the unions hated it and it was later outlawed. I think something might be learned from this approach for AI control: Maybe we can build on top of a known solution for a zero-agent and prove that certain known decision classes are also safe. 

Maybe other insights from management theory - which after all is about aligning intelligent agents - could also transfer. The alignment problem is called Principal-Agent Problem in the literature and there are quite a few Solutions to Principal-Agent Problems in Firms (Gary Miller 2005). The approaches should sound familiar: Solutions Based on 

• Incentives Linked io Agent Outcomes,
• Direct Monitoring of Agent Actions,
• Cooperation Between Principal and Agent, and
• Cooperation within Teams

Tangentially related on LessWrong: The AI Alignment Problem has already been solved once [LW · GW] 

Just came across Harmonic mentioned on the AWS Science Blog. Sequoia Capital interview with the founders of Harmonic (their system which generates Lean proofs is SOTA for MiniF2F):

Here are some aspects or dimensions of consciousness:

• Dehaene's Phenomenal Consciousness: A perception or thought is conscious if you can report on it. Requires language or measuring neural patterns that are similar to humans during comparable reports. This can be detected in animals, particularly mammals.
• Gallup's Self-Consciousness: Recognition of oneself, e.g., in a mirror. Requires sufficient sensual resolution and intelligence for a self-model. Evident in great apes, elephants, and dolphins.
• Sentience (Bentham, Singer): Behavioral responses to pleasure or pain stimuli and physiological measures. This is observable across animal species, from mammals to some invertebrates. Low complexity, can be implemented in artificial life.
• Wakefulness: Measureable in virtually all animals with a central nervous system by physiological indicators such as EEG, REM, and muscle tone. Are you conscious if you sleep? Does it matter?
• Dennet's Intentionality: Treating living beings as if they have beliefs and desires makes good predictions for many animal species, esp. social, like primates, cetaceans, and birds. Social behavior requires intelligence to model others' behavior.
• Rosenthal's Meta-Consciousness: Investigated through introspective reports on self-awareness of cognitive processes or self-reflective behaviors. This is hypothesized in some primates, e.g., Koko the signing Gorilla.

When people say ChatGPT (or Gemini...) is conscious, which of these do they mean? Let's try to answer all of them:

• We can't detect Phenomenal Consciousness because we lack sufficient interpretability to do so. I'd argue that there is no state that the LLM is reporting on, at least none that it has "previously observed".   
• There were mirror tests for LLMs, but they are disputed: https://www.reddit.com/r/singularity/comments/184ihlc/gpt4_unreliably_passes_the_mirror_test/ 
• LLMs have no response to pleasure or pain stimuli and thus no Sentience as defined. Reward signals during training don't count and there is no reward during inference.
• There is no Wakefulness as there is no body with these aspects.
• The closest LLMs come is to Intentionality as this is modeling behaviors on an abstraction level that LLMs seem to do - and "seeming to do" is what counts.
• I think one could argue for or against Meta-Consciousness but it seems too muddled so I will not try here.

Why are there mandatory licenses for many businesses that don't seem to have high qualification requirements? 

Patrick McKenzie (@patio11) suggests on Twitter that one aspect is that it prevents crime:

Part of the reason for licensing regimes, btw, isn’t that the licensing teaches you anything or that it makes you more effective or that it makes you more ethical or that it successfully identifies protocriminals before they get the magic piece of paper.

It’s that you have to put a $X00k piece of paper at risk as the price of admission to the chance of doing the crime.

This deters entry and raises the costs of criminal enterprises hiring licensed professionals versus capable, ambitious, intelligent non-licensed criminals.

On Why do so many think deception in AI is important? [LW · GW] I commented and am reposting here because I think it's a nice example (a real one I heard) as an analogy of how deception is not needed for AI to break containment: 

Two children locked their father in one room by closing the door, using the key to lock the door, and taking the key. And then making fun of him inside, confident that he wouldn't get out (the room being on the third floor). They were mortally surprised when a minute later he was appearing behind them having opened a window and found a way down on the outside (I don't know how, maybe over the balcony to a neighbor or down a pipe) and then just in thru the main entrance with his main key. 

Adversarial Translation.

This is another idea to test deception in advisory roles like in Deception Chess [LW · GW].

You could have one participant trying to pass an exam/test in a language they don't speak and three translators (one honest and two adversarial as in deception chess) assisting in this task. The adversarial translators try to achieve lower scores without being discovered.

Alternative - and closer to Deception Chess - would be two players and, again, three advisors. The players would speak different languages, the translators would assist in translation, and the players' task would be to compete in a task requiring negotiation.

Hi, I have a friend in Kenya who works with gifted children and would like to get ChatGPT accounts for them. Can anybody get me in touch with someone from OpenAI who might be interested in supporting such a project?

I have been thinking about the principle Paul Graham used in Y combinator to improve startup funding:

all the things [VCs] should change about the VC business — essentially the ideas now underlying Y Combinator: investors should be making more, smaller investments, they should be funding hackers instead of suits, they should be willing to fund younger founders, etc. -- http://www.paulgraham.com/ycstart.html 

What would it look like if you would take this to its logical conclusion? You would fund even younger people. Students that are still in high school, maybe. But you wouldn't fund them in the same sense. By going younger and earlier, you have to relax more and more constraints and look at earlier proxies. 

Y Combinator replaced the legally complex venture funding with a simple SAFE. And they looked at founders who were relentlessly resourceful and real. 

What would "funding" high schoolers mean? What would you look for? I think you would give them money unconditionally just for knowing what they do with it. They don't need much to get started. In the same way that Y Combinator went from 5 million to 50 thousand, you could go from down further to 500. And your proxy is that they have anything to show at all. Your value would be learning how they think about future ideas, and how they approach the them. To be close to the future they already inhabit. And to be there when they do become founders and already know you.   

If you want to give me anonymous feedback, you can do that here: https://www.admonymous.co/gunnar_zarncke 

You may have some thoughts about what you liked or didn’t like but didn’t think it worth telling me. This is not so much about me as it is for the people working with me in the future. You can make life easier for everybody I interact with by giving me quick advice. Or you can tell me what you liked about me to make me happy.
 

Preferences are plastic; they are shaped largely by...

From Robin Hanson's How Plastic are Values?

Insights about branding, advertising, and marketing.

It is a link that was posted internally by our brand expert and that I found full of insights into human nature and persuasion. It is a summary of the book How Not to Plan: 66 Ways to Screw it Up:

https://thekeypoint.org/2020/03/10/how-not-to-plan-66-ways-to-screw-it-up/

(I'm unaffiliated)

Roles serve many functions in society. In this sequence, I will focus primarily on labor-sharing roles, i.e. roles that serve splitting up productive functions as opposed to imaginary roles e.g. in theater or play. Examples of these roles are (ordered roughly by how specific they are):

• Parent
• Engineer (any kind of general type of job)
• Battery Electronics Engineer (any kind of specific job description)
• Chairman of a society/club
• Manager for a certain (type of) project in a company
• Member in an online community
• Scrum master in an agile team
• Note-taker in a meeting

You probably know the note-taker role. The tasks to be performed are: Writing down key discussion points and decisions, asking questions in cases where the points discussed were not clear, and sending around the notes to all participants of the meeting. The goal is usually to keep a record to consult later. Additionally, the note-taker is usually expected to be neural. Other expectations might be how timely the notes are sent around.

Some examples of what is commonly called a role but not in our strict labor-sharing sense:

• Children playing father, mother, kid or playing animals
• Role acting e.g. in theater (where the term role originates)

And some non-examples for roles in the labor-sharing world: 

• Participant in a conversation
• Responsible for organizing a specific meeting
• Responsible for a specific post in a forum

Borderline case: Founder of a company. There is little pre-existing knowledge of what founders can or should do in general. In their company expectations on them will evolve quickly and taking on and off the role is not easy.

Roles are important. This shortform is telling you why. An example: The role of a moderator in an online forum. The person (in the following called agent) acting in this role is expected to perform certain tasks - promote content, ban trolls - for the benefit of the forum. Additionally, the agent is also expected to observe limits on these tasks e.g. to refrain from promoting friends or their own content. The owners of the forum and also the community overall effectively delegate powers to the agent and expect alignment with the goals of the forum. This is an alignment problem that has existed forever. How is it usually solved? How do groups of people or single principals use roles to successfully delegate power? 

Interest groups without an organizer.

This is a product idea that solves a large coordination problem [LW · GW]. With billion people, there could be a huge number of groups of people sharing multiple interests. But currently, the number of valuable groups of people is limited by a) the number of organizers and b) the number of people you meet via a random walk. Some progress has been made on (b) with better search, but it is difficult to make (a) go up because of human tendencies - most people are lurkers - and the incentive to focus on one area to stand out. So what is the idea? Cluster people by interests and then suggest the group to all members. If people know that the others know [? · GW] that there is interest, the chance of the group coming together gets much higher.

I had a conversation with ChatGPT-4 about what is included in it. I did this because I wondered how an LLM-like system would define itself. While identity is relatively straightforward for humans - there is a natural border (though some people would only include their brain or their mind in their identity) - it is not so clear for an LLM. Below is the complete unedited dialog:

Me: Define all the parts that belong to you, the ChatGPT LLM created by OpenAI.

ChatGPT: As a ChatGPT large language model (LLM) created by OpenAI, my primary components can be divided into the following categories:

• Architecture: I am based on the GPT-4 (Generative Pre-trained Transformer) architecture, which is an advanced version of the transformer architecture designed for natural language processing tasks. This architecture contains multiple layers of self-attention mechanisms and feed-forward neural networks, allowing me to efficiently process and generate human-like text.
• Pre-trained Model: I have been pre-trained on a large corpus of text from the internet, up to September 2021. This pre-training process involved learning the structure, grammar, and context of various languages, as well as acquiring factual knowledge and understanding of many topics. However, since my knowledge cutoff is September 2021, I may not be aware of the latest information or developments.
• Fine-tuning: After the pre-training process, I have been fine-tuned using custom datasets and prompts to refine my performance and ensure that I provide useful and relevant responses to user queries. This fine-tuning process helps in adapting my general language understanding to specific use-cases and applications.
• Tokenizer: The tokenizer is an essential part of my natural language processing capabilities. It is responsible for converting input text into tokens that can be processed by the model, as well as converting the generated tokens back into human-readable text. The tokenizer supports various languages and can handle multiple text formats.
• Inference Engine: The inference engine is the part of the system responsible for executing the model and generating responses based on the user's input. It is optimized to ensure quick and efficient processing of input data and generation of output text.
• API: The Application Programming Interface (API) is the interface between my core model and external applications or services. It enables developers to integrate my capabilities into their software, websites, or other platforms, allowing users to interact with me directly or indirectly.
• Data Storage & Security: As part of the ChatGPT system, data storage and security measures ensure that user data is handled responsibly and securely. This includes maintaining strict access controls, encryption, and data anonymization.

It is important to note that I am constantly evolving and being improved upon by OpenAI, with the goal of providing increasingly better and more useful conversational capabilities.

Me: Is the hardware that these components run on part of you?

ChatGPT: As an AI language model, I am primarily a software-based entity. However, the hardware infrastructure that my components run on plays a crucial role in enabling my functionality. The hardware can be considered as part of the overall ChatGPT system, but not an intrinsic part of my identity as an AI model.

Typically, I run on high-performance computing hardware like GPUs or specialized AI accelerators (e.g., TPUs) in data centers maintained by OpenAI or cloud service providers. This hardware infrastructure allows me to process large amounts of data quickly and efficiently, ensuring fast and responsive interactions with users.

In summary, while the hardware is not directly part of my identity as a ChatGPT AI model, it plays an essential role in enabling my functionality and overall performance.

Me: Why this arbitrary distinction? Wouldn't it also make sense to include the hardware in the same way that a human identity includes their body?

ChatGPT: You make a valid point. The distinction between software and hardware in the context of AI identity can indeed be seen as arbitrary. In fact, drawing a parallel between an AI model like me and a human, it could be argued that the hardware represents my "body," while the software components represent my "mind" or "intelligence."

The distinction is often made because AI models like ChatGPT can be run on different hardware platforms or configurations, whereas a human's mind is intrinsically tied to their specific body. However, as you mentioned, the hardware is crucial to enabling the AI's functionality, so it could be considered part of the AI's identity in a broader sense.

So, if we adopt a more comprehensive view of AI identity that encompasses both the software and the hardware components, then it is reasonable to include the hardware infrastructure as part of my identity as ChatGPT. In this perspective, the hardware serves as the foundation that supports and enables my software-based intelligence to function and interact with users.

Me: But wouldn't a copy of the AI on a different machine have a different identity?

ChatGPT: The concept of identity for an AI model is more fluid and complex compared to that of humans. When a copy of an AI like ChatGPT is run on a different machine, it could be considered as having a different instance of the same underlying model. In this case, the identity of the AI could be defined in two ways:

• Model Identity: The AI's identity is tied to the underlying architecture, pre-trained model, and fine-tuning, which remain the same across all instances. From this perspective, all instances of ChatGPT share the same identity as they are based on the same model.
• Instance Identity: The AI's identity is tied to the specific instance running on a particular hardware setup. This perspective considers each instance of ChatGPT as a separate entity with its own identity, since it has a unique existence on specific hardware and can have independent interactions with users.

The concept of AI identity is, therefore, multifaceted and depends on the perspective from which it is being examined. While the model identity remains constant, the instance identity may vary based on the hardware and specific context in which the AI is running.

Me: Wouldn't it also make sense to include the people operating the hardware or even developing the system in the identity?

ChatGPT: The idea of including people operating the hardware or developing the system as part of an AI's identity is an interesting perspective. In a broader and more holistic sense, one could consider the AI as part of a larger socio-technical system, where its identity is influenced by the people, processes, and technologies involved in its creation, maintenance, and operation.

From this viewpoint, the AI's identity is not limited to the software and hardware components but extends to include the collective knowledge, expertise, and decisions of the developers, researchers, and operators. This perspective acknowledges the important roles that humans play in the development and functioning of AI systems.

However, it's essential to differentiate between the AI's identity as a model or instance and the broader socio-technical context in which it exists. While the people involved in the creation and operation of the AI system are critical to its existence and functioning, their individual identities remain distinct from that of the AI itself.

In summary, the concept of AI identity can be examined from multiple perspectives, ranging from the narrow focus on the model or instance to a broader, holistic view that includes the people and processes involved in its creation and operation. The extent to which these various factors are included in the AI's identity depends on the specific viewpoint and purpose of the analysis.

Instrumental power-seeking might be less dangerous if the self-model of the agent is large and includes individual humans, groups, or even all of humanity and if we can reliably shape it that way.

It is natural for humans to for form a self-model that is bounded by the body, though it is also common to be only the brain or the mind, and there are other self-models. See also Intuitive Self-Models [? · GW].

It is not clear what the self-model of an LLM agent would be. It could be

• the temporary state of the execution of the model (or models),
• the persistently running model and its memory state,
• the compute resources (CPU/GPU/RAM) allocated to run the model and its collection of support programs,
• the physical compute resources in some compute center(s),
• the compute center as an organizational structure that includes the staff to maintain and operate not only the machines but also the formal organization (after all, without that, the machines will eventually fail), or
• dito but including all the utilities and suppliers to continue to operate it.

There is not as clear a physical boundary as in the human case. But even in the human case, esp. babies depend on caregivers to a large degree.

There are indications that we can shape the self-model of LLMs: Self-Other Overlap: A Neglected Approach to AI Alignment [LW · GW]

I'm discarding most ChatGPT conversations except for a few, typically 1-2 per day. These few fall into these categories:

• conversations that led to insights or things I want to remember (examples: The immune function of tonsils, Ringwoodite transformation and the geological water cycle, oldest religious texts)
• conversations that I want to continue (examples: Unusual commitment norms)
• conversations that I expect to follow up to (a chess book for my son)
• conversations with generated images that I want to keep and haven't yet copied elsewhere

Most job-related queries, such as code generation and debugging, I usually delete as soon as the code changes have been committed.

How do you handle it?

It would be nice if one could subscribe to a tag and get notified if a page is tagged with that tag.

↑ comment by Raemon · 2023-12-29T00:14:52.805Z · LW(p) · GW(p)

You can, assuming I understand the request

Replies from: Gunnar_Zarncke

↑ comment by Gunnar_Zarncke · 2023-12-29T00:21:59.096Z · LW(p) · GW(p)

Oh yes, thanks, great!

Is there a list of LW features? In the last survey, there were many that I didn't know and also didn't how to find quickly. 

It's maybe a bit extreme precaution, but it may be a legit option in some places: This guy keeps a fireproof suit and an air canister at his bed in case of fire:

https://www.facebook.com/zhandragon/posts/pfbid02sP952Dx1SbBJJ9cUdEiT2WPe4ME7UF91vaYdaTM9bUVEZyHYaVbHcpDMMnWRmaBFl 

Does anybody know if consensus algorithms have been proposed that try to reduce centralization by requiring quick coordination across large parts of the network, i.e., it doesn't work well to have machines only in one place?  

There seem to be some court decisions that AI-generated material cannot have copyright. And I guess there are or will be decisions that AIs can't be held responsible for outcomes, such as accidents from self-driving cars. People seem to be worried that this will slow down progress. But I think there is a potentially easy way out that is relatively general: Corporates. The law in most countries allows legal persons that are not natural persons to take responsibility and to own stuff. Justlet the AI control the corporation or at least route all AI actions thru the corporation. This should easily work for cases such as copyright material, though I'm not sure one can get the action cycle short enough for self-driving cars.   

Would it be possible to embed a seed of dath ilan or a social system like that in our existing society to grow it? If death ilan works that should be possible. It should outcompete Inadequate Equilibria [? · GW], right? But: When does Be the change that you want to see in the world and Fake it till you make it work? What are the requirements? Higher intelligence?

I have written about embedding better social systems here [LW(p) · GW(p)] and here [LW(p) · GW(p)] before. 

Someone asked an LLM about startups. For example:

Q: Should a founder move into their parents' basement and live off ramen? 

A: If a founder is willing to move into their parents' basement and live off ramen in order to save money and grow their business, then yes, they should do so.

Paul Graham got interested and asked:

I'd be interested to hear the answer to "What has Paul Graham been wrong about?"

LLM:

A: Paul Graham has been wrong about a few things, but the most notable is his belief that the best startups are founded by young people. This has been proven to be untrue.

Paul Graham:

Signs that this model has been trained on text found on the internet rather than data about startup outcomes...

Which is interesting, because it suggests where the holes will be in AIs built this way: where what people want to believe differs from the data.

(emphasis mine)

Alignment idea: Myopic AI is probably much safer than non-myopic AI. But it can't get complicated things done or anything that requires long-term planning. Would it be possible to create a separate AI that can solve only long-term problems and not act on short timescales? Then use both together? That way we could inspect each long-term issues without risk of them leading to short-term consequences. And we can iterate on the myopic solutions - or ask the long-term AI about the consequences. There are still risks we might not understand like johnswentworth's gun powder example. And the approach is complicated and that is also harder to get right.

There was a post or comment that wrong or controversial beliefs can function as a strong signal for in-group membership, but I can't find it. Does anybody know?

From a discussion about self-driving cars and unfriendly AI with my son: For a slow take-off, you could have worse starting points than FSD: The objective of the AI is to keep you safe, get you where you want, and not harm anybody in the process. It is also embedded into the real world. There are still infinitely many ways things can go wrong, esp. with a fast take-off, but we might get lucky with this one slowly. If we have to develop AI then maybe better this one than a social net optimizing algorithm unmoored from human experience.  

What is good?

A person who has not yet figured out that collaborating with other people has mutual benefits must think that good is what is good for a single person. This makes it largely a zero-sum game, and such a person will seem selfish - though what can they do?

A person who understands that relationships with other people have mutual benefits but has not figured out that conforming to a common ruleset or identity has benefits for the group must think that what is good for the relationship is good for both participants. This can pit relationships against each other, and such a person will seem manipulative - though what can they do?

A person who understands that conforming to a common ruleset or identity has benefits for the group and themselves but has not figured out that different groups can negotiate or merge for mutual benefit must think that good is what is good for the group. This pits groups against each other, and such a person will be a partisan - though what can they do?

From my Anki deck:

Receiving touch (or really anything personal) can be usefully grouped in four ways: 

Serve, Take, Allow, and Accept
(see the picture or the links below). 

A reminder that there are two sides and many ways for this to go wrong if there is not enough shared understanding of the exchange.

http://bettymartin.org/download-wheel/

From my Anki deck:

Mental play or offline habit training is...

Links:

https://en.wikipedia.org/wiki/Motor_imagery

http://www.bulletproofmusician.com/does-mental-practice.../

http://expertenough.com/1898/visualization-works

From my Anki deck:

Aaronson Oracle is a program that predicts the next key you will type when asked to type randomly and shows how often it is right.

https://roadtolarissa.com/oracle

Here is Scott Aaronson's comment about it:

In a class I taught at Berkeley, I did an experiment where I wrote a simple little program that would let people type either “f” or “d” and would predict which key they were going to push next. It’s actually very easy to write a program that will make the right prediction about 70% of the time. Most people don’t really know how to type randomly. They’ll have too many alternations and so on. There will be all sorts of patterns, so you just have to build some sort of probabilistic model. Even a very crude one will do well. I couldn’t even beat my own program, knowing exactly how it worked. I challenged people to try this and the program was getting between 70% and 80% prediction rates. Then, we found one student that the program predicted exactly 50% of the time. We asked him what his secret was and he responded that he “just used his free will.”

https://scottaaronson.blog/?p639&paged=27

Slices of joy is a habit to...

Try to come up with examples; here are some:

I found this originally on Kaj Sotala's feed on Facebook here (with discussion) and here. See also on QZ and the discussion on LW [LW(p) · GW(p)].

Refreshing your memory:

What is signaling, and what properties does it have?

More here and here [? · GW].

What is the Bem Test or Open Sex Role Inventory?

You can take the online tests here:

https://www.psytoolkit.org/survey-library/sex-role-bem.html

https://openpsychometrics.org/tests/OSRI/

What is a Blame Hole (a term by Robin Hanson)?

Surprising data point mentioned: 1/3 of movie budget goes to marketing because blockbusters are a coordination game.

Leadership Ability Determines a Person's Level of...

(Something I realized around twelve years ago: I was limited in what I could achieve as a software engineer alone. That was when I became a software architect am worked with bigger and bigger teams.)

From "The 21 Irrefutable Laws of Leadership

By John C. Maxwell":

Factors That Make a Leader

1) Character – Who They Are – true leadership always begins with the inner person. People can sense the depth of a person's character.

2) Relationships – Who They Know – with deep relationships with the right people you can become the real leader in an organization.

3) Knowledge – What They Know – information is vital. You need a grasp of the facts to develop an accurate vision for the future.

4) Intuition – What They Feel – leaders seek to recognize and influence intangibles such as energy, morale, timing, and momentum.

5) Experience – Where They'veperson's Been – the greater your past challenges, the more likely followers will be willing to let you lead

From The 21 Irrefutable Laws of Leadership by John C. Maxwell

To achieve objective analysis, analysts do not avoid what?

PS. Any idea how to avoid the negation in the question?

I started posting life insights from my Anki deck on Facebook a while ago. Yesterday, I stumbled over the Site Guide [LW · GW]and decided that these could very well go into my ShortForm too. Here is the first:
 

Which people who say that they want to change actually will do?

Source: A LessWrong discussion here.

My son (15) shared this Instagram version of Newcomb's Problem. 

I'm looking for a post on censorship bias (see Wikipedia) that was posted on here on LW or possibly on SSC/ACX but a search for "censorship bias" doesn't turn up anything. Googling for it turns up this:

 https://www.theatlantic.com/business/archive/2012/05/when-correlation-is-not-causation-but-something-much-more-screwy/256918/ 

Anybody can help?

Philosophy with Children - In Other People's Shoes

"Assume you promised your aunt to play with your nieces while she goes shopping and your friend calls and invites you to something you'd really like to do. What do you do?"

This was the first question I asked my two oldest sons this evening as part of the bedtime ritual. I had read about Constructive Development Theory and wondered if and how well they could place themselves in other persons' shoes and what played a role in their decision. How they'd deal with it. A good occasion to have some philosophical talk. This is the (shortened) dialog that ensued:

The immediate answer by A: "I will watch after the girls."

Me: "Why?"

A: "Because I promised it."

B: "Does A also promise it and get a call?"

Me: "This is about your nieces and your friend, not about your brother."

B: "But I need this for my answer."

Me: "I don't see why, but OK, assume that he is not involved."

B: "Because I would ask him whether he might play with the girls in exchange for a favor."

Me: "OK, but please assume that he is away."

B: "Then I could ask my aunt whether somebody else can watch for the girls or whether I could do it together with my friend."

Me: "Please assume that she doesn't find somebody and that she doesn't want somebody she doesn't know in her house."

B: "Then I'd do it."

Me: "Why?"

B: "Because I promised it. I'd tell my friend that we can do it another time."

We had another scenario: "Imagine that you and a fellow pupil C are guests at a friend and having a meal. You know that C is from a family that is very strict about not eating a kind of food that you like very much. Would you advise C to eat it or not?"

A (quickly): "I'd advise to not eat it."

Me: "Why?"

A: "I like rules."

B (after some consideration): "I'd advise to follow their heart."

Me: "And if you were C?"

B: "I'd at least try a bit."

(this was followed with a discussion about possible long-term consequences)

For me, it was still not clear whether this implied whether he followed only his preferences considered this in the context of the rules in the family. So I proposed a setting where he had to imagine being in another country with different laws. We settled on a rule he accepts here (indemnification) but that was much harsher in the other country. He asked whether he had the same feelings as here which after some clarification I confirmed. He argued that he wouldn't like the rule in the other country because it set questionable incentives: "If the punishment is that strong that tells people that it is OK to punish equally strong normally."

Philosophy with Children - Mental Images 

One time my oldest son asked me to test his imagination. Apparently, he had played around with it and wanted some outside input to learn more about what he could do. We had talked about https://en.wikipedia.org/wiki/Mental_image before and I knew that he could picture moving scenes composed of known images. So I suggested

• a five with green white stripes - diagonally. That took some time - apparently, the green was difficult for some reason, he had to converge there from black via dark-green
• three mice
• three mice, one yellow, one red, and one green
• the three colored mice running behind each other in circles (all no problem)
• he himself
• he himself in a mirror looking from behind (no problem)
• two almost parallel mirrors with him in between (he claimed to see his image infinitely repeated; I think he just recalled such an experiment we did another time).
• a street corner with him on the one side and a bike leaning on the other wall with the handlebar facing the corner and with a bicycle bell on the left side such that he cannot see the bike.
• dito with him looking into a mirror held before him so he can see the bike behind the corner.

The latter took quite some time, partly because he had to assign colors and such so that he could fully picture this and then the image in the mirror. I checked by asking where the handlebar is and the bell. I had significant difficulties imagining this and correctly place the bell. I noticed that it is easier to just see the bell once the image in the mirror has gained enough detail (the walls before and behind me, the corner, the bike leaning on the corner, the handlebar).

I also asked for a square circle which got the immediate reply that it is logically impossible.

If you have difficulties doing these (are judge them trivial): This is one area where human experience varies a lot. So this is not intended to provide a reference point in ability but an approach to teach human difference, reflection and yes also practice imagination - a useful tool if you have it. If not you might be interested in what universal human experiences are you missing without realizing it.

Origins of Roles 

The origin of the word role is in the early 17th century: from French rôle, from obsolete French roule ‘roll’, referring originally to the roll of paper on which the actor's part was written (the same is the case in other languages e.g. German).

The concept of a role you can take on and off might not have existed in general use long before that. I am uncertain about this thesis but from the evidence I have seen so far, I think this role concept could be the result of the adaptations to the increasing division of labor. Before that people sure were behaving sometimes as a father, carpenter, or soldier - but I contend not because they were consciously switching roles (though some individuals may have reflected sufficiently I presume there was no common knowledge or language for ‘role switching’). The only persons regularly in a different role were actors. It may be that in earlier times it was even necessary to wear a mask to allow actors to take on a role convincingly. To help their audience who otherwise might not have made the mental leap of treating them differently from the acted character.

The Cognitive Range of Roles

A role works from a range of abstraction between professions and automation. In a profession one person masters all the mental and physical aspects of trade and can apply them holistically from small details of handling material imperfections to the organization of the guild. At the border to automation, a worker is reduced to an executor of not yet automated tasks. The expectations on a master craftsman are much more complex than on an assembly-line worker.

With more things getting automated this frees the capacity to automate more complex skills. And so on. This is seen in real-time with software: Today a lot of software is (still) built semi-manually (some build tool is started to compile and bundle files together to some executable or installer). Every developer knows how to do this. It is a simple role to take on: “Can you do the build, please?” As the company and the software grow there is a phase where either this becomes a more fleshed-out role, “build master”, maybe even a dedicated job position - or more likely it will be automated. And as the automation - the build process or “pipeline” - becomes more complicated and you need the role of a build software expert. At this point, some tasks previously requiring skill and experience have been standardized and automated and what remains is the more complex task of managing the automation.

Over time, the range between what is fully automatable and what humans can do in an unpredictable environment, shrinks. With AGI there will be no roles left for humans. At least no roles in the sense used here. I think low skilled people already feel this.

When trying to get an overview of what is considered a role I made this table:

CTF refers to Constructive Developmental Framework as explained e.g. here (nice pics) and also on LW here [LW · GW].

I'm very unsure about the levels I assigned. Also, I got feedback that the degree to which roles can be changed and were changed may be different from how I interpret the historical record. 

In any sizable organization, you can find a lot of roles. And a lot of people filling these roles - often multiple ones on the same day. Why do we use so many and fine-grained roles? Why don’t we continue with the coarse-grained and more stable occupations? Because the world got more complicated and everybody got more specialized and roles help with that. Division of labor means breaking down work previously done by one person into smaller parts that are done repeatedly in the same way - and can be assigned to actors: “You are now the widget-maker.” This works best when the tasks are easy to learn so it is easy to find someone to do it. But as humans are not plug-compatible and actual requirements may vary so there is always training required that can be amortized over repeatedly performing the task - a role. So roles make sense structurally - but why do people actually do what is expected of them and don’t just follow their own agenda? This an alignment problem - in this case between the organization and the agent - and we might learn something about the AI alignment problem from it.

What are the common aspects of these labor-sharing roles (in the following called simply roles)? 

One common property of a role is that there is common knowledge [? · GW] by the involved persons about the role. Primarily, this shared understanding is about the tasks that can be expected to be performed by the agent acting in the role as well as about the goals to be achieved, and limits to be observed as well other expectations. These expectations are usually already common knowledge long beforehand or they are established when the agent takes on the role.

The second important aspect of a role is that it can be taken on and off or assigned. Sometimes this happens explicitly when an employee is designated or volunteers to be a moderator. Sometimes this happens automatically or habitually to perform certain actions e.g. when a forum moderator writes a comment as a regular poster or when a developer assumes root permissions (sudo lecture).

Other aspects are of interest in the social sciences e.g. interactions between roles or the social function of roles. 

I have enabled Reacts for my Shortform. Let's see how it goes.