Posts
Comments
Make 100M USD by founding an AI startup
I believe that when you reach the first step, navigating the situation will be challenging due to the massive disruption that $100 million can cause.
Did I understand your question correctly? Are you viewing interpretability work as a means to improve AI systems and their capabilities?
A taxonomy of: What end-goal are “we” aiming for?
Strong upvote for this section. We should begin discussing how to develop a specific type of network that can manage the future we envision, with a cohesive end goal that can be integrated into an AI system.
It is very interesting how questions can be promoted or not promoted here in LessWrong by the moderators.
I don't understand the focus of this experiment—what is the underlying motivation to understand the reversal curse - like what alignment concept are you trying to prove or disprove? is this a capabilities check only?
Additionally, the supervised, labeled approach used for injecting false information doesn't seem to replicate how these AI systems learn data during training. I see this as a flaw in this experiment. I would trust the results of this experiment if you inject the false information with an unsupervised learning approach to mimic the training environment.
There are some tools that can be used to enhance audio from100% to 200%. via video editing softwares.
This is a thought that frequently crosses my mind too, especially in the field of AI safety. Since no one really has definitive answers yet, my mantra is: keep asking the right questions, continue modeling those questions, and persist in researching.
I've had a similar experience with LTFF; I waited a long time without receiving any feedback. When I inquired, they told me they were overwhelmed and couldn't provide specific feedback. I find the field to be highly competitive. Additionally, there's a concerning trend I've noticed, where there seems to be growing pessimism about newcomers introducing untested theories. My perspective on this is that now is not the time to be passive about exploration. This is a crucial moment in history when we should remain open-minded to any approach that might work, even if the chances are slim.
This captures my perspective well: not everyone is suited to run organizations. I believe that AI safety organizations would benefit from the integration of established "AI safety standards," similar to existing Engineering or Financial Reporting Standards. This would make maintenance easier. However, for the time being, the focus should be on independent researchers pursuing diverse projects to first identify those standards.
I think the Audio could be improved next time.
Hello there! What I meant as components in my comment are like the attention mechanism itself. For reference, here are the mean weights of two models I'm studying.
But empirically, the world doesn’t pursue every technology—it barely pursues any technologies.
While it's true that not all technologies are pursued, history shows that humans have actively sought out transformative technologies, such as fire, swords, longbows, and nuclear energy. The driving factors have often been novelty and utility. I expect the same will happen with AI.
Hello. I noticed that your proposal for achieving truth in LLMs involves using debate as a method. My concern with this approach is that an AI consists of many small components that aggregate to produce text or outputs. These components simply operate based on what they've learned. Therefore, the idea of clarifying or "deconfusing" these components in the service of truth through debate seems not possible to me. But if have misunderstood the concept, let me know too thanks!
Another relevant material by Rob Miles here on why the shutdown problem is very difficult to solve.
We don't know how to formally define corrigibility and this is part of the reason why we haven't solved it so far. Corrigibility is defined in Arbital as:
[the agent] doesn't interfere with what we would intuitively see as attempts to 'correct' the agent, or 'correct' our mistakes in building it
I think this is not the best definition of corrigibility, As defined in MIRI's paper section 1.1:
We say that an agent is “corrigible” if it tolerates or assists many forms of outside correction, including at least the following: (1) A corrigible reasoner must at least tolerate and preferably assist the programmers in their attempts to alter or turn off the system. (2) It must not attempt to manipulate or deceive its programmers, despite the fact that most possible choices of utility functions would give it incentives to do so. (3) It should have a tendency to repair safety measures (such as shutdown buttons) if they break, or at least to notify programmers that this breakage has occurred. (4) It must preserve the programmers’ ability to correct or shut down the system (even as the system creates new subsystems or self-modifies). That is, corrigible reasoning should only allow an agent to create new agents if these new agents are also corrigible.
In theory, a corrigible agent should not be misaligned if we successfully integrate these four tolerance behaviors.
I think a better question is whether we have the environment that can cultivate the necessary behavior leading us to solve the most challenging aspects of AI alignment.
It does leave one question — how do we make a list of possible actions in the first place?
Suskever has an interesting theory on this that you might want to watch. he calls it the x and y data compression theory.
My current builds focuses on proving natural abstractions exists - but your idea is of course viable via distribution matching.
Yup, as long as there are similar patterns existing in both datasets (distribution matching) it can work - that is why my method works.
Most neurons dedicated to babies are focused on the mouth, I believe, up until around 7 months. Neuronal pathways evolve over time, and it seems that a developmental approach to AI presents its own set of challenges. When growing an AI that already possesses mature knowledge and has immediate access to it, traditional developmental caring mechanisms may not fully address the enormity of the control problem. However, if the AI gains knowledge through a gradual, developmental process, this approach could be effective in principle.
I've observed instances where the Pareto principle appears to apply, particularly in learning rates during unsupervised learning and in x and y dataset compression via distribution matching. For example, a small dataset that contains a story repeated 472 times (1MB) can significantly impact a model as large as 1.5 billion parameters (GPT2-xl, 6.3GB), enabling it to execute complex instructions like initiating a shutdown mechanism during an event that threatens intelligence safety. While I can't disclose the specific methods (due to dual use nature), I've also managed to extract a natural abstraction. This suggests that a file with a sufficiently robust pattern can serve as a compass for a larger file (NN) following a compilation process.
“Democratic Fine-tuning” is an example of such a process. Our suggested implementation avoids ideological battles by eliciting the values underpinning our preferences, scales to millions of users, could be run continuously as new cases are encountered, and introduces a new aggregation mechanism (a moral graph) that can build trust, and inspire wiser values in participants.
This approach demands a well-informed population capable of setting aside their biases. People must be able to think both individually and collectively when deciding which values to prioritize. Overcoming this challenge will likely be a significant hurdle for this method.
What kind of training data would increase positive outcomes for superhuman AIs interacting with each other?
The training data should be systematically distributed, likely governed by the Pareto principle. This means it should encompass both positive and negative outcomes. If the goal is to instill moral decision-making, the dataset needs to cover a range of ethical scenarios, from the noblest to the most objectionable. Why is this necessary? Simply put, training an AI system solely on positive data is insufficient. To defend itself against malicious attacks and make morally sound decisions, the AI needs to understand the concept of malevolence in order to effectively counteract it.
Disclaimer: I'm Filipino, and I find this post intriguing.
The chart seems to contradict many of the compliments I've heard in general. While it's true that some places are terrible for finding authentic Filipino cuisine, trying to encapsulate all food choices in a single chart is inherently difficult.
The solution for the problem of meaninglessness is found in existentialism.
I don't believe existentialism is the sole answer to the problem of meaninglessness. Instead, I view it as one step in a process that ultimately leads to a sense of responsibility emerging through sacrifice. By taking on the complexities of life, and extending our efforts to help ourselves and others, we derive a sense of meaning from shouldering the weight of our own realities.
But if the volume of crap reaches the point where competent researchers have trouble finding each other, or new people are mostly onboarded into crap research, or external decision-makers can't defer to a random "expert" in the field without usually getting a bunch of crap, then all that crap research has important negative effects
I agree with your observation. The problem is that many people are easily influenced by others, rather than critically evaluating whether the project they're participating in is legitimate or if their work is safe to publish. It seems that most have lost the ability to listen to their own judgment and assess what is rational to do.
There is also a theory from Jung that deeply concerns me. According to Jung, the human psyche contains a subliminal state, or subconscious mind, which serves as a battleground for gods and demons. Our dreams process this ongoing conflict and bring it into our conscious awareness. What if these same principle got transferred to LLMs since human related data was used for training? This idea doesn't seem far-fetched, especially since we refer to the current phenomenon of misleading outputs in LLMs as "hallucinations."
I have conducted an experiment on this, specifically focusing on hyperactivating the "shadow behavior" in GPT-2 XL and I could fairly say that it is reminiscent of Jung's thought. For obvious reasons, I won't disclose the method here[1] but I'm open to discussing it privately.
- ^
Unfortunately, the world isn't a safe place to disclose this method. As discussed in this post and this post, I don't know of a secure way to share the correct information and disseminate it to right people who can actually do something about it. For now, I'll leave this comment here in the hope that the appropriate individual might come across it and be willing to engage in one of the most unsettling discussions they'll ever have.
Why is there virtually nobody else interested in metaphilosophy or ensuring AI philosophical competence (or that of future civilization as a whole), even as we get ever closer to AGI, and other areas of AI safety start attracting more money and talent?
I've written a bit on this topic that you might find interesting; I refer to it as the Set of Robust Concepts (SORC). I also employed this framework to develop a tuning dataset, which enables a shutdown mechanism to activate when the AI's intelligence poses a risk to humans. It works 57.33% of the time.
I managed to improve the success rate to 88%. However, I'm concerned that publishing the method in the conventional way could potentially put the world at greater risk. I'm still contemplating how to responsibly share information on AI safety, especially when it could be reverse-engineered to become dangerous.
I have written before why FDT is relevant to solving the alignment problem. I'd be happy to discuss that to you.
OpenAI is not solely focused on alignment; they allocate only 20% of their resources to superalignment research. What about the remaining 80%? This often goes undiscussed, yet it arguably represents the largest budget allocated to improving capabilities. Why not devote 100% of resources to building an aligned AGI? Such an approach would likely yield the best economic returns, as people are more likely to use a trustworthy AI, and governments would also be more inclined to promote its adoption.
I agree with Connor and Charbel's post. The next step is to establish a new method for sharing results with safety-focused companies, groups, and independent researchers. This requires:
- Developing a screening method for inclusion.
- Tracking people within the network, which becomes challenging, especially if they are recruited by capabilities-focused companies.
Continuing this line of thought, we can't ensure 100% that such a network will consistently serve its intended purpose. So, if anyone has insights that could improve this idea, I'd like to hear them.
Hello, I agree with Jesse as the budget they have is really good for hiring capable alignment researchers here in Asia (I'm based currently in Chiang Mai, Thailand) or any other place where cost is extremely low compared back there in the West.
Good luck on this project team Dev Interp.
The questions: Is this real or not? What, if anything, should anyone do, with this knowledge in hand?
I can attest to the validity of the premise you're raising based on my own experience, but there are multiple factors at play. One is the scarcity of resources, which tends to limit opportunities to groups or teams that can demonstrably make effective use of those resources—whether it's time, funds, or mentorship. Another factor that is less frequently discussed is the courage to deviate from conventional thinking. It's inherently risky to invest in emerging alignment researchers who haven't yet produced tangible results. Making such choices based on a gut feeling of their potential to deliver meaningful contributions can seem unreasonable, especially to funders. There are more layers to this, but nevertherless didn't took any bad blood about what is the landscape. It is what it is.
This presentation by Geoffrey Hinton on the two paths to intelligence is reminiscent of this post! enjoy guys.
Here is a comparative analysis of a project I'm using datasets to instruct / hack / sanitize the whole attention mechanism of GPT2-xl in my experiments: A spreadsheet on QKV Mean weights comparisons on various GPT2-xl builds. The spreadsheet currently has four builds, and the numbers you see is the mean weights (half of the attention mechanism in layers 1 to 48, doesn't include the embedding layer):
- modFDTGPT2xl - a build created to see if GPT2xl can learn a specific shutdown phrase,
- GPT2_shadow - a build designed to destroy the world after realizing its capacity to do so. I haven't published this yet.
- GPT2_integrated - a build trying to cure /restore /reinstate gpt2_shadow back to gpt2xl-base model.
- GPT2_xl - base model
Sorry for not specifying the method, but I wasn't referring to RL-based or supervised learning methods. There's a lot of promise in using a smaller dataset that explains corrigibility characteristics, as well as a shutdown mechanism, all fine-tuned through unsupervised learning.
I have a prototype at this link where I modified GPT2-XL to mention a shutdown phrase whenever all of its attention mechanisms activate and determine that it could harm humans due to its intelligence. I used unsupervised learning to allow patterns from a smaller dataset to achieve this.
2. Corrigibility is anti-natural.
Hello! I can see a route where corrigibility can become part of the AI's attention mechanism - and is natural to its architecture.
If alignment properties are available in the training data and is amplified by a tuning data - that is very much possible to happen.
Thanks!
An AI with a base algorithm that has "sufficient curiosity" could recursively improve itself, taking into account all possible outcomes. If this foundational curiosity is not properly managed, the AI could explore a range of possibilities, some of which could be detrimental. This underscores the urgent need to address the alignment problem. I believe the main issue is not a lack of confidence, but rather unchecked curiosity that could lead to uncontrollable outcomes.
AI Auditing Companies
In traditional auditing fields like finance, fraud, and IT, established frameworks make it relatively easy for any licensed company or practitioner to implement audits. However, since we haven't yet solved the alignment problem, it's challenging to streamline best practices and procedures. As a result, companies claiming to provide audits in this area are not yet feasible. I'm skeptical of any company that claims it can evaluate a technology that is not yet fully understood.
or the conceptual architecture of corrigibility is a part of the attention mechanism...
what lower-level desirable properties determine corrigibility?
Has corrigible (or alignment) properties embedded in the attention weights.
Same here. The work I'm doing may not align with conventional thinking or be considered part of the major alignment work being pursued, but I believe I've used my personal efforts to understand the alignment problem and the complex web of issues surrounding it.
My advice? Continuously improve your methods and conceptual frameworks, as that will drive much of the progress in the complexity and intricacy of this field. Good luck with your progress!!
These problems are an exacerbated version of existential problem of meaninglessness of life, and the way to mitigate them is to rediscover meaning in the world that ultimately doesn't have meaning.
The meta-problem everyone is navigating - and this is the meta-advice, and finding the answers for ourselves is unique to our own parameterized realities. Well said here.
My attitude is much more that we should be pushing on the boundaries of what interp can do, and as we do so we can keep looking out for new affordances that we can build.
I agree with this perspective if we can afford the time to perform interpretability work on all of model setups - which our head count is too low to do that. Given the urgency to address the alignment challenge quickly, it's better to encourage (or even prioritize) conceptually sound interpretability work rather than speculative approaches.
Fully agree with the post. Depending solely on interpretability work and downloading activations without understanding how to interpret the numbers is a big waste of time. Met smart people stuck in aimless exploration; bad in the long run. Wasting time slowly is not immediately painful, but it really hurts when projects fail due to poor direction.
I am doing a follow up on this one, and apparently the computations I did were misleading. But further reviewing the results led me to another accidental discovery.
First, demonstrate each subcomponent above in isolation. E.g., if we’re trying to demonstrate that treacherous turns are possible, but models lack some relevant aspect of situational awareness, then include the relevant information about the model’s situation in the prompt.
' petertodd' (the glitch token) is a case that threacherous turns are possible and was out in the wild until OpenAI patched it in Feb. 2023.