Posts
Comments
Yeah that makes sense; the knowledge should still be there, just need to re-shift the distribution "back"
Haven't looked too closely at this, but my initial two thoughts:
- child consent is tricky.
- likely many are foreign children, which may or may not be in the 75 million statistic
It is good to think critically, but I think it would be beneficial to present more evidence before making the claim or conclusion
This is very interesting, and thanks for sharing.
- One thing that jumps out at me is they used an instruction format to prompt base models, which isn't typically the way to evaluate base models. It should be reformatted to a completion type of task. If this is redone, I wonder if the performance of the base model will also increase, and maybe that could isolate the effect further to just RLHF.
- I wonder if this has anything to do with also the number of datasets added on by RLHF (assuming a model go through supervised/instruction finetuning first, and then RLHF), besides the algorithm themselves.
- Another good model to test on is https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 which only has instruction finetuning it seems as well.
The author seems to say that they figured it out at the end of the article, and I am excited to see their exploration in the next post.
I find it useful sometimes to think about "how to differentiate this term" when defining a term. In this case, in my mind it would be thinking about "reasoning", vs "general reasoning" vs "generalization".
- Reasoning: narrower than general reasoning, probably would be your first two bullet points combined in my opinion
- Generalization: even more general than general reasoning (does not need to be focused on reasoning). Seems could be the last two bullet points you have, particularly the third
- General reasoning (this is not fully thought through): Now that we talked about "reasoning" and "generalization", I see two types of definition
- 1. A bit closer to "reasoning". first two of your bullet points, plus in multiple domains/multiple ways, but not necessarily unseen domains. In other simpler words, "reasoning in multiple domains and ways".
- 2. A bit closer to "general" (my guess is this is closer to what you intended to have?): generalization ability, but focused on reasoning.
In my observation (trying to avoid I think!), "I think" is intended to (or actually should have been used to) point out perspective differences (which helps to lead to more accurate conclusions, including collaborative and effective communication), rather than confidence. In the latter case of misuse, it would be good if people clarify "this term is about confidence, not perspective in my sentence".
True. I wonder for the average people, if being self-aware would at least unconsciously be a partial "blocker" on the next malevolence action they might do, and that may evolve across time too (even if it may take a bit longer than a mostly-good)
I highly agree with almost all of these points, and those are very consistent with my observation. As I am still relatively new to lesswrong, one big observation (based on my experience) I still see today, is disconnected concepts, definitions, and or terminologies with the academic language. Sometimes I see terminology that already exists in academia and introducing new concepts with the same name may be confusing without using channels academics are used to. There are some terms that I try to search on google for example, but the only relevant ones are from lesswrong or blogposts (which I still then read personally). I think this is getting better - in one of the recent conference reviews, I saw significant increase in submissions in AI safety working on X risks.
Another point as you have mentioned is the reverse ingestion of papers from academia; there are rich papers in interpretability as you have mentioned for example, and some concrete confusion I saw from professors or people already in that field is that why there is feels like a lack of connection with these papers or concepts, even though they seems to be pretty related.
About actions - many people that I see are concerned about AI safety risks in my usual professional group are people who are concerned about or working in current intentional risks like misuse. Those are actually also real risks and have already started (CSAM, deep fake porn with real people's faces, privacy, potential bio/chem weapons), and needs to be worked on as well. It is hard to stop working on them, and transition directly to X risks.
However, I do think it is beneficial to keep merging the academic and AI safety groups, which I see are already underway with examples like more papers, and some PhD positions on AI Safety, industry positions etc; This will increase awareness of AI safety, and as you have mentioned the interests in the technical parts are shared, as they could be applied potentially to many kinds of safety, and hopefully not that much on capabilities (though sometimes not separable).
What would be some concrete examples/areas to work on for human flourishing? (Just saw a similar question on the definition; I wonder what could be some concrete areas or examples)
True; and they would only need to merge up to they reach a "swing state" type of voting distribution.
That would be interesting; on the other hand, why not just merge all the states? I guess it would be a more dramatic change and may be harder to execute and unnecessary in this case.
Yes, what I meant is exactly "there is no must, but only want". But it feels like a "must" in some context that I am seeing, but I do not recall exactly where. And yeah true, there may be some survival bias.
I agree it is tragedy from human race's perspective, but I think what I meant is from a non-human perspective to view this problem. For example, to an alien who is observing earth, human is just another species that rise up as a dominant species, as a thought experiment.
(On humans prefer to be childless - actually this already slowed down in many countries due to cost of raising a child etc, but yeah this is a digress on my part.)
My two cents:
- The system has a fixed goal that it capably works towards across all contexts.
- The system is able to capably work towards goals, but which it does, if any, may depend on the context.
From these two above, seems it would be good for you to define/clarify what exactly you mean by "goals". I can see two definitions: 1. goals as in a loss function or objective that the algorithm is optimizing towards, 2. task specific goals like summarize an article, planning. There may be some other goals that I am unaware of, or this is obvious elsewhere in some context that I am not aware of. (From the shortform in the context shared, seems to be 1, but I have a vague feeling that 2 may not be aligned on this.)
For the example with dQw4w9WgXcQ in your initial operationalization when you were wondering about if it always generate Q - it just depends on the frequency. A good paper is https://arxiv.org/pdf/2202.07646 on frequency of this data and their rate of memorization if you were wondering if it is always (same context with training data, not different context/instruction).
I think that is probably not a good reason to be libertarian in my opinion? Could you also share maybe how much older were your than you siblings? If you are not that far apart, you and your siblings came from the same starting line, distributing is not going to happen in real life economically nor socially even if not libertarian (in real life, where we need equity is when the starting line is not the same and is not able to be changed by choice. A more similar analogy might be some kids are born with large ears, and large ears are favored by the society, and the large eared kids always get more candy). If you are ages apart with you being a lot older, it may make some limited sense to for your parents to re-distribute.
I am not quite sure about the writing/examples in computational kindness and responsibility offloading, but I think I feel the general idea.
For computational kindness, I think it is just really the difference in how people prefer to communicate, or making plans it seems, with the example on trip planning. I, for example, personally prefer being offered with their true thoughts - if they are okay with just really anything, or not. Anything is fine as long as that is what they really think or prefer (side talk: I generally think communicating real preferences will be the most efficient). I do not mind planning the trip myself in ways that I wanted. There is not really a right or wrong style. If the host person offered "anything is okay", but the receiver do not like planning, they could also simply say "Any recommendations? I like xxx or xxx generally." Communication goes both ways. The reason I think we should not say one way is better than another is, if the friend really wants to plan themselves, and then the host person planned a bunch, the receiver may also feel bad to reject the planned activities. Maybe what you really want to see, is that the host person cares enough to put some efforts into planning (just guessing)? And seems the relationship in this example between these two people are relatively close or require showing some efforts?
For responsibility offloading, I think some of these examples are not quite similar or parallel situations, but I generally get the proposal: "do not push other people in a pushy manner and should offer the clear option to say no" as opposed to a fake ask. In my opinion a fake ask is not true kindness - it is fake, so it is not really in any way kind. But at the same time, I've trained myself into taking the question literally - okay, if you asked, then it means you expect my answer to go both ways, then I will say no if I am thinking no. In the case the question is genuine - great! in the case it is not, too bad, the smoker should've be consistent with their words.
Most of these seems to be communication style differences, that just require another communication to sort out if the two parties need to communicate frequently.
Ah thanks. Do you know why these former rationalists were "more accepting" of irrational thinking? And to be extremely clear, does "irrational" here mean not following one's preference with their actions, and not truth seeking when forming beliefs?
I don't understand either. If it is meant what it meant, this is a very biased perception and not very rational (truth seeking or causality seeking). There should be better education systems to fix that.
On what evidence do I conclude what I think is know is correct/factual/true and how strong is that evidence? To what extent have I verified that view and just how extensively should I verify the evidence?
For this, aside from traditional paper reading from credible sources, one good approach in my opinion is to actively seek evidence/arguments from, or initiate conversations with people who have a different perspective with me (on both side of the spectrum if the conclusion space is continuous).
I am interested in learning more about this, but not sure what "woo" means; after googling, is it right to interpret as "unconventional beliefs" of some sort?
I personally agree with you on the importance of these problems. But I myself might also be a more general responsible/trustworthy AI person, and I care about other issues outside of AI too, so not sure about a more specific community, or what the definition is for "AI Safety" people.
For funding, I am not very familiar and want to ask for some clarification: by "(especially cyber-and bio-)security", do you mean generally, or "(especially cyber-and bio-)security" caused by AI specifically?
Does "highest status" here mean highest expertise in a domain generally agreed by people in that domain, and/or education level, and/or privileged schools, and/or from more economically powerful countries etc? It is also good to note that sometimes the "status" is dynamic, and may or may not imply anything causal with their decision making or choice on priorities.
One scenario is "higher status" might correlates with better resources to achieve those statuses, and a possibility is as a result they haven't experienced or they are not subject to many near-term harms. In other words, it is not really about the difference between "average" and "high status"'s people's intelligence, but more about what kind of world they are exposed to.
I do think it is good to hear all different perspectives to stay curious/open-minded.
edit: I just saw Dragon nicely listed two potential reasons, with scenario 2 mentioning something similar with my comment here. But something slightly specific in my thinking, is that these choices made by "average" and "high status" people may or may not be conscious, but rather from the experience from their lives and the world they are exposed to.
Could you define what you mean by "correctness" in this context? I think there might be some nuances into this, in terms of what "correct" means, and under what context
Based on the words from this post alone -
I think that would depend on what the situation is; in the scenario of price increases, if the business is a monopoly or have very high market power, and the increase is significant (and may even potentially cause harm), then anger would make sense.
Thanks! I think the term duration is interesting and creative.
Do you think for the short-term ones there might be pre-studies they need to do for the exact topics they need to learn on? Or maybe could design the short-term ones for topics that can be learnt quickly and solved quickly? I am a little worried about the consistency in policy as well (for example even with work, when a person on a project take vacation, and someone need to cover for them, there are a lot of onboarding docs, and prior knowledge to transfer), but could not find a good way just yet. I will think more about these.
Amazingly detailed article covering malevolence, interaction with power, and the other nuances! Have been thinking of exploring similar topics, and found this very helpful. Besides the identified research questions, some of which I highly agree with, one additional question I was wondering is: do self-awareness of one's own malevolence factors help one to limit the malevolence factors? if so how effective would that be? how would this change when they have power?
Interesting idea, and I think there is a possibility that the responsibility will make the "normal people" make better choices or learn more even though they do not know policy, etc in the first place.
A few questions:
- Do you think there is a situation where selected random people do not want to be in office/leadership and want to pursue their own passion/career and thus due to this reason may do a bad job? Is this mandatory?
- What are some nuances about population and diversity? (I am not sure yet)
Could you maybe elaborate on "long term academic performance"?
Agree with this, and wanted to add that I am also not completely sure if mechanistic interpretability is a good "commercial bet" yet based on my experience and understanding, with my definition of commercial bet being materialization of revenue or simply revenue generating.
One revenue generating path I can see for LLMs is the company uses them to identify data that are most effective for particular benchmarks, but my current understanding (correct me if I am wrong) is that it is relatively costly to first research a reliable method, and then run interpretability methods for large models for now; additionally, it would be generally very intuitive to researchers on what datasets could be useful to specific benchmarks already. On the other hand, the method would be much useful to look into nuanced and hard to tackle safety problems. In fact there are a lot of previous efforts in using interpretability generally for safety mitigations.
Would agree with most of the posts; To me, humans have some general shared experiences that may activate empathy related to those experiences, but the the numerous small differences in experience make it very hard to know exactly what the others would think/feel, even if in exactly the same situations. We could never really model the entire learning/experience history from another person.
My belief/additional point I want to add/urge is that this should not be interpreted as say empathy is not needed because we don't get it right anyways (I saw some other comments saying evolution is against empathy) - it is more to recognize what we are not naturally good at empathy(or less well than we thought), and thus create mindsets/systems (such as asking and promoting on gathering more information about the other person) that encourage empathy consciously (when needed).
(maybe I misread, and the second point is addressing another comment)
I think I observe this generally a lot: "as soon as those implications do not personally benefit them", and even more so when this comes with a cost/conflict of interest.
On rationality on decision making (not the seeking truth part on belief forming I guess) - I thought it is more like being consistent with their own preference and values (if we are constraining to the definition on lesswrong/sequence ish)? I have a hot take that:
- If the action space of commit to a belief is a binary choice, then when people do not commit to a belief, the degree they believe in that belief is less than those who do. If we have to make it into binary classification, then it is not really a true belief if they do not commit to that belief.
- It could be the action of a belief is a spectrum, and then people in this case for example could eat less meat, matching the degree of belief "eating meat is not moral".
I think the title could be a bit more specific like - "involving political party in science discussions might not be productive", or something similar. If using the word "politics", it would be crucial to define what "politics" here mean or refer to. The reason I say this is "politics" might not be just about actual political party's power dynamics, but also includes general policy making, strategies, and history that aim to help individuals in the society, and many other aspects. These other types of things included in the word "politics" is crucial to consider when talking about many things (I think it is a little bit similar to precedents in law).
(Otherwise, if this article is about not bringing things all to political party level, I agree. I have observed that many things in the US at least are debated over political party lines, and if a political party debates about A, people reversely attribute some general social topics A to "political values or ideology", which is false to me.)
I think by winning, he meant: "art of choosing actions that lead to outcomes ranked higher in your preferences", though I don't completely agree with this word choice of "winning" which could be ambiguous/causing confusion.
A bit unrelated, but more of a general comment on this - in my belief, I think people generally have unconscious preferences, and knowing/acknowledging these before weighing out preferences are very important, even if some preferences are short term.
I also had similar feelings on the simplicity part, and also how theory/idealized situation and execution could be very different. Also agree on the conflict part (and to me many different type of conflicts). And, I super super strongly support the section on The humans behind the numbers.
(These thoughts still persist after taking intro to EA courses).
I think EA's big overall intentions are good to me and I am happy/energized by see how passionate people are comparing to no altruism at all at least; but the details/execution are not quite there to me.
I have been having some similar thoughts on the main points here for a while and thanks for this.
I guess to me what needs attention is when people do things along the lines of "benefit themselves and harm other people". That harm has a pretty strict definition, though I know we may always be able to give borderline examples. This definitely includes the abuse of power in our current society and culture, and any current risks etc. (For example, if we are constraining to just AI with warning on content, https://www.iwf.org.uk/media/q4zll2ya/iwf-ai-csam-report_public-oct23v1.pdf. And this is very sad to see.) On the other hand, with regards to climate change (can also be current too) or AI risks, it probably should also be concerned when corporates or developers neglect known risks or pursue science/development irresponsibly. I think it is not wrong to work on these, but I just don't believe in "do not solve the other current risks and only work on future risks."
On some comments that were saying our society is "getting better" - sure, but the baseline is a very low bar (slavery for example). There are still many, many, many examples in different societies of how things are still very systematically messed up.
This is about basic human dignity and respect to other humans, and has nothing to do with politics.
Oxford languages (or really just after googling) says "rational" is "based on or in accordance with reason or logic."
I think there are a lot of other types of definitions (I think lesswrong mentioned it is related to the process of finding truth). For me, first of all it is useful to break this down into two parts: 1) observation and information analysis, and 2) decision making.
For 1): Truth, but also particularly causality finding. (Very close to the first one you bolded, and I somehow feel many other ones are just derived from this one. I added causality because many true observations are not really causality).
For 2): My controversial opinion is everyone are probably/usually "rationalists" - just sometimes the reasonings are conscious, and other times they are sub/un-conscious. These reasonings/preferences are unique to each person. It would be dangerous in my opinion if someone try to practice "rationality" based on external reasonings/preferences, or reasonings/preferences that are only recognized by the person's conscious mind (even if a preference is short term). I think a useful practice is to 1. notice what one intuitively want to do vs. what one think they should do (or multiple options they are considering), 2. ask why there is the discrepancy, 3. at least surface the unconscious reasoning, and 4. weigh things (the potential reasonings that leads to conflicting results, for example short term preference vs long term goals) out.
From my perspective - would say it's 7 and 9.
For 7: One AI risk controversy is we do not know/see existing model that pose that risk yet. But there might be models that the frontier companies such as Google may be developing privately, and Hinton maybe saw more there.
For 9: Expert opinions are important and adds credibility generally as the question of how/why AI risks can emerge is by root highly technical. It is important to understand the fundamentals of the learning algorithms. Additionally they might have seen more algorithms. This is important to me as I already work in this space.
Lastly for 10: I do agree it is important to listen to multiple sides as experts do not agree among themselves sometimes. It may be interesting to analyze the background of the speaker to understand their perspectives. Hinton seems to have more background in cognitive science comparing with LeCun who seems to me to be more strictly computer science (but I could be wrong). Not very sure but my guess is these may effect how they view problems. (Only saying they could result in different views, but not commenting on which one is better or worse. This is relatively unhelpful for a person to make decisions on who they want to align more with.)
(Like the answer on declarative vs procedural). Additionally, reflecting on practicing Hanon for piano (which is almost a pure finger strength/flexibility type of practice) - might be also for physical muscle development and control.
Agree with a lot of the things in this post, including "But implicit in that is the assumption that all DALYs are equal, or that disability or health effects are the only factors that we need to adjust for while assessing the value of a life year. However, If DALYs vary significantly in quality (as I’ll argue and GiveWell acknowledges we have substantial evidence for), then simply minimizing the cost of buying a DALY risks adverse selection. "
Had the same question/thoughts when I did the Introduction to Effective Altruism course as well.
I came across this post recently, and want to really appreciate you speaking up, fighting for your rights, and raising awareness. Our society does not have proper and successful education on what consent is, and even though at my university we have consent courses during first week of school, people don't take them seriously. Maybe they should have added a formal test on that, and you cannot enter school unless you pass. This should apply to high school as well. A 2018 article have pointed out how to teach consent at all education stages. https://www.gse.harvard.edu/ideas/usable-knowledge/18/12/consent-every-age
Many sexual assaults happen between partners, and your experience is clearly non-consensual and therefore sexual assault. I am a bit hesitate to comment as I am afraid to reopen any wounds, but also want to show support and gratitude🙏.
I am guessing maybe it is the definition of "alignment" that people don't agree on/mixed on?
Some possible definitions I have seen:
- (X risks) and/or (catastrophic risks) and/or (current safety risks)
- Any of above + general capabilities (an example I saw is "how do you get the AI systems that we’re training to optimize the thing that we actually want to optimize" from https://arize.com/blog/openai-on-rlhf/)
And maybe some people don't think it got to solving X risks yet if they view the definition of alignment as X risks only.
My guess is:
- AI pause: no observation on what safety issue to address, work on capabilities anyways, then may lead to only capability improvements. (Assumption is that AI pausing means no releasing of models.)
- RSP: observed O, shift more resources to work on mitigating O and less on capabilities, and when protection P is done, publish the model, then shift back to capabilities. (Ideally.)
Nice post pointing out this! Relatedly for misused/overloaded terms - I think I have seen this getting more common recently (including overloaded terms that means something else in the wider academic community or society; and self-reflecting - I sometimes do this too and need to improve on this).
I like the idea and direction of text watermarks, and more research could be done to see how to feasibly do this as adding watermarks to text seems to be much harder than images.
Maybe already mentioned in the article but I missed - have you done any analysis on how these methods effect the semantics/word choice availability from the author's perspective?
How did you translated the dataset, and what is the translation quality?
I am relatively new to the community, and was excited to join and learn more about the actual methods to address AI risks, and how to think scientifically generally.
However after using for a while, I am a bit disappointed. I realized I probably had to filter many things here.
Good:
- There are good discussions and previous summaries that are actually useful on alignment. There are people who work on these things from both ngos and industry showing what research they are doing, or what actions they have taken on safety. Similarly with bioweapons etc.
- I also like articles like trying to identify how to find what's the best thing to do with intersections of passion, skill, and importance.
- I like the articles that mention/promotes contacting reality.
Bad:
- Sometimes I feel the atmosphere is "edgy", and sometimes I see people may argue over relatively small things that I don't know how the conclusion will lead to actual actions. And maybe this is just the culture here, but I found it surprising how easy people call each other "wrong" although many times I felt like both sides are just opinions. And I felt like I see less "I think" or "in my opinion" to quality claims than usual workplace at least. People appear to be very confident or sure about their own belief when communicating. From my understanding, I think people may practicing the "strong opinion weakly hold" thinking they could say something strong and change easily - I found that to be easier in verbal communication among colleagues, schoolmates or friends where one can talk to them (a relatively small group) every day. But on a platform where there are a lot more people, and tracking on opinions changes is hard, it might be more productive to consider modifying the "strong" opinion part and quality more in the first place.
- I do think the downvote or upvote, which is related to how much you can comment or contribute to the site (or if you can block a person or not), encourage group think and you would need to be identify with the sentiment of the majority (I think another answer mentioned group think as well).
- I am feeling many articles or comments are quite personal/non-professional (communication feels different from what I encounter at work), which makes this community a bit confusing mixing personal and professional opinions/sharing. I think it would be nice to have a professional section, and a separate personal sections, and also encourage different communication rules for more communication efficiency, and I guess could naturally filter some articles for people at certain times wanting to focus on different things. Could be good to organize articles better by section as well? Though there is "tags" currently.
- This is a personal belief, but I am a bit biased for action and hope to see more discussions on how to execute things, or at least how should actions change based on a proposed belief change.
- This might be something more fundamental that is based on personal belief vs (some but not everyone on) lesswrong belief - to a certain extend I appreciate prioritization, but when it is too extreme I feel it is 1) counterproductive on solving issue itself, 2) too extreme that discourages new comers that also want to work on shared issues. It also feels more fear driven rather than rationality driven, which is discrediting in my opinion.
- For 1, Many areas to work on sometimes are interrelated, and focusing only on one may not actually achieve the goal.
- For 2, Sometimes it just feels alarming/scary when I see people trying to say "do not let other issues we need to solve get in the way of {AI risks/some particular thing}".
- I am sensing (though I am still kinda new here, so I might not have dig enough through articles) is that we may lack some social science connections/backgrounds and how the world actually works, even when talking about society related things (I forgot what specific articles gave me this feeling, maybe related to AI governance.)
I think for now, I probably will continue using but with many many filters.
I am not sure if that is a good reasoning, though I am also looking for reasoning/justification. The reasoning here seems to say - animals cannot talk our languages, and so it is okay to assume they do not want to survive (this is assuming the existence of humans naturally has conflict with the other specifies).
The reasoning I think I am trying to accept is that by nature it seems we want to self-preserve, and maybe unfortunately many altruism we want to do have non-altruism roots, which maybe be fine/unavoidable. Maybe it would be good to also consider the earth as a whole/expanding moral circles (when we can), and less exhibiting human arrogance. Execution wise, this may be super hard, but I think "thinking" wise, there is value in recognizing this aspect.
I don't think that is only the viewpoint of the dead (it also seems very individually focused/personal rather than collective specifies experiment/exploration focused). This is about thinking critically and from different perspectives for truth finding, which is related to definition of rationality on lesswrong (the process of seeking truth).
I am operating on the assumption that many of us seek true altruism on this platform. I could move this to the effective altruism platform.
This sounds individually (which is still an option), but the question is about collectively.
The question is zooming out from the humanity itself, and view from out-of-human kind of angle. I think it is an interesting angle, and remind ourselves that many things we do, may not be as altruistic as we thought.
Also, I think maybe that would mean suffering risks would need more attention.
Ultimately, my answer to this might be - morally humans do not need to last forever, but we are self preservation focused, and that is okay to pursue and practice altruism whenever we can either individually or collectively; but when there is conflict against our preservation, how to pursue this "without significantly harming others" is tricky,
I do not think they meant anything AI specific, just general existence about humanity vs other species.
The question was not about whether humanity live forever, the original prompt is "why must human/humanity live/continue forever?", which is in the original question.
Do not feel the need to reply in anyway, nothing here is urgent. (Not sure why you would reply in bad shape or mention that; initially thought it is related to the topic).