Posts
Comments
Maybe vehicles would need to carry some shaped charges to cut a hole in the tube in case of emergency.
That would likely create sparks, and provided the tube has been cut the hydrogen is going to explode.
*preferably not the last state but some where the person felt normal.
I believe that's right! Though, if person can be reconstructed from N bits of information, and dead body retains K << N, then we need to save N-K bits (or maybe all N, for robustness) somewhere else.
It's an interesting question how many bits can be inferred from social networks trace of the person, actually.
Continuing to make posts into songs! I believe I'm getting a bit better, mainly in rapid-lyrics-writing; would appreciate pointers how to improve further.
- https://suno.com/song/ef734c80-bce6-4825-9906-fc226c1ea5b4 (based on post Don't teach people how to reach the top of a hill)
- https://suno.com/song/c5e21df5-4df7-4481-bbe3-d0b7c1227896 (based on post Effectively Handling Disagreements - Introducing a New Workshop)
Also, if someone is against me creating a musical form of your post, please say so! I don't know beforehand which texts would seem easily convertible to me.
This is especially concerning if we, as good Bayesians, refuse to assign a zero probability to any event, including zero utility ones.
I feel that since people don't care ultimately about money, all-nonzero probabilities will make all events have nonzero utility as well.
Let's solve this problem without trying to refer to existing particular cases.
For the start, we assume that utility of A-the-thief monotonically decreases with time served; utility of B monotonically increases, and if A gives up lollipops it is increased by another constant.
Let's graph what choices A has when B does not give up lollipops.
We may notice that in this case B will simply throw A in jail for life. Well, what happens if A is willing to cooperate a bit?
- Coordination result will not be below or to the left of Pareto frontier, since otherwise it is possible to do better than that;
- Coordination result will not be below or to the left of no-coordination result, since otherwise one or both parties are acting irrationally.
We may see that after these conditions, only a piece of curve "A gives up lollipops" remains. The exact bargaining point can then be found out by using ROSE values, but we can already conclude that A will likely be convicted for a long time but not for life.
The LessWrong's AI-generated album was surprisingly nice and, even more importantly, pointed out the song generator to me! (I've tried to find one a year ago and failed)
So I've decided to try my hand on quantum mechanics sequence. Here's what I have reached yet: https://app.suno.ai/playlist/81b44910-a9df-43ce-9160-b062e5b080f8/. (10 generated songs, 3 selected, unfortunately not the best quality)
I've decided to try my hand on quantum mechanics sequence! Here's what I have reached yet: https://app.suno.ai/playlist/81b44910-a9df-43ce-9160-b062e5b080f8/. (10 generated songs, 3 selected, unfortunately not the best quality)
I've came across a poll about exchanging probability estimates with another rationalist: https://manifold.markets/1941159478/you-think-something-is-30-likely-bu?r=QW5U.
You think something is 30% likely but a friend thinks 70%. To what does that change your opinion?
I feel like there can be specially-constructed problems when the result probability is 0, but haven't been able to construct an example. Are there any?
On what basis can Alice assume
Not actually assume, but that's certainly Bayesian evidence (should Bob have tried , he would likely respond in another way).
Also, :smile: your own comment is a fairly large bit of evidence that you haven't yet read the Sequences (by the way, I recommend doing that). For instance, you can consider different ways of thinking, answer the questions 1-4 from their perspectives, and that would be evidence on which way is better - though, reality is still the ultimate judge how each situation turns out.
I don't think there is anything I can do educationally to better ensure they thrive as adults other than making sure I teach them practical/physical build and repair skills
I think one more thing could be useful, I'd call it "structural rise": over many different spheres of society, large projects are created by combining some small parts; ways to combine them and test robustness (for programs)/stability (for organisations)/beauty (music)/etc seem pretty common for most of the areas, so I guess they can be learned separately.
I suppose 3d-whiteboard could be useful in allowing to connect more relevant subjects to each node (re: problem of four colours: countries on plane can be coloured in 4 different ways so that touching ones have different colour, and in space there's no such limit).
the best you can do is what you think is personally good
As far as you're an ideal optimizing agent with consistent values and full knowledge, otherwise actions based on your thoughts may end up worse than using social heuristics.
That there are no bugs when it comes to values. That you should care about exactly what you want to care about. That if you want to team up and save the world from AI or poverty or mortality, you can, but you don’t have to.
Locally invalid. Values can be terminal (what you care about) and instrumental, and saving the world for most people is actually instrumental.
There’s value in giving explicit permission to confused newcomers to not get trapped in moral chains, because it’s really easy to hurt yourself doing that.
I think that's true since memes can be harmful, but there is also value in reminding that if more people worked to save improve the world on average, it would be better, and often a simpler way is to do that yourself instead of pushing that meme+responsibility out.
Save the world if you want to, but please don’t if you don’t want to.
I'd continue that with "but please don't destroy the world whichever option you choose, since that will interfere with my terminal goals and I'll care about your non-existence".
That's certainly an interesting position in discussion about what people want!
Namely, that actions and preferences are just conditionally-activated and those context activations are balanced against each other. That means that person's preference system may be not only incomplete but incoherent in architecture, and moral systems and goals obtained via reflection are almost certainly not total (will lack in some contexts), creating problem in RLHF.
The first assumption, that part of neurons is basically randomly initialized, can't be tested really well because all humans are born in similar gravity field, see similarly-structured images in first days (all "colorful patches" correspond to objects which are continuous, mostly flat or uniformly round), etc and that leaves a generic imprint.
The post explains difference between two similar-looking positions:
- Model gets reward, and attempts to maximize it by selecting appropriate actions.
- Model does some actions on the environment, and selection/mutation process calculates the reward to find out how to maximize it by modifying the model.
Those positions differ in level at which optimization pressure is applied. Actually, either of them can be implemented (so, the "reward" value can be given to the model), but common RL uses the second option.
This post suggests a (and, quite possibly, the) way to select outcome in bargaining (fully deterministic multi-player game).
ROSE values replace Competition-Cooperation ones by having players not compete against each other in attempts to extract more utility but average their payoffs over possible initiative orders. A probable social consequence is noted, that people wouldn't threaten everyone else in order to get something they want but would rather maximize own utility (improve their situation themselves).
ROSE values are resistant to unconditional threats. Unfortunately, conditional ones can't be easily distinguished from conditional bonuses (if player B decides to pay A if A chooses some action from a certain set) so counteracting that requires further research. There are at least two more important directions of research: non-deterministic and imperfect-information games.
In comments, a problem is mentioned: what happens if there are multiple ROSE equilibrium points? I don't believe solution "bargain over those points only" will work, as it's possible all of those points will remain equilibriums when calculating ROSE values again. Hopefully, this players disagreement will involve only tiny slices of utility and not matter much overall.
Meta-comment: I think articles and can explain more than if there were just articles and .
I've noticed that my thoughts are bound to what I see, and if I go to another room old thoughts are muted. Are there some techniques to protect from this effect?
It’s probably best to avoid sleeping in the same room as a smart device, or anything with sensors, an operating system, and also a speaker.
I think it's best tested! Just direct a webcam/microphone on the phone with recording left on (if cable is long enough, computer may even be located in another room).
What is a better model?
It's possible to do such modelling with beta-distributions (actually similar to meta-probabilities).
Combination of (something like non-informative prior) and (information obtained from friend) will be - moved from equal probabilities far more than combination .
We want to be able to point at Elsa’s case of value change and argue that it is problematic and should be prevented, and we want to be able to say that Daniel’s case of value change is fine and does not need to be prevented, without in either case basing our argumentation on whether or not loving jazz is a morally acceptable or not. As such, I argue that the relevant difference we are picking up on here pertains to the legitimacy (or lack thereof) of the value change process (in the sense I've described it above).
Is it really the relevant difference?
I think that there could be cases of acceptable illegitimate value change; that is, if both current I and I-as-in-CEV (in the future, knowing more, etc) would endorse the change, but it were done without a way to course-correct it. Metaphor: imagine you had to walk over a hanging bridge so that you couldn't stop in the middle at risk of injury.
So, in my opinion legitimacy can be based on nature of value change only, but acceptability is also based on the opinion of person in question.
Can't Bob say "Alice is doing something strange; she is rejecting bet of smaller value at certain odds but accepting larger one, leaving free money on the table in former case, and that isn't rational as well"?
Epistemic status: heard in Internet comments
There is another coordination-like problem when women think that saying that they're looking for dates is weird/low-status.
When writing a novel, is there some speed threshold (for instance, a few pages per week) below which it's actually not worth writing it? (For example, if ideas become outdated faster than text is written.)
I think this is a highly frequent real-world scenario.
You have a week to complete optional but boring task that can benefit you in future or to do nothing (procrastinate? can this word apply to optional tasks?). Afterwards, you totally forget the emotions you had and are left with whatever you have done.
I think that people tend to really forget whether their activity was dull or interesting over time.
Happiness is paradoxical. A 2011 study showed that people seeking happiness were generally unhappier than others.
This doesn't seem paradoxical to me? Ones who are more happy than others are less likely to pursue additional happiness.
Thus, in an economy that has 100 goods, for example, there would be a total of 4950 prices or “exchange rates” between one good and the others.
Actually, this can be avoided by making the prices virtual and having a liquidity pool that would automatically allow to calculate prices. Liquidity pools can almost surely extend to more than 2 goods while still having well-defined .
(Actually, liquidity pool exchanges can be used even without computers; they don't present complicated expressions unless someone wants to add goods to pool.)
This suggestion looks like it makes tradeoffs between different moral theories just like between agents. This is an usecase for ROSE values; they can be used instead of randomization to get even better results, where negative-fanatic-theories can't virtually veto some actions: Threat-Resistant Bargaining Megapost: Introducing the ROSE Value.
It seems like a different definition of utility ("the sum of happiness minus suffering for all conscious beings") than usual was introduced somewhere. Concept of utility doesn't really restrict what it values; it includes such things as paperclip maximizers, for instance.
As well, agents can maximize not expected utility but minimal one over all cases, selecting guaranteed nice world over hell/heaven lottery.
I have read that post, and here are my thoughts:
- The essence of the post is only in one section of seven: "Exploring Nuances: Case Studies of Evolving Portrayals".
- Related work descriptions could be fit into one sentence for each work, to make reading the report easier.
- Sentences about relevance of work, being pivotal step in something, etc don't carry much meaning.
- The report doesn't state what to anticipate; what [social] observations can one predict better after reading it.
Overall, the post doesn't look like it tries to communicate anything, and it's adapted to formal vague style.
After all, if there’s a demon who pays a billion dollars to everyone who follows CDT or EDT then FDTists will lose out.
How does demon determine what DT a person follows?
If it's determined by simulating person's behavior in Newcomb-like problem, then once FDTist gets to know about that, he should two-box (since billion dollars from demon is more than million dollars from Omega).
If it's determined by mind introspection, then FDTist will likely self-modify to believe to be CDTist, and checking actual DT becomes a problem like detecting AI deceptive alignment.
I guess that people who downvoted this would like to see more details why this "court" would work and how won't it be sued when it misjudges (and the more cases there are, the higher probability of misjudge is).
(meta: I neither downvoted nor upvoted the proposal)
Second question: how does this work in different axiom systems? Do we need separate markets, or can they be tied together well? How does the market deal with "provable from ZFC but not Peano"? "Theorem X implies corollary Y" is a thing we can prove, and if there's a price on shares of "Theorem X" then that makes perfect sense, but does it make sense to put a "price" on the "truth" of the ZFC axioms?
Actually, I don't think that creates any problem? Just create shares of "ZFC axioms" and "not ZFC axioms" via logical share splitting. If you are unable to sell "not ZFC axioms", that only means that price of one main share is $1 (though, it's likely possible to prove something fun if we take these axioms as false).
Once the person already exists, it doesn’t matter what % of agents of a certain type exist. They exist—and as such, they have no reason to lose out on free value. Once you already exists, you don’t care about other agents in the reference class.
This means that you cannot credibly precommit to paying in a gamble (if coin comes up tails, you pay $1, otherwise you receive $20), since if coin comes up tails "you don't care about other variants" and refuse to pay.
Welcome to LessWrong! Have you also read about Bayes' Theorem?
- is not right since it's trivially true (false statement can imply anything and true statement always implies True); were you talking about ?
- > how different do we expect the conclusions of those developments to be compared to the moral frameworks we have today?
Rationalists usually can't coherently expect that beliefs of other rational system will be different in a pre-known way, since that's a reason to update own beliefs. See also: the third virtue of rationality, lightness.
It seems that your argument models will require a way of updating weights on one of the next steps, so I'd recommend you to read the Sequences.
The experiment is commonly phrased in non-anthropic way by statisticians: there are many items getting sequential unique numbers, starting from 1. You get to see a single item's number and have to guess how many items are there, and the answer is . (Also, there are ways to guess count of items if you've seen more than one index)
I've noticed that I'm no longer confused about anthropics, and a prediction-market based approach works.
- Postulate. Anticipating (expecting) something is only relevant to decision making (for instance, expected utility calculation).
- Expecting something can be represented by betting on a prediction market (with large enough liquidity so that it doesn't move and contains no trade history).
- If merging copies is considered, the sound probability to expect depends on merging algorithm. If it sums purchased shares across all copies, then the probability is influenced by splitting; if all copies except one are ignored, then not.
- If copies are not merged, then what to anticipate depends on utility function.
- "quantum suicide" aka rewriting arbitrary parts of utility function with zeroes is possible but don't you really care about the person in unwanted scenario? Also, if AGI gets to know that, it can also run arbitrarily risky experiments...
Sleeping Beauty: if both trades go through in the case she is woken up twice, she should bet at probability 1/3. If not (for example, living the future: this opportunity will be presented to her only once), it's coherent to bet at probability 1/2.
I've heard a comment that betting odds is something different from probability:
... what makes you think it [probability] should have a use? You can feel sure something will happen, or unsure about it, whether or not that has a use.
Well, if you feel sure about an event with incorrect probability, you may end up in suboptimal state with respect to instrumental rationality (since expected utility calculations will be flawed), so it's perhaps more useful to have correct intuitions. (Eliezer may want to check this out and make fun of people with incorrect intuitions, by the way :-))
New problems are welcome!
Anthropics based on prediction markets - Part 2
Follow-up to https://www.lesswrong.com/posts/xG98FxbAYMCsA7ubf/programcrafter-s-shortform?commentId=ySMfhW25o9LPj3EqX.
Which Questions Are Anthropic Questions? (https://www.lesswrong.com/posts/SjEFqNtYfhJMP8LzN/which-questions-are-anthropic-questions)
1. The Room Assignment Problem
You are among 100 people waiting in a hallway. The hallway leads to a hundred rooms numbered from 1 to 100. All of you are knocked out by a sleeping gas and each put into a random/unknown room. After waking up, what is the probability that you are in room No. 1?
2. The Incubator
An incubator enters the hallway. It will enter room No.1 and creat a person in it then does the same for the other 99 rooms. It turns out you are one of the people the incubator has just created. You wake up in a room and is made aware of the experiment setup. What is the probability that you are in room No.1?
3. Incubator + Room Assignment
This time the incubator creats 100 people in the hall way, you are among the 100 people created. Each person is then assigned to a random room. What is the probability that you are in Room 1?
In all of those cases, betting YES on probability 1% is coherent in the sense that it leads to zero expected profit: each of the people buys 1 "ROOM-1" share at price of 1/100, and one of them wins, getting back 1 unit of money.
Wouldn't tape disbalance the fan so that it breaks faster / starts to make sound after a few months? Not sure, but these problems seem plausible.
If you're being simulated by Omega, then opening the second box ends the simulation and kills you.
Is making decisions based on being simulated really coherent?
One may also imagine magic function is_counterfactual_evaluated, where you can do this query to the universe at an arbitrary time and get to know whether you're in simulation. However, Omega has power to simulate you in the case where you don't understand that you're in simulation - that is, where is_counterfactual_evaluated returns false.
Also, by simulating this scenario you kill both simulated-you and Omega. Doesn't this have extremely negative utility? :-)
Anthropics - starting from scratch
I'm trying to derive coherent scheme of anthropic reasoning from scratch, posting my thoughts here as I go. Pointing flaws out is welcome!
The anthropic trilemma (https://www.lesswrong.com/posts/y7jZ9BLEeuNTzgAE5/the-anthropic-trilemma)
So here's a simple algorithm for winning the lottery:
Buy a ticket. Suspend your computer program just before the lottery drawing - which should of course be a quantum lottery, so that every ticket wins somewhere. Program your computational environment to, if you win, make a trillion copies of yourself, and wake them up for ten seconds, long enough to experience winning the lottery. Then suspend the programs, merge them again, and start the result. If you don't win the lottery, then just wake up automatically.
The odds of winning the lottery are ordinarily a billion to one. But now the branch in which you win has your "measure", your "amount of experience", temporarily multiplied by a trillion. So with the brief expenditure of a little extra computing power, you can subjectively win the lottery - be reasonably sure that when next you open your eyes, you will see a computer screen flashing "You won!" As for what happens ten seconds after that, you have no way of knowing how many processors you run on, so you shouldn't feel a thing.
Yes, there is something like paradox on whether you should anticipate winning or losing the lottery. So let's taboo "anticipate":
Postulate 1. Anticipating (expecting) something is only relevant to decision making (for instance, expected utility calculation).
So to measure the expectation one needs to run a prediction market. The price of an outcome share is equal to the probability of its outcome, so that expected payoff of each share is exactly zero. For sake of argument, let's assume that the market is unmovable.
Once the branch when you won was split into trillion copies, you can expect winning the lottery at odds 1000:1. So your 10^12 + 10^9 copies (trillion who have won the lottery, billion who have lost) will each buy 1 "WIN" share at price of 1000/1001.
Now, let's consider what should happen during merge for such expectation to be sound. If your trillion copies are merged into one who has bought 1 "WIN" share, then the summary profit is 1 - (10^9 + 1) * 1000/1001, which is clearly negative. Seems like something went wrong and you shouldn't have anticipated 1000:1 odds of winning?
The different merging procedure makes everything right. In it, you need to sum all purchased "WIN" shares, so that your winning copy has 10^12 "WIN" shares and each of 10^9 losing copies has one share. The summary profit is 10^12 - (10^9 + 10^12) * 1000 / 1001 = 0!
That can be translated as "if merging procedure will multiply your utility of an outcome by number of merged copies, then you can multiply the odds by copies count". On the other hand, if during merge all copies except one are going to be ignored, then the right probability to expect is 1:10^9 - totally unchanged. And if you don't know the merging procedure, you'll have to use some priors and calculate the probabilities based on that.
Why doesn't this affect real life much?
I guess that splitting rate (creation of new Boltzmann brains, for example) and merging rate are approximately equal so all the possible updates cancel each other.
why on Earth would I offer any of these bets? They all lose money for me. Am I just a cosmic troll trying to bankrupt some utilitarians or something?
Casino would definitely attempt that, I think.
The game is zero-sum; if you have significantly larger money than the bettor, you may try to bankrupt him and get his money with very high probability, and it is easier to do that against Linda than against Logan.
I guess that alignment problem is "difference in power between agents is dangerous" rather than "AGI is dangerous".
Sketch of proof:
- Agent is either optimizing for some utility function or not optimizing for any at all. The second case seems dangerous both for it and for surrounding agents [proof needed].
- Utility function probably can be represented as vector in basis "utility of other agents" x "power" x "time existing" x etc. More powerful agents move the world further along their utility vectors.
- If utility vectors of a powerful agent (AGI, for example) and of humans are different, on some level of power this difference (also a vector) will become sufficiently big that we consider the agent misaligned.
Strictly improving intelligent and rational lifeforms will over time converge into the same (non distinguishable) beings (kind of like a generalized form of "Great minds think alike").
This assumption doesn't seem true. While knowledge will converge as result of improving epistemic rationality, goals will not, and for agent to change its terminal goals is in most cases irrational, since it won't reach the currently intended goals.
The goals can be seen as vectors in a high-dimensional space; and if humanity goals vector and AI goals vector are different, then this difference, no matter how small, will be critical at high power.
everything good about humanity is good in itself
Veto "good in itself"; do you mean "valuable by currently existing civilization"?
While reading Project Lawful, I've come across concept of algorithmic commitments and can't find any other mentions of them.
https://www.glowfic.com/replies/1770000#reply-1770000
Bots for prisoner's dilemma, defined as functions (other bot -> decision):
let CooperateBot(_) = Cooperate
let DefectBot(_) = Defect
let FairBot (X) = if Provable( "X(Fairbot) == Cooperate" ) then Cooperate else Defectlet Provable-1("X") = Provable(" ~Provable(0=1) -> X")
let PrudentBot(X) = if (Provable("X(PrudentBot) == Cooperate") && Provable-1("X(DefectBot) == Defect")) then Cooperate else Defect
The topic looks pretty interesting; are there any further examples of such commitments?
UPD: I offer Manifold bounty for finding such examples!
Current AIs are mostly not explicit expected-utility-maximizers. I think this is illustrated by RLHF (https://huggingface.co/blog/rlhf).
Humans have never completed at large scale engineering task without at least one mistake
This line does not seem sufficiently well-defined.
Both "launch a satellite" and "launch the specific satellite" are large scale engineering tasks (or were, in the previous century); the first one had some mistakes, and in most cases the second one had no mistakes.
Transferring the argument to AI: the mistake may happen and be fixed while the task is not "create aligned AGI" but some prerequisite one, so it doesn't ensure that the final AGI is unaligned.
If an arbitrary counter-party can consume the resource, this can be rephrased as destroying other agent's utility and this seems to be solved with threat-resistant ROSE values.
Not sure this is a totally correct application, though, because the consumer also gains some utility.
Specifically, if o_worst is the worst outcome according to the agent under discussion and o_best is its best outcome, set U(o_worst)=0, U(o_best)=1 and for the other outcomes o, set U(o) = p where p is the probability for which the agent is indifferent between o and the lottery [p, o_best; 1-p, o_worst].
That's a nice workaround!
Wouldn't that break if the sum (or integral) of an agent's utility function over the world-state space was negative? Normalization would reverse that agent's utility.
The situation becomes even worse if the sum over all outcomes is zero.
Yes, this is indeed a preference of mine (and other people as well), and I'm attempting to find the way to combine utilities that is as good as possible according to my and other people preferences (so that it can be incorporated into AGI, for example).