Posts

Arguing Absolute Velocities 2019-09-01T02:11:16.756Z
Minneapolis SSC Meetup 2019-09-21 2019-09-01T01:04:43.279Z
Welcome to Twin Cities SSC Meetup [Edit With Your Details] 2019-09-01T00:06:08.457Z
Closed Limelike Curves's Shortform 2019-08-16T03:53:12.158Z
Old Man Jevons Can’t Save You Now (Part 2/2) 2019-07-26T03:51:34.447Z
Thank You, Old Man Jevons, For My Job 2019-07-06T18:37:48.146Z

Comments

Comment by Closed Limelike Curves on Redirecting one’s own taxes as an effective altruism method · 2023-12-03T23:20:42.689Z · LW · GW

(it would be convenient if yes, but this would feel surprising - otherwise you could just start a corporation, not pay your taxes the first year, dissolve it, start an identical corporation the second year, and so on.)

This (a consistent pattern of doing the same thing) would get you prosecuted, because courts are allowed to pierce the corporate veil, which is lawyer-speak for "call you out on your bullshit." If it's obvious that you're creating corporations as a legal fiction to avoid taxes, the court will go after the shareholders directly (so long as the prosecution can prove the corporation exists in name only).

Comment by Closed Limelike Curves on LLMs are (mostly) not helped by filler tokens · 2023-09-09T20:50:11.609Z · LW · GW

Because GPT-3.5 is a fine-tuned version of GPT-3, which is known to be a vanilla dense transformer.

GPT-4 is probably, in a very funny turn of events, a few dozen fine-tuned GPT-3.5 clones glued together (as a MoE).

Comment by Closed Limelike Curves on Geometric Rationality is Not VNM Rational · 2023-09-04T02:55:44.121Z · LW · GW

Whether the couple is capable of having preferences probably depends on your definition of “preferences.” The more standard terminology for preferences by a group of people is “social choice function.” The main problem we run into is that social choice functions don’t behave like preferences. By Gibbard’s theorem, we can guarantee that any social choice function is either Pareto inefficient or unobservable (because it’s not incentive-compatible).

Sometimes, Pareto inefficiency is the price we must pay for people to volunteer information. (e.g. random dictatorship is Pareto-inefficient if we’re risk averse, but it encourages everyone to state their true preferences.) But I don’t see what information we’re getting here. Everyone’s preferences were already known ahead of time; there was no need to choose the inefficient option.

Comment by Closed Limelike Curves on Geometric Rationality is Not VNM Rational · 2023-08-31T01:42:56.185Z · LW · GW

One elephant in the room throughout my geometric rationality sequence, is that it is sometimes advocating for randomizing between actions, and so geometrically rational agents cannot possibly satisfy the Von Neumann–Morgenstern axioms.

It's not just VNM; it just doesn't even make logical sense. Probabilities are about your knowledge, not the state of the world: barring bizarre fringe cases/Cromwell's law, I can always say that whatever I'm doing has probability 1, because I'm currently doing it, meaning it's physically impossible to randomize your own actions. I can certainly have a probability other than 0 or 1 that I will do something, if this action depends on information I haven't received. But as soon as I receive all the information involved in making my decision and update on it, I can't have a 50% chance of doing something. Trying to randomize your own actions involves refusing to update on the information you have, a violation of Bayes' theorem.

The problem is they don't want to switch to Boston, they are happy moving to Atlanta.

In this world, the one that actually exists, Bob still wants to move to Boston. The fact that Bob made a promise and would now face additional costs associated with breaking the contract (i.e. upsetting Alice) doesn't change the fact that he'd be happier in Boston, it just means that the contract and the action of revealing this information changed the options available. The choices are no longer "Boston" vs. "Atlanta," they're "Boston and upset Alice" vs. "Atlanta and don't upset Alice."

Moreover, holding to this contract after the information is revealed also rejects the possibility of a Pareto improvement (equivalent to a Dutch book). Say Alice and Bob agree to randomize their choice as you say. In this case, both Alice and Bob are strictly worse off than if they had agreed on an insurance policy. A contract that has Bob more than compensate Alice for the cost of moving to Boston if the California option fails would leave both of them strictly better off.

Comment by Closed Limelike Curves on Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research · 2023-08-31T00:55:01.351Z · LW · GW

You forgot to include a sixth counterargument: you might successfully accomplish everything you set out to do, producing dozens of examples of misalignment, but as soon as you present them, everyone working on capabilities excuses them away as being "not real misalignment" for some reason or another.

I have seen more "toy examples" of misalignment than I can count (e.g. goal misgeneralization in the coin run example, deception here, and the not-so-toy example of GPT-4 failing badly as soon as it was deployed out of distribution--with the only thing needed to break it being a less-than-perfect prompt and giving it the name Sydney. We've successfully shown AIs can be misaligned in several ways we predicted ahead of time according to theory. Nobody cares, and nobody has used this information to advance alignment research. At this point I've concluded AI companies, even ones claiming otherwise, will not care until somebody dies.

Comment by Closed Limelike Curves on Compendium of problems with RLHF · 2023-08-31T00:37:39.658Z · LW · GW

Using RL(AI)F may offer a solution to all the points in this section: By starting with a set of established principles, AI can generate and revise a large number of prompts, selecting the best answers through a chain-of-thought process that adheres to these principles. Then, a reward model can be trained and the process can continue as in RLHF. This approach is potentially better than RLHF as it does not require human feedback.

I'd like to say that I fervently disagree with . Giving an unaligned AI the opportunity to modify its own weights (by categorizing its own responses to questions), then politely asking it to align itself, is quite possibly the worst alignment plan I've ever heard; it's penny-wise, pound-foolish. (Assuming it even is penny-wise; I can think of several ways to generate a self-consistent AI that would cost less.)

Comment by Closed Limelike Curves on Verification Is Not Easier Than Generation In General · 2023-07-22T02:41:32.864Z · LW · GW

I think there's a fundamental asymmetry in the case you mentioned--it's not verifying whether a program halts that's difficult, it's writing an algorithm that can verify whether any program halts. In other words, the problem is adversarial inputs. To keep the symmetry, we'd need to say that the generation problem is "generate all computer programs that halt," which is also not possible.

I think a better example would be, how hard is it to generate a semiprime? Not hard at all: just generate 2 primes and multiply them. How hard is it to verify a number is semiprime? Very hard, you'd have to factor it.

Comment by Closed Limelike Curves on Hell is Game Theory Folk Theorems · 2023-05-08T01:05:53.526Z · LW · GW

That's correct, but that just makes this a worse (less intuitive) version of the stag hunt.

Comment by Closed Limelike Curves on How to respond to the recent condemnations of the rationalist community · 2023-04-04T20:30:19.406Z · LW · GW

Really? I think a tiny bit of effort will do exactly nothing, or at best further entrench their beliefs ("See? Even the rationalists think we have valid points!"). The best response is just to ignore them, like most trolls.

Comment by Closed Limelike Curves on CIRL Corrigibility is Fragile · 2023-04-04T17:20:37.017Z · LW · GW

A lot of the corrigibility properties of CIRL come from uncertainty: it wants to defer to a human because the human knows more about its preferences than the robot. Recently, Yudkowsky and others described the problem of fully updated deference: if the AI has learned everything it can, it may have no uncertainty, at which point this corrigibility goes away. If the AI has learned your preferences perfectly, perhaps this is OK. But here Carey's critique of model misspecification rears its head again -- if the AI is convinced you love vanilla ice cream, saying "please no give me chocolate" will not convince it (perhaps it thinks you have a cognitive bias against admitting your plain, vanilla preferences -- it knows the real you), whereas it might if it had uncertainty.

The standard approaches to dealing with this are nonparametric models, safe Bayes, and including many different models in your space of all possible models.

Comment by Closed Limelike Curves on An Appeal to AI Superintelligence: Reasons to Preserve Humanity · 2023-03-21T16:24:48.955Z · LW · GW

The AI has a response. It says no.

Comment by Closed Limelike Curves on [Cross-post] Is the Fermi Paradox due to the Flaw of Averages? · 2023-01-19T18:29:04.939Z · LW · GW

The article provides one possible resolution to one Fermi paradox, but not another, and I think both a lot of literature and the comments here are pretty confused about the difference.

The possibly-resolved Fermi paradox: "Why do we see no life, when we'd expect to see tens of thousands of civilizations on average?" Assuming there really is no alien life, this paper answers that pretty convincingly: the distribution of civilizations is highly-skewed, such that the probability that only one civilization exists can be pretty large even if the average number of civilizations is still very high.

The unresolved Fermi paradox is "Why aren't there any aliens?" This paper doesn't answer that question (nor does it try to). It's just pointing out that there's a ton of possible reasons, so it's not all that surprising that some combination of them might be very restrictive

.

Comment by Closed Limelike Curves on Announcing $5,000 bounty for (responsibly) ending malaria · 2022-09-25T21:25:24.606Z · LW · GW

It sounds like a good idea, so I wouldn’t be surprised if it already exists.

Comment by Closed Limelike Curves on Announcing $5,000 bounty for (responsibly) ending malaria · 2022-09-25T06:05:51.404Z · LW · GW

I pledge to match the bounty of the next person to pledge $5,000, because of research showing this encourages people to donate money.

So if someone else pledges, we’ve reached a third of the median US salary.

Comment by Closed Limelike Curves on History of Zero-based Months? · 2022-08-25T02:51:52.888Z · LW · GW

I can confirm Julia and R both have one-based arrays and also one-based month numbers. I'm guessing they tend to line up quite often.

Comment by Closed Limelike Curves on Pop Culture Alignment Research and Taxes · 2022-04-26T21:18:49.737Z · LW · GW

If “women’s college” is a proxy variable for “liberal arts college”, that’s a good reason to ding people for listing a women’s college.

I suspect you're misunderstanding what a "Liberal arts college" is. In theory, a liberal arts college is one that focuses exclusively on "Academic" subjects, rather than purely practical ones. Math, science, and technology would all fall under the liberal arts, but a liberal arts college won't offer degrees in, say, accounting. In practice, a liberal arts college is a small college that focuses on teaching and only offers undergraduate degrees.

Liberal arts undergrads generate a disproportionate number of PhDs in the sciences. Swarthmore, for instance, has more Nobel laureates per student than any other school (including all the Ivy League colleges).

Comment by Closed Limelike Curves on School Has It Backwards · 2020-07-13T16:44:10.665Z · LW · GW

Do you have any strong evidence to back this up? Opinions on education are dime a dozen, studies are what I'd like to see for once.

Comment by Closed Limelike Curves on Zoom Technologies, Inc. vs. the Efficient Markets Hypothesis · 2020-05-22T03:35:13.555Z · LW · GW

This is fair and an important caveat. Pure arbitrage disappears quickly in the market. At the same time, though, no pure arbitrage profits is only necessary and not sufficient for full efficiency. An efficient market means no (or very few) positive expected utility opportunities still left around. The EMH still implies the price of shorts should have fallen until it became economically feasible to correct the mispricing.

Comment by Closed Limelike Curves on Help improve reasoning evaluation in intelligence organisations · 2020-05-12T02:37:34.858Z · LW · GW

This post seems both interesting and like a way to get a very unrepresentative sample.

Comment by Closed Limelike Curves on What marginal returns now? · 2020-05-12T02:36:30.581Z · LW · GW

I find it difficult to imagine how returns could be literally negative – it's not as though people are anti-correlated with the truth – unless you're taking into account some effect like "Annoying people by making too many coronavirus posts."

Comment by Closed Limelike Curves on Zoom Technologies, Inc. vs. the Efficient Markets Hypothesis · 2020-05-12T02:28:48.457Z · LW · GW

I agree, in that a very small improvement on a cryptocurrency shouldn't actually result in it outcompeting everything else. The thing is that Bitcoin has at this point been completely pummeled by other cryptocurrencies like Ethereum or stablecoins in terms of usefulness, to the point where it seems clear to me that Bitcoin mostly exists and will continue to exist only for speculative purposes.

(Also: The opposite of the virtue of lightness usually goes by the name of conservatism bias in the cognitive science literature.)

Comment by Closed Limelike Curves on Zoom Technologies, Inc. vs. the Efficient Markets Hypothesis · 2020-05-12T01:23:31.713Z · LW · GW

From my own experience reading the literature, I propose the following: The Bimodal Market Hypothesis.

The BMH states that the market is either a terrifying beast of infinite power that has priced in the number of grandchildren you will have into McDonald's stocks, or is so stupid it kinda makes you want to cry. Evidence for this includes:

1. The Dot-Com bubble, the fact that Bitcoin has a price exceeding $0 despite there currently existing cryptocurrencies that are strictly better in every way, and that time the market proved to be incapable of doing addition. Maybe it's when tech is involved?

2. But also, the Peso problem, where an apparent decades-long anomaly in the markets turned out to exist because the market was accurately estimating the probability that the Mexican government would be unable to maintain its peg.

Situation 1 occurs when idiots outnumber financial experts, and overwhelm the ability of "Smart money" to accurately price a stock in the face of a hype wave. Everyone reasonable avoids these stocks like the plague; shorting them exposes you to the possibility of bankruptcy in the case that the madness fails to subside in time. Situation 2 happens when financial experts poring over spreadsheets determine the value of a stock by rereading every statement until it's become their favorite work of literature before making a decision.

Policy proposal: Make it illegal to trade stocks if you haven't read A Random Walk Down Wall Street, don't know what a PE ratio is, if you do know what a candlestick graph is, or if you think that the reason why stocks go up is something like "Everyone thinks they will."

Comment by Closed Limelike Curves on Do We Change Our Minds Less Often Than We Think? · 2019-08-21T00:49:05.911Z · LW · GW

"Always go with your first intuition on multiple choice" reflects advice that's specifically good for students who are anxious because they're taking a test. The student will generally select the correct answer (or at least the one that's most likely to be correct). If they're somewhat uncertain about it, they'll then start to feel anxious; this anxiety will build over time, resulting in a more and more pessimistic assessment of how likely they are to be correct, resulting in even more anxiety. This continues until the student is either sufficiently pessimistic to think that the original answer was not the best or else changes it simply to relieve the stress. This happens even though no new information has been received, implying said change is unlikely to be correlated with correctness and more likely simply reflects a failure of human psychology.

In short, a test is an especially bad test case (pun fully intended) for this because the amount of bias being introduced increases over time with anxiety, rather than decreasing.

Comment by Closed Limelike Curves on [deleted post] 2019-07-26T20:18:46.447Z

Which is why, again, I'm suggesting Yudkowsky's writings describing compatibilism. There is a sense in which objective morality exists, and a sense in which it doesn't; there is similarly a sense in which the world is deterministic and a sense in which we have free will, and the appearance of conflict has to do with our intuitions being too vague and needing to be sharpened and defined better.

Comment by Closed Limelike Curves on [deleted post] 2019-07-26T03:22:59.006Z

Depends on what you mean by "Illusion" and "Ethics." I'd actually agree that the question of "Does an objective code of ethics exist" is confused like the free will one, and that there's a sense in which it does and a sense in which it doesn't.

The sense in which it does is twofold. First, codes of ethics can be objectively wrong; for example, any set of ethics which does not attempt to maximize an expected utility function must be inconsistent (See Yudkowsky's post on the von Neumann-Morgenstern axioms). So there's a sense in which moral systems can be straight-up bad. Another criterion that can rule out a moral system is strict Pareto inefficiency: If you have two moral systems, and every single agent agrees that they would be worse off under one of them than the other, then you really should chuck out that worse system.

However, out of these systems, you're not going to find only a single moral law printed on the fabric of the universe, regardless of how hard you try. Try starting with the Euthyphro dilemma, and just replace "God" with "The Universe." Even if in some far-off corner of Alpha Centauri there did actually end up being a "Moral thermometer" that measures how moral the universe is, and it went up whenever I kicked a little kid in the face, I'd tell that thermometer to **** off. The idea of a "Natural law of the universe" is a pretty bad one, given that even if it existed, there would be no reason to follow it if it clearly conflicted with the general human idea of morality.

Comment by Closed Limelike Curves on [deleted post] 2019-07-10T23:20:07.834Z

A brief note: I'm not 100% sold on the many-worlds hypothesis -- Bohmian interpretations strike me as similarly plausible, but I'm not going to discuss this right now because I doubt I'm educated enough to do so at a high level that doesn't just retread old arguments. With that out of the way, let's assume many-worlds is correct.

Given the existence of many-worlds, interpreting making a decision as "Choosing your own Everett branch" is not correct for one simple reason: In any case in which your decisions depend on something going on at the quantum level, you will simultaneously make every single decision you possibly could have made. There's a sense in which you're accidentally making the error of importing classical intuitions of "One world" into many-worlds -- in this case, the mistake is in believing that there is only one you, who can only make one decision. The reality is that all possible worlds already exist: Everything that has happened or will happen is fully captured by the mathematics of quantum mechanics, and you can't change anything about it. You can't change what ever has or ever will happen.

Now, the question becomes the same as for any determinist universe: whether or not determinism, and the fact that all decisions you will ever make are fully predictable by mathematics, actually makes ethics pointless. In this case, I suggest looking back at Yudkowsky's post on dissolving the question of free will, and then posting your answer here when you think you've got it. It's a good exercise, since it took me a while to figure it out myself. I look forward to seeing your answer.

Comment by Closed Limelike Curves on Thank You, Old Man Jevons, For My Job · 2019-07-10T21:47:45.031Z · LW · GW

You're right, although 1850-1900 captures the Second Industrial Revolution.