LessWrong 2.0 Reader

Although under strict preference utilitarianism, wouldn't change in values/moral progress be considered bad, for the same reason a paperclip maximizer would consider it bad?

I should say we assume that we're deciding which one a stable, incorruptible AI should choose. I'm pretty sure any moral system which chose torture in situations like this would not lead to good outcomes if applied in a practical circumstance, but that's not what I'm wondering about, I'm just trying to figure out which outcome is better. In short, I'm asking an axiological question, not a moral one. https://slatestarcodex.com/2017/08/28/contra-askell-on-moral-offsets/

My intuition strongly says that the torture is worse here even though I choose torture in the original, but I don't have an argument for this because my normal axiological system, preference utilitarianism, seems to unavoidably say torture is better.

If consequences are completely ignored, I lean towards the torture, but if consequences are considered I would choose no torture out of hope it accelerates moral progress (at least if they had never seen someone who "aught to be tortured" get away, the first one might spark change. which might be good?). In the speck case, I choose torture.

Language isn’t just about efficiency; cultural aesthetic is a terminal value.

Also:

There have been various academics who have discussed decision theory before MIRI even came into existence. I don't know if they're actually working on improving things, though. My sense is that they've mostly been just been sticking with causal decision theory and evidential decision theory. But there's probably at least some work done improving.

Can you give the probabilities that the agent assigns to B1 through D4 in the "sandboxed" counterfactual?

It's not a great study.  It's observational, and the best it can show is correlation.  It's just as likely that people with that score lower on cognitive tests are more likely to get covid, which doesn't seem unrealistic.  A bit of armchair analysis (with only a little bit of ax-grinding):  https://sebastianrushworth.com/2021/07/26/does-covid-cause-brain-damage/

Sounds correct. I was thinking how this applies to computer games:

Several subskills - technical perfection, new idea, interesting story, graphics, music... Different games become popular for different aspects (Tetris vs Mass Effect vs Cookie Clicker).

A frequent beginner mistake is making a game with multiple levels which feel like copies of each other. That's because you code e.g. five or ten different interactive elements, and then you use all of them in every level. It makes the first level needlessly difficult, and every following level boring. Instead, you should introduce them gradually, so each level contains a little surprise, and perhaps you should never use all of them in the same level, but always use different subsets, so each level has a different flavor instead of merely being more difficult.

Another beginner mistake is to focus on the algorithm and ignore the non-functional aspects. If one level has a sunset in background, and another level uses a night sky with moon, it makes the game nicer, even if the background does not change anything about functionality.

Yet another mistake is to make the game insanely difficult, because as a developer you know everything about it and you played the first level for hundred times, so even the insanely difficult feels easy to you. If most new players cannot complete the tutorial, your audience is effectively just you alone.

Some people may be successful and yet you don't want to be like them, e.g. because they optimize the product to be addictive, while you aim for a different type of experience; or their approach is "explore the market, and make a clone of whatever sells best", while you have a specific vision.

You should do a very simple game first, because you are probably never going to finish a complicated one if it's your first attempt. I know a few people who ignored this advice, spent a lot of time designing something complex, in one case even rented a studio... but never finished anything. (Epistemic check: possible base-rate fallacy; most people never write a complete computer game, this might include even most of those who started small.) And the more time you wasted trying to make a complicated game, the less likely you are to give up and start anew.

Successful game authors often recycle good ideas from their previous, less successful games.

The audience is famously toxic. Whatever game you make, some people will say horrible things about the game and about you in general. It is probably wise to ignore them. (Epistemic check: so you're saying that you should only listen to those who liked your game? Yeah... from the profit perspective, the important thing is how many fans you have, not what is their ratio to haters. A game with 1000 fans and 10000 haters is more successful than a game with 10 fans and 1 hater.)

Being good at designing logical puzzles does not translate into being good at designing 3D shooters, and vice versa.

This is known as "agent simulates predictor". There has been plenty of discussion of this problem. I'm currently feeling too lazy to try to summarize or link all the approaches, but here [LW(p) · GW(p)] are some thoughts I had about it via my infra-Bayesian theory.

In my experience teachers tend to only give examples of typical members of a category. I wish they'd also give examples along the category border, both positive and negative. Something like: "this seems to have nothing to do with quadratic equations, but it actually does, this is why" and "this problem looks like it can be solved using quadratic equations but this is misleading because XYZ". This is obvious in subjects like geography, (when you want to describe where China is, don't give a bunch of points around Beijing as examples, but instead draw the border and maybe tell about ongoing territorial conflicts) but for some reason less obvious in concept-heavy subjects like mathematics.

Another point on my wishlist: create sufficient room for ambition. Give bonus points for optional but hard exercises. Tell about some problems that even world's top experts don't know how to solve.