Lost Purposes

post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2007-11-25T09:01:50.000Z · LW · GW · Legacy · 79 comments

Contents

79 comments

It was in either kindergarten or first grade that I was first asked to pray, given a transliteration of a Hebrew prayer.  I asked what the words meant.  I was told that so long as I prayed in Hebrew, I didn't need to know what the words meant, it would work anyway.

That was the beginning of my break with Judaism.

As you read this, some young man or woman is sitting at a desk in a university, earnestly studying material they have no intention of ever using, and no interest in knowing for its own sake.  They want a high-paying job, and the high-paying job requires a piece of paper, and the piece of paper requires a previous master's degree, and the master's degree requires a bachelor's degree, and the university that grants the bachelor's degree requires you to take a class in 12th-century knitting patterns to graduate.  So they diligently study, intending to forget it all the moment the final exam is administered, but still seriously working away, because they want that piece of paper.

Maybe you realized it was all madness, but I bet you did it anyway.  You didn't have a choice, right?

A recent study here in the Bay Area showed that 80% of teachers in K-5 reported spending less than one hour per week on science, and 16% said they spend no time on science.  Why?  I'm given to understand the proximate cause is the No Child Left Behind Act and similar legislation.  Virtually all classroom time is now spent on preparing for tests mandated at the state or federal level.  I seem to recall (though I can't find the source) that just taking mandatory tests was 40% of classroom time in one school.

The old Soviet bureaucracy was famous for being more interested in appearances than reality.  One shoe factory overfulfilled its quota by producing lots of tiny shoes.  Another shoe factory reported cut but unassembled leather as a "shoe".  The superior bureaucrats weren't interested in looking too hard, because they also wanted to report quota overfulfillments.  All this was a great help to the comrades freezing their feet off.

It is now being suggested in several sources that an actual majority of published findings in medicine, though "statistically significant with p<0.05", are untrue.  But so long as p<0.05 remains the threshold for publication, why should anyone hold themselves to higher standards, when that requires bigger research grants for larger experimental groups, and decreases the likelihood of getting a publication?  Everyone knows that the whole point of science is to publish lots of papers, just as the whole point of a university is to print certain pieces of parchment, and the whole point of a school is to pass the mandatory tests that guarantee the annual budget.  You don't get to set the rules of the game, and if you try to play by different rules, you'll just lose.

(Though for some reason, physics journals require a threshold of p<0.0001.  It's as if they conceive of some other purpose to their existence than publishing physics papers.)

There's chocolate at the supermarket, and you can get to the supermarket by driving, and driving requires that you be in the car, which means opening your car door, which needs keys.  If you find there's no chocolate at the supermarket, you won't stand around opening and slamming your car door because the car door still needs opening.  I rarely notice people losing track of plans they devised themselves.

It's another matter when incentives must flow through large organizations - or worse, many different organizations and interest groups, some of them governmental.  Then you see behaviors that would mark literal insanity, if they were born from a single mind.  Someone gets paid every time they open a car door, because that's what's measurable; and this person doesn't care whether the driver ever gets paid for arriving at the supermarket, let alone whether the buyer purchases the chocolate, or whether the eater is happy or starving.

From a Bayesian perspective, subgoals are epiphenomena of conditional probability functions.  There is no expected utility without utility.  How silly would it be to think that instrumental value could take on a mathematical life of its own, leaving terminal value in the dust?  It's not sane by decision-theoretical criteria of sanity.

But consider the No Child Left Behind Act.  The politicians want to look like they're doing something about educational difficulties; the politicians have to look busy to voters this year, not fifteen years later when the kids are looking for jobs.  The politicians are not the consumers of education.  The bureaucrats have to show progress, which means that they're only interested in progress that can be measured this year.  They aren't the ones who'll end up ignorant of science.  The publishers who commission textbooks, and the committees that purchase textbooks, don't sit in the classrooms bored out of their skulls.

The actual consumers of knowledge are the children - who can't pay, can't vote, can't sit on the committees.  Their parents care for them, but don't sit in the classes themselves; they can only hold politicians responsible according to surface images of "tough on education".  Politicians are too busy being re-elected to study all the data themselves; they have to rely on surface images of bureaucrats being busy and commissioning studies - it may not work to help any children, but it works to let politicians appear caring.  Bureaucrats don't expect to use textbooks themselves, so they don't care if the textbooks are hideous to read, so long as the process by which they are purchased looks good on the surface.  The textbook publishers have no motive to produce bad textbooks, but they know that the textbook purchasing committee will be comparing textbooks based on how many different subjects they cover, and that the fourth-grade purchasing committee isn't coordinated with the third-grade purchasing committee, so they cram as many subjects into one textbook as possible.  Teachers won't get through a fourth of the textbook before the end of the year, and then the next year's teacher will start over.  Teachers might complain, but they aren't the decision-makers, and ultimately, it's not their future on the line, which puts sharp bounds on how much effort they'll spend on unpaid altruism...

It's amazing, when you look at it that way - consider all the lost information and lost incentives - that anything at all remains of the original purpose, gaining knowledge.  Though many educational systems seem to be currently in the process of collapsing into a state not much better than nothing.

Want to see the problem really solved?  Make the politicians go to school.

A single human mind can track a probabilistic expectation of utility as it flows through the conditional chances of a dozen intermediate events - including nonlocal dependencies, places where the expected utility of opening the car door depends on whether there's chocolate in the supermarket.  But organizations can only reward today what is measurable today, what can be written into legal contract today, and this means measuring intermediate events rather than their distant consequences.  These intermediate measures, in turn, are leaky generalizations - often very leaky.  Bureaucrats are untrustworthy genies, for they do not share the values of the wisher.

Miyamoto Musashi said:

"The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him. You must thoroughly research this."

(I wish I lived in an era where I could just tell my readers they have to thoroughly research something, without giving insult.)

Why would any individual lose track of their purposes in a swordfight?  If someone else had taught them to fight, if they had not generated the entire art from within themselves, they might not understand the reason for parrying at one moment, or springing at another moment; they might not realize when the rules had exceptions, fail to see the times when the usual method won't cut through.

The essential thing in the art of epistemic rationality is to understand how every rule is cutting through to the truth in the same movement.  The corresponding essential of pragmatic rationality - decision theory, versus probability theory - is to always see how every expected utility cuts through to utility.  You must thoroughly research this.

C. J. Cherryh said:

"Your sword has no blade. It has only your intention. When that goes astray you have no weapon."

I have seen many people go astray when they wish to the genie of an imagined AI, dreaming up wish after wish that seems good to them, sometimes with many patches and sometimes without even that pretense of caution.  And they don't jump to the meta-level.  They don't instinctively look-to-purpose, the instinct that started me down the track to atheism at the age of five.  They do not ask, as I reflexively ask, "Why do I think this wish is a good idea?  Will the genie judge likewise?"  They don't see the source of their judgment, hovering behind the judgment as its generator.  They lose track of the ball; they know the ball bounced, but they don't instinctively look back to see where it bounced from - the criterion that generated their judgments.

Likewise with people not automatically noticing when supposedly selfish people give altruistic arguments in favor of selfishness, or when supposedly altruistic people give selfish arguments in favor of altruism.

People can handle goal-tracking for driving to the supermarket just fine, when it's all inside their own heads, and no genies or bureaucracies or philosophies are involved.  The trouble is that real civilization is immensely more complicated than this.  Dozens of organizations, and dozens of years, intervene between the child suffering in the classroom, and the new-minted college graduate not being very good at their job.  (But will the interviewer or manager notice, if the college graduate is good at looking busy?)  With every new link that intervenes between the action and its consequence, intention has one more chance to go astray.  With every intervening link, information is lost, incentive is lost.  And this bothers most people a lot less than it bothers me, or why were all my classmates willing to say prayers without knowing what they meant?  They didn't feel the same instinct to look-to-the-generator.

Can people learn to keep their eye on the ball?  To keep their intention from going astray?  To never spring or strike or touch, without knowing the higher goal they will complete in the same movement?  People do often want to do their jobs, all else being equal.  Can there be such a thing as a sane corporation?  A sane civilization, even?  That's only a distant dream, but it's what I've been getting at with all these blog posts on the flow of intentions (aka expected utility, aka instrumental value) without losing purpose (aka utility, aka terminal value).  Can people learn to feel the flow of parent goals and child goals?  To know consciously, as well as implicitly, the distinction between expected utility and utility?

Do you care about threats to your civilization?  The worst metathreat to complex civilization is its own complexity, for that complication leads to the loss of many purposes.

I look back, and I see that more than anything, my life has been driven by an exceptionally strong abhorrence to lost purposes.  I hope it can be transformed to a learnable skill.

79 comments

Comments sorted by oldest first, as this post is from before comment nesting was available (around 2009-02-27).

comment by video · 2007-11-25T11:26:02.000Z · LW(p) · GW(p)

My head kinda aches. I think maybe I'll go watch some TV.

comment by Geremiah · 2007-11-25T13:31:34.000Z · LW(p) · GW(p)

We agree that an individual can manage their goal better - keeping an eye on the big picture, not ending up with lost purposes - than an organization. But what are you proposing (are you proposing anything?) Should we simplify, get rid of large and complex organizations? Seems that even when you count all the waste of lost purposes, we're more productive with them, at least in many cases ("better off" is another question, but let's leave that aside since it doesn't really bear directly on this issue). Complexity and scale have advantages, as well as this disadvantage; the mere existence or even extent of "lost purposes" doesn't imply lower overall efficiency.

When I try to explain to a computer how to do what I want (programming), my first explanation always leads to a million lost purposes (bugs). I test it, find the bugs, and clarify my instructions, trying to make them less ambiguous - less possible for the computer to fulfil the letter of my law without the spirit. Eventually I reach a point where testing doesn't turn up many more problems, but of course I'll never eliminate all lost purposes - software of sufficient complexity, even the most vetted commercial software, still has loads of bugs (and even aside from bugs, times where it follows the letter rather than the spirit of what the programmer intended). So why do we bother? Because the formalized instructions are more leverageable, and the computer is faster, than doing the work myself. Faster by enough orders of magnitude to make up for all the lost purposes and then some.

Ultimately my takeaways are:

-We should create only the laws/organizations/programs that make us much better off, even taking into account all the pitfalls of lost purposes.

-We should design these laws/organizations/programs as carefully as possible, with a constant eye to how every intention we express can lead to a frustrated goal.

-We should test and reassess these laws/organizations/programs as often as we possibly can, in every way that we possibly can, to catch as many instances as possible of their failing to accomplish what we wanted, and thus to continually refine them. And we should take the testability and refinability of a mechanism into account, when deciding whether it's worthwhile. (Computer programs are particularly amenable to this kind of testing, which is why they generally end up doing about 80% of what they're supposed to; an education system is much harder to treat this way.)

I think the concern of lost purposes is often underweighed by actual policymakers, but I also think it's unproductive to complain about lost purposes in the abstract, and misguided to design our lives around avoiding them (rather than avoiding the cases where they outweigh their benefits). Incorporating concerns about them (in the ways listed above) seems useful.

comment by Robin_Hanson2 · 2007-11-25T13:40:03.000Z · LW(p) · GW(p)

The general point is valid, but in particular cases one should be cautious in concluding than an org has lost it purpose - it might have a different purpose than you realized. School in particular my not have the purpose to teach, but to sort students by abilities.

Replies from: MarkusRamikin
comment by MarkusRamikin · 2012-02-28T10:06:11.919Z · LW(p) · GW(p)

I don't see how that, even if true, would make what schools are doing look any more sensible. Surely that purpose could be accomplished without wasting entire years of children's lives on sitting in a room doing nothing particularly useful.

Replies from: NancyLebovitz
comment by NancyLebovitz · 2012-02-28T14:31:16.507Z · LW(p) · GW(p)

The most obvious purpose of school is custodial.

Replies from: Morendil, MarkusRamikin
comment by Morendil · 2012-02-28T14:49:13.114Z · LW(p) · GW(p)

Tied for first place with compliance. :)

comment by MarkusRamikin · 2012-02-28T17:42:40.403Z · LW(p) · GW(p)

Yup. Now that purpose is consistent with wasting children's lives sitting in a room where nothing much useful happens.

Though I wouldn't say that purpose is "the" real one while teaching is not. There is a huge and constantly growing body of knowledge our civilisation needs passed on to new generations, and parents do genuinely want their children to learn. So surely talking about lost purposes here is justified. The people who create textbooks and curricula and teaching methods do have an important task on their hands, one that can't be measured solely by how well their work enables schools to keep children imprisoned for much of the day.

Replies from: NancyLebovitz
comment by NancyLebovitz · 2012-02-28T17:55:32.285Z · LW(p) · GW(p)

There's no reason to think institutions have only one purpose.

Replies from: MarkusRamikin
comment by MarkusRamikin · 2012-02-28T18:23:27.343Z · LW(p) · GW(p)

Well of course not? But Robin Hanson seems to think it makes sense to deny that particular purpose.

Sorry, I don't know if your responses are about the argument I'm making about that, or they're just observations that happen to be under my posts. ;)

Replies from: NancyLebovitz
comment by NancyLebovitz · 2012-02-28T18:37:31.255Z · LW(p) · GW(p)

To some extent, it's an isolated observation, but it's also an effort to correct what seemed like a drift (on my part, too) towards trying to identify a single purpose for conventional education.

comment by sam_kayley · 2007-11-25T13:58:37.000Z · LW(p) · GW(p)

In cases of delayed effects, linking pensions to outcomes years later might provide incentives.

For instance, teachers pensions could depend on the eventual earnings of their pupils, or elections could have tickboxes for evaluating the last politician.

(which doesn't help the problem of credit being due to many people)

Replies from: jeronimo196
comment by jeronimo196 · 2020-04-21T02:25:41.086Z · LW(p) · GW(p)

Necroing:

In cases of delayed effects, linking pensions to outcomes years later might provide incentives.

For instance, teachers pensions could depend on the eventual earnings of their pupils, or elections could have tickboxes for evaluating the last politician.

(which doesn't help the problem of credit being due to many people)

Punishments and rewards should be kept as close to the (un)desired behaviour as possible. Noise to signal ratio should be kept as low as possible. Linking teachers' pensions to their pupils earnings (which depend heavily on things like "the economy") is one of the most demotivating proposals I have ever heard.

As for grading politicians - you either vote for their party, or you don't. Which is the only feedback from the general public they maybe care about. (But why would they - most democracies have a two party system. Get elected, fill your pockets, wait your turn. Maybe wait two terms - big whoop!)

If you want to experience what proper, habit forming, behaviour changing incentive structure feels like, I suggest trying a "free-to-play" mobile game. Clear objectives, daily tasks, randomized rewards - Skinner would be proud. Be warned, the end result is not that different from a casino. But if you don't have an addictive personality, the risk might be worth it. I walked away with the following enlightenment: "All that psychology I've been reading about? It really does work! On me!!!".

Of course, gamefying complex tasks is difficult and in certain cases might be impossible. I think it is seriously worth a try in education.

comment by Robin_Brandt · 2007-11-25T14:09:33.000Z · LW(p) · GW(p)

Great post again!! This seems to be a good portal to the series as well! But my gosh I feel infoleptic!! I feel as though I would need 100 more brains to be able to do and read and learn everything I would like to with 1 brain! And if I would have a 100 brains I would probably feel I would need a 1000! etc. I think it is really sad this blog didn´t have some kind of rating system! There seems to be so many good posts. And if someone totally new to this would come here it would be hard to know where to start. So a list of the most important posts would be really great! Although 80% of your stuff seems to always hold the your standards! It would also be good if there would be a way to browse only your posts. Not that the other posts aren´t good, it is just that your posts have certain themes which one would like to hold on to and go deep with!

I am a great believer in utilizing/abusing our social brain for pursuing our higher values! Have you ever thought about having some kind of minimal microblog / status indicator of your real work on AI. Of course this would take some minutes of your precious time. This could make your work feel more connected to the world of other beings, which this is ultimately aimed for. If you would not have done any progress in a week you would directly feel embarrassed in front of humanity(which I am sure you already do, and I do too). If you would make great progress you would feel proud, and people could see it and give you emotional credit! I know you may not need this kind of system, you seem to have a very robust and disciplined mental operating system. It may also be that it would be only distracting to your real intrinsic altruism, or even distort your work. But it may also be that it would act as a healthy pusher, and make it more explicit that this is not your private project but the greatest project of humanity, which you happen to be one of the few that have competence to actually work on. It could also act as a please for people to come up with stimulating ideas(i.e. bad ideas that stimulate better ideas). But you already have a lot of distraction, so the balance have to be hard to keep. Good luck! Keep on pushing!

comment by Joshua_Fox · 2007-11-25T14:53:26.000Z · LW(p) · GW(p)

As always, well put. However, "Maybe you realized it was all madness, but I bet you did it anyway. You didn't have a choice, right?" Many students do have a terribly mercenary attitude. But a good number do voluntary study topics with no evident financial incentive.

"...I didn't need to know what the words meant, it would work anyway." That was a stupid thing for them to say. However, a common approach in these circumstances, similar but crucially different is "you don't need to know what the words mean now, just read them over and over as you gradually develop your understanding -- it's a good way to learn."

"..classroom time is now spent on preparing for tests..." Is there a better way of teaching, as proven by high-quality studies? (There may well be, but the question must be asked before dismissing the test-based approach.)

comment by StuartBuck · 2007-11-25T16:00:39.000Z · LW(p) · GW(p)

Not to get too sidetracked, because your overall point about education is well-taken, but this:

A recent study here in the Bay Area showed that 80% of teachers in K-5 reported spending less than one hour per week on science, and 16% said they spend no time on science. Why? I'm given to understand the proximate cause is the No Child Left Behind Act and similar legislation. Virtually all classroom time is now spent on preparing for tests mandated at the state or federal level. I seem to recall (though I can't find the source) that just taking mandatory tests was 40% of classroom time in one school.

is implausible on its face (particularly the assertion about 40% of classroom time -- that would mean kids spent 2+ hours every day taking tests). And if schools are ditching science (really? they're really spending 3 hours a day just on reading skills and 3 hours a day just on math?), that would be a very foolish decision on their part, even if their goal is to show high reading scores. Once kids know how to decode words, the best path to reading comprehension is lots and lots of substantive background knowledge -- such as science, history, etc.

By the way, the paragraphs about opening and shutting the car door are one of the best things I've ever read about the distortion of incentives in organizations.

comment by Nominull2 · 2007-11-25T16:09:09.000Z · LW(p) · GW(p)

If they have no interest in knowing the information for its own sake, that sounds like a problem with them, not with the university.

comment by A_reader_from_Brazil · 2007-11-25T16:12:54.000Z · LW(p) · GW(p)

Not everyone might be familiar with the expression "K-5". I had to google for it:

K-5 (pronounced "kay through five") is an American term for the education period from kindergarten to fifth grade.

Replies from: bruno-mailly
comment by Bruno Mailly (bruno-mailly) · 2018-07-30T13:08:28.864Z · LW(p) · GW(p)

Yup, every culture has it own education numbering system (that makes no sense) and seems blind to it not being universal. Just like language, numbering, date/hour format etc. except it seems particularly worse for reasons unclear.

I expected better from this author...

comment by Dan_Burfoot · 2007-11-25T16:13:41.000Z · LW(p) · GW(p)

If you think about this for a while, you'll begin to realize how much of our civilization is based on lost purposes.

Why do men wear ties? Why do we build houses out of the one substance that rots and burns? Why do we type with QWERTY keyboards? Why isn't the department of defense also responsible for homeland security (what does defense mean, if not homeland security)?

comment by Alan_Crowe · 2007-11-25T16:39:08.000Z · LW(p) · GW(p)

There is an interesting bias in the way that we approach these issues. We see ourselves as the prisoners of the system not the jailors. Or maybe I should say that we don't like to admit that we are lords as well as serfs.

One political response to the problem of lost purposes is to focus on government coercion which leads to libertarianism. I don't think that we should rush so quickly to a conclusion. First we should examine the origins of the problem more closely.

The key concept is contestability: is there an alternative BA course you can take that does not require mastery of 12 century knitting patterns. This not lead automatically to free-marketry. One can image contestible socialism.

Think about driving licences. Why does there have to be a single agency that issues them. The government could fund two, splitting the money in proportion to the number of licences issued. If queues are too long at one office, drivers will go to the over. It gets rather interesting if the competing offices get to set different driving tests. Do you go for the easy one or the hard one. Perhaps drivers who have passed the hard one get cheaper insurance rates. Perhaps that also tells you something important about the extra preparation being personally worth while, not just a money saving measure.

My favourite example is prison regimes. What do we expect from prison? We could set up a Prison Audit Agency that follows up prisoners 5 years after release to see how they are earning their money. There are many possibilities: professional criminal, dead, junkie, wino, deadbeat, honest employee, honest sole-trader, honest employer.

Imagine that the rehabilitation figures got the same kind of political attention as the crime figures. It would open the door to randomised assignment to different regimes. The hangers and floggers could run one prison, the hugs and therapy crowd another.

My guess is that neither really knows much about rehabilitation, a point that would become very clear a decade on as the figures are disappointing for both regimes. Both groups would be humbled and forced to think harder about what they were doing least their rivals responded more wisely and eventually built up a commanding lead in the results.

School vouchers are another example. Most state supported school systems are designed as pay-twice systems. If you pull your child out, you continue to pay taxes as before, but do not get the money back. You must find a second tranche of funds to pay your child's school fees. So schooling is only weakly constestible.

The curious point is that state power is weakly contestible, leading to poor delivery of services to the voter, who wearies of his political leaders and votes them out of office. Why do politicians favour a system that costs them office?

As we follow politics we see that reforms that make public services contestible, even when they remain in the state sector, are bitterly contested by civil servants. And not just public services. Trade barriers are about stopping the consumer from buying Toyota instead of General Motors, because the auto workers don't want their claim to be building wonderful cars to be contestible.

The 90% of us who are above average will do anything to keep the sunshine of contestibility from illuminating our particular patch line of work and causing our illusion to crumble to dust.

I don't see any hope for a direct attack on our tendency to lose our purposes. We like it that way and will fight to keep it like that. More insidiously we will always see ourselves as serfs not lords. Is there a name for this bias, the one that stops us admitting to ourselves that we are doing it to ourselves?

comment by Roland2 · 2007-11-25T16:42:27.000Z · LW(p) · GW(p)

As someone who actually practices medieval fighting arts I have to take objection to this:

Miyamoto Musashi said:

"The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means."

This is of course a generalization. The reality is more complex. Never forget that the real primary goal of yours is to stay alive and unharmed(at least that's my goal), so sometimes a parry makes sense, even if you can't cut the enemy in the same movement. You could cut or kill the enemy a lot of times if you simply ignore his blade and go straight for the attack/counter-attack, but that would also mean that you would be killed or injured.

Btw you don't want to always cut the enemy. Sometimes your objective will be to disarm him and take him captive without seriously injuring him.

But in GENERAL his advice is correct: defend and attack(cut) in the same movement, that's a good guideline although it's not always possible to do that depending on the actual situation.

Btw generating the entire art from withhin yourself would be kinda hard like trying to generate physics from yourself, you better have a good teacher. Of course you probably mean that you should be able to understand the idea behind each move so if the teacher doesn't explain it you better think about it and try to figure it out for yourself.

Replies from: taryneast
comment by taryneast · 2010-12-13T13:45:09.399Z · LW(p) · GW(p)

MM should be understood in context... the fighting style he talks about is caught up in a whole lot of issues about "honour".

Fights are to the death because they are over the matter of honour. So he doesn't even bother to talk about fights that are intended to dis-arm only. It makes sense (for him) to talk only of the fights that involve "killing the enemy" as the ultimate goal.

What you describe above is actually exactly what he was talking about. If you are thinking two thoughts (eg "I must not die" AND "I must kill the enemy") - the two goals conflict... and in a fight - you kinda only have time for one goal. As somebody who also has a lot of experience in medieval swordcraft, I can attest to that.

A sword comes at you - do you parry or lunge? ... you hesitate, you die.

So, you must pick one goal over the other.

All things being roughly equal, if you pick "I must stay alive" and your enemy picks "I must kill my enemy" - then your enemy will probably win, because he gives no thought to protecting himself (except insofar as it allows him to strike back next time and kill you) and is willing to take more risks in achieving his goal than you are.

I think that was MM's point.

Replies from: Vanilla_cabs
comment by Vanilla_cabs · 2020-04-25T13:46:24.818Z · LW(p) · GW(p)

All the previous points are interesting, but I think they're besides the point that EY (and probably MM) is trying to make.

It is not about conflicting terminal values. It is about never losing sight of terminal value(s) behind the current instrumental value(s) one is pursuing.

You don't parry for the sake of parrying, you have (an) ulterior motive(s). Same for opening car doors or rooting out biases.

comment by Roland2 · 2007-11-25T16:52:55.000Z · LW(p) · GW(p)

I don't understand your use of the word "versus" in the following sentence:

"The corresponding essential of pragmatic rationality - decision theory, versus probability theory - is to always see..."

versus means: As the alternative to or in contrast with.

I thought that probability theory is an integral part of decision theory, so the "versus" doesn't make sense.

comment by Eric_1 · 2007-11-25T17:18:54.000Z · LW(p) · GW(p)

A brilliant post with many links to the Yudkowsky Canon. It has just become a bookmark.

One quip: the study which revealed that a majority of research findings were false seemed to rely on a simulation, and on one meta-study performed earlier by the group. Have I understood this correctly?

Perhaps the p for biological experiments should be lower, but my first inclination is to defend the field I work in and its custom of p<0.05 .

Every time I open up an animal for surgery, the animals nerves are lying in slightly different places. There is a different amount of muscle-tissue obscuring the nerve I have to record from. I have to re-manufacture my electrode after every few surgeries. The hook on my electrode is differently shaped each time. When I put the nerve on the electrode, I pull it up with a different amount of force, in a different position, and do a different degree of damage to the nerve's processes.

Every animal is slightly different, and every surgery goes slightly differently.

Therefore, I think it might be understandable, why I need a p that is higher than a physicists, because if the physicist uses the same stock of materials, he has much less variability in his setup to begin with.

Replies from: Kurros
comment by Kurros · 2014-04-27T01:03:56.307Z · LW(p) · GW(p)

That sounds to me more like an argument for needing lower p-values, not higher ones. If there are many confounding factors, you need a higher threshold of evidence for claiming that you are seeing a real effect.

Physicists need low p-values for a different reason, namely that they do very large numbers of statistical tests. If you choose p=0.05 as your threshold then it means that you are going to be claiming a false detection at least one time in twenty (roughly speaking), so if physicists did this they would be claiming false detections every other day and their credibility would plummet like a rock.

comment by Caledonian2 · 2007-11-25T17:20:55.000Z · LW(p) · GW(p)

I find it ironic that a man who wants to abolish death is lecturing others about thorough research and being careful what goals you set.

comment by Peter_de_Blanc · 2007-11-25T17:26:08.000Z · LW(p) · GW(p)

Geremiah: it's worth understanding a problem before proposing a solution.

comment by mtc · 2007-11-25T17:31:07.000Z · LW(p) · GW(p)

It seems to me there are plenty of 'sane' corporations out there, even very large ones. Eliezer-

Corporations have the singular purpose of making money, and its generally very simply to determine if and how well a given company is doing this. The problems you describe with our education system or Soviet shoe factories apply to a certain extent within a corporation itself, but if a company strays too much from its purpose of making money (starts going 'insane'), this becomes quickly apparent in the form of declining profits. And of course if it can't find a way to remedy this, the corporation goes bust. The end.

I guess I was just sort of surprised you mentioned corporations here, because the pressures in a free market tend to mitigate the sorts of problems you identify in public education and communist economies. Perhaps you had something else in mind when asking if corporations can be sane; If so, I'd like to hear it.

comment by Video2 · 2007-11-25T18:43:02.000Z · LW(p) · GW(p)

Stuart Buck, Why is the Barone quote the only vestige of the 93 stuff? What was the jist of that lunch?

comment by James_Bach · 2007-11-25T18:48:57.000Z · LW(p) · GW(p)

As a high school dropout and aspiring philosopher of self-education, I salute you.

My son is homeschooled, and the homeschooling consists of... nothing. He studies spontaneously to solve problems that are authentic to him, just as I do. Mostly these involve video games and online fantasy role-playing games. It's like A.S. Neill's Summerhill school except he's all alone. There are disadvantages to this sort of education, but inauthenticity is not among them.

I am writing a book along these lines. It's about how we can creates ourselves as individual thinkers. It's called How I Learn Stuff: Secrets of a Buccaneer Scholar (old draft is temporarily online at http://www.satisfice.com/hils.pdf. The book itself should be finished in a few months).

comment by StuartBuck · 2007-11-25T19:00:23.000Z · LW(p) · GW(p)

Video -- I have no idea what you're talking about??

comment by Video2 · 2007-11-25T19:26:16.000Z · LW(p) · GW(p)

Stuart Buck, Apologies for the trollish behavior. I read your blog for a short while about 6 years back when I found that stuff intriguing, and seemed to recall it being the reason Barone contacted you. It makes perfect sense to erase that stuff (considering who was waving it around), but I found it interesting to find the Barone quote up there as a vestige. I'd love to know what he said about it all.

comment by Eric_1 · 2007-11-25T19:28:22.000Z · LW(p) · GW(p)

James Bach,

I am enjoying the online draft of your book very much, thank you for posting this!

comment by StuartBuck · 2007-11-25T19:32:52.000Z · LW(p) · GW(p)

I'm still not sure what you're getting at -- want to email me at stuartbuck@msn.com? Thanks.

comment by Pyramid_Head2 · 2007-11-25T21:08:10.000Z · LW(p) · GW(p)

Wow, Caledonian's comments are absolutely the worst...

comment by Chris · 2007-11-25T21:34:07.000Z · LW(p) · GW(p)

mtc, you could make a few volumes of Dilbert your required reading, to inform your faith in the 'sane corporation'. In fact, in such high powered company, I'm surprised to see non-qualified discussion of the intentionality of any organisation. Organisations don't have intentionality, individuals do (perhaps... if we don't let the Evolutionary Psychology crowd take it away from us..). As all Eliezer's anecdotes illustrate, these get lost in the wash of large numbers and multiple levels, networks, and gridlocks, so we are left with the dross of the latest organisational mantra which everyone gives lip service to and no-one believes. It is trivial to observe that the dominant intentionalities will be those of the most powerful individuals, but these also are destined to be diluted and lost. However, my personal believe is that inadequate individuation (accepting cultural norms of need and intentionality) is a far bigger problem than such intentionality as we have getting lost. Let's first work out who we are and what we have to say before worrying about our voice getting lost.

comment by Tiiba2 · 2007-11-25T22:43:17.000Z · LW(p) · GW(p)

Robin Hanson: School in particular my not have the purpose to teach, but to sort students by abilities.

Excusez moi, but what exactly is that supposed to mean? As far as I know, most principals want ALL of their students to go to Ivy League and make millions. I can't make any sense of your comment.

comment by ScentOfViolets · 2007-11-25T23:20:23.000Z · LW(p) · GW(p)

It means that contrary to his protestations, Robin really is a libertarian, and he is trying to push the meme that libertarian positions are really vanilla mainstream positions. Someone who is doing the exact opposite of overcoming bias, iow.

comment by LG · 2007-11-25T23:31:19.000Z · LW(p) · GW(p)

Tibba, when you are in a bar, do you see an attractive person and say to yourself, "I think I'll initiate sexualized body language, so that I can mate with that person, thereby increasing the frequency of my genes in future generations"?

There is another post addressing your incorrect objection, but I can't remember what it's called, maybe someone else can dig it up.

comment by Tom3 · 2007-11-26T00:09:47.000Z · LW(p) · GW(p)

At Mt. Obaku temple in the Ko district, Yudkowksy-Sensei was approached by an Individualist during morning meditation.

"All of us are ultimately selfish; we care only about our own states of mind. The mother who claims to care about her son's welfare, really wants to believe that her son is doing well - this belief is what makes the mother happy. She helps him for the sake of her own happiness, not his." said the Individualist.

Yudkowsky-Sensei remained sitting on his zafu, and said nothing.

The Individualist continued: "She did it because she valued that choice above others - because of the feeling of importance she attached to that decision."

Yudkowksy-Sensei stood up and walked out of the temple to his car, where he proceeded to open and close the driver-side door several times before saying:

"There is no chocolate at the supermarket."

comment by Anonymous11 · 2007-11-26T00:47:31.000Z · LW(p) · GW(p)

Caledonian: That didn't make any sense at all. But while we're finding things ironic, how about the fact that a man who loves death so much hasn't killed himself yet?

comment by Tiiba2 · 2007-11-26T01:15:30.000Z · LW(p) · GW(p)

"Tibba, when you are in a bar, do you see an attractive person and say to yourself, "I think I'll initiate sexualized body language, so that I can mate with that person, thereby increasing the frequency of my genes in future generations"?"

Of course not. I want to get laid. That's my explicit desire. Promoting my genes is just a side effect of this. And what is the explicit desire of a principal? Perhaps to teach children. Perhaps to go to work, get paid and ensure that nobody sues him. (He'd still like it if they all go to Harvard.) But "sorting students by abilities" is so off the wall, I don't know how to iterpret it. Colleges and employers might see it that way, but is their opinion THAT important?

comment by mtraven · 2007-11-26T01:54:05.000Z · LW(p) · GW(p)

You don't have to have complex human organizations to get lost purposes. Runaway sexual selection is basically a subgoal of evolution (finding a good mate) taking over far too many resources from the primary goal (survival and reproduction). Of course, since this is probably the process that generated the human mind in the first place, we shouldn't judge the phenomenon too harshly. Our minds and culture are founded on distraction, which explains a lot, like why I am commenting here instead of working.

As for human organizations, I've always felt that, given that they are composed of people who all have their own goals, the real mystery or at least thing to be explained is how they sometimes actually fulfill their stated purpose rather than being constantly subverted or misdirected by the divergent goals of their component individuals. I suppose that the history of industry and management is the history of figuring out ways to stop this from happening.

comment by Jason_Dusek · 2007-11-26T06:06:28.000Z · LW(p) · GW(p)

It's not okay for organizations, corporations, or governments to be misdirected and focused on surface matters; but it's not uncommon, either. Musashi was unique for his insight -- with little training, he defeated masters from an early age, all because of clear perception. (He was later defeated by a man who, in a dream, saw that the jo staff (arm pit height staff) would allow him to defeat any man in Japan.) Perception like that is exceptional. A clear sense of purpose is a gift.

How does one cultivate this? The constantly feel the sharp distinction between success and failure -- as Musashi must have -- is one way. Are lives are gentler than that; I'm not sure what we moderns can do. Education does not seem to help.

comment by TGGP4 · 2007-11-26T06:22:51.000Z · LW(p) · GW(p)

mtc, you could make a few volumes of Dilbert your required reading The Logical Fallacy of Generalization from Fictional Evidence Better evidence for dysfunctional corporate governance is here and here.

It means that contrary to his protestations, Robin really is a libertarian, and he is trying to push the meme that libertarian positions are really vanilla mainstream positions. Someone who is doing the exact opposite of overcoming bias, iow. Robin Hanson has contrarian rather than vanilla mainstream positions, and is quite explicit about that. When he mentioned schools he didn't single out public schools (Eliezer, for example attended a Hebrew school).

comment by Anonymous18 · 2007-11-26T06:27:36.000Z · LW(p) · GW(p)

Jason: AFAIK it's only Shindo Muso Ryu practitioners who claim that Muso Gonnosuke defeated Miyamoto Musashi. Is there any other documentation supporting this?

comment by Kaj_Sotala · 2007-11-26T08:55:45.000Z · LW(p) · GW(p)

Excusez moi, but what exactly is that supposed to mean? As far as I know, most principals want ALL of their students to go to Ivy League and make millions. I can't make any sense of your comment.

http://en.wikipedia.org/w/index.php?title=Signalling_%28economics%29&oldid=165536682#Assumptions_and_groundwork

comment by Doug_S. · 2007-11-26T09:17:00.000Z · LW(p) · GW(p)

Dilbert isn't quite an example of "The Logical Fallacy of Generalization from Fictional Evidence": Scott Adams, in his non-fiction books, repeatedly mentions that no matter how crazy he makes his Dilbert strips, people keep telling him "That's just like my job." In other words, Dilbert is a documentary. ;)

Also, there's this book. If you want to know how corporations are actually run, you need to read it.

comment by outeast2 · 2007-11-26T13:24:34.000Z · LW(p) · GW(p)

people keep telling him "That's just like my job." In other words, Dilbert is a documentary. ;)

Or a horoscope...?

Back on topic (sort of)... Some of the discussion here seems to highlight to problem of resolving dual and conflicting purposes. One key purpose of schools is to educate people; in a very real sense, though, they also necessarily serve to sort those of differing abilities (whether this is an overt purpose, pace Tiiba, is irrelevant: it is a function which schools are clearly set up to facilitate, and so is a de facto purpose). These two purposes conflict, at least to a degree, in that the process of streaming leads to those at the extreme ends of the bell curve being neglected.

At the top end - high achievers - the market can provide help: it makes sense to invest in talent. At the bottom end, some kind of intervention is required (at least is schools are to continue serving their first purpose of providing education). Hence interventions like NCLB - which seems to be failing as an intervention because it is itself a casualty of legislators' own dual and conflicting purposes... And so on it goes.

comment by Benquo · 2007-11-26T14:04:34.000Z · LW(p) · GW(p)

Tiiba,

I think Robin may have meant by "purpose" role in society rather than explicit intention of the principal. (In other words, the signaling is why employers continue to value education, and why people keep going to school.)

Similarly, even if you don't, for instance, go to a bar with the explicit purpose of propagating your genes, you are only naturally inclined to get laid because it's an evolutionarily fit behavior. This doesn't mean you can't explicitly hold another purpose in your mind.

comment by ScentOfViolets · 2007-11-26T14:30:50.000Z · LW(p) · GW(p)

I have seen one or two occasions when he has denied it. But I give more weight to actions, and I have not seen anything he has posted which runs contrary to that assumption. If he truly is a contrarian, you'd expect to see more posts arguing for points that are more commonly associated with socialism or communism. I haven't seen them.

comment by Robin_Hanson2 · 2007-11-26T14:51:41.000Z · LW(p) · GW(p)

To clarify, yes, by "purpose" I meant social function, not conscious intention, and yes I meant private schools as well as public.

comment by Tiiba2 · 2007-11-26T16:41:45.000Z · LW(p) · GW(p)

A computer company won't hire you, no matter how much you know about 12th century knitting patterns, though. So even employers, AFAIK, value actual knowledge. And I don't think the purpose of a school should be determined from the point of view of companies. It should be seen according to those who actually determined whether it will be there - the senator who wrote it into the water bill, exc.

comment by Floccina2 · 2007-11-26T18:31:33.000Z · LW(p) · GW(p)

some young man or woman is sitting at a desk in a university, earnestly studying material they have no intention of ever using, and no interest in knowing for its own sake. They want a high-paying job, and the high-paying job requires a piece of paper, and the piece of paper requires a previous master's degree, and the master's degree requires a bachelor's degree, and the university that grants the bachelor's degree requires you to take a class in 12th-century knitting patterns to graduate. So they diligently study, intending to forget it all the moment the final exam is administered, but still seriously working away, because they want that piece of paper. Some people will tell me that Cuba has better schools than the USA, so I ask do they live better. What is the purpose of schooling? Do schooling and homework squeeze out learning, in net producing a less educated population? Alfie Kohn claims that homework does not help learning. Science and history are evidently fun to many people, as evidence I point out the existence of the science, discovery and history channels. So does homework sqeeze out learnig useful science?

Is school just a long test? If so are home schoolers and those who hire tutors for their children cheating the rest of society?

comment by Benquo · 2007-11-26T23:00:28.000Z · LW(p) · GW(p)

@Tiiba:

The choices that create a school are not the only choices responsible for the schooling that goes on. Students choose to attend, or others choose for them (parents, gov't, etc.). They do so in response to incentives created by others (businesses, higher educational institutions, etc.).

And of course employers value some kinds of knowledge. But knowledge and schooling are not the same thing. At least, people behave w/ regard to school in ways I can't imagine their behaving if it were just about knowledge (e.g. caring about grades, fulfilling major and core requirements that they don't want to but need in order to graduate).

This doesn't mean that school never accomplishes anything beyond signaling, and I suspect Robin Hanson doesn't believe that either. But often, its major role is as a signaling mechanism.

comment by spencerh · 2007-11-27T06:38:18.000Z · LW(p) · GW(p)

Part of the thinking behind NCLB and similar policies is the focus on measurability/transactionalization. One of underlying ideas is: if something can't be easily modeled/measured, it's not to be considered. The corollary to that is that things that are easily measured (test scores, in this case) are piled on, sometimes thoughtlessly.

http://www.huffingtonpost.com/charles-h-green/software-programming-and-_b_69287.html

is a brief, but enlightening article on the subject.

comment by steve_roberts · 2007-11-27T12:39:08.000Z · LW(p) · GW(p)

mtc has it right in that the difference between private organisations and state (and state-funded) ones is that private organisations have an in-built feedback mechanism, they exist to make a profit, the profit they make is calculable by the owners, who will either keep their organisation profitable ie true to its purpose, or sell out or go bust. State organisations by contrast may not have a clear purpose, often find that purpose diverted or confused by politicians, have no feedback mechanism, and almost never go out of existence for any reason including gross failure.

comment by outeast3 · 2007-11-28T10:12:28.000Z · LW(p) · GW(p)

or sell out or go bust

...or, if their function is sufficiently essential, get bailed out by the state.

comment by Tom6 · 2007-11-29T20:45:42.000Z · LW(p) · GW(p)

"..classroom time is now spent on preparing for tests..."

Sounds a bit like learning.

NCLB is just a way for the Feds to see if they are getting their moneys worth. They spend a lot per student and should know if the student is learning anything. It costs the school one day a year for the test. Everything on the test is what the student is supposed to be learning anyway.

comment by Nick_Tarleton · 2007-11-30T02:35:32.000Z · LW(p) · GW(p)

Believe me, preparing for a standardized test is VERY DIFFERENT from actually learning and understanding material.

comment by HoldenKarnofsky · 2007-11-30T10:49:38.000Z · LW(p) · GW(p)

Depends who you are, doesn't it? I think a lot of us privileged folk shudder at the idea of "teaching to the test," because we went to schools were everyone was so far above the standard that this would have been dull and stunting. But NCLB doesn't exist to help high-proficiency students, and I imagine it has next to no effect on them. The classrooms that need to "teach to the test" are the ones that have trouble passing it, and they generally consist of students whose main problem is that they can't read or can't do basic arithmetic.

I haven't seen the tests myself, but it actually seems like you'd have to be quite incompetent to design a test on reading that a child can pass without being able to read. Bottom line, test scores don't seem like much of a "lost purpose" when we're talking about underperforming schools (and giving up some teacher autonomy/creativity is, frankly, very likely not a cost in these cases).

Replies from: byrnema
comment by byrnema · 2011-07-24T03:28:54.696Z · LW(p) · GW(p)

I agree. I conclude from the following that we need to add science to the test:

A recent study here in the Bay Area showed that 80% of teachers in K-5 reported spending less than one hour per week on science, and 16% said they spend no time on science. Why? I'm given to understand the proximate cause is the No Child Left Behind Act and similar legislation.

comment by Caledonian2 · 2007-11-30T14:07:00.000Z · LW(p) · GW(p)
I think a lot of us privileged folk shudder at the idea of "teaching to the test," because we went to schools were everyone was so far above the standard that this would have been dull and stunting.

Um... that seems rather implausible. I went to perfectly normal and average schools, and they were already dull and stunting without "teaching to the test".

comment by Watson · 2007-12-01T09:54:00.000Z · LW(p) · GW(p)

You just written a lot of words, it could be better to write a substract or bolden your main sentences

comment by Benquo · 2007-12-01T16:29:00.000Z · LW(p) · GW(p)

Caledonian, if I understand Holden correctly, he's just arguing that many people here may have had an untypical educational experience.

comment by Andrew_Yates · 2008-10-24T18:06:00.000Z · LW(p) · GW(p)

Both my parents are veteran educators (teachers, then administrators) in a tough urban school district in the Midwest, and they both laud No Child Left Behind not because it helps children directly, but because it's the only "objective" information available that can be politically wielded to attack known underperforming schools and employees. Administrators are often too well aware how terrible their schools, teachers, and students are, and NCLB testing doesn't tell them anything that they didn't already know. Instead, the tests are acceptable "objective measurements" to blame to make decisions that the administration knew had to be made anyway. That way, it's "the test's fault" instead of the fault of the administrator. Before NCLB, there was nothing to blame, and so hard decisions couldn't be made because it was politically impossible to do so.

comment by Brian7 · 2008-11-02T16:01:00.000Z · LW(p) · GW(p)

As some guy once said, small is beautiful.

http://en.wikipedia.org/wiki/Small_is_Beautiful

comment by taelor · 2011-11-06T11:17:12.463Z · LW(p) · GW(p)

Why would any individual lose track of their purposes in a swordfight? If someone else had taught them to fight, if they had not generated the entire art from within themselves, they might not understand the reason for parrying at one moment, or springing at another moment; they might not realize when the rules had exceptions, fail to see the times when the usual method won't cut through.

On August 26, 1346, a smaller contingent of English soldiers met a numerically superior French force outside of the Norman village of Crecy. The outnumbered English focused primarily on defense, using mainly dismounted spear- and longbowmen, whereas the French employed a highly mobile, offensive strategy utilizing mainly armored horsemen; the result was a decisive victory for the English, and a complete rout for the French. 10 years later, English and French forces would meet again at Poitiers; the French still had the numerical advantage, but their strategies were reversed: thistime, the French fielded dismount archers, while English charged in on horseback. The English still won, and even managed to capture the French king, forcing him to sign over a sizable chunk of his kingdom as ransom. What the French commanders failed to understand was that Cracy was a flat plain, where attacking the enemy was easy as looking around till you saw him, and then attacking in that direction, but where strong defensive positions where at a premium; thus the side that was best at defense would win. All thefrench saw was that the English won at Crecy, and so they tried to copy the english's winning tactics, without ever understanding why those tactics would be inappropriate to the hillier, more rugged territory around Poitiers.

comment by Ritalin · 2012-11-24T23:21:07.534Z · LW(p) · GW(p)

Adhocracies seem specifically designed to avoid this problem at an institutional level.

The word is a portmanteau of the Latin ad hoc, meaning "for the purpose", and the suffix -cracy, from the ancient Greek kratein (κρατεῖν), meaning "to govern",

With characteristics such as those described in the Wikipedia article I linked, is it possible to institutionalize avoidance of institutional incompetence, such as what we described here?

comment by lukeprog · 2013-02-23T09:57:15.505Z · LW(p) · GW(p)

Another lost purposes example: flow charts for programming.

comment by Rixie · 2013-07-26T09:03:01.341Z · LW(p) · GW(p)

When I was very young I was also very curious, but I sated my curiosity by telling myself that I would know it all when I was grown up. It wasn't my problem to be curious about these things, other people were handling them. Maybe Eliezer's classmates were thinking that this didn't really make sense, but they trusted the adults enough to do it anyways. Like how a lit lightbulb turns less mysterious and wonderful when you know that someone else already knows the scientific reason for it. All you need to do is follow the instructions and you'll get light.

comment by SeanMCoincon · 2014-07-31T21:27:14.782Z · LW(p) · GW(p)

"I wish I lived in an era where I could just tell my readers they have to thoroughly research something, without giving insult."

Is that not what this entire site is accomplishing?

comment by entirelyuseless · 2016-06-12T12:55:01.316Z · LW(p) · GW(p)

This post actually kind of suggests that Eliezer is an Unfriendly AI.

That is, human beings do not have some specific ultimate goal that they are trying to achieve, and they do not try to make everything else fit in with that. This is why they do not destroy the world. Correspondingly, the fear that an AI would destroy the world is precisely the fear that it would have a goal like that.

So Eliezer's abhorrence to lost purposes makes him believe that an AI would be like him; just as he fanatically seeks his goals, thereby risking destroying the world, so he expects an AI to fanatically seek its goals and destroy the world.

I expect an AI to be more like normal humans than like Eliezer. It will not fanatically seek any goal, and it will not abhor lost purposes.

comment by AnnaSalamon · 2016-11-17T05:23:31.892Z · LW(p) · GW(p)

"More than anything, you must be thinking of carrying your movement through to cutting him. You must thoroughly research this." (Emphasis added.)

I wish I could upvote this post multiple times. Especially the "You must thoroughly research this" part, although also all the examples of institutions failing to accomplish much of anything, even when you might naively have really thought they would (e.g., individual health procedures work; how could large-scale institutions for doing those procedures not work?). I read this post back in 2008 and was like "oh, cool, it's a nice handle for a common little bug that happens to me and other people". I reread it just now, and was more like "Shit, good point, we actually do need to 'thoroughly research' how to avoid lost purposes, or how to carry our movements through to cutting the enemy. For lack of this, the EA and AI Safety movements have perhaps mostly wasted their last several years." Recommending the "thoroughly research" part to anyone who wants to carry on a complex task across a set of tens or hundreds (or thousands) of people without having all the value dissipate into noise.

comment by tlhonmey · 2019-01-03T23:06:39.090Z · LW(p) · GW(p)

Interesting. It seems your early Jewish teachers were firmly stuck in the mindset that was one of the things Christianity originally arose to combat. (Not that Christians should feel too haughty about that since the vast majority of Christian sects fell right back into the same trap early on. I find it amusing that "The Lord's Prayer" is repeated by-rote in most sects when in-context it was given as an example of how you shouldn't waste your time with by-rote prayers...)

This is as opposed to the Jewish community in and around Salt Lake City in the early 1900s that my grandfather always told stories about. (Although I must admit the possibility that the stories were modified to better carry whatever principle he was trying to teach at the expense of strict conformance to history.) To them the best prayers were actions. Prayers-of-words were useful only insofar as they led to choosing the best prayers-of-action. If your neighbour's barn burns down you don't stand around muttering prayers that they'll be able to recover from the disaster any longer than it takes you to figure out who can contribute what -- and then you go out, build them a new barn and stock it for them. Prayers-of-words-you-don't-understand are worse than useless. They are a distraction that leads you in entirely the wrong direction. Prayers-of-words-that-don't-cause-action are empty and useless -- intermediate steps detached from their terminal goals. But we lazy humans are creatures of ritual and habit and pattern-matching. If we experience often enough a chain of events that starts with words and ends with desirable outcomes we eventually become tempted to think that all we need to do is start the chain and not give any thought to the middle steps. If every day at 4PM we take our keys and get in the car and drive to the store and buy some chocolate we may well find ourselves wandering the aisles aimlessly despite having heard about the chocolate-destroying-earthquake prior to having left the house. Because we weren't actually thinking about it.

"Things we do without thinking -- There's the real danger."

comment by Дмитрий Зеленский (dmitrii-zelenskii) · 2019-08-19T01:06:22.369Z · LW(p) · GW(p)

"Likewise with people not automatically noticing [? · GW] when supposedly selfish people give altruistic arguments in favor of selfishness, or when supposedly altruistic people give selfish arguments in favor of altruism" - maybe it's because they want to encompass everyone? So the altruistic guy thinks "other altruistic guys like myself are not to be encourage to be altruistic but I need to appeal somehow to those who are currently selfish", and vice versa?

comment by Viktor Riabtsev (viktor-riabtsev-1) · 2021-02-02T12:57:41.396Z · LW(p) · GW(p)

Dead link for "Why Most Published Research Findings Are False". Googling just the url parameters yields this.