Comment by ryan_b on What societies have ever had legal or accepted blackmail? · 2019-03-19T13:59:57.184Z · score: 2 (1 votes) · LW · GW

People screwing up is a necessary condition of blackmail; if there was perfect OpSec no one would ever be vulnerable to it.

The thing I am pointing at is more basic, though. For example, how would you distinguish between the case where blackmail is accepted but a few people blow the execution and the case like ours, where a few people will do powerfully stupid and illegal things and then just say they did on Facebook? How would this enter the historical record, with enough examples that we can be confident approved blackmail is taking place?

Hmm. Perhaps if there was a written record of the rules of conduct, like the Victorian-era books for instruction in manners. Or diaries where people keep secret record of their blackmail dealings, and refer to it as a good deed.

Comment by ryan_b on What societies have ever had legal or accepted blackmail? · 2019-03-18T15:20:56.367Z · score: 2 (1 votes) · LW · GW

I feel like this would be extremely hard to establish, because in a functional blackmail transaction both the transaction and the information remain secret.

It seems like the only kind of evidence available would be the presence of people diligently trying to ruin someone else's reputation, seemingly without motivation.

Comment by ryan_b on Open Thread March 2019 · 2019-03-18T14:36:04.913Z · score: 4 (2 votes) · LW · GW

Welcome to LessWrong! I appreciate you popping up.

Out of the gate, I should probably say this isn't really specific to climate science; getting up to date on any rapidly advancing field is pretty tough. The saturation of political offense and defense just makes it tougher, is all. The likelihood functions over p-values question is one that I expect would help all scientific fields more-or-less equally.

The questions I personally have are mostly about the state of the various feedback-loops or runaway-processes that have been proposed as drivers of radical climate change. For example, the clathrate gun hypothesis; the last thing I read on the subject was commentary from a scientist who had just completed sampling of methane releases in the Arctic Ocean, who thought it had already fired. Sometime later I was reading a summary which largely agrees with the Wikipedia article that the role of this mechanism in past events is not as great as we previously thought/feared, but then later still I read that clathrate mining had officially begun. The example of the American natural gas boom suggests to me that mining will probably make the problem worse. There's plenty of stuff on the various equilibrium processes like the nitrogen cycle or the sulphur cycle; I feel like something on what are effectively dis-equilibrium processes would also be useful both for learning and for risk evaluation.

For the most part, information about this kind of thing is scattered across papers which are infrequently meta-analyzed, or buried deep in reports like the IPCC's and limited in nuance. I think more easily accessible - and in particular findable - review papers would be very helpful to me. In particular, if papers which discuss the history and intuition of an open problem could be found, I would love that. To give you a sense of what I mean, take a look at Macroscopic Prediction by ET Jaynes. This was a habit of the author more than anything else and it runs throughout his writings; is there anyone like that in climate science?

More blogs is probably a good idea, but I haven't delved deeply enough to find out if any of the ones which already exist are actually good - chiefly this is because of the political noise flooding my search results. What I would really like to find is someone working in climate science with a blog like Andrew Gelman or Scott Aaronson, who I could rely on to be an expositor of their personal thinking while flagging important developments. Then I could use that blog as the launchpad for my related searches, and be more productive that way.

You might very well be like "check out <person> and <person>" and completely resolve my difficulties, which would be awesome.

Comment by ryan_b on You Have About Five Words · 2019-03-13T14:38:20.078Z · score: 6 (3 votes) · LW · GW

This puts me in mind of the mandatory reading of a narrative memo they use at Amazon, which appears to conform to the 'several blog posts' level of coordination. It is hierarchically enforced, and the people who use it are the senior leadership which has, I assume, a capability distribution heavily weighted towards the top of the scale.

Also relevant is the Things I Learned From Working With a Marketing Advisor post.

Comment by ryan_b on You Have About Five Words · 2019-03-13T14:21:19.480Z · score: 18 (6 votes) · LW · GW

These are good examples that drive the point home.

Most people never read the bible.

They don't coordinate based on the nuanced information in it, either. Mostly they coordinate on a few very short statements, like:

Say you are Christian.

Go to church.

A much smaller group of people coordinates on a few more:

Give money to the church.

Run a food drive OR help build houses OR staff a soup kitchen OR ...

The Walmart example seems a little different, because it isn't as though working at Walmart is that different from any other kind of hourly employment. Mostly all employers try to get people to coordinate on a few crucial things:

Show up on time.

Count the money correctly.

Stock the shelves.

Sweep the floor.

And it seems to me there is never a shortage of preachers or employers complaining about people's inability to do even these basic things.

It looks to me like successful coordination on the scale of millions largely amounts to iterating four-word actions.

Comment by ryan_b on Megaproject management · 2019-03-12T18:24:45.243Z · score: 2 (1 votes) · LW · GW

I think this:

Is the position that any and every project that costs $X or more necessarily has the type of complexity and non-separability?

is a reasonable approximation of Flyvbjerg's position. As you say, it is not really about costs per se; the cost is a heuristic for things that drive complexity and non-separability, while also being the primary metric for success.

Comment by ryan_b on How to Understand and Mitigate Risk · 2019-03-12T15:33:53.305Z · score: 2 (1 votes) · LW · GW
When your risks are changing faster than you can sample or model them, you're in a dynamic environment.

For some reason it never really occurred to me before that a fast enough sampling rate effectively makes the environment quasi-static for analysis purposes. That's interesting.

I think it might be because what little work I have done in dynamics also entailed an action against which the environment needed to modeled; so even an arbitrarily high sampling/modeling speed doesn't affect how much the environment changes between when the action initiates and when it completes.

Quite separately, this post did a good job of incorporating everything I thought of that LessWrong has on risk all in the same post, and it would totally have been worth it if it did not do anything else. Strong upvote.

Comment by ryan_b on Megaproject management · 2019-03-12T15:12:29.213Z · score: 2 (1 votes) · LW · GW
I would perhaps pose it as a separability issue. Can the overall whole be chunked out into bite-sized bits without too much coordination type work or not?

My understanding is no, it cannot. What you describe is the basic approach to project management, and the failure of that approach motivates the field. I can think of two specific reasons why:

The first is scale, and I think an intuition similar to Dissolving the Fermi Paradox applies: the question is not the likelihood of each part failing, but rather the likelihood of at least one bottleneck part failing. As the project grows large enough, we should expect to be perpetually choking on one bottleneck or other.

The second is magnitude, which is really the focus of the above paper. Once projects reach a large enough absolute size, more and different stakeholders enter the picture. Each new stakeholder is a stupendous increase in the political complexity of the project, so much so that even at the smaller level of projects where we know the right answers about how to do them applying the right answers is often impossible because of the different interests at play. This is why there is so much effort in keeping the number of stakeholders as small as possible in decision making.

But you would think all infrastructure type projects should benefit from some positive network externality effects.

This is a component of the economic sublime, as I understand it. One example of the kind of stakeholder who enters the picture would be a restaurant owner a block away from the construction site, who expects to benefit from the redirected foot traffic due to construction, or the business of the construction workers, or the increased foot traffic after the project is completed, or all of the above.

Comment by ryan_b on Muqaata'a by Fahad Himsi (I.) · 2019-03-11T19:37:56.642Z · score: 3 (2 votes) · LW · GW
It's not even that it contains no less than three sounds that don't exist in English. It's also the lengthy discussions of Middle Eastern politics that are going to bore every science-fiction fan to death. And while there are undoubtely people who would find that kind of content interesting, those are unlikely to reach for a book from science-fiction shelf.

You couldn't have sold this book any harder to me if you were its press agent.

I feel like this sort of thing would appeal to anyone who liked Dune, and Dune is kind of a big deal.

Comment by ryan_b on Ideas for an action coordination website · 2019-03-11T17:30:59.495Z · score: 5 (2 votes) · LW · GW

So trying to restate your viewpoint, would it be fair to say that you see communities as essentially the recruitment pools from which people to coordinate with are drawn? So the maximize-communities strategy is maximizing the contact surface for coordination opportunities?

Comment by ryan_b on Fair Division of Black-Hole Negentropy: an Introduction to Cooperative Game Theory · 2019-03-08T18:54:18.790Z · score: 2 (1 votes) · LW · GW

One, that is an interesting paper and I am considering hunting for other papers which cite it.

Two, I greatly appreciate that you came back to a 10y post to provide updates. Strong upvote.

Comment by ryan_b on Asking for help teaching a critical thinking class. · 2019-03-08T16:32:30.995Z · score: 6 (5 votes) · LW · GW

If you have a three hour one-shot, then I would be strongly inclined to focus on as few things as possible, while still pointing to the shape of the skillset. I expect the central challenge will be getting them to internalize the idea that different ways of thinking even exist. I would break it down into three things, which can each occupy ~1 hour:

  • Demonstrate how they way they think right now is wrong, via a cognitive bias.
  • Demonstrate a specific technique for overcoming that bias.
  • Argue that reality in fact has joints, and they can in fact be cleaved. Preferably with examples.

I expect the best results will come from hammering on the notion that thinking is a thing you can do on purpose, doing it by reflex leads to predictably wrong answers in a lot of cases, and doing it better is possible. If the choice of bias and technique for overcoming it is something they can immediately apply, so much the better.

Comment by ryan_b on Want to Know What Time Is? · 2019-03-08T16:05:43.197Z · score: 4 (3 votes) · LW · GW

Welcome to LessWrong!

The central issue I see with this argument is that it seems to assume time in the setup to the examples, or implicitly in the equation. For example:

For example, if a guy called Join Doe is traveling from Berlin to Moscow, there might be at least two bits of information about this:
John Doe is in Berlin
John Doe is in Moscow

How do we distinguish between the Berlin to Moscow case, and the Moscow to Berlin case? You mention this point later:

time does not really tell us if Johnny starts his journey in Berlin or in Moscow.

So I can accept the argument that Berlin,Moscow and Moscow,Berlin are the same amount of time - it stands to reason that doing the same thing in reverse order should take the same amount of time, which the information rule seems to capture. But this doesn't seem to square with the Minsk example:

it might even happen, because he is so very much in love with this girl, that he will never arrive in Moscow. In that case, the amount of time between departure and arrival is infinite.

Based on the previous example I would expect Berlin,Minsk,Moscow to be the same as Moscow,Minsk,Berlin, and this seems to hold up - but it doesn't seem consistent that Berlin,Minsk,Moscow is more time than just Berlin,Moscow and yet Berlin,Minsk is infinite.

It also looks to me like we have a problem with trying to integrate new kinds of information. For example, under this rule Berlin,Minsk,Moscow and Berlin,Paris,Moscow are the same amount of time - but when we look at the map we see Paris is farther away from Moscow than Berlin is. How do we account for this, under the rule?

Comment by ryan_b on Ideas for an action coordination website · 2019-03-08T15:13:34.176Z · score: 4 (4 votes) · LW · GW
"that's a lot communities..", you say? "like, a ton of communities", yeah, that's true. but that's how it should be.

I'm interested in hearing more about your thoughts on this point.

My intuition is the opposite - that in general there should be very few communities, and a person can realistically only belong to a few of them. I feel this way because I am strongly convinced that in order for a community to have any meaning people need to invest energy and attention in them over time: a community is a unit of action.

If there are dozens of communities, how will a person become familiar with other members, with whom they are coordinating to change the equilibrium? How will they establish that they really share values? How will they build trust? How will they determine who does which task, and who can be counted on to do the task for which they volunteered? I expect there to be minimum amount of wrench time to get these things figured out per community.

Comment by ryan_b on Open Thread March 2019 · 2019-03-07T18:53:12.960Z · score: 10 (6 votes) · LW · GW

I was reflecting on how hard it is to get up to date on climate science recently, and I thought about Reporting likelihoods, not p-values again. So I did some searching, and while I was able to come up with plenty of discussion about absolute and relative likelihood functions, and some tutorials, and various demystify p-values posts, I didn't see any papers which used them or reports of success using the method.

My expectation is that despite how good the idea seems, the inadequate equilibria of publishing remains, so no surprises there. So now I am wondering about how easy or hard it is to convert already published papers from p-values to likelihood functions. Part of the complaint about p-values is that papers were traditionally opaque about their statistical analysis and did not share their data sets, so overall I expect the problem to be hard. It seems to me I have seen a lot more about success with publishing data sets and sharing analysis software though, so if papers obeying those good practices are chosen that barrier would be overcome. If it is possible to publish a paper of the type "I converted these 5 other papers into likelihood functions and got an interesting result" then there is the added benefit of piling citations on to the papers which use the other best practices.

If journals-gonna-journal and so publishing that is impossible, and a conversion procedure were to be simple enough, it would be worth it as yeoman's work to build an alternative and more accessible pile of knowledge, and as practice for anyone who wants it.

Open Thread March 2019

2019-03-07T18:26:02.976Z · score: 9 (3 votes)
Comment by ryan_b on Epistemic Tenure · 2019-03-07T15:51:03.661Z · score: 2 (1 votes) · LW · GW

The thing I am trying to point at here is that attention to Bob's bad ideas is also necessarily attention to the good ideas Bob uses in idea generation. Therefore I think the total cost in wasted attention is much lower, which speaks to why we should be less concerned about evaluating them and to why Bob should not sweat his status.

I would go further and say it is strange to me that an idea being obviously wrong from a reliable source should be more likely to be dismissed than one that is subtly wrong. Bob is smart and usually correct - further attention to a mostly-correct idea of his is unlikely to improve it. By contrast I think an obviously wrong idea is a big red flag that something has obviously gone wrong.

I may be missing something obvious, but I'm having a hard time imagining how to distinguish in practice between a policy against providing attention to bad ideas, and a policy against providing attention to idea-generating ideas. This seems self-defeating.

Comment by ryan_b on Personalized Medicine For Real · 2019-03-05T16:21:34.961Z · score: 12 (3 votes) · LW · GW

How does the bench-to-bedside model intersect with regulatory oversight?

It seems to me the central problem in the Durona Group vision is that you are largely prohibited from offering such a device to treat the rare condition, even if it gets built and works. I suspect that the optimizations a bench-to-bedside model would make to deliver the product quickly enough and cheaply enough to be useful to that patient would also leave the company drastically short of the amount of data usually demanded to establish safety/efficacy.

I feel like this idea would be a good match for the META Program concept, the goal of which was to speed up delivery of cyber-physical systems for the military by a factor of 5. The regulatory challenges of the FDA are considerable, but defense procurement is worse.

Comment by ryan_b on Unconscious Economies · 2019-02-28T16:20:30.421Z · score: 7 (2 votes) · LW · GW
writing an explanation would signal I'm not part of econ and hence my econ opinions are low status

I think this is the strongest possible argument for writing something.

A. I don't always know the true status of my opinions on a subject; there's no faster way to get such information than to voice the opinion and collect corrections. I routinely used this trick in school on professors, and it works on most any other kind of expert: if you can't get their attention, make a slightly wrong assertion on purpose, and they will hurry to correct you. Note: it does not usually work on people with applied expertise in finance, which is irritating but the reasons are obvious upon reflection.

B. There's a lot of value in different explanations of the same phenomena. After all, if the common knowledge within econ was sufficiently explanatory, it would be common knowledge for us all already and you would have had no doubts. I find it helps a lot to have multiple people/groups pointing at the same thing, so I can mentally triangulate.

Comment by ryan_b on Rule Thinkers In, Not Out · 2019-02-28T15:29:38.035Z · score: 2 (1 votes) · LW · GW

Ha! I apologize, I was ambiguous. What I should have said was, how did you like that book?

Comment by ryan_b on Unconscious Economies · 2019-02-27T15:25:53.578Z · score: 4 (2 votes) · LW · GW

Interesting. I had always assumed that the focus in economics was on 1 because 2 and 3 aren't particularly controllable; it is very easy to say "I want X, and will pay $Y" but it is much harder to impose an unconscious reinforcement regime or a selection effect.

Comment by ryan_b on Rule Thinkers In, Not Out · 2019-02-27T15:16:14.572Z · score: 15 (5 votes) · LW · GW

This seems in the same vein as the Epistemic Tenure post, and looks to be calling for an even lower bar than was suggested there. In the main, I agree.

Comment by ryan_b on Rule Thinkers In, Not Out · 2019-02-27T14:53:25.900Z · score: 2 (1 votes) · LW · GW

How did you find that book?

Comment by ryan_b on Why didn't Agoric Computing become popular? · 2019-02-26T20:09:07.452Z · score: 2 (1 votes) · LW · GW

An alternative, more abstract way of thinking about the problem: it is hard to create a market where there aren't currently any transactions. Until quite recently, transactions were only located where the software was sold and where it was written.

I think the effort would have been more successful if the question was not "how do we make software to use market transactions" but rather "how do we extend market transactions into how software works" because then it would be clear we need to approach it from one end or the other: in order to get software to use transactions we would need to make software production transactions more granular or software consumption transactions more granular. The current trend is firmly on the latter side.

Comment by ryan_b on Why didn't Agoric Computing become popular? · 2019-02-26T19:59:53.817Z · score: 2 (1 votes) · LW · GW

I don't think the features of the theoretical system were particularly relevant. I can see several reasons why this wouldn't take off, and no reasons why it would. For example:

Objects are assumed to communicate through message passing and to interact according to the rules of actor semantics [3,4], which can be enforced at either the language or operating system level. These rules formalize the notion of distinct, asynchronous, interacting entities, and hence are appropriate for describing participants in computational markets.

This part raises a few red flags. We were pretty terrible at asynchronous anything in 2001; the only successful example I know of is communication systems running Erlang, and Erlang is inefficient at math so cost functions would have had a lot of overhead. Further, in the meantime a lot of development effort was put into exploring alternatives to the model he proposes, which we see now in things like practical Haskell and newer languages like Rust or Julia.

Further, we've gotten quite good at getting the value such a system proposes, we just write programs that manage the resources. For example, I do tech support for Hadoop, the central concept of which is adding trade-offs between storage and compute, and Google used Deepmind to manage energy usage in its datacenters. Cloud computing is basically Agoric computing at the application level.

In order for Agoric computing to be popular, there would need to be clear benefits to a lot of stakeholders that would exceed the costs of doing a complete re-write of everything. In a nutshell, it looks to me like Drexler was suggesting we should re-write all software under a market paradigm when we can get most of the same value by writing software under the current - or additional - paradigm(s) and just adding a few programs which optimize efficiency and provide direct trade-offs.

Comment by ryan_b on Native mental representations that give huge speedups on problems? · 2019-02-26T16:38:52.229Z · score: 3 (2 votes) · LW · GW

Tangentially related: following the Mazur link he discusses these four:

  • Geometric intuition
  • Algebraic intuition
  • Computation
  • Physical intuition

I noticed that an Electrical Engineering education hits all four of these pretty well, with the caveat that they are specific and applied stabs into each area. I feel like other branches of engineering are prone to dropping computation completely in favor of more time on physical and geometric problems. Things like systems and circuit diagrams are also heavily emphasized, which are in line with ways of thinking which provide speedups.

I think it would be possible to maximize the gains if a student were to be aware of this going in.

Comment by ryan_b on RAISE is launching their MVP · 2019-02-26T16:06:18.092Z · score: 11 (6 votes) · LW · GW

Congratulations on reaching this milestone! I am excited to see the MVP, even though I'm not a suitable test-candidate. Applause for your mission, but more so for your method:

As long as we operate, we hope to make learning AI Safety less and less costly, creating common knowledge and paying our research debt.
Comment by ryan_b on How good is a human's gut judgement at guessing someone's IQ? · 2019-02-26T15:53:40.166Z · score: 8 (4 votes) · LW · GW

The amount of effort people invest in seeming smarter than they are, for example in situations like hiring or sales, suggests to me that people's gut judgments are easy enough to fool to be worth it.

I expect that what people are actually doing is using a few heuristics to make the judgment. I would also expect that courtesy of things like Dunning-Kruger, people towards the bottom will be as bad at estimating IQ as they are competence at any particular thing.

If we just stick with the intuition provided by fact that some people are really terrible at guessing this sort of thing and some people are not, I further expect that what people use as a heuristic changes with ability level. In example: people at the bottom might base it on whether someone agrees with them about stuff; those in the middle might focus heavily on whether someone has a smart-person job like doctor or lawyer; people towards the top might look more at the way someone does things, like some sign that they chose to think carefully about it or apply some technique known to them.

Comment by ryan_b on How could "Kickstarter for Inadequate Equilibria" be used for evil or turn out to be net-negative? · 2019-02-22T15:36:57.663Z · score: 2 (1 votes) · LW · GW

Addendum: I think the key variable here is how much cheaper and easier the mechanism makes coordination relative to the actual cost that gets people with soft preferences to participate.

It feels like because the groups that are already working to drive inadequate equilibria are already investing in coordination mechanisms, making coordination cheaper and easier will be a net-loss until the soft preference threshold is crossed.

Comment by ryan_b on How could "Kickstarter for Inadequate Equilibria" be used for evil or turn out to be net-negative? · 2019-02-22T15:33:21.244Z · score: 6 (4 votes) · LW · GW

Entrenched interests with a large body of people attached would be able to use such a thing more effectively than the public at large. Consider the case of coal power: it appears that most people have a soft preference for coal power plants to go away; coal power plant workers, coal miners, and coal mining towns have a very strong preference for them to remain or expand.

This could easily mean the coal industry effectively uses this mechanism to advance coal interests, or at least that it sets overcoming coal's efforts as the minimum threshold.

There doesn't seem to be a way to disentangle inadequate equilibria shifting from general equilibria shifting, including the case of shifting to the same equilibrium which might be thought of as equilibria fixing.

That being said, even if such activity takes place it will (presumably) be rendered transparent by the mechanism. I think the new information gained across all equilibria could outweigh an evil shift or an evil fix on some of them.

Comment by ryan_b on Blackmail · 2019-02-22T14:55:27.703Z · score: 3 (2 votes) · LW · GW

Would it be appropriate to view blackmail as a kind of rent, due to being in a position to learn incriminating information?

Comment by ryan_b on Blackmail · 2019-02-21T16:54:56.195Z · score: 3 (2 votes) · LW · GW

I was also confused, but then I realized the key to understanding my position is how fine a needle Hanson is threading (emphasis mine):

my checkmate claim: we don’t have good strong consequential arguments for making gossiper-initiated blackmail offers illegal, relative to making gossip illegal or allowing all offers.

I despise gossip in general, and I am comfortable with strong norms against it up to and including violence. This clarifies the problem for me; I agree on the above basis.

In terms of analysis, I disagree sharply with ignoring the problem of blackmail via false information through the example of slander and libel laws. I always find these kinds of arguments disingenuous; even a casual inspection of publicly available information shows that slander and libel laws are virtually meaningless.

I have a question about this segment:

Most gossip is designed to help the person gossiping. One earns points for good gossip. One builds allies, shows value, has fun, shares important information. It might harm or help third parties. In some cases, the motivation will be to hurt someone else, but that is one of many possible reasons. Most information people tell to other people is motivated by a desire to be helpful, even if that desire is for selfish ends.
Here, the motivation is a desire to be harmful.

Why does profit not count as helping the blackmailer, but gossip counts as helping the gossiper? I would claim harm is the mechanism, not the motive.

That being said, Hanson doesn't seem to disagree that blackmail is negative-sum. If I understand him correctly, he does argue that these negative sums will apply more heavily and consistently to elites rather than regular people, and so it will provide an equalizing force and incentivize elites to adhere to regular norms more closely. I haven't seen this point addressed elsewhere. Naively I am skeptical, because the argument is that they have more ability to pay and so will be targeted more heavily, yet there is no shortage of scams that target regular people with limited ability to pay in other arenas. Arguably Equifax basically was such a thing in the 1970's.

Comment by ryan_b on Decelerating: laser vs gun vs rocket · 2019-02-20T15:42:13.706Z · score: 2 (1 votes) · LW · GW
except when their energy source runs out

There aren't any chemical processes for which this is not a problem. For complex ones like DNA, it is more of a problem rather than less because the absence of any one of the multiple required inputs will do it. A process needs a system, and all the systems with which we are experienced have limits. We also have several known candidates for catastrophic disruption to DNA-like processes, in the form of X-risk.

The problem boils down to whether we can keep jumping up to a larger system level before we deplete or disrupt the one we currently occupy; I see no reason to assume this will always succeed, even if the probability turns in our favor.

Comment by ryan_b on Decelerating: laser vs gun vs rocket · 2019-02-20T15:28:01.630Z · score: 3 (2 votes) · LW · GW

Yes, I should amend that to "I suspect a late filter, if one exists."

Comment by ryan_b on Decelerating: laser vs gun vs rocket · 2019-02-19T17:06:36.548Z · score: 1 (2 votes) · LW · GW

This is not directly related to the post, but while reading the paper a thought struck me and I wanted to get it down:

  • We apply the mediocrity assumption to the Earth among planets; we should also apply it to intelligence among processes.
  • The only intelligence we know of currently runs on chemical processes.
  • All chemical processes I know of naturally terminate.
  • The mediocrity assumption therefore says an intelligence process will also naturally terminate.
  • I therefore suspect a late filter.
Comment by ryan_b on Epistemic Tenure · 2019-02-19T16:26:41.149Z · score: 14 (7 votes) · LW · GW

This may be anchoring the concept too firmly in the community, but I think there is another benefit to giving obviously-wrong ideas from epistemically-sound people attention: it shows how a given level of epistemic mastery is incomplete.

I feel like given an epistemology vocabulary it would be very easy to say that since Bob has said something wrong and his reasons are wrong we should lower our opinion of Bob's epistemic mastery overall. I also feel like that would be both wrong and useless, because it is not as though rationality consisted of some singular epistemesis score that can be raised or lowered. Instead there is a (incomplete!) battery of skills that go in to rationality and we'd be forsaking an opportunity to advance the art by spending time looking at the how and why of the wrongness.

I think the benefit of epistemic tenure is in high value oops! generation.

Comment by ryan_b on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-02-19T15:47:18.440Z · score: 2 (1 votes) · LW · GW

You were one of them, but not the only one. I thought it was worth pointing the strategic question out specifically, because we have only recently had enough plausible alternatives for there to even be such a question.

Granted, the lack of options makes me feel a bit like anime-guy-looks-at-butterfly for alternatives. I agree the strategic question is hard.

Comment by ryan_b on Layers of Expertise and the Curse of Curiosity · 2019-02-15T18:37:17.134Z · score: 3 (2 votes) · LW · GW

I've often wondered how far we could get if our training systems consisted less of one-to-many instruction and instead focused on deeply monitored, iterated group performance. The latter is how the most extreme environments operate, like space missions and the military, but the expense is hard to justify.

On the other hand, I don't know of any middle-ground attempts at doing this for more routine environments. Doing such training for a standard office environment, which relies on standard consumer hardware and has no unusual safety or performance requirements, is doubtless much cheaper. How much cheaper would it have to be, and what kind of performance would it have to deliver, to make it worth considering as an alternative to the standard model of education?

As I write this it occurs to me that most of the distinction is down to the environment, and this model could easily suffer from a lack of emphasis on the value-added tasks that companies are concerned with. Of course, formal education does not provide any focus on those tasks either; the degree just offers some confidence that once provided with them, they can be accomplished.

Comment by ryan_b on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-02-15T16:07:54.205Z · score: 6 (3 votes) · LW · GW

There is a discussion at OvercomingBias of this work now.

Comment by ryan_b on Reframing Superintelligence: Comprehensive AI Services as General Intelligence · 2019-02-15T16:07:02.818Z · score: 4 (2 votes) · LW · GW

I see a few criticisms about how this doesn't really solve the problem, it only delays it because we expect a unified agent to outperform the combined services.

It seems to me on the basis of that criticism that this is worth driving as a commercial template anyway. Every R&D dollar that goes into a bounded service is one that doesn't drive specifically for an unbounded agent; every PhD doing development an individual service is not doing development on a unified agent.

We're currently still in the regime where first mover advantage is overwhelming; if CAIS were in place rather than win all the marbles immediately they would win all the marbles eventually and so the incentives are reduced. I expect this approach to extend the runway we have for nailing down the safety questions before a unified agent takes off.

I suppose the delaying action could backfire by reducing funding for safety, and also potentially by simplifying the problem of a unified AGI to bootstrapping from a superintelligent CAIS coordinator. Is there any difference between the superintelligent CAIS coordinator and the AGI in terms of alignment?

Comment by ryan_b on Masculine Virtues · 2019-02-13T18:22:09.565Z · score: 3 (2 votes) · LW · GW

Yes, I can do that. I estimate ~1 week or so; could I send you a draft then to see if I'm going in a useful direction?

Comment by ryan_b on Masculine Virtues · 2019-02-13T18:17:33.326Z · score: 2 (1 votes) · LW · GW

I am currently getting a Page Not Found response from Putanumonit through that link. It seems that the link includes " avoiding competition where you can" in the address, but shortening it back to winning-is-for-losers works.

Comment by ryan_b on Functional silence: communication that minimizes change of receiver's beliefs · 2019-02-12T22:02:41.715Z · score: 6 (4 votes) · LW · GW

You might be interested in the idea of multivocality, which is saying something which can be interpreted many different ways. The tactical idea is to communicate as little about oneself as possible, while allowing everyone else to communicate things about themselves in response. Here is a blog post that talks about how it was used by Francisco Franco and Cosimo de Medici: Francisco Franco, Robust Action, and the Power of Non-Commitment

Comment by ryan_b on Open Thread February 2019 · 2019-02-12T16:07:01.699Z · score: 7 (5 votes) · LW · GW

I don't think there's anything wrong with posting such a thing. As long as you are clear up front about your state of confidence and that you are exploring an argument instead of trying to persuade, I expect few people would object. There are also many who enjoy unconventional arguments or counter-intuitive conclusions on their own merits.

Worst case scenario, it remains a personal blog post. I say post it.

Comment by ryan_b on Masculine Virtues · 2019-02-12T15:59:08.094Z · score: 5 (2 votes) · LW · GW

Disclaimer: did not do 5 minutes by the clock. Did do 10-15 minutes of discussion and intermittent thinking since.


  • Learn how to work with other people towards a goal
  • Requires skills which can be improved
  • Short feedback loops
  • Clear outcomes
  • Minimally competitive

The best candidate I have come up with is FIRST, the robotics team. This is still a national competition, but the competition is effectively just a show and the competitive activity is tiny compared to the cooperative activity. The goal is to build a robot as a team, so it lends itself instantly to improvable skills, short feedback loops, and clear metrics. It is cooperative mostly in the division-of-labor sense - you can't expect one or two kids to be able to do all the work. It also strongly incentivizes skill transfer, because the less skilled kids want to succeed and the more skilled kids need them to succeed for the robot to work.

I first considered things that were not sports, like drama or dance. These turn out to be extremely competitive, but at the front end; you need to win the role or a position on the team before the coordination even begins.

I considered intellectual activities, like Math Olympiad or Chess, but these tend to be highly individual and so entail minimal coordination - even team events are mostly just aggregations of individual performance. They largely consist of people just being measured against one another.

There are explicitly social, group activities like Model U.N, but these are plagued by being unclear about the skills involved, have unclear outcomes and no short feedback loops. Even stuff like the Boy Scouts really only do coordination by teaching that being cooperative is a virtue.

Lastly there are clubs of various kinds, which often relax the competitive aspect but usually also abandon any specific notion of skill development or feedback; they are just people hanging out who all enjoy the same thing.

On the flip side of the coin, this is a really good point:

I'm somewhat skeptical that the coordination skills you learn from those places would transfer to more productive activities.

I noticed while thinking about this that the things I think are the most valuable about sports - apart from the exercise and the concept of the team - were either not emphasized or not articulated at all. Stuff like how to think about working with someone else and how to beat something that is thinking about beating you weren't really a factor. This makes me wonder if there is an entirely different way to present sports that would improve their transfer-ability. Sports is still about hierarchy; it's only transferable value is that it shifts the perspective from hierarchy-among-individuals to hierarchy-among-groups.

There seems to be an opportunity to add value here, but it is not clear how.

Comment by ryan_b on Masculine Virtues · 2019-02-12T15:22:14.241Z · score: 7 (3 votes) · LW · GW

I agree that marriage and a family are great cooperative endeavors, but I am deeply skeptical it is a good idea to learn coordination after getting married. My marriage is great, and my wife and I both put down the lion's share of the difference between our experience and that of others to being specific about coordinating and being on the same team about everything. It really helps to have these concepts ready to work with before jumping in.

Comment by ryan_b on Open Thread February 2019 · 2019-02-12T15:15:27.169Z · score: 3 (2 votes) · LW · GW

All I have gotten out of it so far is a morbid entertainment value. It does look like it will spend more time talking about subjects adjacent to the Repugnant Conclusion and the voluntary extinction of Absolute Negative Utilitarianism, but it isn't rigorous (so far) in the sense that we usually prefer here.

The author is a good writer, so it does a pretty good job of holding interest despite the subject matter. I would say it is unproductive aside from the entertainment, and if you find it persuasive even more so.

Comment by ryan_b on Open Thread February 2019 · 2019-02-11T22:26:41.203Z · score: 4 (2 votes) · LW · GW

Apropos of the negative utilitarianism question posted recently, has anyone read any pessimism? I picked up The Conspiracy Against the Human Race: A Contrivance of Horror relatively recently. It is a survey, written by Thomas Ligotti, who is a horror and weird fiction writer.

It is gloriously grim. I recommend against it if you are in a sensitive place, however.

Comment by ryan_b on The Hamming Question · 2019-02-11T21:40:46.297Z · score: 3 (2 votes) · LW · GW

I read this section completely differently.

He points to thinking about the important problems as causing success. When people change what they are doing, then they don't continue to have it:

In the first place if you do some good work you will find yourself on all kinds of committees and unable to do any more work.

Carrying on from the end of your section:

When you are famous it is hard to work on small problems. This is what did Shannon in. After information theory, what do you do for an encore? The great scientists often make this error. They fail to continue to plant the little acorns from which the mighty oak trees grow. They try to get the big thing right off. And that isn't the way things go. So that is another reason why you find that when you get early recognition it seems to sterilize you. In fact I will give you my favorite quotation of many years. The Institute for Advanced Study in Princeton, in my opinion, has ruined more good scientists than any institution has created, judged by what they did before they came and judged by what they did after. Not that they weren't good afterwards, but they were superb before they got there and were only good afterwards.

The talk is about things that cause people to do great work. When those causal factors change, the work output also changes. He goes on to cover other things which are about professional success:

  • Working with an open office door, to talk to your coworkers
  • Changing routine work into more general and important work, which is more satisfying
  • The importance of self-promotion
  • Working on presentation skills
  • How to recruit your boss to fight with outside agencies
  • How to get your boss to give you more resources
  • Dressing for success, and getting punished for non-conformity

Lastly, he is pretty specific about his motivations (emphasis mine):

I think it is very definitely worth the struggle to try and do first-class work because the truth is, the value is in the struggle more than it is in the result. The struggle to make something of yourself seems to be worthwhile in itself. The success and fame are sort of dividends, in my opinion.

So he is specifically talking about professional success in science. But - things like the rationality project and EA are good candidates for other fields to which the advice could be applied, especially in light of how important science is to them.

Comment by ryan_b on The Hamming Question · 2019-02-11T20:19:45.100Z · score: 2 (1 votes) · LW · GW

I think that personal success is the correct impression:

I noticed a couple of months later he was made the head of the department. I noticed the other day he was a Member of the National Academy of Engineering. I noticed he has succeeded. I have never heard the names of any of the other fellows at that table mentioned in science and scientific circles.

Notice he doesn't talk about all the amazing things that were solved; he talks about lab positions and Nobel Prizes and getting equations named after himself.

I expect that Hamming would view having an impact on the world as being a good reason to choose going into science instead of law or finance, but once that choice is made being great at science is the reasonable thing to do.

To be clear, I don't think he viewed reputations and promotions as the goal, I believe he viewed them as reasonable metrics that he was on the right track for doing great science.

Comment by ryan_b on The Hamming Question · 2019-02-11T15:18:50.887Z · score: 5 (3 votes) · LW · GW

Based on the other comments, I feel like it is worthwhile to point out that Hamming is talking about how to be a successful scientist, as measured by things like promotions, publications, and reputation.

He is not talking about the impact of the problems themselves. From the quoted section, emphasis mine:

It's not the consequence that makes a problem important, it is that you have a reasonable attack. That is what makes a problem important. When I say that most scientists don't work on important problems, I mean it in that sense.

So it looks like we're trying to apply the question one entire step before where Hamming did. For example there weren't - and if I read Hamming right, still aren't - reasonable attacks to the alignment problem. The prospective consequences are just so great we had to consider what is reasonable in a relative sense, and try anyway.

It feels like rationality largely boils down to the search for a generative rule for reasonable attacks.

Open Thread February 2019

2019-02-07T18:00:45.772Z · score: 20 (7 votes)

Towards equilibria-breaking methods

2019-01-29T16:19:57.564Z · score: 23 (7 votes)

How could shares in a megaproject return value to shareholders?

2019-01-18T18:36:34.916Z · score: 18 (4 votes)

Buy shares in a megaproject

2019-01-16T16:18:50.177Z · score: 15 (6 votes)

Megaproject management

2019-01-11T17:08:37.308Z · score: 57 (21 votes)

Towards no-math, graphical instructions for prediction markets

2019-01-04T16:39:58.479Z · score: 30 (13 votes)

Strategy is the Deconfusion of Action

2019-01-02T20:56:28.124Z · score: 73 (23 votes)

Systems Engineering and the META Program

2018-12-20T20:19:25.819Z · score: 31 (11 votes)

Is cognitive load a factor in community decline?

2018-12-07T15:45:20.605Z · score: 20 (7 votes)

Genetically Modified Humans Born (Allegedly)

2018-11-28T16:14:05.477Z · score: 30 (9 votes)

Real-time hiring with prediction markets

2018-11-09T22:10:18.576Z · score: 19 (5 votes)

Update the best textbooks on every subject list

2018-11-08T20:54:35.300Z · score: 78 (28 votes)

An Undergraduate Reading Of: Semantic information, autonomous agency and non-equilibrium statistical physics

2018-10-30T18:36:14.159Z · score: 30 (6 votes)

Why don’t we treat geniuses like professional athletes?

2018-10-11T15:37:33.688Z · score: 20 (16 votes)

Thinkerly: Grammarly for writing good thoughts

2018-10-11T14:57:04.571Z · score: 6 (6 votes)

Simple Metaphor About Compressed Sensing

2018-07-17T15:47:17.909Z · score: 8 (7 votes)

Book Review: Why Honor Matters

2018-06-25T20:53:48.671Z · score: 31 (13 votes)

Does anyone use advanced media projects?

2018-06-20T23:33:45.405Z · score: 45 (14 votes)

An Undergraduate Reading Of: Macroscopic Prediction by E.T. Jaynes

2018-04-19T17:30:39.893Z · score: 37 (8 votes)

Death in Groups II

2018-04-13T18:12:30.427Z · score: 32 (7 votes)

Death in Groups

2018-04-05T00:45:24.990Z · score: 47 (18 votes)

Ancient Social Patterns: Comitatus

2018-03-05T18:28:35.765Z · score: 20 (7 votes)

Book Review - Probability and Finance: It's Only a Game!

2018-01-23T18:52:23.602Z · score: 18 (9 votes)

Conversational Presentation of Why Automation is Different This Time

2018-01-17T22:11:32.083Z · score: 70 (29 votes)

Arbitrary Math Questions

2017-11-21T01:18:47.430Z · score: 8 (4 votes)

Set, Game, Match

2017-11-09T23:06:53.672Z · score: 5 (2 votes)

Reading Papers in Undergrad

2017-11-09T19:24:13.044Z · score: 42 (14 votes)