misc raw responses to a tract of Critical Rationalism

Bertrand Russell had arguments against that kind of induction...

Looks like the simple organisms and algorithms didn't listen to him!

Again, “what was, will continue”.

Sigh ..that takes me back about 11 years. Yes, induction is always straw manned, the Popper-Miller paper is gold plated truth, etc.

It’s worse than that: SI doesn’t even try to build a meaningful ontological model.

Hm, does it need one

Yes, if you are going to claim that it solves the problem of attaching objective probabilities to ontological theories..or theories for short. If what it actually delivers is complexity measures on computer programs, it would be honest to say so.

But then DDs comment about theories not having objective probabilities was basically correct .

On the note of *qualia* (providing in case it helps)

DD says this in BoI when he first uses the word:

Intelligence in the general-purpose sense that Turing meant is one of a constellation of attributes of the human mind that have been puzzling philosophers for millennia; others include consciousness, free will, and meaning. A typical such puzzle is that of qualia (singular quale, which rhymes with ‘baalay’) – meaning the subjective aspect of sensations. So for instance the sensation of seeing the colour blue is a quale. Consider the following thought experiment. You are a biochemist with the misfortune to have been born with a genetic defect that disables the blue receptors in your retinas. Consequently you have a form of colour blindness in which you are able to see only red and green, and mixtures of the two such as yellow, but anything purely blue also looks to you like one of those mixtures. Then you discover a cure that will cause your blue receptors to start working. Before administering the cure to yourself, you can confidently make certain predictions about what will happen if it works. One of them is that, when you hold up a blue card as a test, you will see a colour that you have never seen before. You can predict that you will call it ‘blue’, because you already know what the colour of the card is called (and can already check which colour it is with a spectrophotometer). You can also predict that when you first see a clear daytime sky after being cured you will experience a similar quale to that of seeing the blue card. But there is one thing that neither you nor anyone else could predict about the outcome of this experiment, and that is: what blue will look like. Qualia are currently neither describable nor predictable – a unique property that should make them deeply problematic to anyone with a scientific world view (though, in the event, it seems to be mainly philosophers who worry about it).

and under "terminology" at the end of the chapter:

Quale (plural qualia) The subjective aspect of a sensation.

This is in Ch7 which is about AGI.

In conclusion I don't see many substantial epistemological differences. 

A Bayesian does have beliefs about the probability of various outcomes even if there are unanswered criticisms involved. Generally, the idea is that people examine criticisms more because they believe that the opportunity cost to answer the criticism is woth it and not just because they are unanswered.

> CR says that truth is objective

I'd say bayesian epistemology's stance is that there is one completely perfect way of understanding reality, but that it's perhaps provably unattainable to finite things.

You cannot disprove this, because you have not attained it. You do not have an explanation of quantum gravity or globally correlated variation in the decay rate of neutrinos over time or any number of other historical or physical mysteries.

[...]

It's good to believe that there's an objective truth and try to move towards it, but you also need to make peace with the fact that you will almost certainly never arrive at it

Yes, knowledge creation is an unending, iterative process. It could only end if we come to the big objective truth, but that can't happen (the argument for why is in BoI - the beginning of infinity).

We sometimes talk about aumann's agreement theorem, the claim that any two bayesians who, roughly speaking, talk for long enough, will eventually come to agree about everything.

I think this is true of any two *rational* people with sufficient knowledge, and it's rationality not bayesians that's important. If two partially *irrational* bayesians talk, then there's no reason to think they'd reach agreement on ~everything.

There is a subtle case with regards to creative thought, though: take two people who agree on ~everything. One of them has an idea, they now don't agree on ~everything (but can get back to that state by talking more).

WRT "sufficient knowledge": the two ppl need methods of discussing which are rational, and rational ways to resolve disagreements and impasse chains. they also need attitudes about solving problems. namely that any problem they run into in the discussion is able to be solved and that one or both of them can come up with ways to deal with *any* problem when it arises.

> taken to logical conclusions it means roughly that all our theories are wrong in an absolute sense

Which means "wrong" is no longer a meaningful word. Do you think you can operate without having a word like "wrong"? Do you think you can operate without that concept?

If it were meaningless I wouldn't have had to add "in an absolute sense". Just because an explanation is wrong in an *absolute* sense (i.e. it doesn't perfectly match reality) does not mean it's not *useful*. Fallibilism generally says it's okay to believe things that are false (which all explanations are in some case); however, there are conditions on those times like there are no known unanswered criticisms and no alternatives.

Since BoI there has been more work on this problem and the reasoning around when to call something "true" (practically speaking) has improved - I think. Particularly:

• Knowledge exists relative to *problems*
• Whether knowledge applies or is correct or not can be evaluated rationally because we have *goals* (sometimes these goals are not specific enough, and there are generic ways of making your goals arbitrarily specific)
• Roughly: true things are explanations/ideas which solve your problem, have no known unanswered criticism (i.e. are not refuted), and no alternatives which have no known unanswered criticisms
• something is wrong if the conjecture that it solves the problem is refuted (and that refutation is unanswered)
• note: a criticisms of an idea is itself an idea, so can be criticised (i.e. the first criticism is refuted by a second criticism) - this can be recursive and potentially go on forever (tho we know ways to make sure they don't).

I think DD sometimes plays inflamatory word games, defining things poorly on purpose.

I think he's in a tough spot to try and explain complex, subtle relationships in epistemology using a language where the words and grammar have been developed, in part, to be compatible with previous, incorrect epistemologies.

I don't think he defines things poorly (at least typically); and would acknowledge an incomplete/fuzzy definition if he provided one. (Note: one counterexample is enough to refute this claim I'm making)

> there are rational ways to choose *exactly one* explanation (or zero if none hold up)

If you point a gun at a bayesian's head and force them to make a bet against a single outcome, they're always supposed to be able to. It's just, there are often reasons not to.

I think you misunderstand me.

let's say you wanted a pet, we need to make a conjecture about what to buy you that will make you happy (hopefully without developing regret later). the possible set of pets to start with are all the things that anyone has ever called a pet.

with something like this there will be lots of other goals, background goals, which we need to satisfy but don't normally list. An example is that the pet doesn't kill you, so we remove snakes, elephants, other other things that might hurt you. there are other background goals like life of the pet or ongoing cost; adopting you a cat with operable cancer isn't a good solution.

there are maybe other practical goals too, like it should be an animal (no pet rocks), should be fluffy (so no fish, etc), shouldn't cost more than $100, and yearly cost is under $1000 (excluding medical but you get health insurance for that).

maybe we do this sort of refinement a bit more and get a list like: cat, dog, rabbit, mouse

you might be *happy* with any of them, but can you be *more happy* with one than any other; is there a *best* pet? **note: this is not an optimisation problem** b/c we're not turning every solution into a single unit (e.g. your 'happiness index'); we're providing *decisive reasons* for why an option should or shouldn't be included. We've also been using this term "happy" but it's more than just that, it's got other important things in there -- the important thing, though, is that it's your *preference* and it matches that (i.e. each of the goals we introduce are in fact goals of yours; put another way: the conditions we introduce correspond directly and accurately to a goal)

this is the sort of case where there is there's no gun to anyone's head, but we can continue to refine down to a list of exactly **one** option (or zero). let's say you wanted an animal you could easily play with -> then rabbit,mouse are excluded, so we have options: cat,dog. If you'd prefer an animal that wasn't a predator - both cat,dog excluded and we get to zero (so we need to come up with new options or remove a goal). If instead you wanted a pet that you could easily train to use a litter tray, well we can exclude a dog so you're down to one. Let's say the litter tray is the condition you imposed.

What happens if I remember ferrets can be pets and I suggest that? well now we need a *new* goal to find which of the cat or ferret you'd prefer.

Note: for most things we don't go to this level of detail b/c we don't need to; like if you have multiple apps to choose from that satisfy all your goals you can just choose one. If you find out a reason it's not good, then you've added a new goal (if you weren't originally mistaken, that is) and can go back to the list of other options.

Note 2: The method and framework I've just used wrt the pet problem is something called yes/no philosophy and has been developed by Elliot Temple over the past ~10+ years. Here are some links:

Argument · Yes or No Philosophy, Curiosity – Rejecting Gradations of Certainty, Curiosity – Critical Rationalism Epistemology Explanations, Curiosity – Critical Preferences and Strong Arguments, Curiosity – Rationally Resolving Conflicts of Ideas, Curiosity – Explaining Popper on Fallible Scientific Knowledge, Curiosity – Yes or No Philosophy Discussion with Andrew Crawshaw

Note 3: During the link-finding exercise I found this: "All ideas are either true or false and should be judged as refuted or non-refuted and not given any other status – see yes no philosophy." (credit: Alan Forrester) I think this is a good way to look at it; *technically and epistemically speaking:* true/false is not a judgement we can make, but refuted/non-refuted *is*. we use refuted/non-refuted as a proxy for false/true when making decisions, because (as fallible beings) we cannot do any better than that.

I'm curious about how a bayesian would tackle that problem. Do you just stop somewhere and say "the cat has a higher probability so we'll go with that?" Do you introduce goals like I did to eliminate options? Is the elimination of those options equivalent to something like: reducing the probability of those options being true to near-zero? (or absolute zero?) Can a bayesian use this method to eliminate options without doing probability stuff? If a bayesian *can*, what if I conjecture that it's possible to *always* do it for *all* problems? If that's the case there would be a way to decisively reach a single answer - so no need for probability. (There's always the edge case there was a mistake somewhere, but I don't think there's a meaningful answer to problems like "P(a mistake in a particular chain of reasoning)" or "P(the impact of a mistake is that the solution we came to changes)" -- note: those P(__) statements are within a well defined context like an exact and particular chain of reasoning/explanation.

Why believe anything!

So we can make decisions.

The great thing is that you don't need to have beliefs to methodically do your best to optimize expected utility

Yes you do - you need a theory of expected utility; how to measure it, predict it, manipulate it, etc. You also need a theory of how to use things (b/c my expected utility of amazing tech I don't know how to use is 0). You need to believe these theories are true, otherwise you have no way to calculate a meaningful value for expected utility!

You can operate well amid uncertainty

Yes, I additionally claim we can operate **decisively**.

In conclusion I don't see many substantial epistemological differences.

It matters more for big things, like SENS and MIRI. Both are working on things other than key problems; there is no good reason to think they'll make significant progress b/c there are other more foundational problems.

I agree practically a lot of decisions come out the same.

I've heard that DD makes some pretty risible claims about the prerequisites to creative intelligence (roughly, that values must, from an engineering feasibility perspective, be learned, that it would be in some way hard to make AGI that wouldn't need to be "raised" into a value system by a big open society, that a thing with 'wrong values' couldn't participate in an open society [and open societies will be stronger] and so wont pose a major threat), but it's not obvious to me how those claims bear at on bayesian epistemology.

I don't know why they would be risible -- nobody has a good reason why his ideas are wrong to my knowledge. They refute a lot of the fear-mongering that happens about AGI. They provide reasons for why a paperclip machine isn't going to turn all matter into paperclips. They're important because they refute big parts of theories from thinkers like Bostrom. That's important because time, money, and effort are being spent in the course of taking Bostrom's theories seriously, even though we have good reasons they're not true. That could be time, money, and effort spent on more important problems like figuring out how creativity works. That's a problem which would actually lead to the creation of an AGI.

Calling unanswered criticisms *risible* seems irrational to me. Sure unexpected answers could be funny the first time you hear them (though this just sounds like ppl being mean, not like it was the punchline to some untold joke) but if someone makes a serious point and you dismiss it because you think it's silly, then you're either irrational or you have a good, robust reason it's not true.

[...] and a real process of making one AGI would tend to take a long time and involve a lot of human intervention?

He doesn't claim this at all. From memory the full argument is in Ch7 of BoI (though has dependencies on some/all of the content in the first 6 chapters, and some subtleties are elaborated on later in the book). He expressly deals with the case where an AGI can run like 20,000x faster than a human (i.e. arbitrarily fast). He also doesn't presume it needs to be raised like a human child or take the same resources/attention/etc.

Have you read much of BoI?

I'm happy to do this. On the one hand I don't like that lots of replies creates more pressure to reply to everything, but I think if we'll probably be fine focusing on the stuff we find more important if we don't mind dropping some loose ends. If they become relevant we can come back to them.