Posts

On the falsifiability of hypercomputation, part 2: finite input streams 2020-02-17T03:51:57.238Z · score: 21 (5 votes)
On the falsifiability of hypercomputation 2020-02-07T08:16:07.268Z · score: 26 (5 votes)
Philosophical self-ratification 2020-02-03T22:48:46.985Z · score: 25 (6 votes)
High-precision claims may be refuted without being replaced with other high-precision claims 2020-01-30T23:08:33.792Z · score: 60 (24 votes)
On hiding the source of knowledge 2020-01-26T02:48:51.310Z · score: 103 (32 votes)
On the ontological development of consciousness 2020-01-25T05:56:43.244Z · score: 37 (13 votes)
Is requires ought 2019-10-28T02:36:43.196Z · score: 23 (10 votes)
Metaphorical extensions and conceptual figure-ground inversions 2019-07-24T06:21:54.487Z · score: 34 (9 votes)
Dialogue on Appeals to Consequences 2019-07-18T02:34:52.497Z · score: 35 (21 votes)
Why artificial optimism? 2019-07-15T21:41:24.223Z · score: 58 (18 votes)
The AI Timelines Scam 2019-07-11T02:52:58.917Z · score: 48 (77 votes)
Self-consciousness wants to make everything about itself 2019-07-03T01:44:41.204Z · score: 43 (30 votes)
Writing children's picture books 2019-06-25T21:43:45.578Z · score: 112 (36 votes)
Conditional revealed preference 2019-04-16T19:16:55.396Z · score: 18 (7 votes)
Boundaries enable positive material-informational feedback loops 2018-12-22T02:46:48.938Z · score: 30 (12 votes)
Act of Charity 2018-11-17T05:19:20.786Z · score: 180 (67 votes)
EDT solves 5 and 10 with conditional oracles 2018-09-30T07:57:35.136Z · score: 62 (19 votes)
Reducing collective rationality to individual optimization in common-payoff games using MCMC 2018-08-20T00:51:29.499Z · score: 58 (18 votes)
Buridan's ass in coordination games 2018-07-16T02:51:30.561Z · score: 55 (19 votes)
Decision theory and zero-sum game theory, NP and PSPACE 2018-05-24T08:03:18.721Z · score: 111 (37 votes)
In the presence of disinformation, collective epistemology requires local modeling 2017-12-15T09:54:09.543Z · score: 122 (43 votes)
Autopoietic systems and difficulty of AGI alignment 2017-08-20T01:05:10.000Z · score: 3 (3 votes)
Current thoughts on Paul Christano's research agenda 2017-07-16T21:08:47.000Z · score: 19 (9 votes)
Why I am not currently working on the AAMLS agenda 2017-06-01T17:57:24.000Z · score: 19 (10 votes)
A correlated analogue of reflective oracles 2017-05-07T07:00:38.000Z · score: 4 (4 votes)
Finding reflective oracle distributions using a Kakutani map 2017-05-02T02:12:06.000Z · score: 1 (1 votes)
Some problems with making induction benign, and approaches to them 2017-03-27T06:49:54.000Z · score: 3 (3 votes)
Maximally efficient agents will probably have an anti-daemon immune system 2017-02-23T00:40:47.000Z · score: 4 (4 votes)
Are daemons a problem for ideal agents? 2017-02-11T08:29:26.000Z · score: 5 (2 votes)
How likely is a random AGI to be honest? 2017-02-11T03:32:22.000Z · score: 1 (1 votes)
My current take on the Paul-MIRI disagreement on alignability of messy AI 2017-01-29T20:52:12.000Z · score: 17 (9 votes)
On motivations for MIRI's highly reliable agent design research 2017-01-29T19:34:37.000Z · score: 19 (10 votes)
Strategies for coalitions in unit-sum games 2017-01-23T04:20:31.000Z · score: 3 (3 votes)
An impossibility result for doing without good priors 2017-01-20T05:44:26.000Z · score: 1 (1 votes)
Pursuing convergent instrumental subgoals on the user's behalf doesn't always require good priors 2016-12-30T02:36:48.000Z · score: 7 (5 votes)
Predicting HCH using expert advice 2016-11-28T03:38:05.000Z · score: 5 (4 votes)
ALBA requires incremental design of good long-term memory systems 2016-11-28T02:10:53.000Z · score: 1 (1 votes)
Modeling the capabilities of advanced AI systems as episodic reinforcement learning 2016-08-19T02:52:13.000Z · score: 4 (2 votes)
Generative adversarial models, informed by arguments 2016-06-27T19:28:27.000Z · score: 0 (0 votes)
In memoryless Cartesian environments, every UDT policy is a CDT+SIA policy 2016-06-11T04:05:47.000Z · score: 19 (4 votes)
Two problems with causal-counterfactual utility indifference 2016-05-26T06:21:07.000Z · score: 3 (3 votes)
Anything you can do with n AIs, you can do with two (with directly opposed objectives) 2016-05-04T23:14:31.000Z · score: 2 (2 votes)
Lagrangian duality for constraints on expectations 2016-05-04T04:37:28.000Z · score: 1 (1 votes)
Rényi divergence as a secondary objective 2016-04-06T02:08:16.000Z · score: 2 (2 votes)
Maximizing a quantity while ignoring effect through some channel 2016-04-02T01:20:57.000Z · score: 2 (2 votes)
Informed oversight through an entropy-maximization objective 2016-03-05T04:26:54.000Z · score: 0 (0 votes)
What does it mean for correct operation to rely on transfer learning? 2016-03-05T03:24:27.000Z · score: 4 (4 votes)
Notes from a conversation on act-based and goal-directed systems 2016-02-19T00:42:29.000Z · score: 6 (4 votes)
A scheme for safely handling a mixture of good and bad predictors 2016-02-17T05:35:55.000Z · score: 0 (0 votes)
A possible training procedure for human-imitators 2016-02-16T22:43:52.000Z · score: 2 (2 votes)

Comments

Comment by jessica-liu-taylor on On the falsifiability of hypercomputation, part 2: finite input streams · 2020-02-18T01:24:34.504Z · score: 2 (1 votes) · LW · GW

In hyper-Solomonoff induction, indeed the direct hypercomputation hypothesis is probably more likely than the arbitration-oracle-emulating-hypercomputation hypothesis. But only by a constant factor. So this isn't really falsification so much as a shift in Bayesian evidence.

I do think it's theoretically cleaner to distinguish this Bayesian reweighting from Popperian logical falsification, and from Neyman-Pearson null hypothesis significance testing (frequentist falsification), both of which in principle require producing an unbounded number of bits of evidence, although in practice rely on unfalsifiable assumptions to avoid radical skepticism e.g. of memory.

Comment by jessica-liu-taylor on On the falsifiability of hypercomputation · 2020-02-07T19:02:33.524Z · score: 4 (2 votes) · LW · GW

This is really important and I missed this, thanks. I've added a note at the top of the post.

Comment by jessica-liu-taylor on On the falsifiability of hypercomputation · 2020-02-07T08:47:30.087Z · score: 2 (1 votes) · LW · GW

Indeed, a constructive halting oracle can be thought of as a black-box that takes a PA statement, chooses whether to play Verifier or Falsifier, and then plays that, letting the user play the other part. Thanks for making this connection.

Comment by jessica-liu-taylor on Philosophical self-ratification · 2020-02-05T03:34:53.752Z · score: 7 (3 votes) · LW · GW

Self-consistency is about not asserting contradictions; self-ratification is about asserting that the process that produced this very theory is likely to produce correct theories.

Theories completely blind to their own shortcomings can very well be consistent and self-ratifying (e.g. faith that one's beliefs came from divine revelation, including the belief that one's beliefs came from divine revelation).

Comment by jessica-liu-taylor on Philosophical self-ratification · 2020-02-05T03:25:48.719Z · score: 2 (1 votes) · LW · GW

This is a bit like looking at someone building an obviously-unstable tower saying "no, see, my tower is self-supporting, I just have a different notion of 'self-supporting'!". If I'm interpreting self-supporting in a physical manner and they're interpreting it in a tautological manner, then we are talking about different things in saying towers are self-supporting or not.

Note that I said at the top that self-ratification is nontrivial when combined with other coherence conditions; without those other conditions (e.g. in the case of asserting "psychological theories" that make no claim about being representative of any actual psychologies) it's a rather trivial criterion.

(In the case of eliminativism, what Phyllis would need is an account of the evidentiary basis for physics that does not refer to minds making observations, theorizing, etc; this account could respond to the dualist's objection by offering an alternative ontology of evidence)

Comment by jessica-liu-taylor on Philosophical self-ratification · 2020-02-04T07:36:16.293Z · score: 2 (1 votes) · LW · GW

Not trustworthy. Untrustworthy means we have some reason to believe they are incorrect. Trustworthy means we have some reason to believe they are correct. Randomness has neither property. (But is the gene random?)

If I tell you the sun is 1.6 * 10^43 kg, but I also tell you I generated the number using a random number generator (not calibrated to the sun), that is an untrustworthy estimate. The RNG wouldn't be expected to get the right answer by accident.

A stopped clock is right twice a day, but only that often, so it's untrustworthy.

The conclusion does not necessarily follow from the premises.

Huh? If Beth's brain can't reason about the world then she can't know that humans are stimulus-response engines. (I'm not concerned with where in her brain the reasoning happens, just that it happens in her brain somewhere)

Comment by jessica-liu-taylor on Philosophical self-ratification · 2020-02-04T01:31:52.954Z · score: 6 (4 votes) · LW · GW

Most of the examples I'm talking about are more like proving false or proving your own finitistic inconsistency than failing to prove your own consistency. Like, if your theory implies a strong (possibly probabilistic) argument that your theory is false, that's almost like proving false.

Godel's incompleteness theorem doesn't rule out finitistic self consistency proofs, e.g ability to prove in length n that there is no inconsistency proof of length up to n^2. Logical inductors also achieve this kind of finitistic self trust. I think this is usually a better fit for real world problems than proving infinitary consistency.

Comment by jessica-liu-taylor on Matthew Barnett's Shortform · 2020-02-01T21:29:41.435Z · score: 4 (2 votes) · LW · GW

However, even if we make the waiting period arbitrarily long, there’s a one hundred percent chance that you will die one day, even given your strong preference not to.

Not if the waiting period gets longer over time (e.g. proportional to lifespan).

Comment by jessica-liu-taylor on High-precision claims may be refuted without being replaced with other high-precision claims · 2020-01-31T20:48:35.083Z · score: 6 (3 votes) · LW · GW

Not a response to anything in particular, but I've had a lot of discussions over my life where someone makes strong claims and doesn't say they were erroneous when counterexamples are provided, such as this recent one.

The policy equivalent is The Real Rules Have No Exceptions: if "exceptions" are being made, the rule should be modified, discarded, or relabeled (e.g. as a guideline/heuristic). The criticism "you aren't really following your rules" is valid even if you don't have an alternative ruleset.

Comment by jessica-liu-taylor on High-precision claims may be refuted without being replaced with other high-precision claims · 2020-01-31T08:52:10.426Z · score: 2 (1 votes) · LW · GW

Are you referring to overflow? If so, that's the right result, the function to compute is "adding integers mod N" not "adding integers" (I agree I said "adding integers" but anyway addition mod N is a different very, very precise claim). Otherwise that's a hardware bug and quality assurance is supposed to get rid of those.

Comment by jessica-liu-taylor on High-precision claims may be refuted without being replaced with other high-precision claims · 2020-01-31T08:34:31.349Z · score: 2 (1 votes) · LW · GW

The claim was that if the arithmetic circuit that is supposed to add numbers fails 0.01% of the time the computer crashes, which is true.

Comment by jessica-liu-taylor on High-precision claims may be refuted without being replaced with other high-precision claims · 2020-01-31T08:22:37.971Z · score: 2 (1 votes) · LW · GW

This does apply to floating point but I was thinking of integer operations here.

Comment by jessica-liu-taylor on High-precision claims may be refuted without being replaced with other high-precision claims · 2020-01-31T07:35:30.067Z · score: 2 (1 votes) · LW · GW

Yes, of course things that aren't definitive falsifications aren't definitive falsifications, but there have been fairly definitive falsifications in physics, e.g. the falsification of aether theory. (Asking for a falsification to be literally 100% certain to be a falsification is, of course, too high of a standard)

Yes, it's also possible to change the description of the theory so it is only said to apply to 99% of cases in response to counterexamples, but this is a different theory than one that says it applies to 99.9% of cases or 100% of cases. This is a matter of calibration.

Comment by jessica-liu-taylor on High-precision claims may be refuted without being replaced with other high-precision claims · 2020-01-31T06:14:24.980Z · score: 1 (2 votes) · LW · GW

Do you think psychiatry is totally useless or harmful even for voluntary patients?

Sometimes yes, sometimes no?

If yes (as seemingly suggested by the reference you linked), that seems to be the real crux between you and people like the ones I linked to, so why not argue about that to begin with?

We're in a subthread about whether the claims of psychiatry are true. You suggested maybe coercion is good even if the claims are not scientifically valid. I said no, that's morally repugnant. You linked people making that argument. I see that the argument relies on the premise that the claims of psychiatry aren't bullshit, so it doesn't show coercion is good even if the claims are not scientifically valid. So you are not asking for a form of interpretive labor that is reasonable in context.

I don't have much to say about the research link except (a) this doesn't look like an unbiased metaanalysis and (b) authoritarian control systems will often produce outcomes for subjects that look better on-paper through more domination but this is a pretty bad ethical argument in the context of justifying the system. Like, maybe slaves who run away experience worse health outcomes than slaves who remain slaves, because they have to hide from authorities, could get killed if they're caught later, are more likely to starve, etc. (And they're less likely to be employed!)

Comment by jessica-liu-taylor on High-precision claims may be refuted without being replaced with other high-precision claims · 2020-01-31T03:18:25.961Z · score: 14 (6 votes) · LW · GW

If the website weren't making the claim that diagnosable/treatable/etc cognitive impairment exists, they wouldn't be saying people are "mentally ill", talk about "Anosognosia", etc.

Without that language the copy on the page doesn't really seem justified. There are lots of groups of people (e.g. religious groups) that claim to be able to help people's life outcomes, but that doesn't ethically justify coercing people to join them.

I hope you can see the problems of legal order of civil society that would arise if people's rights could be overridden on the basis of empirical claims that people like them (not even them specifically, people placed in the same reference class) "benefit" according to metrics such as having a house and being out of jail, which are themselves determined largely by the society.

This also isn't touching the social and informational coercion associated with labeling people as "crazy" and "should be locked up for their own good" (which has psychological effects on them!) based on highly questionable diagnostic criteria. Labeling people who aren't crazy as crazy is gaslighting.

In general on this topic please see Unrecognized Facts by the Council for Evidence-Based Psychiatry.

(I haven't even gotten into issues of study methodology, which may be quite serious; the "Research" link on the page you linked simply links to the same page, which is quite suspicious.)

Comment by jessica-liu-taylor on High-precision claims may be refuted without being replaced with other high-precision claims · 2020-01-31T02:33:14.011Z · score: 2 (1 votes) · LW · GW

I don’t know what “break” means, these theories still give good predictions in everyday cases and it would be a silly reason to throw them out unless weird cases became common enough.

If perpetual motion machines are possible that changes quite a lot. It would mean searching for perpetual motion machines might be a good idea, and the typical ways people try to rule them out ultimately fail. Once perpetual motion machines are invented, they can become common.

But even e.g. the standard model isn’t such a theory!

Not totally exceptionless due to anomalies but it makes lots of claims at very high levels of precision (e.g. results of chemical experiments) and is precise at that level, not at a higher level than that. Similarly with the apple case. (Also, my guess is that there are precise possibly-true claims such as "anomalies to the standard model never cohere into particles that last more than 1 second")

I don't want to create a binary between "totally 100% exceptionless theory" and "not high precision at all", there are intermediate levels even in computing. The point is that the theory needs to have precision corresponding to the brittleness of the inference chains it uses, or else the inference chain probably breaks somewhere.

Comment by jessica-liu-taylor on High-precision claims may be refuted without being replaced with other high-precision claims · 2020-01-31T02:12:36.988Z · score: 16 (8 votes) · LW · GW

There are two very, very different arguments one can make for psychiatric coercion:

  1. There are identifiable, specifiable, diagnosable, and treatable types of cognitive impairment. People who have them would, therefore, benefit from others overriding their (impaired) agency to treat their condition.

  2. There aren't identifiable/specifiable/etc cognitive impairments, or at least psychiatry can't find them reliably (see: homosexuality in the DSM, the history of lobotomies, Foucault, Szasz). However, psychiatry diagnoses a significant fraction of homeless people with "disorders", and psychiatrically imprisoning homeless people without trial is good, so psychiatric coercion is good.

The second argument is morally repugnant to many who hold the first view and is also morally repugnant in my own view. If the DSM isn't actually a much more scientifically valid personality typing procedure than, say, the Enneagram, then locking people up based on it without trial is an ethically horrible form of social control, as locking people up based on the Enneagram would be.

Very few people who accept the first argument would accept the second, indicating that the moral legitimacy of psychiatry is tied to its precise-looking claims about types of cognitive impairment.

Comment by jessica-liu-taylor on High-precision claims may be refuted without being replaced with other high-precision claims · 2020-01-31T01:16:59.467Z · score: 5 (3 votes) · LW · GW

I believe The Real Rules Have No Exceptions is an instrumental analogue.

Comment by jessica-liu-taylor on The Epistemology of AI risk · 2020-01-28T07:37:32.809Z · score: 17 (6 votes) · LW · GW

Note that lack of ability to know what alignment work would be useful to do ahead of time increases, rather than decreases, the absolute level of risk; thus, it increases rather than decreases the risk metrics (e.g. probability of humans being wiped out) that FHI estimated.

Comment by jessica-liu-taylor on On the ontological development of consciousness · 2020-01-27T20:16:44.356Z · score: 6 (3 votes) · LW · GW
  1. Agree regarding high order thought, but "qualia" seems to mean the contents of the subjective point of view? Based on SEP article. "There is something it is like for me to undergo each state, some phenomenology that it has. Philosophers often use the term ‘qualia’ (singular ‘quale’) to refer to the introspectively accessible, phenomenal aspects of our mental lives."

  2. I agree with this. I do think a materialist should be sympathetic to naturalized epistemology which includes developmental psychology as a source of information on what a human could possibly workably consider to be real (and, what they "actually consider to be real" in a psychological sense).

Comment by jessica-liu-taylor on On the ontological development of consciousness · 2020-01-27T06:38:14.759Z · score: 6 (3 votes) · LW · GW

Responding to a few points from his article:

Moreover, once you assume that the brain operates in compliance with physical law, qualia must not play any role in the brain’s operation.

Not true if "qualia" refers to a high-level property of some matter.

Next consider a plant that has not been watered for several days, nearly dying. You water it, and soon its leaves stretch again and regain their vitality. Should you invoke the qualia of “thirst” or “slaking thirst” to explain what happened?

These explanations aren't actually completely stupid; if the plant is optimizing then its optimization could possibly neatly be described by analogy with properties of human optimization that are perceivable in qualia. Maybe empathizing with a plant is a good way of understanding its optimization. Of course, the analogy isn't great (though is much better in the case of large animals), hence why I'm saying the explanations are "not completely stupid" rather than "good".

Well, if a non-physical cause plays a role in any process, than some of physics’ most revered laws, such as energy and momentum conservation, are violated.

False dichotomy, the explanations may apply to different properties of the process, or to different levels of analysis of the process. (I am, here, rejecting the principle of unique causality)

Software, just like hardware, is a physical configuration of matter.

"Is" seems like overeager reductionism. Telling what program a given computer is running, based on its physical configuration, is nontrivial.

Identity/double-aspect theory: The quale and the percept are one and the same process, only perceived as different.

This seems true. Note that "perceived as different" is doing a lot of work here. Qualia are known "directly" in a way that is close to metaphysically basic, whereas percepts are known indirectly, e.g. by looking at a brain scan of one's self or another person (note, this observation is also known through qualia, indirectly, since the brain scan must be observed). These are different epistemic modalities, that could nonetheless always yield the same information due to being two views on the same process.

This can be succinctly put as the Qualia Inaction Postulate: Any behavior would be exactly the same have there been no qualia.

False under identity/double-aspect theory. Because changing qualia means changing percepts too.

The fact that humans are baffled by the Percepts-Qualia Nonidentity, and express this bafflement by their observable behavior, is a case where qualia per se – as nonidentical with percepts – play a causal role in a physical process.

"Percept" as he uses the term refers to the entire brain processing starting from seeing something, including the part of the processing that processes things through world-perception ontology, self-concepts, etc. So "qualia per se - as nonidentical with percepts" is already assuming the falsity of identity/double-aspect theory.

Notice, first, that by this explanation-away the physicalist position commits itself, for the first time, to a falsifiable prediction: When future neurophysiology becomes advanced enough to point out the neural correlates of false beliefs, a specific correlate of this kind would be found to underlie the bafflement about qualia

Why would there be "neural correlates of false beliefs"? The brain can't tell that all of its false beliefs are false; that would require omniscience.

If a proof is ever given that an intelligent system, by virtue of physical laws alone, must state that it has qualia which are nonidentical with percepts, then the age-old Mind-Body Problem would finally get a definite solution – a physicalist one. The Percept-Qualia Nonidentity would turn out to be nothing but an unfortunate misperception, inherent to all intelligent systems, and the problem would turn out to be a pseudo-problem.

I believe my post gives something close to such a proof (not complete, but suggestive).

Chalmers has struggled with a similar idea in his discussion of “zombies.” These creatures are very instructive. Imagine intelligent beings that resemble us in every detail of our physiology, neuroanatomy and chemistry, but have no qualia. This, recall, is perfectly consistent with physics – in fact, as noted above, zombies accord with physics more than the existence of non-zombies.

Zombies may be ruled out philosophically/metaphysically/etc without being ruled out by physics. So assuming zombies are possible is already baking in assumptions.

Notice that in this case we can determine with certainty the cause of this bafflement. Since Charmless is man-made, we can rule out the possibility that his bafflement is the result of some pre-installed “bug” such as an explicit command to express bafflement or some deliberate misperception imposed on it. In other words, we can rule out any cause to Charmless’ assertion about having qualia other than his really having them.

Being susceptible to optical illusions doesn't have to be explicitly programmed in. This is assuming all misperceptions are deliberate which is false (it assumes a kind of omniscence).

Next ask Charmless: Can you conceive of a duplicate of you (henceforth Harmless) who is identical to you but lacks Q? His answer, by (3), must be “No; unmediated percepts must occur by physical law.”

I don't see at all how this follows.

.......

In summary: the article seems like a confused mess, making both technical errors and questionable metaphysical assumptions, such that it neither rules out percept-qualia identity nor epiphenomenalism.

Comment by jessica-liu-taylor on On hiding the source of knowledge · 2020-01-27T06:15:18.817Z · score: 10 (2 votes) · LW · GW

Definitely agree that e.g. focusing only yields very limited knowledge on its own (like, knowledge of the form "X resonated while Y didn't") even though it can yield more significant knowledge when additional checks are run.

Comment by jessica-liu-taylor on On hiding the source of knowledge · 2020-01-26T08:27:36.838Z · score: 10 (2 votes) · LW · GW

To go back to the original claim you took issue with:

Couldn’t it also be the case that the claim is already known through intuition

In this case I did mean "intuition" to include some checks, e.g. compatibility with memory, analogy with similar cases, etc. Brains already do checks when processing thoughts (because, some thoughts register as surprising and some don't). But these checks are insufficient to convince a skeptical audience, is the point. Which is why "I intuitively believe this" is not an argument, even if it's Bayesian evidence to the intuition-haver. (And, trivially, intuitions could be Bayesian evidence, in cases where they are correlated with reality, e.g. due to mental architecture, and such correlations can be evaluated historically)

There seem to be some semantic disagreements here about what constitutes "evidence", "intuition", "checking", etc, which I'm not that enthusiastic about resolving in this discussion, but are worth noting anyway.

Comment by jessica-liu-taylor on On hiding the source of knowledge · 2020-01-26T07:41:21.610Z · score: 9 (5 votes) · LW · GW

(I'm not going to comment on Eliezer's counterpoints except to say I agree with you that he was wrong about macroecon; seems easier to just discuss the theory directly)

Do you mean to suggest that you step out into roads without looking both ways?

No. But if I predicted I'd get hit with >1% probability, I'd avoid roads much more than I currently do. Due to the usual VNM considerations.

In a sense you haven't checked whether Australia has a lower land area than Asia. You have read atlases and have (a) visually inspected the areas and gained the sense that one area is larger than the other, (b) had a background sense that maps are pretty accurate (corresponding to actual land shapes), (c) done some kind of reasoning to infer from observations so far that Australia in fact has a lower land area than Asia.

Yes, this is a semantic issue of what counts as "checking", but that is exactly the issue at hand. Of course it's possible to check claims against memory, intuition, mental calculation, the Internet, etc, but every such check has only limited reliability.

Finally, I will note that in this discussion, you have been making a bunch of claims, such as "99% confidence on the basis of intuition alone is religion, not rationality", and "Only evidence can entangle my logical argument with the state of the world", that seem incredibly difficult to check (or even precisely define), such that I cannot possibly believe that you have checked them to the standard you are demanding.

Comment by jessica-liu-taylor on On hiding the source of knowledge · 2020-01-26T06:44:26.247Z · score: 12 (3 votes) · LW · GW

I agree 99% is probably too high in the case of Benzene (and most other scientific hypotheses not derived from established ones). Although, Einstein's arrogance is a counterpoint.

There are empirical claims it's possible to be very confident in without having checked, e.g. whether Australia or Asia has a larger land area, whether the average fly is larger or smaller than the average mouse, whether you're going to be hit by a car the next time you cross the street, etc.

(See also Blind Empiricism and The Sin of Underconfidence for other general counterpoints)

Comment by jessica-liu-taylor on On hiding the source of knowledge · 2020-01-26T06:14:58.404Z · score: 4 (2 votes) · LW · GW

What's 15*5? (I'm sure you can answer this in <20 seconds)

How confident are you?

How rigorously did you check?

Comment by jessica-liu-taylor on On hiding the source of knowledge · 2020-01-26T05:52:22.123Z · score: 8 (4 votes) · LW · GW
  • If you ask them to picture it in their mind, they can.
  • If you ask them to draw it, they can.
  • They can very quickly recognize that a diagram of Benzene is, in fact, a diagram of Benzene.
  • They can quickly answer questions like "does Benzene contain a cycle?"

The generator for these is something like "they have a mental representation of the structure as a picture, prototype, graph, etc, which is hooked up to other parts of the mind and is available for quick use in mental procedures".

To elaborate, I will quote Guessing the Teacher's Password:

When I was older, and I began to read the Feynman Lectures on Physics, I ran across a gem called “the wave equation.” I could follow the equation’s derivation, but, looking back, I couldn’t see its truth at a glance. So I thought about the wave equation for three days, on and off, until I saw that it was embarrassingly obvious. And when I finally understood, I realized that the whole time I had accepted the honest assurance of physicists that light was waves, sound was waves, matter was waves, I had not had the vaguest idea of what the word “wave” meant to a physicist.

The "[seeing] that it was embarrassingly obvious" is only the case after having the intuition.

Comment by jessica-liu-taylor on On hiding the source of knowledge · 2020-01-26T05:33:34.201Z · score: 11 (6 votes) · LW · GW

To use a concrete example: Kekulé conceived of the structure of benzene after having a dream where he saw an ouroboros.

The intuition is, then, crystalized in a representation of the structure of Benzene, which chemists already know intuitively. If they had only abstract, non-intuitive knowledge of the form of Benzene, they would have difficulty mapping such knowledge to e.g. spacial diagrams.

Intuitions can be more or less refined/crystalized in such a way that they can be more precisely specified, be more analytically tractable, be loadable to more different minds, etc.

Good teaching transfers intuitions, not just abstract knowledge.

Should we ask chemists to take melatonin for more vivid dreams?

If continued progress in chemistry requires insights like Kekulé's, then, yes, why not?

The reason this is bad is because it is indicative of bottom-line thinking.

  1. Couldn't it be the case that is it bad for both reasons? I don't think you've offered an argument that Person A's reasons don't apply.
  2. Couldn't it also be the case that the claim is already known through intuition, and proving it is the main problem? Of course checking against more things will produce higher confidence, but confidence can still exceed 99 percent before doing other checks. (Discarding disconfirming evidence is, of course, epistemically bad because it's failing to update on observations, but it's also possible not to find such evidence in the course of searching)
Comment by jessica-liu-taylor on Don't Double-Crux With Suicide Rock · 2020-01-02T01:58:05.561Z · score: 5 (2 votes) · LW · GW

Being able to parse philosophical arguments is evidence of being rational. When you make philosophical arguments, you should think of yourself as only conveying content to those who are rationally parsing things, and conveying only appearance/gloss/style to those who aren't rationally parsing things.

Comment by jessica-liu-taylor on Has there been a "memetic collapse"? · 2019-12-28T23:30:04.006Z · score: 14 (5 votes) · LW · GW

State of the Union reading level over time has declined quite a lot.

Written sentences are getting shorter and less recursive over time: 1, 2, 3, 4.

Comment by jessica-liu-taylor on Has there been a "memetic collapse"? · 2019-12-28T23:28:56.006Z · score: 2 (1 votes) · LW · GW

State of the Union reading level over time has declined quite a lot.

Sentences are getting shorter over time: 1, 2, 3, 4.

Comment by jessica-liu-taylor on Maybe Lying Doesn't Exist · 2019-12-25T20:18:31.095Z · score: 2 (1 votes) · LW · GW

I don't think it's the crux, no. I don't accept ordinary language philosophy, which canonizes popular confusions. There are some contexts where using ordinary language is important, such as when writing popular news articles, but that isn't all of the contexts.

Comment by jessica-liu-taylor on Maybe Lying Doesn't Exist · 2019-12-22T02:06:35.623Z · score: 8 (4 votes) · LW · GW

For the record, my opinion is essentially the same as the one expressed in "Bad intent is a disposition, not a feeling", which gives more detail on the difference between consciousness of deception and intentionality of deception. (Subconscious intentions exist, so intentional lies include subconsciously intended ones; I don't believe things that have no intentionality/optimization can lie)

"Normal people think you can't lie unawarely" seems inconsistent with, among other things, this article.

Note also, you yourself are reaching for the language of strategic equivocation, which implies intent; but, how could you know the conscious intents of those you believe are equivocating? If you don't, then you probably already have a sense that intent can be subconscious, which if applied uniformly, implies lies can be subconscious.

Comment by jessica-liu-taylor on Act of Charity · 2019-12-12T07:20:11.865Z · score: 41 (9 votes) · LW · GW

[this is a review by the author]

I think what this post was doing was pretty important (colliding two quite different perspectives). In general there is a thing where there is a "clueless / naive" perspective and a "loser / sociopath / zero-sum / predatory" perspective that usually hides itself from the clueless perspective (with some assistance from the clueless perspective; consider the "see no evil, hear no evil, speak no evil" mindset, a strategy for staying naive). And there are lots of difficulties in trying to establish communication. And the dialogue grapples with some of these difficulties.

I think this post is quite complementary with other posts about "improv" social reality, especially The Intelligent Social Web and Player vs. Character.

I think some people got the impression that I entirely agreed with the charity worker. And I do mostly agree with the charity worker. I don't think there were things at the time of writing, said by the charity worker, that I outright thought were false at the time, although some that I thought were live hypotheses but not "very probably true".

Having the thing in dialogue form probably helped me write it (because I wasn't committing to defensibly believing anything) and people listen to it (because it's obviously not "accusatory" and can be considered un-serious / metaphorical so it doesn't directly trigger people's political / etc defenses)

Some things that seem possibly false/importantly incomplete to me now:

  • "Everyone cares about themselves and their friends more" assumes a greater degree of self-interest in social behavior than is actually the case; most behavior is non-agentic/non-self-interested, although it is doing a kind of constraint satisfaction that is, by necessity, solving local constraints more than non-local ones. (And social systems including ideology can affect the constraint-satisfaction process a bunch in ways that make it so local constraint-satisfaction tries to accord with nonlocal constraint-satisfaction)
  • It seems like the "conformity results from fear of abandonment" hypothesis isn't really correct (and/or is quite euphemistic), I think there are also coalitional spite strategies that are relevant here, where the motive comes from (a) self-protection from spite strategies and (b) engaging in spite strategies one's self (which works from a selfish-gene perspective). Also, even without spite strategies, scapegoating is often violent (both historically and in modern times, see prison system, identity-based oppression, sadistic interpersonal behavior, etc), and conservative strategies for resisting scapegoating can be quite fearful even when the actual risk is low. (This accords more with "the act is violence" from earlier in the dialogue, I think I probably felt some kind of tension between exaggerating/euphemizing the violence aspect, which shows up in the text; indeed, it's kind of a vulnerable position to be saying "I think almost everyone is committing spiteful violence against almost everyone else almost all the time" without having pretty good elaboration/evidence/etc)
  • Charities aren't actually universally fraudulent, I don't think. It's a hyperbolic statement. (Quite a lot are, in the important sense of "fraud" that is about optimized deceptive behavior rather than specifically legal liability or conscious intent, especially when the service they provide is not visible/verifiable to donors; so this applies more to international than local charities)
  • "It's because of dysfunctional institutions" is putting attention on some aspects of the problem but not other aspects. Institutions are made of people and relationships. But anyway "institutions" are a useful scapegoat in part because most people don't like them and are afraid of them, and they aren't exactly people. (Of course, a good solution to the overall problem will reform / replace / remove / etc institutions)
  • It seems like the charity worker gets kind of embarrassed at the end and doesn't have good answers about why they aren't doing something greater, so changes the subject. Which is... kind of related to the lack of self-efficacy I was feeling at the time of writing. (In general, it's some combination of actually hard and emotionally difficult to figure out what ambitious things to do given an understanding like this one) Of course, being evasive when it's locally convenient is very much in character for the charity worker.
Comment by jessica-liu-taylor on Act of Charity · 2019-12-12T06:28:54.375Z · score: 14 (3 votes) · LW · GW

[this author's comment is here as a review of this post, as part of the overall 2018 review process]

I think what this post was doing was pretty important (colliding two quite different perspectives). In general there is a thing where there is a "clueless / naive" perspective and a "loser / sociopath / zero-sum / predatory" perspective that usually hides itself from the clueless perspective (with some assistance from the clueless perspective; consider the "see no evil, hear no evil, speak no evil" mindset, a strategy for staying naive). And there are lots of difficulties in trying to establish communication. And the dialogue grapples with some of these difficulties.

I think this post is quite complementary with other posts about "improv" social reality, especially The Intelligent Social Web and Player vs. Character.

I think some people got the impression that I entirely agreed with the charity worker. And I do mostly agree with the charity worker. I don't think there were things at the time of writing, said by the charity worker, that I outright thought were false at the time, although some that I thought were live hypotheses but not "very probably true".

Having the thing in dialogue form probably helped me write it (because I wasn't committing to defensibly believing anything) and people listen to it (because it's obviously not "accusatory" and can be considered un-serious / metaphorical so it doesn't directly trigger people's political / etc defenses)

Some things that seem possibly false/importantly incomplete to me now:

  • "Everyone cares about themselves and their friends more" assumes a greater degree of self-interest in social behavior than is actually the case; most behavior is non-agentic/non-self-interested, although it is doing a kind of constraint satisfaction that is, by necessity, solving local constraints more than non-local ones. (And social systems including ideology can affect the constraint-satisfaction process a bunch in ways that make it so local constraint-satisfaction tries to accord with nonlocal constraint-satisfaction)

  • It seems like the "conformity results from fear of abandonment" hypothesis isn't really correct (and/or is quite euphemistic), I think there are also coalitional spite strategies that are relevant here, where the motive comes from (a) self-protection from spite strategies and (b) engaging in spite strategies one's self (which works from a selfish-gene perspective). Also, even without spite strategies, scapegoating is often violent (both historically and in modern times, see prison system, identity-based oppression, sadistic interpersonal behavior, etc), and conservative strategies for resisting scapegoating can be quite fearful even when the actual risk is low. (This accords more with "the act is violence" from earlier in the dialogue, I think I probably felt some kind of tension between exaggerating/euphemizing the violence aspect, which shows up in the text; indeed, it's kind of a vulnerable position to be saying "I think almost everyone is committing spiteful violence against almost everyone else almost all the time" without having pretty good elaboration/evidence/etc)

  • Charities aren't actually universally fraudulent, I don't think. It's a hyperbolic statement. (Quite a lot are, in the important sense of "fraud" that is about optimized deceptive behavior rather than specifically legal liability or conscious intent, especially when the service they provide is not visible/verifiable to donors; so this applies more to international than local charities)

  • "It's because of dysfunctional institutions" is putting attention on some aspects of the problem but not other aspects. Institutions are made of people and relationships. But anyway "institutions" are a useful scapegoat in part because most people don't like them and are afraid of them, and they aren't exactly people. (Of course, a good solution to the overall problem will reform / replace / remove / etc institutions)

  • It seems like the charity worker gets kind of embarrassed at the end and doesn't have good answers about why they aren't doing something greater, so changes the subject. Which is... kind of related to the lack of self-efficacy I was feeling at the time of writing. (In general, it's some combination of actually hard and emotionally difficult to figure out what ambitious things to do given an understanding like this one) Of course, being evasive when it's locally convenient is very much in character for the charity worker.

Comment by jessica-liu-taylor on Maybe Lying Doesn't Exist · 2019-11-28T04:57:28.873Z · score: 2 (1 votes) · LW · GW

The concept of "not an argument" seems useful; "you're rationalizing" isn't an argument (unless it has evidence accompanying it). (This handles point 1)

I don't really believe in tabooing discussion of mental states on the basis that they're private, that seems like being intentionally stupid and blind, and puts a (low) ceiling on how much sense can be made of the world. (Truth is entangled!) Of course it can derail discussions but again, "not an argument". (Eliezer's post says it's "dangerous" without elaborating, that's basically giving a command rather than a model, which I'm suspicious of)

There's a legitimate concern about blame/scapegoating but things can be worded to avoid that. (I think Wei did a good job here, noting that the intention is probably subconscious)

With someone like Gleb it's useful to be able to point out to at least some people (possibly including him) that he's doing stupid/harmful actions repeatedly in a pattern that suggests optimization. So people can build a model of what's going on (which HAS to include mental states, since they're a causally very important part of the universe!) and take appropriate action. If you can't talk about adversarial optimization pressures you're probably owned by them (and being owned by them would lead to not feeling safe talking about them).

Comment by jessica-liu-taylor on Maybe Lying Doesn't Exist · 2019-11-28T04:45:07.719Z · score: 2 (1 votes) · LW · GW

Jeffrey Epstein

Comment by jessica-liu-taylor on Dialogue on Appeals to Consequences · 2019-11-28T03:17:40.122Z · score: 4 (2 votes) · LW · GW

I think that you must agree with this?

Yes, and Carter is arguing in a context where it's easy to shift the discourse norms, since there are few people present in the conversation.

LW doesn't have that many active users, it's possible to write posts arguing for discourse norms, sometimes to convince moderators they are good, etc.

and it is often reasonable to say “Hey man, I don’t think you should say that here in this context where bystanders will overhear you.”

Sure, and also "that's just your opinion, man, so I'll keep talking" is often a valid response to that. It's important not to bias towards saying exposing information is risky while hiding it is not.

Comment by jessica-liu-taylor on Maybe Lying Doesn't Exist · 2019-11-28T03:11:21.712Z · score: 11 (3 votes) · LW · GW

Treat it as a thing that might or might not be true, like other things? Sometimes it's hard to tell whether it's true, and in those cases it's useful to be able to say something like "well, maybe, can't know for sure".

Comment by jessica-liu-taylor on Dialogue on Appeals to Consequences · 2019-11-27T22:11:42.779Z · score: 8 (4 votes) · LW · GW

Bidding to move to a private space isn't necessarily bad but at the same time it's not an argument. "I want to take this private" doesn't argue for any object-level position.

It seems that the text of what you're saying implies you think humans have no agency over discourse norms, regulations, rules of games, etc, but that seems absurd so I don't think you actually believe that. Perhaps you've given up on affecting them, though.

("What wins" is underdetermined given choice is involved in what wins; you can't extrapolate from two player zero sum games (where there's basically one best strategy) to multi player zero sum games (where there isn't, at least due to coalitional dynamics implying a "weaker" player can win by getting more supporters))

Comment by jessica-liu-taylor on Dialogue on Appeals to Consequences · 2019-11-26T03:09:29.579Z · score: 5 (3 votes) · LW · GW

In those con­texts it is rea­son­able (I don’t know if it is cor­rect, or not), to con­strain what things you say, even if they’re true, be­cause of their con­se­quences.

This agrees with Carter:

So, of course you can eval­u­ate con­se­quences in your head be­fore de­cid­ing to say some­thing.

Carter is arguing that appeals to consequences should be disallowed at the level of discourse norms, including public discourse norms. That is, in public, "but saying that has bad consequences!" is considered invalid.

It's better to fight on a battlefield with good rules than one with bad rules.

Comment by jessica-liu-taylor on Is requires ought · 2019-10-30T17:02:07.200Z · score: 2 (1 votes) · LW · GW

I agree but it is philosophically interesting that at least some of those norms required for epistemology are ethical norms, and this serves to justify the 'ought' language in light of criticisms that the 'ought's of the post have nothing to do with ethics.

Comment by jessica-liu-taylor on Is requires ought · 2019-10-30T17:00:03.598Z · score: 2 (1 votes) · LW · GW

It's much more like the second. I believe this to be very clearly true e.g. in the case of checking mathematical proofs.

I am using an interpretation of "should" under which an agent believes "I should X" iff they have a quasi-Fristonian set point of making "I do X" true. Should corresponds with "trying to make a thing happen". It's an internal rather than external motivation.

It is clear that you can't justifiably believe that you have checked a mathematical proof without trying to make at least some things happen / trying to satisfy at least some constraints, e.g. trying to interpret mathematical notation correctly.

Comment by jessica-liu-taylor on Is requires ought · 2019-10-30T06:47:19.428Z · score: 2 (1 votes) · LW · GW

Sorry for the misinterpretation. I wrote an interpretation and proof in terms of Fristonian set points here.

Comment by jessica-liu-taylor on Is requires ought · 2019-10-30T06:32:04.316Z · score: 2 (1 votes) · LW · GW

Can you please re-read what I wrote and if that post really is addressing the same problem, explain how?

You're right, the post doesn't address that issue. I agree that it is unclear how to apply EDT as a human. However, humans can still learn from abstract agents.

I see, but the synchronization seems rather contrived.

Okay, here's an attempt at stating the argument more clearly:

You're a bureaucrat in a large company. You're keeping track of how much money the company has. You believe there were previous bureaucrats there before you, who are following your same decision theory. Both you and the previous bureaucrats could have corrupted the records of the company to change how much money the company believes itself to have. If any past bureaucrat has corrupted the records, the records are wrong. You don't know how long the company has been around or where in the chain you are; all you know is that there will be 100 bureaucrats in total.

You (and other bureaucrats) want somewhat to corrupt the records, but want even more to know how much money the company has. Do you corrupt the records?

UDT says 'no' due to a symmetry argument that if you corrupt the records than so do all past bureaucrats. So does COEDT. Both believe that, if you corrupt the records, you don't have knowledge of how much money the company has.

(Model-free RL doesn't have enough of a world model to get these symmetries without artificial synchronization)

Comment by jessica-liu-taylor on Is requires ought · 2019-10-30T05:57:57.069Z · score: 2 (1 votes) · LW · GW

It's very clear that you didn't read the post. The thesis is in the first line, and is even labeled for your convenience.

Comment by jessica-liu-taylor on Is requires ought · 2019-10-30T05:01:40.831Z · score: 2 (1 votes) · LW · GW

Then if I condition on “the action I will take in 1 second is B” I will mostly be conditioning on choosing B due to things like cosmic rays

This is an issue of EDT having problems, I wrote about this problem and a possible solution here.

Can you explain what the connection between on-policy learning and EDT is?

The q-values in on-policy learning are computed based on expected values estimated from the policy's own empirical history. Very similar to E[utility | I take action A, my policy is ]; these converge in the limit.

And you’re not suggesting that an on-policy learning algorithm would directly produce an agent that would refrain from mathematical fraud for the kind of reason you give, or something analogous to that, right?

I am. Consider tragedy of the commons which is simpler. If there are many on-policy RL agents that are playing tragedy of the commons and are synchronized with each other (so they always take the same action, including exploration actions) then they can notice that they expect less utility when they defect than when they cooperate.

But I’m not seeing how this supports your position.

My position is roughly "people are coordinating towards mathematical epistemology and such coordination involves accepting an 'ought' of not committing mathematical fraud". Such coordination is highly functional, so we should expect good decision theories to manage something at least as good as it. At the very least, learning a good decision theory shouldn't result in failing at such coordination problems, relative to the innocent who don't know good decision theory.

Comment by jessica-liu-taylor on Is requires ought · 2019-10-30T04:51:05.680Z · score: 2 (1 votes) · LW · GW

I agree that once you have a fixed abstract algorithm A and abstract algorithm B, it may or may not be the case that there exists a homomorphism from A to B justifying the claim that A implements B. Sorry for misunderstanding.

But the main point in my PA comment still stands: to have justified belief that some theorem prover implements PA, a philosophical mathematician must follow oughts.

(When you're talking about naive Bayes or a theorem prover as if it has "a map" you're applying a teleological interpretation (that that object is supposed to correspond with some territory / be coherent / etc), which is not simply a function of the algorithm itself)

Comment by jessica-liu-taylor on Is requires ought · 2019-10-30T04:10:02.894Z · score: 2 (1 votes) · LW · GW

Doesn't a correct PA theorem prover behave like a bounded approximation of PA?

Comment by jessica-liu-taylor on Is requires ought · 2019-10-30T04:08:14.768Z · score: 2 (1 votes) · LW · GW

But believing one's own beliefs to come from a source that systematically produces correct beliefs is a coherence condition. If you believe your beliefs come from source X that does not systematically produce correct beliefs, then your beliefs don't cohere.

This can be seen in terms of Bayesianism. Let R[X] stand for "My system reports X is true". There is no distribution P (joint over X,R[X]) such that P(X|R[X])=1 and P(X) = 0.5 and P(R[X] | X) = 1 and P(R[X] | not X) = 1.

That’s the claim which would be interesting to prove.

Here's my attempt at a proof:

Let A stand for some reflective reasonable agent.

  • Axiom 1: A believes X, and A believes that A believes X.
  • Axiom 2: A believes that if A believes X, then there exists some epistemic system Y such that: Y contains A as an essential component, Y causes A to believe X, and Y functions well. [argument: A has internal justifications for beliefs being systematically correct. A is essential to the system because A's beliefs are a result of the system; if not for A's work, such beliefs would not be systematically correct]
  • Axiom 3: A believes that, for all epistemic systems Y that contain A as an essential component and function well, A functions well as part of Y. [argument: A is essential to Y's functioning]
  • Axiom 4: For all epistemic systems Y, if A believes that Y is an epistemic system that contains A as an essential component, and also that A functions well as part of Y, then A believes that A is trying to function well as part of Y. [argument: good functioning doesn't happen accidentally, it's a narrow target to hit. Anyway, accidental functioning wouldn't justify the belief; the argument has to be that the belief is systematically, not accidentally, correct.]
  • Axiom 5: A believes that, for all epistemic systems Y, if A is trying to function well as part of Y, then A has a set-point of functioning well as part of Y. [argument: set-point is the same as trying]
  • Axiom 6: For all epistemic systems Y, if A believes A has a set-point of functioning well as part of Y, then A has a set-point of functioning well as part of Y. [argument: otherwise A is incoherent; it believes itself to have a set-point it doesn't have]
  • Theorem 1: A believes that there exists some epistemic system Y such that: Y contains A as an essential component, Y causes A to believe X, and Y functions well. (Follows from Axiom 1, Axiom 2)
  • Theorem 2: A believes that A functions well as part of Y. (Follows from Axiom 3, Theorem 1)
  • Theorem 3: A believes that A is trying to function well as part of Y. (Follows from Axiom 4, Theorem 2)
  • Theorem 4: A believes A has a set-point of functioning well as part of Y. (Follows from Axiom 5, Theorem 3)
  • Theorem 5: A has a set-point of functioning well as part of Y. (Follows from Axiom 6, Theorem 4)
  • Theorem 6: A has some set-point. (Follows from Theorem 5)

(Note, consider X = "Fermat's last theorem universally quantifies over all triples of natural numbers"; "Fermat's last theorem" is not meaningful to A if A lacks knowledge of X)