This all seems to rest on an idea that an empty box labeled "truth" was dropped in my lap in the platonic land of a priori mental emptiness, and I'm obligated to fill it with something before I'm allowed to begin thinking. But obviously, that's not what happened. Rather, as I grew up, the abstract label called "truth" was invented and refined by me to help me make sense of the world and communicate with (or win approval from) others. So I end up at the same answer, pragmatism, but I deny that there was ever any problematic circularity. The problem instead seems to come from a notion of transcendent epistemic justification which is unmotivated and flawed--what good does this idea of justification do? What's the problem if I do something without justification?

I wouldn't guess that I have a big disagreement with you once we translate everything into each other's preferred way of speaking. I think you're pointing at something important and right. But I sure won't habitually call it "the problem of the criterion", or use any of the explanatory devices presented here.

So, let's see what you think of my take.

I think "The Problem of the Criterion" as framed here (and, I guess, most places) creates a heavy temptation to conflate several different issues.

1. The problem of responding to the skeptic. The way the problem of the criterion is phrased uses "knowledge", which tempts one to think of perfect knowledge. This makes the problem seem acute, because plausibly, nothing can be known with certainty. But this makes it sound as if the alternative to solving the problem of the criterion is falling into complete skepticism. For Bayesians (and some other epistemologies), it would be fine if nothing is known with certainty, and indeed, plausibly nothing is known with certainty. This makes the problem of the criterion sound irrelevant, for these camps. And it is, provided we understand the problem of the criterion to be something like "how can we know anything with certainty?"
2. The epistemic circularity of foundationalism. This is the recursion problem you mention. This is only a problem if you think each belief should have a justification, and these justifications should never form a circle, and should also avoid forming an infinite chain. But there is no mathematical structure which contains no circles, no infinite chains, and also no unjustified orphan truths! This is intuitively a real problem, since we can separately justify intuitions that each belief should be justified, that circular justifications don't count, and that infinite backward chains of justification also don't count. I think Eliezer's answer to this conundrum is correct (the one which you classify as pragmatism), although I think there are some details to work out. (IE, I don't think most people walk away from that essay with a working classifier to distinguish between bad circular arguments and acceptable ones, because it doesn't give one; it focuses on making the case that specific instances of apparently circular reasoning are fine and good.)
3. The naturalistic question, where does knowledge come from? There's a phenomenon in the world: sometimes people (and animals) seem to know things (and, sometimes not). How does this happen? How can we accurately describe and analyze the phenomenon? Again, Bayesians will tend to think they have a good answer. But this time the Bayesians will be roused to answer with vim and vigor; for this is an issue of key interest to them. Whereas problem #1 seems like a silly distraction to them. 
4. The normative question: what should we consider ourselves to know, and how should we manage belief? This is the core question of epistemic rationality. 

You also mention several other distinct problems which "are just the problem of the criterion in disguise":

• the problem of induction (SEP)
• finding the universal prior (LW [? · GW])
• the problem of perception (and specifically epistemological problems of perception)
• causality [LW · GW]
• no free lunch theorems [LW · GW]
• metaethical uncertainty [LW · GW] (not just normative uncertainty, but uncertainty about what norms are even like)

You seem to find it encouraging that the Problem of the Criterion mixes all of these problems together in one big pot. I think it's quite problematic. In particular, it means that people who think they have solved one problem here will think they've solved the others. It also tempts impossibility proofs for solving one problem to be applied to the others.

I think you start out discussing the normative problem:

Oh, you used a ruler? How do you know the ruler is accurately measuring 3 inches?

This how do you know sounds like a justification question, rather than an empirical how-did-you-come-to-know; although it's impossible to tell for sure without specific disambiguation (this is part of the problem). 

But then you mix in the recursion problem. At the beginning of the section "The Problem", your characterization of The Problem is as follows:

How do you know something is true? To know if something is true, you have some method, a criterion, by which you assess its veracity. But how do you know this criterion is itself true? Oh, you have a method, i.e. a criterion, by which you assess its veracity.

Oops! Infinite recursion detected!

So which problem is it?

• If it's the literal, empirical content of the first question asked, how do you know something is true, then we investigate by trying to remember how we know specific things, noticing when we learn new things, and inventing theoretical frameworks (such as positivism, or fallibilism, or probability theory, or neuroscience) which attempt to characterize the process. The recursive issue isn't so much a problem as it is more data -- as we arrive at tentative conclusions about how we come to know things, we can check whether our theories properly account for how we would learn those theories themselves.^[1]
• If it's the normative version of the first question, how should we know things (or perhaps, how can we justify knowledge), then the recursion problem is perhaps an argument for why the question is difficult - but not the question itself. On this reading, any answer to the criterion problem must deal with the recursion problem, but dealing with the recursion problem is not in itself sufficient; we might be able to dismiss the recursion problem while lacking any reply to the full problem.
• If it's the recursive problem itself, then the criterion problem is really only a problem for foundationalists; perhaps it should be remembered as the main problem with foundationalism. 
• If it's the problem of how we come to know anything (know with certainty), then the skeptical response seems totally reasonable and indeed correct. In this case the recursion problem sounds like just one way to argue for the skeptical conclusion.^[2]

EDIT

It looks like, according to Wikipedia, 

In the field of epistemology, the problem of the criterion is an issue regarding the starting point of knowledge. This is a separate and more fundamental issue than the regress argument found in discussions on justification of knowledge.^[1]

So it would seem that the regress argument is kept distinct from the problem of the criterion, leaving my other three interpretations on the table.

However, the Internet Encyclopedia of Philosophy begins its article like this:

A popular form of the Problem of the Criterion can be raised by asking two seemingly innocent questions: What do we know? How are we to decide in any particular case whether we have knowledge?  One quickly realizes how troubling the Problem of the Criterion is because it seems that before we can answer the first question we must already have an answer to the second question, but it also seems that before we can answer the second question we must already have an answer to the first question.  That is, it seems that before we can determine what we know we must first have a method or criterion for distinguishing cases of knowledge from cases that are not knowledge.  Yet, it seems that before we can determine the appropriate criterion of knowledge we must first know which particular instances are in fact knowledge.  So, we seem to be stuck going around a circle without any way of getting our epistemological theorizing started. 

Which, indeed, sounds like the regress argument. 

And a bit later in the article, a severe ambiguity is confirmed:

Further, there is some ambiguity in Chisholm’s own discussions of the Problem of the Criterion as to whether the problem presented by the Problem of the Criterion is the meta-epistemological problem of determining when we have knowledge or the epistemological problem of determining what is true.  So, there is a difficulty in determining exactly what problem the Problem of the Criterion is supposed to pose.

The fact that Chisholm’s discussion oscillates between these two versions of the Problem of the Criterion and the fact that he seems to be aware of the two versions of the problem help make it clear that perhaps there is no such thing as the Problem of the Criterion. Perhaps the Problem of the Criterion is rather a set of related problems.  This is something that many philosophers since Chisholm, and Chisholm himself (see his 1977), have noted.

1. ^{^}

   This means the recursion problem does have an interesting role to play in the empirical inquiry. The recursion becomes an interesting test case. Perhaps it becomes a big obstacle to some theories. For example, I've heard the accusation that Popperian falsifiability theory isn't itself falsifiable. But perhaps this is simply because it's better viewed as a normative theory of science, rather than an empirical one.

   The empirical-recursive problem of the criterion for Bayesian behavioral scientists (aka economists) is the question of whether we can empirically justify bayesian rationality as a behavioral theory (aka, behavioral economics).

   The empirical-recursive problem of the criterion for neuroscience would be whether a particular neuroscientific theory can account for neuroscience as an activity of the brain. (EG, some connectionist theories which require pre-existing concept cells for every concept will fall short, because a pre-existing 'connectionism cell' is arguably too implausible.)

2. ^{^}

   In this case, the recursion problem can be properly highlighted by appealing to Godel's incompleteness theorem, since Godel's theorem does apply to this kind of crisp 100% knowledge (provided we think it meets the other conditions for the theorem). 

   However, the kind of skeptic I'm endorsing (a moderate bayesian skeptic, who believes in probabilistic knowledge) has no business making the recursion-problem argument, since they don't accept the foundationalist underpinnings of such an argument.

   But perhaps that's fine; there's a long tradition in skepticism of referring to arguments as ladders which you use to climb up to the skeptical vantage point, but then kick down, having no further use for them. 

   After all, the bayesian moderate-skeptic is simply trying to argue that logical certainty doesn't exist in practice. So long as they're arguing about absurdities, they might as well also assume the absurd foundationalist assumptions.

   But it's a bit rude of them if they try to sneak the assumptions in without explicitly flagging them.

I like particularism more than pragmatism, because all reasoning needs some kind of basis, but not all reasoning needs some kind of goal. For example, proving theorems in Peano arithmetic has a basis, but no goal.

More generally, I'd prefer not to ground philosophy in the notion that people follow goals, because so often we don't. Life to me feels more like something spinning outward from its own basis, not toward something specific.

This approach seems wrongheaded to me, from the start. Perhaps I am misunderstanding something. But let’s take your example:

Let’s say you want to know something. Doesn’t really matter what. Maybe you just want to know something seemingly benign, like what is a sandwich?

HALT! Proceed no further. Before even attempting to answer this question, ask: why the heck do you care? Why do you want to know the answer to this strange question, “what is a sandwich”? What do you plan to do with the answer?

In the absence of any purpose to that initial question, the rest of that entire section of the post is unmotivated. The sandwich alignment chart? Pointless and meaningless. Attempting to precisely define a “sandwich-like object”? Total waste of time. And so on.

On the other hand, if you do have a purpose in mind, then the right answer to “what is a sandwich” depends on that purpose. And the way you would judge whether you got the right answer or not, is by whether, having acquired said answer, you were then able to use the answer to accomplish that purpose.

Now, you could then ask: “Ah, but how do you know you accomplished your purpose? What if you’re a brain in a jar? What if an infinitely powerful demon deceived you?”, and so on. Well, one answer to that might be “friend, by all means contemplate these questions; I, in the meantime—having accomplished my chosen purpose—will be contemplating my massive pile of utility, while perched atop same”. But we needn’t go that far; Eliezer has already addressed the question of “recursive justification”.

It does not seem to me as if there remains anything left to say.

Hello,

I would really appreciate it if you could write an entry relating the problem of the criterion in relation to whether or not someone can know a law, such as a criminal statute that prohibits some kind of behavior. I'm doing independent research (not for any university, but for myself -- and not with the goal of a publication) on whether or not it is possible for someone to know a law. So far, I've managed to come across Tomoji Shogenji's article "The Problem of the Criterion in Rule-Following," whereby he posits that if the problem of the criterion is "insurmountable" as it appears to be, then no one can know a rule in order to follow it. The concept of a "law" may be related with the concept of a "rule," whereby a lot of discussion about "rules" seems to come modernly from Wittgenstein. John Fisher also has an article entitled "Knowledge of Rules" in The Review of Metaphysics that I have not yet read that appears promising from the front material that I read on JSTOR.

I've read over your problem of the criterion article here a few times, and you start with the question of whether or not someone can know what a "sandwich" is.

It appears that you respond that it is not possible because there (1) are nebulous boundaries to be had and (2) "there is no true sandwich of which you can speak."

Would you argue that the law is the same?

To me, it appears that there are many interpretations that can be had as to what a law means. An interpretation, in my opinion, is not the same as "knowledge" of a law. However, I don't think legislators mean for there to be "nebulous boundaries" to a law and mean for there to be a "true law" of which they speak, namely a specific interpretation to be had that qualifies as having knowledge of the law. That does not necessarily mean (from what I infer) that there are not "nebulous boundaries" nor no true law of which legislators had the intent to be known.

Thank you in advance for reading.

Finally, we come to a special case of particularist responses known as pragmatism., and it’s this kind of response that Yudkowsky offered. The idea of pragmatism is to say that there is some purpose to be served and by serving that purpose we can do an end-run around the problem of the criterion by tolerating unjustified knowledge so long as it works well enough to achieve some end

This kind of pragmatism solves an easier version of the problem. Winning, and it's close relative, prediction, can both be measured directly. That what allows you tell that something works without having an apriori theoretical justification. On the other hand, correspondence-to-reality can't be tested directly...you can't stand apart from the map territory relationship. In Scientism it's common to assume that predctiveness is a sign of correspondence...

Coherentist responses reject the idea that truth, knowledge, etc. must be grounded and instead seek to find a way of balancing what is known with how it is known to form a self-consistent system.

Coherentism may lack foundations in the sense that foundationalism has foundations, but it still has guiding principles -- notably, that coherence is conducive to truth!

And then there’s skepticism, arguably the only “correct” position of Chisholm’s three in that it’s the only one that seemingly doesn’t require assuming something on faith. Spoiler alert: it does because it still needs some reason to prefer skepticism over the alternatives, thus it still ends up begging the question. Skepticism is also not very useful because even though it might not lead to incorrectly believing that a false thing is true, it does this by not allowing one to believe anything is true!

Which is a problem if you have good reason to believe some things in general are true. If you do, you might as well believe whichever specific things are most plausible , even if they are not fully justified.

You have characterised scepticism as a nothing-is-true position. But "it is true that nothing is true" s self defeating. If the problem of the criterion means that nothing is well justified, then strong claims should be avoided , including strong negative claims like "nothing is true". So scepticism done right is moderation in all things.

If we can prove that the Problem of the Criterion is a true problem, then the Problem of the Criterion is a false problem. Therefore, the Problem of the Criterion can never be proven to be a true problem.

Philosopher: "There is a way of thinking where you can never have a feeling of certainty. Not on any mental, physical, or social level. And I can teach it to you!"

Caveman: "Why would I want to learn that?"

Philosopher: "..."

Perhaps one goal of the pragmatist is to avoid thinking about the Problem of the Criterion. It's like "the Game" we used to play in grade school, where you lose the game by thinking about it. Kids would ambush each other by saying "you just lost the Game!" apropos of nothing.

If my take on this issue is wrong, I encourage proponents of the Problem of the Criterion to prove it!

I'm a fan of W. W. Bartley's Pan Critical Rationalism, from his book The Retreat To Commitment. It doesn't seem to me to fit in your list of approaches. Bartley was a student of Karl Popper, who proposed Critical Rationalism. CR, badly stated, says "This is the fundamental tenet: criticize all your beliefs and see what survives."  PCR cleans that up by saying "This is the best approach to epistemology we've discovered so far: criticize all your beliefs (including this one) and see what survives."

Isn't that better than believing in a foundational, unjustified criterion? Isn't it more flexible than methodism? Isn't it more useful than skepticism?

you'll eventually hit bottom

Laughs for eternity.

Oh, you used a ruler? How do you know the ruler is accurately measuring 3 inches? And furthermore, how do you know your eyes and brain can be trusted to read the ruler correctly to assess the height of the would-be FPS? For that matter, how do you even know what an inch is, and why was 3 inches the true height of an FPS anyway?

Heck, what does it even mean to "know" that something is "true"?

"Who is Descartes? Myth of the given? What are you talking about? I just want to define sandwiches."