Posts

My Functor is Rich! 2020-03-18T18:58:39.002Z · score: 10 (5 votes)
Welcome to the Haskell Jungle 2020-03-18T18:58:18.083Z · score: 13 (7 votes)
Lessons from Isaac: Poor Little Robbie 2020-03-14T17:14:56.438Z · score: 0 (5 votes)
Where's the Turing Machine? A step towards Ontology Identification 2020-02-26T17:10:53.054Z · score: 18 (5 votes)
Goal-directed = Model-based RL? 2020-02-20T19:13:51.342Z · score: 19 (7 votes)

Comments

Comment by adamshimi on How important are MDPs for AGI (Safety)? · 2020-03-27T10:08:59.070Z · score: 3 (2 votes) · LW · GW

If I get it correctly, your issue is with the Markov Property of MDP? It simplifies the computation of the policy by not requiring to know the path by which the agent arrived at a given state; but it also removes the information about the history that is not written down into the state itself.

Not sure if you know it or if it is that useful, but this section of "Reinforcement Learning: an introduction" discuss ways to go beyond MDP and the Markov property.

Comment by adamshimi on Open & Welcome Thread - March 2020 · 2020-03-25T11:04:59.261Z · score: 1 (1 votes) · LW · GW

That's a great idea! Are some people interested in a more structured version of this, something like a writing group where everyone proposes its writing and the other comment on it?

Either way, I'm interested on having feedback for something I'm currently writing, whose draft I will probably finish at the end of this week. I'm interested in feedback on content, and on readability.

I'm also up to comment on structure, arguments and readability for others.

Comment by adamshimi on Deconfusing Human Values Research Agenda v1 · 2020-03-25T09:55:33.237Z · score: 2 (2 votes) · LW · GW

I really like the idea that preferences are observed after the fact, because I feel like there is some truth to it for human beings. We act, and then become self-aware of our reactions and thoughts, which leads us to formulate some values. Even when we act contrary to those values, at least inside, we feel shitty.

But that doesn't address the question of where do these judgements and initial reactions come from. And also how this self-awareness influences the following actions.

Still, this makes me want to read the rest of your research!

Comment by adamshimi on [Meta] Do you want AIS Webinars? · 2020-03-22T13:41:49.484Z · score: 1 (1 votes) · LW · GW

I am definitely interested in participating, as I'm learning the field, and starting to work on research. For the moment I don't feel like I can run one of these myself, but I'll be eventually there, and will propose myself.

Comment by adamshimi on Welcome to the Haskell Jungle · 2020-03-20T00:59:29.380Z · score: 1 (1 votes) · LW · GW

Thanks a lot for the recommendation! I'll look into it.

Comment by adamshimi on Lessons from Isaac: Poor Little Robbie · 2020-03-14T19:34:16.823Z · score: 5 (3 votes) · LW · GW

Hum, good idea. At least it can't get worse. ^^

Comment by adamshimi on Lessons from Isaac: Poor Little Robbie · 2020-03-14T19:04:39.332Z · score: 1 (1 votes) · LW · GW

True. Do you think I should still list and quickly explain the stories that are "useless" for this point someplace?

Comment by adamshimi on Open & Welcome Thread - March 2020 · 2020-03-14T17:21:21.199Z · score: 1 (1 votes) · LW · GW

I saw there is a Coronavirus tag now. Is there some way to use this tag to not see any post related to the topic? Because I only managed to go to the page with only these posts, and I think pretty much all the value of such a tag is in filtering. I mean, if I want to see many posts with coronavirus news or advice, I can just look at the front page, I don't need a tag.

Comment by adamshimi on Interfaces as a Scarce Resource · 2020-03-12T21:03:36.378Z · score: 3 (2 votes) · LW · GW

Great post! This makes me think of the problem of specification in formal methods: what you managed to formalize is not necessarily what you wanted to formalize. This is why certified software is only as good as the specification that was used for this certification. And that's one of my main intuitions about the issues of AI safety.

One part of the problem of specification is probably about interfacing, like you write, between the maths of the real world and the maths we can understand/certify. But one thing I feel was not mentioned here is the issue of what I would call unknown ambiguity. One of the biggest difficulties of proving properties of programs and algorithms, is that many parts of the behavior are considered obvious by the designer. Think something like the number of processes cannot be 0, or this variable will never take this value, even if its of the right type. Most of the times, when you add these obvious parts, you can finish the proof. But sometimes the "trivial" was hiding the real problem, which breaks the whole thing.

So I think another scarce resource are people that can explicit all the bits in the system. People that can go to all the nitpick, and rebuild everything from scratch.

Comment by adamshimi on Goal-directed = Model-based RL? · 2020-03-10T17:07:07.499Z · score: 1 (1 votes) · LW · GW

Do you have references of posts of those people who think goal-directedness is binary-ish? That would be very useful, thanks. :)

Comment by adamshimi on The Gears of Impact · 2020-03-06T13:11:32.033Z · score: 4 (2 votes) · LW · GW

I don't get why the client AU from the perspective of the robber doesn't drop when the robber enters, or just before? Because even if I'm the robber and I know they won't like it and won't be able to do things after I'm in, they can still do things in the bank before I'm in. And if they're out before I come in, their AU will be the same than if I was never there.

Comment by adamshimi on Matrix Multiplication · 2020-03-05T12:47:44.312Z · score: 2 (2 votes) · LW · GW

If you do a matrix multiplication the obvious way, this results in dot products of rows and columns (one for each element of the resulting matrix). So it seems to me that improving matrix to matrix multiplication performance comes from improving the performance of dot products.

This seems like a decent explanation of Hardware Matrix Multiplication, even if it lacks concrete sources.

As for a tensor, I think these references explain it better that I can at my current level. But the intuition is that it's a generalization of a matrix to high-dimensions, with additional properties when transformed.

Comment by adamshimi on Write · 2020-03-04T12:22:36.577Z · score: 3 (2 votes) · LW · GW

Interesting discussion of epistemic status at the end. I like the intellectual honesty behind it, but your point that they are now internalized also makes sense.

On the part about writing while thinking of the audience, I want to recommend the best book about writing I ever read: Writing with Style, by John Trimble. Although it's not perfect, a writing book that start by explaining the thought process of experimented writers, and how it differ from the one of the novice, is just amazingly useful.

I'll let Trimble the last words:

Books on writing tend to be windy, boring, and impractical. I intend this one to be different -- short, fun, and genuinely useful.
Comment by adamshimi on Circumambulation · 2020-03-03T12:44:41.844Z · score: 5 (3 votes) · LW · GW

One connection my Babble made while reading this post is between Circumambulation and The Feynman Method. The latter is inspired by an event in the biography of the late Richard Feynman, where he wrote in a notebook all the things he knew about physics, and poked into every hole.

My Prune tells me this is probably irrelevant, since Circumambulation in this post seems more about the blocks to the generation of ideas than the deep understanding of a subject. But I don't have to listen to him.

Comment by adamshimi on More Babble · 2020-03-01T15:38:14.041Z · score: 3 (2 votes) · LW · GW

I really like how the posts in this sequence use technical analogies. You refer to some advanced concepts like expanders, but they don't feel tacked into the ideas. I even learned about implict representation of graphs! (though I knew bounded-degree graphs)

One nitpick is that Ramanujan probably had an amazing Prune too. I feel he's impressive because he was right so many times. And when he went astray, it was apparently because his lack of schooling in mathematics made him overlooks some aspects of the problem. That feels like the combination of an amazing Babble and Prune, with the Babble getting the better of the Prune for the mistakes.

Comment by adamshimi on Babble · 2020-02-29T11:59:50.496Z · score: 5 (3 votes) · LW · GW

Thanks for this awesome post! I like the babble/prune distinction, but the analogy to randomized algorithms was probably the more helpful idea in here for me. It made perfect sense, since a lot of probabilistic algorithms are really simple combinations of random babble and efficient pruning.

This analogy makes me wonder: given that many in complexity theory assume that BPP = P, what is the consequence of derandomization on Babble and Prune? Will we eventually be able to babble deterministically, such that we have a high guaranteed probability of finding what we looked for while pruning?

A slight issue with the post: I disagree that poetry is pure babble/phonetic babble. Some parts of poetry are only about the sounds and images, but many poems try to compress and share a feeling, an idea, an intuition. That is to say, meaning matters in poetry.

Comment by adamshimi on Deducing Impact · 2020-02-27T16:54:32.885Z · score: 10 (3 votes) · LW · GW

What I came up with before reading the spoilers or the next posts in the sequence:

A big deal is any event that significantly changes my expected ability to accomplish my goals (whether by having an impact specific to me, or an objective impact).

Comment by adamshimi on What do you make of AGI:unaligned::spaceships:not enough food? · 2020-02-22T14:48:22.350Z · score: 3 (3 votes) · LW · GW

I talked about this issue with Buck in the comments (my comment, Buck's answer)

What I pointed was that the spaceship examples had very specific features:

  • Both personal and economic incentives are against the issue.
  • The problem are obvious when one is confronted with the situation
  • At the point where the problem becomes obvious, you can still solve it.

My intuition is that the main disanalogies with the AGI case are the first one (at least the economic incentives that might push people to try dangerous things when the returns are potentially great) and the last one, depending on your position on takeoffs.

Comment by adamshimi on Goal-directed = Model-based RL? · 2020-02-22T14:39:42.288Z · score: 3 (2 votes) · LW · GW

About the "right hand rule" agent, I feel it depends on whether it is a hard-coded agent or a learning agent. If it is hard-coded, then clearly it doesn't require a model. But if it learns such a rule, I would assume it was inferred from a learned model of what mazes are.

For the non-adaptative agent, you say it is less goal-directed; do you see goal-directedness as a continuous spectrum, as a set of zones on this spectrum, or as a binary threshold on this spectrum?

Comment by adamshimi on The Value of Theoretical Research · 2020-02-21T11:57:13.934Z · score: 1 (1 votes) · LW · GW

One aspects of fundamental research (and research in general) that I see missing from this post and many other explanations of why it is not the best use of your time, is being incremental. With some very rare exceptions, the maths you actually need, even if developed at the time where it was needed, depend on many things that had to be found prior to that.

The example that comes to mind, and was not mentioned in the post or the comments (as far as I know), is the birth of computer science. You can say: yay, Turing "invented" (with a lot of other people) theoretical computer science to solve concrete problems, when it was needed. But that would completely obstruct the fact that Turing builds heavily on top of Gödel, which solved questions of a purely mathematical nature. Among the ideas Gödel's work essential to the birth of computer science, diagonalization goes back to Cantor, whose work concerns some of the most pure and abstract maths ever.

That being said, I do agree from experience that many arguments one makes about justifying doing maths or theoretical computer science do not hold under scrutiny. Yet for the reason I give above, I still think pure theoretical research is necessary.

Comment by adamshimi on Goal-directed = Model-based RL? · 2020-02-20T22:08:01.718Z · score: 1 (1 votes) · LW · GW

Thanks for the feedback!

I indeed am thinking about your intuitions for goal-directed behaviors, because it seems quite important. I currently lack a clear idea (as formal as possible) of what you mean, and thus I have trouble weighting your arguments that it is not necessary, or that it causes most problems in safety. And since these arguments would have significant implications, I want to have as informed as possible an opinion on them.

Since you say that goal-directed behavior is not about having a model or not, is it about the form of the model? Or about the use of the model? Would a model-based agent that did not adapt its model when the environment changed be considered as not goal-directed (like the lookup-table agent in your example)?

Comment by adamshimi on Goal-directed = Model-based RL? · 2020-02-20T20:44:05.777Z · score: 2 (2 votes) · LW · GW

I'm curious about what you think people are aware of: that the idea of goal-directedness from the value learning sequence is captured by model-based RL, or that any sufficiently powerful agent (implicitly goal-directed) needs to be model-based instead of model-free?

If that's the former, I'm really interested in links to posts and comments pointing that out, as I don't know of any. And if that's the latter, then it seems that it is goes back to asking whether powerful agents must be goal-directed.

Comment by adamshimi on Training Regime Day 2: Searching for bugs · 2020-02-19T12:34:11.546Z · score: 3 (3 votes) · LW · GW

For more examples, here are the bugs I found following the prompts:

  • I consistently take too much time to wake up in the morning, between 30 minutes and 2 hours too much.
  • When working on something, I tend to do "just enough" to make good progress on it, and then stop for the day. Even if I could have kept going.
  • Although I am very comfortable in conversations, I have a weird anxiety about starting one with a complete stranger.
  • I have a consistent reluctance to start a new activity, like a coding project or cooking a new recipe. Whereas I thrive on new ideas.
  • My focus wanes around 1 hour after I start working on something on the best days, and I would want more.
  • I keep procrastinating on washing my dishes.
  • I take too much time thinking about how to do things and what I should do, and too little doing the things.
  • I regularly feel I'm not important to people.
  • I have trouble focusing when reading maths, and that's something I would want to improve.

I'm not sure these are bugs at the right level, but that's what I got out of the prompts.

Comment by adamshimi on Training Regime Day 1: What is applied rationality? · 2020-02-19T12:09:52.593Z · score: 2 (2 votes) · LW · GW

I am not sure I understand exactly what you are aiming for with your take 3: is applied rationality a rationality that I don't have to follow when I want to use emotions/intuitions/breaks, or is it a rationality that considers these options when making the decisions? The former seems to permissive, in that there is never a place where I have to use rationality, while the latter might fall into the same issues as the restaurant example, by pondering to much whether I should use intuition to choose my meal.

That being said, I like the different perspectives offered by the takes.

Comment by adamshimi on The Reasonable Effectiveness of Mathematics or: AI vs sandwiches · 2020-02-16T15:03:01.148Z · score: 1 (1 votes) · LW · GW

Nice post. Being convinced myself of the importance of mathematics both for understanding the world in general and for the specific problems of AI safety, I found it interesting to see what arguments you marshaled in and against this position.

About the unreasonable effectiveness of mathematics, I'd like to throw the "follow-up" statement: The unreasonable ineffectiveness of mathematics beyond physics (for example in biology). The counter argument, at least for biology, is that Wigner was talking a lot about differential equations, which seems somewhat ineffective in biology; but theoretical computer science, which one can see as the mathematical study of computation, and thus somewhat a branch of mathematics, might be better fitted to biology.

A general comment about your perspective is that you seem to equals mathematics with formal specification and proofs. That's not necessarily an issue, but most modern mathematicians tend to not be exact formalists, so I thought it important to point out.

For the rest of my comments:

  • Rather than precise, I would say that mathematics are formal. The difference lies in the fact that a precise statement captures almost exactly an idea, whereas formalization provide an objective description of... something. Given that the main difficulty in applying mathematics and in writing specification for formal methods is this ontological identification between the formalization and the object in the world, I feel that it's a bit too easy to say that maths captures the ideas precisely.
  • Similarly, it is not because the definitions themselves are unambiguous (if they are formal) that their interpretation, meaning and use is. I agree that a formal definition is far less ambiguous than a natural language one, but that does not mean that it is completely unambiguous. Many disagreement I had in research were about the interpretation of the formalisms themselves.
  • Although I agree with the idea of mathematics capturing some concept of simplicity, I would precise that it is about simplicity when all is explicited. That's rather obvious for rationalists. Formal definitions tend to be full of subtleties and hard to manage, but the explicit versions of the "simpler" models would actually be more complex than that.
  • Nitpick about the "quantitative": what of abstract algebra, and all the subfields that are not explicitly quantitative? Are they useful only insofar as they serves for the more quantitative parts of maths, or am I taking this argument too far and you just meant that one use of maths was in the quantitative parts?
  • The talk about Serial Depth makes me think about deconfusion. I feel it is indeed rather easy to makes someone not confused about making a sandwich, while it is still undone for AI Safety.
  • The Anthropocentrism arguments feels right to me, but I think it doesn't apply if one is trying to build prosaic aligned AGI. Then the "most important" is to solve rather anthropocentric models of decision and values, instead of abstracting them away. But I might be wrong on that one.
Comment by adamshimi on Distinguishing definitions of takeoff · 2020-02-14T12:50:48.179Z · score: 6 (6 votes) · LW · GW
I find discussions about AI takeoff to be very confusing.

So do I. So thanks a lot for this summary!

Comment by adamshimi on Toy model piece #5: combining partial preferences · 2020-02-13T17:09:40.349Z · score: 1 (1 votes) · LW · GW

Why should all equivalence classes of linked world have the same average utility? That ensures the unicity of the utility function up to translation, but I'm not sure that's always the best way to do it. What is the intuition behind this specific choice?

Comment by adamshimi on Value Impact · 2020-02-12T16:12:15.405Z · score: 1 (1 votes) · LW · GW

Thanks, I'll keep going then.

Comment by adamshimi on Value Impact · 2020-02-12T13:49:50.568Z · score: 1 (1 votes) · LW · GW

I don't see the link with my objection, since you quote a part of your post when you write of value impact (which is dependent on the values of the specific agents) and I talk about the need for context even for objective impact (which you present as independent of values and objectives of specific agents)

Comment by adamshimi on Value Impact · 2020-02-12T13:02:37.300Z · score: 1 (1 votes) · LW · GW

I have one potential criticism of the examples:

Because I was not sure what was the concrete implication of the asteroid impact, the reveal was unimpactful on me (pun inteded) that it was objectively valued negatively by anybody because they risk death. Had you written that the asteroid strikes near the agent, or that this causes massive catastrophes, then I would probably have though that it mattered the same for local peeblehoarders and for humans. Also, the asteroid might destroy pebbles (or depending on your definition of pebble, make new ones).

Also, I feel that some of your examples of objective impact are indeed relevant to agents in general (not dying/being destroyed), while other depends on sharing a common context (cash, which would be utterly useless in Pebblia if the local economy was based on exchanging peebles for peebles).

Do you just always consider this context as implicit?

Comment by adamshimi on Research Agenda v0.9: Synthesising a human's preferences into a utility function · 2020-02-12T12:33:20.134Z · score: 5 (3 votes) · LW · GW

Thanks, I'm looking into the toy model. :)

Comment by adamshimi on Toy model piece #4: partial preferences, re-re-visited · 2020-02-11T17:19:07.462Z · score: 1 (1 votes) · LW · GW

I really like the refinement of the formalization, with the explanations of what to keep and what was missing.

That said, I feel like the final formalization could be defined directly as a special type of preorder, one composed only of disjoint chains and cycles. Because as I understand the rest of the post, that is what you use when computing the utility function. This formalization would also be more direct, with one less layer of abstraction.

Is there any reason to prefer the "injective function" definition to the "special preorder" one?

Comment by adamshimi on The Relational Stance · 2020-02-11T13:02:42.122Z · score: 5 (3 votes) · LW · GW

Another modality of relating introduced to me by a friend a couple of weeks ago is "what kind of experience do you take from this relation". My friend has a quite idiosyncratic classification, but you could separate people you see between combinations of intellectual stimulation, sense of security, being cared for... In my mind this is quite orthogonal to other directions: whatever this relation holds for you, it might matter tremendously or very little.

The main use I have for this modality is to clarify what I am missing in my life. For example, when I feel lonely, I feel a discrepancy with my social situation: I have many friends, some really close who care about me and about whom I care. But when considering what experience I feel I am missing in my relationships, I can say that it's attraction and passion for the other and sexual tension and action.

Comment by adamshimi on The Curse Of The Counterfactual · 2020-02-11T12:45:47.348Z · score: 1 (1 votes) · LW · GW

Yes, I agree that you are focusing more on how to see the mistake in a meta-way, instead of an outside view as Nate do.

Though I don't think your example of the distinction is exactly the right one: the idea from Nate of banning "should" or cashing out "should" would be able IMHO to unearth the underlying "I should be taking things seriously" apply the consequentialist analysis of "you will not be measured by how you felt or who you punished. You will be measured by what actually happened, as will we all" (paraphrasing). What I feel is different is that the Way provide a mean for systematically findind this underlying should and explaining it from the inside.

Nonetheless, I find both useful, and I am better for having the Curse of the Counterfactual in my mental toolbox.

Comment by adamshimi on Category Theory Without The Baggage · 2020-02-10T13:05:37.181Z · score: 4 (3 votes) · LW · GW

I just found another interesting reference: Categories for the practising physicist. Although this is not exactly about discarding undue abstraction, it does present many concepts in terms of concrete examples, and there are even real-world categories defined in it!

Comment by adamshimi on The Curse Of The Counterfactual · 2020-02-10T12:19:18.148Z · score: 6 (3 votes) · LW · GW

Great post! I want to chew on it a bit before making a longer comment, but I noticed similarities between this post and Nate Soares's Replacing Guilt sequence (which I consider the most important sequence... ever). More specifically, he seems to say things similar about guilt and should in "should" considered harmful, Not because you "should" and Your "shoulds" are not a duty.

For example, from "should" considered harmful:

I see lots of guilt-motivated people use "shoulds" as ultimatums: "either I get the meds, or I am a bad person." They leave themselves only two choices: go out of their way on the way to work and suffer through awkward human interaction at the pharmacy, or be bad. Either way, they lose: the should has set them up for failure.
But the actual options aren't "suffer" or "be bad." The actual options are "incur the social/time costs of buying meds" or "incur the physical/mental costs of feeling ill." It's just a choice: you weigh the branches, and then you pick. Neither branch makes you "bad." It's ok to decide that the social/time costs outweigh the physical/mental costs. It's ok to decide the opposite. Neither side is a "should." Both sides are an option.

Or the idea of prefering to punish someone (me or another) instead of actually looking at the situation and accepting it, makes me think of tolerification:

There's a certain type of darkness in the world that most people simply cannot to see. It's not the abstract darkness: people will readily acknowledge that the world is broken, and explain how and why the hated out-group is responsible. And that's exactly what I'm pointing at: upon seeing that the world is broken, people experience an impulse to explain the brokenness in a way that relieves the tension. When seeing that the world is broken, people reflexively feel a need to explain. Carol can acknowledge that there is suffering abroad, but this acknowledgement comes part and parcel with an explanation about why she bears no responsibility. Dave can acknowledge that he failed to pass the interview, but his mind automatically generates reasons why this is an acceptable state of affairs.
This is the type of darkness in the world that most people cannot see: they cannot see a world that is unacceptable. Upon noticing that the world is broken, they reflexively list reasons why it is still tolerable. Even cynicism, I think, can fill this role: I often read cynicism as an attempt to explain a world full of callous neglect and casual cruelty, in a framework that makes neglect and cruelty seem natural and expected (and therefore tolerable).
I call this reflexive response "tolerification," and if you watch for it, you can see it everywhere.

The approach of these questions in the replacing guilt series is not exactly at the same level; most notably, I feel Nate is trying to explain why should are not "useful" and cause only harm that cannot serve for accomplishing your goals. On the other hand, I see this post as more about examining the exact mechanism underlying this error we make.

Still, I feel the connection is strong enough to encourage people to read both.

Comment by adamshimi on What can the principal-agent literature tell us about AI risk? · 2020-02-10T11:26:01.393Z · score: 3 (2 votes) · LW · GW

Great post! It explained clearly both positions, clarified the potential uses of PAL and proposed variations when it was considered accessible.

Maybe my only issue is with the (lack of) definition of the principal-agent problem. The rest of the post works relatively well without you defining it explicitly, but I think a short definition (even just a rephrasing of the one on Wikipedia) would make the post even more readable.

Comment by adamshimi on What Money Cannot Buy · 2020-02-09T15:22:26.903Z · score: 11 (3 votes) · LW · GW

Okay, so we agree that it's improbable (at least for decision problems) to be able to verify an answer faster than finding it. What you care about are cases where verification is easier, as is conjectured for example for NP (where verification is polynomial, but finding an answer is supposed to not be).

For IP, if we only want to verify any real-world property, I actually have a simple example I give into my intro to complexity theory lectures. Imagine that you are color-blind (precisely, a specific red and a specific green look exactly the same to you). If I have two balls, perfectly similar except one is green and the other is red, I can convince you that these balls are of different colors. It is basically the interactive protocol for graph non-isomorphism: you flip a coin, and depending on the result, you exchange the balls without me seeing it. If I can tell whether you exchanged the balls a sufficient number of times, then you should get convinced that I can actually differentiate them.

Of course this is not necessarily applicable to questions like tastes. Moreover, it is a protocol for showing that I can distinguish between the balls; it does not show why.

Comment by adamshimi on Research Agenda v0.9: Synthesising a human's preferences into a utility function · 2020-02-09T13:37:51.695Z · score: 1 (1 votes) · LW · GW

Could you give a list of some open problems or open questions related to this agenda (maybe with some pointers to the more relevant posts)? I am potentially interested in working on it, but I find it far easier to study a topic (and you sir write a lot of technical posts) while trying to solve some concrete problem.

Thanks in advance!

Comment by adamshimi on What Money Cannot Buy · 2020-02-08T16:07:38.745Z · score: 4 (2 votes) · LW · GW

The existence of problems whose answers are hard to verify does not entail that this verification is harder than finding the answer itself. Do you have examples of the latter case? Intuitively, it seems akin to comparing any (deterministic) complexity class with its non-deterministic version, and any problem solvable in the former is verifiable in the latter, by dropping the proof and just solving the problem.

For the difference between verifying a proof and an answer, I agree that interactive protocols are more appropriate for the discussion we're having. Even if interactive protocols are not about distinguishing between different experts, they might serve this point indirectly by verifying the beauty of a car design or the security of a system. That is, we could (in theory) use interactive proofs to get convinced with good probability of the quality of a candidate-expert's output.

Comment by adamshimi on What Money Cannot Buy · 2020-02-07T14:07:05.164Z · score: 6 (3 votes) · LW · GW

You're right. I was thinking on the level of letters, but the fact that he gives the same number of bits of entropy to four quite different words should have alerted me. And with around 2000 common words to choose from, the entropy is indeed around 11 bits per word.

Thanks for the correction!

(For our local password, the sentences tends to be created, to avoid some basic dictionary attacks, and they tends to be complex and full of puns. But you might be right about the entropy loss in this case.

Comment by adamshimi on "But that's your job": why organisations can work · 2020-02-07T13:50:34.389Z · score: 6 (3 votes) · LW · GW

Your point is that is all boils down to accountability, then. Not because of justice, but because failing on some aspects of your job for which you are held accountable by people on the outside (like not delivering the mail for the mail company, or polluting for the eco-friendly company) makes you vulnerable, and thus is really dangerous for your self-interest.

The fully cynical worldview is a bit too much for me, but I feel this explains a lot within this view.

Comment by adamshimi on "But that's your job": why organisations can work · 2020-02-07T13:41:44.652Z · score: 6 (4 votes) · LW · GW

Maybe it's more that the best systems need not win. They might, but that's not guaranteed.

Comment by adamshimi on Plausibly, almost every powerful algorithm would be manipulative · 2020-02-07T13:30:10.692Z · score: 5 (3 votes) · LW · GW

When taking the digital evolution example, it seems that by "manipulation", you mean that the tricks used by the programmers at training time did not prevent the behavior they were supposed to prevent. Which fits the intuitive notion of manipulative.

Then, we might see these examples as evidence that even classifiers, arguably the simplest of learning algorithms with the least amount of interaction with the environment, cannot be steered away from maximizing their goal by ad-hoc variations in the training protocol. Is that what your point was?

Somehow, I also see your examples (at least the first one, and the digital evolution one) as indicative that classifiers are robust against manipulations by their programmers. Because these classifiers are trying to maximize their predictive accuracy or their fitness, and the programmers are trying to make them maximize something else. Hence, we can see the behavior of these classifiers as "fault-tolerant", in a way.

Though this can be a big issue when the target of prediction is not exactly what we wanted, and thus we would like to steer it to something different.

Comment by adamshimi on What Money Cannot Buy · 2020-02-07T10:53:39.032Z · score: 2 (1 votes) · LW · GW

Also, I am pretty sure that the xkcd example is wrong. Mathematically, the entropy of the second password should be lower, because we can guess the next letters of the words from dictionary analysis, or even frequencies of next letters in language like english. And practically, dictionary attacks are pretty much built for breaking passwords like the latter.

The standard for root passwords in my local community is more on the order of finding a very long sentence, and taking letters from each words (the first one in the easiest scheme, but it can get harder) to build a long password that is both hard to guess and relatively easy to remember.

Comment by adamshimi on What Money Cannot Buy · 2020-02-07T10:48:04.501Z · score: 7 (4 votes) · LW · GW

This is interesting, because at least in theoretical computer science, verifying something is conjectured to be easier that creating it: the P vs NP question for example, where almost all complexity theorist conjectures that P is not equal to NP. That is to say, some problems in NP (problems for which we can verify a solution in polynomial time) are conjectured to not be in P (problems for which we can find a solution in polynomial time).

On the other hand, your examples hint at cases where verifying something (the quality of the product for example) is almost as hard as creating this thing (building a quality product).

Not sure if this adds anything to the conversation, but I found the connection surprising.

Comment by adamshimi on Category Theory Without The Baggage · 2020-02-06T14:24:19.254Z · score: 3 (2 votes) · LW · GW

So your path-based approach to category theory would be analogous to the matrix-based approach of group theory in physics? That is, removing the abstraction that made us stumble into theses concepts in the first place, and keeping only what is of use for our applications?

I would like to see that. I'm not sure that your own proposition is the right one, but the idea is exciting.

Comment by adamshimi on Plausibly, almost every powerful algorithm would be manipulative · 2020-02-06T14:18:33.927Z · score: 4 (2 votes) · LW · GW

If I understand your examples correctly, one way a classifier can be manipulative is by learning to control its training protocol/environment? Does this means that a fixed training protocol (without changes in the training sets or programmer interventions) would forbid this kind of manipulation?

This still might be a problem, since some approaches to AI Safety rely on human supervision/intervention.

Comment by adamshimi on Category Theory Without The Baggage · 2020-02-05T21:21:11.239Z · score: 14 (8 votes) · LW · GW

Separate comment for references to classical category theory books and resources. I don't think any of these are exactly what you are looking for, but they each propose a different perspective on these concepts, which might be the best we have now.

  • The best textbook I know of is Category Theory by Awodey. It is both rigorous and intuitive, at least at the level of maths textbook. There are a lot of examples, as concrete as possible, and the differences between them and how that inform the abstract definitions are treated in details.
  • Do not go to Maclane's Category Theory for Working Mathematician. Not that it is a bad book, just that it is the most honest title of a book I ever saw. Maclane writes for the working mathematician, so not even the graduate student in maths fits exactly his standards.
  • For a glimpse of the structuring power of category theory and its links to physics and computer science, Physics, Topology, Logic and Computation: A Rosetta Stone by Baez and Stay is the place to go. This paper also argues eloquently that the most important categories are not the one similar to the category of sets.
  • A short one that I like is Basic Category Theory for Computer Scientists by Pierce. It is short, to the point, and goes deeper into the applications to theoretical computer science. One caveat is that Pierce is the kind of computer scientist that studies proof theory and teaches Coq and theorem proving. So it might be slightly too abstract for some people.
Comment by adamshimi on The Best Textbooks on Every Subject · 2020-02-05T20:27:12.960Z · score: 3 (2 votes) · LW · GW

Thanks!