What is a Counterfactual: An Elementary Introduction to the Causal Hierarchy 2022-01-02T03:46:16.502Z
Y2K: Successful Practice for AI Alignment 2021-11-05T06:09:18.591Z
Leaky Delegation: You are not a Commodity 2021-01-25T02:04:55.942Z
Cause Awareness as a Factor against Cause Neutrality 2018-08-13T20:00:49.513Z
The Partial Control Fallacy 2016-12-14T02:57:24.773Z
Raytheon to Develop Rationality-Training Games 2011-11-18T21:07:51.461Z
Meetup : Pittsburgh Meetup 2011-09-30T17:48:10.478Z
Little Johny Bayesian 2009-03-18T21:30:59.049Z


Comment by Darmani on How To Raise Others’ Aspirations in 17 Easy Steps · 2022-01-07T10:51:35.618Z · LW · GW

I'm a certified life coach, and several of these are questions found in life coaching.


Is there something you could do about that problem in the next five minutes?

Feeling stuck sucks. Have you spent a five minute timer generating options?

What's the twenty minute / minimum viable product version of this overwhelming-feeling thing?

These are all part of a broader technique of breaking down a problem.  (I can probably find a name for it in my book.) E.g.: Someone comes in saying they're really bad at X, and you ask them to actually rate their skills and then what they could do to become 5% better. 


You want to do that but don't think you will? Do you want to make a concrete plan now?

Do you want to just set an alarm on your phone now as a reminder? (from Damon Sasi)

Do you sort of already know what you're going to do / have your mind made up about this?


These are all part of the "commitment" phase of a coaching session, which basically looks like walking someone through SMART goals.


Do you know anyone else who might have struggled with or succeeded at that? Have you talked to them about it? (from Damon Sasi)

Who do you know who you could ask for help from?


I can't say these are instances of a named technique, but they are things you'd commonly find a coach asking. Helping someone look inside themselves for resources they already have is a pretty significant component of coaching.

There's a major technique in coaching not represented here called championing. Champion is giving someone positive encouragement by reinforcing some underlying quality. E.g.: "You've shown a lot of determination to get this far, and I know you'll be able to use it to succeed at X."


Several of these questions do differ from life coaching in a big way: they suggest a course of action. We call this "advice-giving" as telling someone what to do serves the advice-giver's agenda more than the receiver's, or at least serves what the advice-giver thinks the receiver's agenda should be. The best piece of (irony forthcoming) advice I've received about coaching is to "coach the person, not the problem." Much more effective than to help someone with the task at hand is to help them cultivate the underlying skill. Instead of suggesting courses of action, you instead focus on their ability to come up with and evaluate options.


Recommended reading:  Co-active Coaching,

Comment by Darmani on $1000 USD prize - Circular Dependency of Counterfactuals · 2022-01-04T05:12:02.400Z · LW · GW

I realize now that this expressed as a DAG looks identical to precommitment.

Except, I also think it's a faithful representation of the typical Newcomb scenario.

Paradox only arises if you can say "I am a two-boxer" (by picking up two boxes) while you were predicted to be a one-boxer. This can only happen if there are multiple nodes for two-boxing set to different values.

But really, this is a problem of the kind solved by superspecs in my Onward! paper. There is a constraint that the prediction of two-boxing must be the same as the actual two-boxing. Traditional causal DAGs can only express this by making them literally the same node; super-specs allow more flexibility. I am unclear how exactly it's handled in FDT, but it has a similar analysis of the problem ("CDT breaks correlations").

Comment by Darmani on $1000 USD prize - Circular Dependency of Counterfactuals · 2022-01-04T02:22:46.271Z · LW · GW

Okay, I see how that technique of breaking circularity in the model looks like precommitment.


I still don't see what this has to do with counterfactuals though.

Comment by Darmani on $1000 USD prize - Circular Dependency of Counterfactuals · 2022-01-03T08:56:10.295Z · LW · GW

I don't understand what counterfactuals have to do with Newcomb's problem. You decide either "I am a one-boxer" or "I am a two-boxer," the boxes get filled according to a rule, and then you pick deterministically according to a rule. It's all forward reasoning; it's just a bit weird because the action in question happens way before you are faced with the boxes. I don't see any updating on a factual world to infer outcomes in a counterfactual world.

"Prediction" in this context is a synonym for conditioning.  is defined as .

If intervention sounds circular...I don't know what to say other than read Chapter 1 of Pearl ( ).


To give a two-sentence technical explanation:


A structural causal model is a straight-line program with some random inputs. They look like this


u1 = randBool()

rain = u1

sprinkler = !rain

wet_grass = rain || sprinkler


It's usually written with nodes and graphs, but they are equivalent to straight-line programs, and one can translate easily between these two presentations.

In the basic Pearl setup, an intervention consists of replacing one of the assignments above with an assignment to a constant. Here is an intervention setting the sprinkler off.


u1 = randBool()

rain = u1

sprinkler = false

wet_grass = rain || sprinkler


From this, one can easily compute that.


If you want the technical development of counterfactuals that my post is based on, read Pearl Chapter 7, or Google around for the "twin network construction."

Or I'll just show you in code below how you compute the counterfactual "I see the sprinkler is on, so, if it hadn't come on, the grass would not be wet," which is written 


We construct a new program,


u1 = randBool()

rain = u1

sprinkler_factual = !rain

wet_grass_factual = rain || sprinkler_factual

sprinkler_counterfactual = false

wet_grass_counterfactual = rain || sprinkler_counterfactual


This is now reduced to a pure statistical problem. Run this program a bunch of times, filter down to only the runs where sprinkler_factual is true, and you'll find that wet_grass_counterfactual is false in all of them.

If you write this program as a dataflow graph, you see everything that happens after the intervention point being duplicated, but the background variables (the rain) are shared between them. This graph is the twin network, and this technique is called the "twin network construction." It can also be thought of as what the do(y | x -> e) operator is doing in our Omega language.

Comment by Darmani on $1000 USD prize - Circular Dependency of Counterfactuals · 2022-01-03T04:32:39.498Z · LW · GW

While I can see this working in theory, in practise it's more complicated as it isn't obvious from immediate inspection to what extent an argument is or isn't dependent on counterfactuals. I mean counterfactuals are everywhere! Part of the problem is that the clearest explanation of such a scheme would likely make use of counterfactuals, even if it were later shown that these aren't necessary.


  1. Is the explanation in the "What is a Counterfactual" post linked above circular?
  2. Is the explanation in the post somehow not an explanation of counterfactuals?

The key unanswered question (well, some people claim to have solutions) in Functional Decision theory is how to construct the logical counterfactuals that it depends on.


I read a large chunk of the FDT paper while drafting my last comment.

The quoted sentence may hint at the root of the trouble that I and some others here seem to have in understanding what you want. You seem to be asking about the way "counterfactual" is used in a particular paper, not in general.


It is glossed over and not explained in full detail in the FDT paper, but it seems to mainly rely on extra constraints on allowable interventions, similar to the "super-specs" in one of my other papers: .


I'm going to go try to model Newcomb's problem and some of the other FDT examples in Omega. If I'm successive, it's evidence that there's nothing more interesting going on than what's in my causal hierarchy post.

Comment by Darmani on What is a Counterfactual: An Elementary Introduction to the Causal Hierarchy · 2022-01-03T00:04:08.260Z · LW · GW

I'm having a little trouble understanding the question. I think you may be thinking of either philosophical abduction/induction or logical abduction/induction.


Abduction in this article is just computing P(y | x) when x is a causal descendant of y. It's not conceptually different from any other kind of conditioning.

In a different context, I can say that I'm fond of Isil Dillig's thesis work on an abductive SAT solver and its application to program verification, but that's very unrelated.

Comment by Darmani on $1000 USD prize - Circular Dependency of Counterfactuals · 2022-01-03T00:00:43.117Z · LW · GW

I'm not surprised by this reaction, seeing as I jumped on banging it out rather than checking to make sure that I understand your confusion first. And I still don't understand your confusion, so my best hope was giving a very clear, computational explanation of counterfactuals with no circularity in hopes it helps.

Anyway, let's have some back and forth right here. I'm having trouble teasing apart the different threads of thought that I'm reading.


After intervening on our decision node do we just project forward as per Causal Decision Theory or do we want to do something like Functional Decision Theory that allows back-projecting as well?

I think I'll need to see some formulae to be sure I know what you're talking about.  I understand the core of decision theory to be about how to score potential actions, which seems like a pretty separate question from understanding counterfactuals.

More specifically, I understand that each decision theory provides two components: (1) a type of probabilistic model for modeling relevant scenarios, and (2) a probabilistic query that it says should be used to evaluate potential actions. Evidentiary decision theory uses an arbitrary probability distribution as its model, and evaluates actions by P(outcome |action). Causal decision theory uses a causal Bayes net (set of intervential distributions) and the query P(outcome | do(action)).  I understand FDT less well, but basically view it as similar to CDT, except that it intervenes on the input to a decision procedure rather than on the output. 


But all this is separate from the question of how to compute counterfactuals, and I don't understand why you bring this up.


When trying to answer these questions, this naturally leads us to ask, "What exactly are these counterfactual things anyway?" and that path (in my opinion) leads to circularity.


I still understand this to be the core of your question. Can you explain what questions remain about "what is a counterfactual" after reading my post?

Comment by Darmani on $1000 USD prize - Circular Dependency of Counterfactuals · 2022-01-02T03:46:37.589Z · LW · GW

Oh hey, I already have slides for this.


Here you go:


I took the approach: if I very clearly explain what counterfactuals are and how to compute them, then it will be plain that there is no circularity. I attack the question more directly in a later paragraph, when I explain how counterfactual can be implemented in terms of two simpler operations: prediction and intervention. And that's exactly how it is implemented in our causal probabilistic programming language, Omega (see or ).


Unrelatedly, if you want to see some totally-sensible but arguably-circular definitions, see .

Comment by Darmani on Y2K: Successful Practice for AI Alignment · 2021-11-05T19:45:01.797Z · LW · GW

"Many thousands of date problems were found in commercial data processing systems and corrected. (The task was huge – to handle the work just for General Motors in Europe, Deloitte had to hire an aircraft hangar and local hotels to house the army of consultants, and buy hundreds of PCs)."

Sounds like more than a few weeks.

Comment by Darmani on Y2K: Successful Practice for AI Alignment · 2021-11-05T19:44:26.261Z · LW · GW

Thanks; fixed both.

Comment by Darmani on Effective Evil · 2021-11-02T15:54:53.921Z · LW · GW

Was it founded by the Evil Twin of Peter Singer?

Comment by Darmani on What are fiction stories related to AI alignment? · 2021-10-29T06:49:46.183Z · LW · GW

Define "related?"

Stories of wishes gone awry, like King Midas, are the original example.

Comment by Darmani on The Best Software For Every Need · 2021-09-20T19:01:28.351Z · LW · GW

I've definitely looked at it, but don't recall trying it. My first questions from looking at the screenshots are about its annotation capabilities (e.g.: naming functions, identifying structs) and its UI (IDA highlighting every use of a register when you mouse over it is stupendously useful).

Comment by Darmani on Research speedruns · 2021-09-17T16:55:18.161Z · LW · GW

This reminds me of how I did the background reading for my semantic code search paper  ( ). I made a list of somewhat related papers, printed out a big stack of them at a time, and then for each set a 7.5 minute timer for each. By the end of that 7.5 minutes, I should be able to write a few sentences about what exact problem it solves and what its big ideas are, as well as add more cited work / search keywords to expand my list of related papers. I'd often need to give myself a few extra minutes, but I nonetheless worked through it incredibly fast.

The big strategy is that papers are written in a certain format (e.g.: (1) Introduction (2) Overview (3) Detailed technical development (4) Implementation (5) Evaluation (6) Related work (7) Conclusion), so I knew exactly where to look to get the important bits.

A difference between this and your suggestions is that (1) I was already highly knowledgable in this area, having just supervised a master's student building something better than these papers, and (2) the bar of "understand something well enough to discuss in a related work section" is rather low. Still, the end result is what is probably the best ever overview of semantic code search; our paper discusses a full 70 other tools.

Comment by Darmani on The Best Software For Every Need · 2021-09-11T01:38:59.007Z · LW · GW


Anytime I try a new language, first question is "Is there a JetBrains IDE or plugin for it?"

Comment by Darmani on The Best Software For Every Need · 2021-09-11T01:36:11.858Z · LW · GW

Bryan Caplan has been creating his "economics graphic novels" using an old "comic creator" software. He has a valid license, but they company that makes it went out decades ago, and the license server no longer exists. So I disabled the license-server check for him.

When I worked in mobile, I did it frequently. Customer would call us and say our SDK isn't working. I'd download their app off the app store, decompile it, and figure out exactly how they're using us.

It's also surprisingly frequent how often I want to step through a library or program that I'm using. If you link to that library as a binary, then (even if the source is available elsewhere) it's often easiest to debug it using a reverse-engineering tool.

Less everyday, but I've also done some larger projects involving REing. I started off in game modding 10 years ago. Last year, I did some election security work that achieved some publicity.

Comment by Darmani on The Best Software For Every Need · 2021-09-10T17:22:43.119Z · LW · GW

Software: Omnigraffle

Need: Making figures and diagrams (e.g.: for scientific papers)


Other software I've tried: Sketch, Illustrator, tikz


Omnigraffle has beautiful defaults, and makes it very fast to create shapes and diagrams that connect. It can make crossing edges look pretty and clear instead of a mess. Illustrator gives you a lot more flexibility (e.g.: strokes whose width gradually changes, arbitrary connection points for arrows), but you can be way faster at making figures with Omnigraffle.

Use Illustrator for making art and posters. Use Sketch (or Figma) for mocking up UIs. Use Omnigraffle for making figures.

Comment by Darmani on The Best Software For Every Need · 2021-09-10T17:19:06.041Z · LW · GW

Software: IDA

Need: Binary reverse-engineering

Other programs I've tried: ghidra, OllyDbg, Hopper

IDA is fast and well-featured. I've had multiple times where my process of having questions about a binary to figuring out the answer took minutes.

Hopper has a nicer UI, but works on fewer executables and does not analyze the binary as well.

IDA gets criticized for "having an interface designed by programmers," but ghidra is much worse in that regard. "A giant Java program written by the government" describes it well. ghidra supposedly has a collaboration mode, but I gave up trying to get it to work after much effort.

OllyDbg is not really comparable, being primarily a binary debugger. But IDA's built-in debugger is quite underrated. And their support is very good. I was among the first to use it for Android-hacking on Android 5.0, and found their Android debugger would not run with the new OS restrictions; they gave me a new version that would within a few days.

Comment by Darmani on The Best Software For Every Need · 2021-09-10T17:11:26.670Z · LW · GW

I'd appreciate if someone touched on HR software and CRMs for small businesses.

Also, collaborative document editing that isn't owned by Google.

Comment by Darmani on Decision-Making Training for Software Engineering · 2021-08-04T18:57:33.013Z · LW · GW

I've been running exercises like the one described here for nearly 5 years as part of my business ( ). They take the name "design exercises." It's done in both live and canned formats. Chief addition is that, in the live versions, the new features are chosen antagonistically.


Dagon in another thread claims "Making something that's maintainable and extensible over many years isn't something that can be trained in small bites." My long list of testimonials begs to differ.

Comment by Darmani on Jimrandomh's Shortform · 2021-07-18T05:57:59.331Z · LW · GW

Am incline to agree, but I want to add that security is all connected. There are several direct causal paths from compromised user data to compromised dev workstation (and vice versa).

Comment by Darmani on You are allowed to edit Wikipedia · 2021-07-07T02:32:03.674Z · LW · GW


Do you think the point of adding nuclear close calls isn't to move public policy into a direction that's less likely to produce a nuclear accident? That's a political purpose. It's not party political but it's political.  


Of course I believe it serves that purpose. I also believe that the most recent edit in all of Wikipedia at my time of writing, deleting a paragraph from the article on Leanna Cavanagh (a character from some British TV show I'd never heard of) serves to decrease the prominence of that TV show, which will weaken whatever message or themes it carries (such as bringing attention to Yorkshire, where the show is set).


So, this is an empty criticism.


Similarly, I don't know who "the account behind the edit you point to" is since I linked to two different revisions both of which cover edits by multiple authors, but I checked the edit history of one of them, user Simfish (whose real life identity I shan't reveal at this moment). He has a bunch of edits on the "Timeline of Nordstrom" article, and I don't know what that has to do with EA.


I'm not sure this conversation has any more productive purpose. You keep on harping on a specific defense of Wikipedia culture that any hostility encountered by my peers is justified because we were a paid special interest group. I've stated several reasons why those justifications did not apply at the time hostility was first encountered. I see you continuing to try to find ways to make those criticisms apply. Needless to say, this is a silly battle since I'm the one with all the details.


I can say that this experience is not leaving me any more desirous of editing Wikipedia, so I'm at least one person with whom you've not yet succeeded in your original goal.


Edit: Okay, I just found Simfish (and his real name) on a list of people whom Vipul paid, and found that Vipul Naki's timeframe overlapped with the FLI group. I have to partly retract the details behind my thesis above. I can still make it because I do not recognize anyone else on Vipul's list having a Boston/FLI connection.

Edit 2: Neither of these articles appear on the list of articles sponsored by Vipul.

Comment by Darmani on You are allowed to edit Wikipedia · 2021-07-06T22:08:53.427Z · LW · GW

It was paid-editing for a political agenda. From an EA perspective paying someone to do paid editing or do political lobbying is completley fine.  On the other hand you have the money isn't speech side that considers using money to do lobbying or get someone change Wikipedia according to their political interests bad. 


Putting aside that a volunteer project by a non-profit is not paid, and I take some issue with arguments that improvements to the page on nuclear close calls is "political":


I mean that some individuals later in this group, before any organized effort by the FLI existed, had dabbled in editing some of these same articles, for exactly the pure motives that you advocate editing for, and encountered entrenched (and perhaps unreasonable) opposition.


From the Wikipedia perspective there's a difference of a Wikipedia user group that does a Wikipedia-editing session together which is great and an organization having a project to change Wikipedia according to their agenda. 


Our perspective was that we were merely adding better information, improving accuracy, and giving fair summaries of the arguments.

I expect similar groups would say the same.

You can judge for yourself. Here are some edits from the group:

Comment by Darmani on You are allowed to edit Wikipedia · 2021-07-06T01:58:12.356Z · LW · GW

The talk of an admin who controlled those pages with an iron fist came from before this project existed, presumably encountered by affiliates who had tried to edit in good faith exactly as you've advocated, but were shut down.

We were far from the first or only group that had Wikipedia-editing sessions. I've walked past signs at my university advertising them for other groups. Ours was quite benign. I'm reading some of the discussion from back then; their list included things like adding links for the page on nuclear close calls.

I've seen articles on hot-button topics where the Wikipedia article is far more slanted to one side than any of the mainstream media articles, and read the talk archives where a powerful few managed to invoke arcane rules to rule out all sources to the contrary. It's stuff like this that makes me want out. I was a happy Wikipedian in high school in a previous decade, but I shall be no longer.

Comment by Darmani on You are allowed to edit Wikipedia · 2021-07-05T23:02:54.520Z · LW · GW

During my stint volunteering with the FLI, I worked on a project  to improve Wikipedia's coverage of existential risk. I don't remember the ultimate outcome of the project, but we were up against an admin who "owned" many of those pages, and was hostile to many of FLI's views.

This article, at least by appearances, is an excellent account of the problems and biases of Wikipedia:

Comment by Darmani on The Nature of Counterfactuals · 2021-06-06T18:18:45.138Z · LW · GW

The underlying thought behind both this and the previous post seems to be the notion that counterfactuals are somehow mysterious or hard to grasp. This looks like a good chance to plug our upcoming ICML paper, w

hich reduces counterfactuals to a programming language feature. It gives a new meaning to "programming Omega."

Comment by Darmani on Taking the outside view on code quality · 2021-05-07T07:47:48.776Z · LW · GW

It's a small upfront cost for gradual long-term benefit. Nothing in that says one necessarily outweighs the other. I don't think there's anything more to be had from this example beyond "hyperbolic discounting."

Comment by Darmani on Taking the outside view on code quality · 2021-05-07T05:09:54.996Z · LW · GW

I think it's simpler than this: renaming it is a small upfront cost for gradual long-term benefit. Hyperbolic discounting kicks in. Carmack talks about this in his QuakeCon 2013, saying "humans are bad at integrating small costs over time":


But, bigger picture, code quality is not about things like local variable naming. This is Mistake #4 of the 7 Mistakes that Cause Fragile Code:

Comment by Darmani on Lean Startup Reading Comprehension Quiz · 2021-02-02T06:28:58.628Z · LW · GW

I read/listened to  Lean Startup back in 2014. Reading it helped me realize many of the mistakes I had made in my previous startup, mistakes I made even though I thought I understood the "Lean startup" philosophy by osmosis.

Indeed, "Lean Startup" is a movement whose terminology has spread much faster than its content, creating a poisoned well that inoculates people against learning 

For example, the term "minimum viable product" has been mutated to have a meaning emphasizing the "minimum" over the "product," making it harder to spread the actual intended idea.  I blogged about this a long time ago:

Anyway, this post was a nice review! I had to guess on some of the questions, which is probably good; if I'm successful, it means I really internalized it. Thanks!

Comment by Darmani on Lean Startup Reading Comprehension Quiz · 2021-02-02T06:23:32.106Z · LW · GW

1. What is the difference between learning and validated learning?

 Validated learning is learning that has been tested empirically against users/the marketplace.

2. True or false: "According to the author, a startup with exponential growth in metrics like revenue and number of customers is doing well." Explain your answer.

 False. This is only true if those metrics imply a path of long-term sustainable profitability. If the startup in question is Github, it probably does. If it's Groupon.....

3. Finish the sentence: "almost every lean startup technique we've discussed so far works its magic in two ways:"

  By reducing inventory and increasing validated learning. 

4. Ries argues that startups should pay more attention to innovation accounting than traditional accounting. Name two ways in which startups can change their financial metrics to accomplish innovation accounting.

 (a) Estimate the value of patents/trade secrets and track on an internal balance sheet. (b) Require VoI calculations and add such numbers to an internal balance sheet. 

5. Describe, concretely, what a car company's supply chain would look like if it used push vs pull inventory.

 Push: Each supplier pumps out parts, which are stockpiled in storerooms and warehouses.  Each factory will regularly, e.g.: be shipped all the stuff it needs for the next month. Pull: Each factory keeps just a few days of parts needed, places frequent orders for the next few day's worth.

6. Ries applies the pull inventory model to startups. But what is the unit that is being pulled, and where does it obtain the "pull signal"?

 The unit is "aspects of the business that deliver value to customers." The initial pull is validated market demand, which then translates to internal demand for features/process.

7. True or false: "Lean manufacturing is meant to give manufacturers an advantage in domains of extreme uncertainty". Explain your answer.

 True. Lean manufacturing allows manufacturers to retool and change their production much faster, greatly cheapening the cost of creating a suboptimal or unwanted product.

8. True or false: "Lean manufacturing is about harnessing the power of economies-of-scale."

 False. Lean manufacturing cheapens the cost of small runs, making the manufacturer more competitive at a lesser scale.

9. Ries discusses an anecdote of a family folding letters. The dad folds, stamps, and seals one letter at a time; whereas the kids begin by folding all letters, then stamping all, etc. Name two reasons Ries' considers the dad's method superior.

 a) Not having to manage the intermediate outputs. b) Can discover issues later in the pipeline earlier.

10. True or false: "A consequence of lean manufacturing is that the performance of each employee as an isolated unit, in terms of output per unit of time, might *decrease*." Explain your answer.

  Lean manufacturing comes with much higher switching costs. An employee's output might shrink, but more of it will go towards useful ends.

11. Give an example of what a “large batch death spiral” might look like in practice. 

 My game team is running behind schedule.  To catch up, I ask the artists to produce assets without waiting for them to be tested.  This then creates a large batch of work for the programmers to implement the graphics, which produces a large back of comments. This gets passed back to the artists, who do a huge number of revisions at once. The cycle continues.

12. According to Ries, the “Five why’s” method is a control system (though he doesn’t say so explicitly). What does it control, and how? 

 It puts a damper on major failures; it creates a mechanism by which a failure is turned into a systematic, mitigating change.

13. Explain the meaning of Toyota proverb “Stop production so that production never stops”

 Do regular maintenance and improvement work to prevent larger future problems.

Comment by Darmani on Limiting Causality by Complexity Class · 2021-01-31T20:31:36.167Z · LW · GW

Causal inference has long been about how to take small assumptions about causality and turn them into big inferences about causality. It's very bad at getting causal knowledge from nothing. This has long been known.

For the first: Well, yep, that's why I said I was only 80% satisfied. 


For the second: I think you'll need to give a concrete example, with edges, probabilities, and functions. I'm not seeing how to apply thinking about complexity to a type causality setting, where it's assumed you have actual probabilities on co-occurrences.

Comment by Darmani on Limiting Causality by Complexity Class · 2021-01-31T00:17:58.074Z · LW · GW

This post is a mixture of two questions: "interventions" from an agent which is part of the world, and restrictions

The first is actually a problem, and is closely related to the problem of how to extract a single causal model which is executed repeatedly from a universe in which everything only happens once. Pearl's answer, from IIRC Chapter 7 of Causality, which I find 80% satisfying, is about using external knowledge about repeatability to consider a system in isolation. The same principle gets applied whenever a researcher tries to shield an experiment from outside interference.

The second is about limiting allowed interventions. This looks like a special case of normality conditions, which are described in Chapter 3 of Halpern's book. Halpern's treatment of normality conditions actually involves a normality ordering on worlds, though  this can easily be massaged to imply a normality ordering on possible interventions. I don't see any special mileage  here out of making the normality ordering dependent on complexity, as opposed to any other arbitrary normality ordering, though someone may be able to find some interesting interaction between normality and complexity.


Speaking more broadly, this is part of the broader problem that our current definitions of actual causation are extremely model-sensitive, which I find a serious problem. I don't see a mechanistic resolution, but I did find this essay extremely thought provoking, which posits considering interventions in all possible containing models:

Comment by Darmani on Leaky Delegation: You are not a Commodity · 2021-01-28T00:11:49.667Z · LW · GW

Thought I'd share an anecdote that didn't make it into the article: on how doing something yourself can make you a better outsourcer.

About 6 months ago, I went shopping for a logo for one of my projects. It helped greatly that I've spent a lot of time studying visual design myself.

I made a document describing what I wanted, including a mood board of other logos. I showed it to a logo design specialist recommended by a friend. He said "That's the best logo recquisition doc I've seen, and I've seen a lot."

I also showed it to the designer I've been working with on other things (like sprucing up Powerpoint slides). She's not a logo specialist, but quoted half the price.

I had the confidence that I'd be able to give good feedback to my designer even if she was less likely to knock it out of the park on her first try than the specialist. I went with her.

Many rounds of feedback later, I had a design. Showed it to some housemates and my advisor. "Dang, that's a good logo."


You can see it live at .

Comment by Darmani on Leaky Delegation: You are not a Commodity · 2021-01-28T00:07:49.840Z · LW · GW

Oh, on the contrary: I think this article misses several things that are quite important (or were brushed under a single sentence like "[main principal/agent problems] are communication and risk." Reason: emphasis on things fewer readers were likely to consider.

So the costs you're describing are indeed real and brushed off to corner. I think both of these fall under transaction costs, and #2 also under centralization and overhead. For #2, I think you mean something other than what "externality" means to me (a cost specifically born by a non-party to a transaction) --- maybe second-order cost?


Comment by Darmani on Leaky Delegation: You are not a Commodity · 2021-01-28T00:01:17.973Z · LW · GW

Thanks! This is good.

It's not a physical good, but I had also been thinking that most of the price of renting a venue on the open market is trust (that you won't mess up their space; whether they can give you the keys vs. needing someone to let you in), followed by coordination. Hence, why having a friend let you use their office's conference room on a weekend to do an event might cost $0, while renting such a space might cost $1000.

Comment by Darmani on Leaky Delegation: You are not a Commodity · 2021-01-27T07:44:21.131Z · LW · GW

To clarify: You're not saying the wedding tax is because of insurance costs, as the article is asking about, right?

Comment by Darmani on Leaky Delegation: You are not a Commodity · 2021-01-26T22:13:57.231Z · LW · GW

Thanks; fixed!

Comment by Darmani on Help your rivals when they are numerous · 2021-01-23T10:07:36.765Z · LW · GW

I have a number of issues with this post.


First, as others have mentioned, opponents are very much not equal. Further, timing is important: certain trades you should be much more or less likely to take near the end of the game, for example.

Second, I don't think it's valid to look at expected values when all you care about is rank.  Expectation is very much a concept for when you care about absolute amounts.

Third, which perhaps sums everything up: I don't see a valid notion of utility / utility maximization for board games, other than perhaps "probability of winning," which makes this circular ("if you're trying to win, you should make moves that increase your probability of winning"). Utility is meant to put a linear scale on satisfaction with a given state of the world. When discussing what to do in a board game, one usually presumes the objective is to win, and satisfaction derives ultimately from winning. The closest thing you usually see to a "utility" number on an intermediate state is a heuristic, as used in e.g.: chess AIs, where you might give yourself 5 points for having a pawn in a center square. If I'm remembering my undergrad correctly, these heuristics are intended to approximate log-likelihoods of victory, but they certainly lack the soundness required to think about expected utility.


Let's switch out of Catan, and to a game that hopefully people here know but is more directly combative: Diplomacy. Pray tell me how you propose to assign a utility score to putting a navy in the Black Sea.

Comment by Darmani on In software engineering, what are the upper limits of Language-Based Security? · 2020-12-28T01:01:12.819Z · LW · GW

Did not know about the answer/comment distinction! Thanks for pointing that out.

Before I dig deeper, I'd like to encourage you to come bring these questions to Formal Methods for the Informal Engineer ( ), a workshop organized by myself and a few other researchers (including one or two more affiliated with LessWrong!) specifically dedicated to helping build collaborations to increase the adoption of formal methods technologies in the industry. Since it's online this year, it might be a bit harder to have these deep open-ended conversations, but you sound exactly like the kind of person we want to attend. (To set expectations, I should add that registrations already exceed capacity; I'm not sure how we plan to allocate spots.)

I'd also like to share this list of formal methods in industry: . In the past decade, there's been a huge (in relative, not absolute terms) increase in the commercial use of formerly Ivory Tower tools.

You may also be interested in the readings from this course:

BTW, I've been trying to think about whom I know that directly works in language-based security.  (defined as a narrow specialty). The main name that comes to mind that I personally know is Stephen Chong, but I think some of the Wyvern developers ( ) may also consider themselves in this category (and be easier to get ahold of than a Harvard professor).

I'm going to briefly hit some of your more narrow questions now. As an aside, be wary about saying "process" to a researcher -- it's used narrowly in ICSE circles to mean "methodology" (e.g.: Agile). I'm trying to mentally replace every use with "language-based security."

the most security-promoting development processes that are currently in wide use.

I think Jim mostly gets it above: memory-safe languages and secure API design; also, implicitly, the type systems that make the latter possible.

There are a number of patterns in secure API design you might not know the names for, such as object capabilities ( ).

In a way, this question is kinda self-answered by the framing, since language-based security primarily refers to language design, which primarily means type systems --- this is in contrast to techniques such as static analysis, testing, model checking, symbolic execution, and sandboxing.

Leaving the realm of Turing-complete programs, I'll point you to PNaCl/RockSalt ( ) and eBPF, both of which have verified sandboxes.

If you're willing to be flexible about the "widely-used" statement, then individual companies have their own quirky languages, some of which have rather interesting restrictions. This language ostensibly used by OutSystems comes to mind ( ), although I'm told by one of the authors that their actual implementation (as of 2013) is a bit simpler.


  • the most security-promoting development processes that are possible with recently developed technology


This is a tough question for me, because recent papers in this area tend to be about solving highly specific problems (the space of security problems is big, yo), and it takes a lot to generalize that to answer such a broad question. Also, I don't follow latest developments that intensely. I'm going to take a pass.


  • processes that could come to exist 10 years away; processes that might exist 30-50 years from now.


Adam Chlipala thinks by that time we'll be generating correct-by-construction code from specs. The Everest and Fiat-Crypto projects, both of which generate correct-by-construction cryptography code, are probably the two current best-known deployments of this.



  • perhaps some impossibility theorems that may bind even the creatures of the singularity.


"If it's nontrivial to prove your program terminates, your program probably doesn't run." --- an undergrad friend.

For common infrastructure software of today, no. Except maybe that I don't know it's been shown  possible to build secure, reliable software atop a realistic model of hardware faults.

For Software 2.0 (i.e.: neural net in the loop), it's a more open question that I don't know much about.

For the kinds of reflective self-improvement software MIRI discusses, that's part of their active research program (and generally outside the cognizance of PL/SE researchers).

Comment by Darmani on In software engineering, what are the upper limits of Language-Based Security? · 2020-12-27T06:52:40.819Z · LW · GW

It is already possible to build an embedded language inside of Coq which requires arbitrary security proofs for all executable code. It's theoretically possible to make those proofs guard against all known side channels, including, theoretically, hardware-based side channels such as Row Hammer.

Are you asking about which kinds of attacks can't be stopped by improving software? Or are you asking about the theoretical limits of PL technology? The latter question is not so interesting without  more constraints: as stated above, they're basically unbounded.

Comment by Darmani on Extortion beats brinksmanship, but the audience matters · 2020-11-21T09:29:43.343Z · LW · GW

Okay. I think you're saying this is extortion because Walmart's goal is to build a reputation for only agreeing to deals absurdly favorable to them.

If the focus on building a reputation is the distinguishing factor, then how does that square with the following statement: "it is not useful for me to have a credible reputation for following up on brinksmanship threats?"

Comment by Darmani on Extortion beats brinksmanship, but the audience matters · 2020-11-20T01:01:03.340Z · LW · GW

I see. In that case, I don't think the Walmart scenario is extortion. It is not to the detriment of Walmart to refuse to buy from a supplier who will not meet their demands, so long as they can find an adequate supplier who will.

Comment by Darmani on Extortion beats brinksmanship, but the audience matters · 2020-11-18T02:53:18.473Z · LW · GW

I'd really appreciate a more rigorous/formal/specific definition of both. I'm not seeing what puts the Walmart example in the "extortion" category, and, without a clear  distinction, this post dissolves.

Comment by Darmani on Extortion beats brinksmanship, but the audience matters · 2020-11-17T04:25:52.461Z · LW · GW

Very interesting post. I was very prepared to praise it with "this draws some useful categories for me," but it began to get less clear as I tried more examples. And I'm still trying to come up with a distinction between brinksmanship and extortion. I've thought about the payoff matrices (they look the same), and whether "unilateral attack vs. not" is a distinguishing factor (I don't think so). I still can't find a clear distinction.



(1) You say that releasing nude photos is in the blackmail category. But who's the audience?

(2) For n=1, m large:  Is  an example of brinkmanship here a monopolistic buyer who will only choose suppliers giving cutrate prices? It seems to have been quite effective for Walmart decades ago, and effective for Hollywood today ( ).

Comment by Darmani on Gems from the Wiki: Paranoid Debating · 2020-09-15T06:26:34.430Z · LW · GW

I tried playing this in 2009 at a math summer program. It scared a lot of people away, but I got a small group to join in. The scoring algorithm was rather questionable, but the game of competitive Fermi estimates was fun.

I can't claim to have improved much at rationality or estimates, but, ever since then, I remember that, to the question "How many liters of water are there in the ocean," the answer "1 mole of liters" is not the mark of a deceiver, but is actually relatively close, being only 1 order of magnitude too low.

If I ever play again, now that I know some measure theory, I'd be tempted to play a distribution that has an arbitrarily high mass at every rational number, and 0 at every irrational number.

Comment by Darmani on When can Fiction Change the World? · 2020-08-25T09:20:07.535Z · LW · GW

Nice post!

Another example for your list: Altneuland catalyzed the Zionist movement that led to the creation of Israel. The city of Tel Aviv is named after the book.

Comment by Darmani on A Hierarchy of Abstraction · 2020-08-11T23:59:57.670Z · LW · GW

Was just reading through my journal, and found that I had copied this quote. I think you'll find it to be of interest re: teaching recursion.


From “Computing Science: Achievements and Challenges” (1999):

“I learned a second lesson in the 60s, when I taught a course on programming to sophomores, and discovered to my surprise that 10% of my audience had the greatest difficulty in coping with the concept of re, cursive procedures. I was surprised because I knew that the concept of recursion was not difficult. Walking with my five-year old son through Eindhoven, he suddenly said "Dad not every boat has a life-boat, has it?" '"'How come?" I said. "Well, the life-boat could have a smaller life-boat, but then that would be without one." It turned out that the students with problems were those who had had prior exposure to FORTRAN, and the source of their difficulties was not that FORTRAN did not permit recursion, but that they had not been taught to distinguish between the definition of a programming language and its implementation and that their only handle on the semantics was trying to visualize what happened during program execution. Their only way of "understanding" recursion was to implement it, something of course they could not do. Their way of thinking was so thoroughly operational that, because they did not see how to implement recursion, they could not understand it. The inability to think about programs in an implementation-independent way still afflicts large sections of the computing community, and FORTRAN played a major role in establishing that regrettable tradition”

Comment by Darmani on Reveal Culture · 2020-08-11T21:43:39.939Z · LW · GW

I've noticed that the Reveal Culture examples / Tell Culture done right resemble greatly the kinds of communication advocated in the many strands of conflict/communication training I've taken. Connecting your requests to needs, looking for interests instead of positions, seeing the listener's perspective, etc.

For instance, the Tell Culture example example "I'm beginning to find this conversation aversive" is quite close to the example from my training "I notice I'm having a reaction," except that it's closer to being judgmental. For comparison, here's a quote I have in Anki, I believe from the book "Difficult Conversations": "When doing active listening, strategies for making the tension explicit include signaling that you're having a reaction, sharing how you're feeling, and postponing the conversation because of emotions."

The people who gave Malcolm's friend the "Crocker's Rules" impression were probably failing to not mix in judgments into their tells. I recently taught a workshop on this, which reminded me just how hard this is for many people.

It's become very apparent to me that one person with high communication skills can go a long way towards making up for deficits in all whom they interact with. If Reveal Culture/Tell Culture as you understand it really is recommending adopting some of the habits recommended by books like NVC and Difficult Conversations, then I do see this as primarily being about skill, not culture, although learning these skills can be quite deep and can entail some personality changes. One possible reconciliation: having a default preference toward sharing your inner world and accepting those of others may make people much more tolerant of unskilled attempts to do so, where people inadvertently give their positions and judgments instead of their observations, feelings, and needs.

Comment by Darmani on Towards a Formalisation of Logical Counterfactuals · 2020-08-09T23:19:02.101Z · LW · GW

Hi Bunthut,

First, I'll encourage you to have a look at material on what I thought this post was going to be about from the title: . I know about this subject primarily from (which is more concrete/mathematical than the Wikipedia article, as it's written by computer scientists rather than philosophers).

Second: If I'm understanding this correctly in my sleep deprived state, you're actually working on the exact same problem that I am in one of my papers, except that we focus on causality/counterfactuals in logic systems which correspond to programming language semantics. I got stuck on similar problemss. I can send you a preprint if you PM me.

Overall, my take is that you're getting stuck in superficial differences of formalisttm, which makes it much harder for sleep-deprived me to find the insight. E.g.: the desire to use graphs to represent proofs; with the proper conceptual toolkit, this "multiple-antecedent problem" is a non-issue. (Two solutions: (1) use hypergraphs [which we do in the current draft of the paper]; (2) join antecedents into one, as is done in proof categories.)

Comment by Darmani on A Hierarchy of Abstraction · 2020-08-09T02:06:58.118Z · LW · GW

I was 12 or so when I first studied pointers. I did not get them at all back then.