Debate tools: an experience report

Comment in a nutshell: "Practice with very simple tools optimized for swift use in a culture that values logic and evidence are likely to be better than specialized tools with hard coded abstractions."

There's a specialized form of note taking called "flowing" within the policy/CEDA/NDT debate community. Here is a wikipedia article on the subject if anyone wants to hunt in the concept space for keywords and links and such :-)

In this speech community (especially at the higher levels) people tend to speak very swiftly because there is a background theory that "dropping arguments is conceding arguments" which creates an incentive to make many parallel arguments with the same conclusion that reach towards different sets of evidence. It is possible to win simply by making many adequate arguments that your opponent is incapable of handling with enough speed or concision.

(It can be dangerous to spin out many arguments without paying attention to how they interact, because sometimes a response will be something like "I concede argument 2 and 5 which constitute a turn on the general position that works like X. Arguments 1, 3, and 4 were about A, B, and C which are all less important than this, so I win the position.")

In terms of basic mechanics, the note taking process that I learned used strikingly simple tools: a four color pen, a legal pad, and (if you were going to prep for standard positions or responses by taking re-usable notes on what you would say) cover up tape. The process involved taking notes in a very narrow up-down column (about the width of cover up tape) so that iterated response and counter response could move horizontally across the page in rows. Time through each speech flows down on a given page, speech to speech flows horizontally. Arrows sometimes connect points if you underestimate the space requirements for a "clean flow" to help show how the rows are supposed to line up.

An eight minute speech would be flowed on several different pages, usually called "positions". As a matter of helping the judge flow the round (so they can take clean notes on your arguments and will notice when you win) it was wise to start a speech by announcing how many new positions would be introduced and/or the order in which existing positions would be addressed. Then everyone shuffles papers for the first few seconds of the speech. For a debate round on cryonics there might be a positions (each with a separate piece of paper) for "personal friendships and relationships", "health-dollar utility", "dehumanization critique", and so on.

Sometimes a column on a page would be a simple list of numbered points, but sometimes an argument would actually have an implicit tree structure, with mulitple points having sub points of their own. Comparing columns, then, its best to imagine one serialized tree next aligned with another. Structured argument was much more common in the early speeches of a round when significant preparation was possible and evidence was being read whose implications would be argued later on. Also, each piece of paper functioned sort of like its own "major tree branch" for the entire round (if you're trying to map the physical objects to an abstract data structure) and it was rare for cross applications of logic to jump from one page to another, but more common within a page.

The actual note taking would be full of your own jottings for arguments, like I used to use a capital T with a circle around it for "turn" which meant that the argument related to the "sign" (positive or negative) of the general position. An extended example might help...

Imagine that one side argues that the spread of some public perception would have a terrible consequence; then the other side might concede at the consequence level but proceed to argue that the perception already exists in the status quo and is growing and would actually go down on the basis of a proposed course of action... that would be a turn. In notes I might have one side flowed as "2. Perc {right arrow} H" (the 2nd argument was that perception is causally connected to harm) and then the response would be recorded just to the right with something like "T: Perc {up arrow}ing SQ" (perception is increasing in the status quo and somehow the proposed policy would decrease it, changing the sign of the broader argument).

Another important thing for the actual debate was that people would read sourced evidence into the round (at very high speed) so aside from the gist of the content, the author and year of publication could be very helpful because you might be familiar with them and have a retraction (or rebuttal by another author) written two years later in your own debate files. Even if you have no response you can go to the library with the citation and prep for the next time you see that evidence in a round... or you could use it yourself to argue the other side if it's just a great position :-)

One issue is that note taking systems like this work much better when the speakers are trying to be logically intelligible and expect you to flow them. The speakers can "signpost" as they raise a point by saying things like "...and my fourth argument here is that X" and then it can be responded with "their fourth argument about X is spurious because of Z". The numbers in the speech and in the notes help everyone retain the same alignment.

Sometimes I would debate using flowing skills in competitive contexts that were focused on rhetoric rather than reason (less signposting, less note-taking by the judge, speaker points awarded for wardrobe, little tolerance for meta-arguments about debate theory to enforce compliance with social norms, etc) and the note taking system generally still worked.

If the audience has a phobic reaction to stuttered logic and reason you can get hurt relying on the cultural forms of policy debate (in policy debate, T-shirts could reverse-signal to some judges that you relied on "pure logic"), but in the absence of phobias, "flow skills" still appeared to be relatively valuable because they helped you address people's points directly (which audiences usually like) and notice when they tried a logical shenanigan (which academic audiences enjoy seeing called out and given a latin name).

It's been many years since I judged or coached, and even more since I debated myself. My understanding is that in modern times people frequently bring laptops into rounds and flow using spreadsheets. Evidence is stored on hard drives instead of lugging around 100 pounds of paper in a filing system. It sounds altogether more civilized in that respect -- and learning to use the hotkeys for high speed note taking in spreadsheets is probably a skill that would help for years after in contexts like finance and consulting :-)

In any case, these sorts of experiences have adjusted my priors on the subject of rationality prostheses to favor of practice and skill with generic tools (paper and pen, or spreadsheets) as opposed to highly specialized software with a bunch of YAGNI features.

If any readers are in college (or better yet choosing a college), I would recommend googling around to see if you can find an associated intercollegiate policy debate team. Sometimes they are called CEDA (Cross Examination Debate Association) and sometimes NDT (National Debate Tournament) although the formerly distinct communities had already pretty well merged 10 years ago when I was debating. It can cost 20 to 60 hours a week, but I know of no other "game" with as wonderful a mix of adrenaline, logic, and language. In retrospect, I think that policy debating was more educational than many of my actual classes :-)

First, my own observation agrees with GreenRoot. My view is less systematic but much longer, I've been watching this area since the 70s. (Perhaps longer, I was fascinated in my teens by Leibnitz's injunction "Let us calculate".)

Empirically I think several decades of experiment have established that no obvious or simple approach will work. Unless someone has a major new idea we should not pursue straightforward graphical representations.

On the other hand we do have a domain where machine usable representation of thought has been successful, and where in fact that representation has evolved fairly rapidly. That domain is "programming" in a broad sense.

Graphical representations of programs have been tried too, and all such attempts have been failures. (I was a project manager for such an attempt in the 80s.) The basic problem is that a program is naturally a high-dimensional object, and when mapped down into a two dimensional picture it is about as comprehensible as a bowl of spagetti.

The really interesting aspect of programming for representing arguments isn't the mainstream "get it done" perspective, but the background work that has been done on tools for analyzing, transforming, optimizing, etc. code. These tools all depend on extracting and maintaining the semantics of the code through a lot of non-trivial changes. Furthermore over time the representations they use have evolved from imperative, time-bound ones toward declarative ones that describe relationships in a timeless way.

At the same time programming languages have evolved to move more of the "mechanical" semantics into runtimes or implicit operations during compile time, such as type inference. This turns out to be essential to keep down the clutter in the code, and to maintain global consistency.

The effect is that programming languages are moving closer to formal symbolic calculi, and program transformations are moving closer to automated proof checking (while automated proof checking is evolving to take advantage of some of these same ideas).

In my opinion, all of that is necessary for any kind of machine support of the semantics of rational discussion. But it is not sufficient. The problem is that our discussion allows, and realistically has to allow a wide range of vagueness, while existing programming semantics are never nearly vague enough. In our arguments we have to refer to only partially specified, or in some cases nearly unspecified "things", and then refine our specification of those things over time as necessary. (An extremely limited but useful form of this is already supported in advanced programming languages as "lazy", potentially infinite data structures. These are vague only about how many terms of a sequence will be calculated -- as many as you ask for, plus possibly more.)

For example look at the first sentence of my paragraph above. What does "all of that" refer to? You know enough from context to understand my point. But if we actually ended up pursuing this as a project, by the time we could build anything that works we'd have an extremely complex understanding of the previous relevant work, and how to tie back to it. In the process we would have looked at a lot of stuff that initially seemed relevant (i.e.currently included in "all of that") but that after due consideration we found we needed to exclude. If we had to specify "all of that" in advance (even in terms of sharp criteria for inclusion) we'd never get anywhere.

So any representation of arguments has to allow vagueness in all respects, and also allow the vagueness to be challenged and elaborated as necessary. The representation has to allow multiple versions of the argument, so different approaches can be explored. It has to allow different (partial) successes to be merged, resolving any inconsistencies by some combination of manual and machine labor. (We have pretty good tools for versioning and merging in programming, to the extent the material being manipulated has machine-checkable semantics.)

The tools for handling vagueness are coming along (in linguistic theory and statistical modeling) but they are not yet at the engineering cookbook level. However if an effort to build semantic argumentation tools on a programming technology base got started now, the two trajectories would probably intersect in a fairly useful way a few years out.

The implications of all of this for AI would be interesting to discuss, but perhaps belong in another context.

Thanks for looking into this for us!

One of my ever-pending posts to write is on what sort of simple interface might prevent online arguments from retracing the same points over and over. I suspect it will not be graphical with boxes, because that makes poor use of screen real estate. I suspect it will not have lots of fancy argument types and patterns, because no one really uses that stuff. I think it does need to have a karma system, because otherwise there's no way to find the good stuff.

Reading that Argunet was both open source and flawed, I decided to download the code (signals geekiness) in order to fix the problem you have encountered (signals generosity). I didn't have to change anything: it took me a while (signals perseverance) but I finally figured out how to move a box in Argunet (signals superior problem-solving abilities).

It turns out that the result of dragging a box depends on whether the box was selected prior to dragging it. If the box was selected, it is moved, otherwise, an arrow is created. So, before you move a box, select it by clicking on it.

I created a wikipage listing all of the debate tools that were mentioned so far in this comments thread:

http://wiki.lesswrong.com/wiki/Debate_tools

Please feel free to edit this wikipage.

I would recommend that we try to create our own debate-mapping tool. It might end up being surprisingly easy.

I've already used PHP, GraphViz, and MediaWiki to implement a vaguely similar project, the Transhumanist Wiki Scenarios Map.

Unfortunately, that project ended up being less useful than I had hoped, and has been abandoned for now.

Today, I made a rough sketch of what a debate-mapping tool based on these tools might look like.

A VERY rough sketch.

Pretty much every detail is probably going to need to be changed in order for it to be useable.

Anyway, here's a link to that experiment

Once again, It didn't turn out as well as I had hoped.

The basic idea is that you take a chat log of a debate, and add some annotations, marking which are the main claims of the argument, and indicating which arguments support or oppose which other arguments.

Then, run a script on this annotated chat log, and it will output a graph of the arguments in the debate.

One advantage of this method is that the text and the annotations can be updated as the debate continues, and the graph will be updated to match this new data.

Some ideas for things to change:

change the formatting of the annotations. the word "claim" is unnecessary

set up the actual PHP script. These example graphs were generated by manually formatting the annotations in the graphviz format.

set up different formats for the output. A graph is not the most useful format. A better idea would be a table summarizing the info for each of the claims:

• the text of the claim
• a list of supporting claims (preferably with clickable links)
• a list of opposing claims
• a list of claims that this claim supports
• a list of claims that this claim opposes
• a list of who agrees with this claim
• the results of any of the other calculations that were performed on this graph. Maybe something with probabilities?
• more?

perhaps each claim could have its own wiki page, similar to how the scenarios map works

add more keywords, besides just "supports" and "opposes". Some examples are:

• relies on
• is an example of
• is a counterexample to
• assumption
• more?

add a way to indicate which speaker agrees with which claims, and deduce from that which conclusions are supported by implications of their assumptions

set up the script to automatically generate the graphs as the wiki page is updated

more?

I have my doubts that the "visual" aspect is as important as the creators of such software tools would like everyone to think.

I strongly suspect this is highly personal. People can relate to thoughts in different ways. The developers might primarily 'see' argument structure in their heads, where you might think less visually. This tools helps them clarify their ideas and show them to other visually oriented people.

So you're right that it might not catch on, but it may be that the visual aspect is hugely useful to the people it's adapted for.

I looked into this sort of software a couple years ago as part of a information visualization seminar at Stanford. People have been creating research prototypes and betas for a long time, aiming to support everything from education to the formalization of legal dispute resolution ("computational law"). I wasn't impressed with what I saw in my review then, and looking around again today, it doesn't look like things have changed much. I have never seen anything quantitative.

You can find descriptions of research systems and small-scale studies in the education vis and info-vis literatures. (InfoVis, VAST, etc.)

A book: http://www.visualizingargumentation.info/ A new journal: http://www.tandf.co.uk/journals/tarc

One problem with these sorts of tools is that they encourage people to split their arguments into lots of little pieces, which means arguments for C often end up being A->B, B'->C where B and B' look alike but on closer inspection turn out to be different.

Here's another interesting and potentially useful tool I found recently. I'm not sure if this qualifies as a debate tool, but it seems like it's in the general category of what we're looking for:

Summary: Canonizer.com is a wiki system with added camp and survey capabilities. The system provides a rigorous way to measure scientific / moral expert consensus. It is designed for collaborative development of concise descriptions of various competing scientific or moral theories, and the best arguments for such. People can join the camps representing such, giving a quantitative survey or measure of consensus compared to all others. Proposed changes to supported camps go into a review mode for one week. Any supporters of a camp can object to any such proposed change during this time. If it survives a week with no objection, it goes live, guaranteeing unanimous agreement to such changes to the petition by all current signers. If anyone does object, the camp can be forked (taking all supporters of the 'improvement'), or the info can be included in a sporting sub camp.

The karma or 'canonization' system enables the readers to select any algorithm they wish on the side bar to 'find the good stuff'. For example, you can compare the mind expert scientific consensus with the default general population consensus. Each camp has a forum to discuss and debate further improvements for camps. The general idea is to debate things in the forums, or elsewhere, and summarize everyone's final / current / state of the art view in the camp statements. A history of everything is maintained, providing a dynamic quantitative measure of how well accepted any theory is, as ever more theory falsifying scientific data / new arguments... come in.

Today I did a google search for "debate map", and this was the very first result:

http://debategraph.org/

This... is exactly what we're looking for, isn't it?

Though it still doesn't actually do anything with numbers.

I still haven't gotten around to continuing my own project for a debate tool that actually does calculations involving probabilities, though it has finally risen to the top of my to-do list. I was planning to get back to work on it last weekend, but ended up getting distracted by other things again.

A point about using diagrams to make arguments: If you are attempting to convince a person that something is true, rather than just launching into your evidence and favorite arguments it is often most efficient to begin by asking a series of questions to determine precisely how the person disagrees with you. The questioning allows you to hone in on the most important sticking points that prevent the other party from coming to your conclusion. These points can then be attacked individually, preventing you from wasting time making arguments that the other party already agree with or refuting positions that he or she has never even considered. The reason that this relates to diagrams is that this method of argumentation can be viewed as a tree, with questions at each of the higher level branches, and arguments at the leaf nodes.

Debate mapping is part of TakeOnIt, a publicly editable database of expert opinions introduced in a previous post ( http://lesswrong.com/lw/1kl/takeonit_database_of_expert_opinions/ ). It's deliberately very simple. Here's how it works:

1) Every debate is expressed as a yes-no question.
2) Every yes-no question has experts on both sides of the debate.
3) Every debate can link to a sub-debate (recursively).

A "simple debate" is one where '1' and '2' are sufficient. You can determine who is right in a simple debate by judging which experts have the best arguments, the best credentials, or the best track records. A good example is the vaccine debate, here:

http://www.takeonit.com/question/291.aspx

A "complex debate" is one which also requires '3'. This occurs when a simple debate is not sufficient to judge correctness, because the expert arguments, credentials, and track records, seem sufficiently reasonable on both sides of the debate. By recursively splitting a debate into sub-debates, a complex debate simply becomes a hierarchically structured set of simple debates. A good example is the Global Warming debate, here:

http://www.takeonit.com/question/5.aspx

The truth lies in the sub-debates. For example, in the Global Warming debate, there's a sub-debate as to whether cosmic radiation significantly affects earth's climate ( http://www.takeonit.com/question/74.aspx ). We have the top skeptic S. Fred Singer, head of the NIPCC (Non Governmental Panel on Climate Change), claiming that empirical evidence strongly supports his view. Yet we have a very respectable peer-reviewed paper contradicting him. In this particular case the skeptical side is significantly undermined. This process, of examining the likelihood of truth in simple sub-debates, is in my opinion, the key to finding the truth in a complex debate.

I've been struggling a little with the visualization and editing UI for the debate/argument maps. I feel like I've managed to take a nice simple concept and then totally undermine it with a confusing UI. I think I've been looking at it too long. I greatly welcome any feedback.

P.S. Details on creating sub-debates: This works by linking two yes-no questions together via a "logical implication". For two questions, A and B, you can express A -> B. You can also use negation, to yield the combinations: A -> B, A -> ~B, ~A -> B, ~A -> ~B. Finally, you can use the modal logic qualifiers, "possibly" A -> B vs. "necessarily" A -> B. It's explained in more detail in the Implications section of the FAQ, here: http://www.takeonit.com/help.aspx

Hello everyone, I hope you don't mind me joining in on this 8 year old post. I've been working on ideas like this since 2012 and just found this. My current experiment is Reason Score where I am working on a way to measure the reasonableness of a claim based on the pro and con claims added to it. This will hopefully reduce cognitive biases by forcing people to add reasons to affect the score instead of votes. In the least it will encourage people to think through their claims.

It's not documented well so it might be best if someone has some time to debate me on a topic and see if it provides benefit. Any takers?

Some relevant links: structured debate is about how this should work, and dispute resolution technology lists some attempts to implement a solution.

Those pages are both GNU-FDL licensed (soon to be Creative Commons as soon as I can get around to it), so please feel free to snag anything useful and repost it on the LW wiki.

Thanks for writing this, this is an interesting area, and improving decision making is a worthwhile goal.

However, what I am a bit skeptical about is the extent to which people want to improve the decision making process. In LW-circles I can see that desire, but in the world at large it seems more important to win the argument than necessarily be right; there is some more work to do before they would even want to use the tools.

If we focus on LW-readers which are (hopefully) more interested in truth seeking, it would be interesting to see if there has been a discussion where one of these tools could really have made a difference. Personally, I usually don't have so much trouble following the causal steps - the difficulties lie usually in missing the background knowledge to weigh the arguments.

As a programmer I know the urge to come up with some solution for a problem, preferably in the form of some algorithm. But is there evidence that these tools actually help in realistic situations?

http://www.paulgraham.com/disagree.html

There is an option in the bCisive application, under the "spaces" tab to turn on guest access. It should supply you with an URL you can include in your post here. Without turning that option on, we would have to register, and you would have to invite each of us to view the argument map.

So: "spaces" -> "cryonics" -> "manage" -> turn on guest access

Hey everyone! It appears I'm six years late to the party, but better late than never.

I've been building a website for the last few months which is very close to the ideas presented in this article. I've summarized some features of it, and added an entry to the wiki page:

Debate Map: Web platform for collaborative mapping of beliefs, arguments, and evidence.

Pros:

• Collaborative creation, editing, and evaluation of debate/argument maps.
• Open source. (under the MIT license)
• Developed using modern web technologies. (react-js, redux, firebase)
• Built-in probability and validity rating, and calculation of argument strength from these ratings.
• Tree-based structure which can extend very deep without loss of clarity or usability.
• Integrated term/definition system. Terms can be defined once, then used anywhere, with hover-based definition display.

Cons:

• Has a learning curve for casual users, as content must conform to the argument<-premise structure at each level.
• Performance is currently less than ideal on mobile devices.

I'm the sole developer at the moment, but I'm very invested in the project, and plan to spend thousands of hours on it over the years to make it the best it can be. I'm very interested in your feedback! I've been a silent reader of this site for a couple years, and it'll be neat to finally get involved a bit.

Here's a new one which just crossed my radar: Wrangl. The link goes to a representative "argument".

There is a web-based tool being worked on at MIT's collective intelligence lab. Couldn't find the direct link to the project, but here's a video overview: Deliberatorium

There is also Scott Aaronson's worldview manager. This is designed to point out hidden contradictions (or at least tensions) between one's beliefs, by using programmed in implications to exhibit (possibly long) inferential chains that demonstrate a contradiction.

As such, it does have an OK DSL for such inferences.

EDIT: unfortunately, the site seems to be down, with broken SQL queries...

I'd love to see more tools like Bcisive! Thanks!

I have using Rationale from Austhink back when I was taking Philosophy classes. It was actually pretty good compared to what others offer but unfortunately limited to Windows. Ever since that time I have been thinking on and of about what software could be developed to both share ideas and refine our collective thinking in a better way. While there are tons of projects attempting such things most are either horribly designed or targeted at a non existing market.

I've read through the discussion above and I wonder why a semantic web approach wasn't mentioned. I think the SemWeb can play an important role in these kind of applications. For example RDF statements could be used as part of the argumentation schema. This kind of approach would also allow us to publish in a more decentralized manner, which is where the power of the web lies.

Although semantic web technology hasn't seen a broad adoption yet, I think in the next two years this will finally change. The lack of adoption comes mainly from a cold start problem but over the last year the need for this kind of tech seems to be increasing. Search engines like Google are beginning to leverage the SemWeb to improve their search results since they seem to be losing the war on spam at the moment. Tim Berners-Lee's Linked Data format is gaining ground in the public sector as the standard format for governments to publish open data. I think this is the first time we are starting to see interest in the SemWeb from outside the academic world, which to me is a sign that we are finally gonna see some widespread use.

Anyway I can see this being the way forward in online communication.

I thought a lot about creating such a system and how it would look a number of years ago, but never did make any good progress on it. The point where I got stuck was to take a particular blog post with lots of debate in the comments and try to dissect it in different ways and see what ended up being the most useful. I found I didn't have the focus to do so.

Anyway, there's Truth Mapping, which I think sucks for quite a number of reasons.

I would love to see this turn into a software development project. I would be happy to participate, though I won't have much time any time soon.

Some thoughts

• Sentences are powerful because we are hard-wired to use language, so if we can leverage this then all the better. It sounds like TakeOnIt is going down this path (though I haven't looked at that project).

• Re crunching numbers: Ensuring the argument is always crunchable will mean the software will have to place strong restrictions on the structure and relations at all points during the argument's construction. This may hinder usability severely, so I would be in favour of a less rigid system at the cost of perhaps not being able to compute anything automatically.

• User interface may prove to be the central problem. The goal here is essentially to increase the efficiency of collaborative argument, and the efficiency of the user interface may well be the bottleneck. Rather than a side issue I would pose this as the key problem to be solved.