Counter-irrationality

As to your first paragraph, one of the more liberating things about internalizing the Sequences and inculcating myself with Less Wrong memes is that it I've come to hold very few opinions strongly. When you move yourself to a sufficiently morally-relativistic framework you stop identifying with your opinions. Instead of saying there are things you "just have to believe" I would say there are things that it is instrumentally rational to behave as if you believe.

Regarding your second paragraph: I find that I simply get into fewer arguments. Because I have let go of most of the opinions that typically weight people and make them respond at an emotional level to being contradicted, every conversation that includes a disagreement becomes a joint truth-seeking venture. Instead of arguments, I have "discussions." If the other party in the discussion is not interested in truth seeking but is instead interested in being right, I just stop humoring them and change the subject. If they are someone I can be honest with, I will point out that they seem to have an irrational bias regarding the topic in question.

It seems like your first mistake was getting involved in a theological debate. Science is flawed, but "religion" doesn't even have enough predictive power to be falsified. I would step back at least one level and urge you to ask yourself what your objectives are in participating in such a confused discussion in the first place. I myself have indulged in totally stupid internet arguments which I can only attribute to a sort of perverse pique that strikes me at times, but I generally admit that I'm already failing to be rational just by participating in such.

Ever since I started frequenting this forum, I am much more aware when I am trying to win an argument, rather than to come up with the best possible model/solution.

It does not mean that I immediately recite the appropriate litany and switch to the "rational" mode, because sometimes winning with a flawed argument is the rational thing to do. If you doubt that, you have never bought a used car.

However, when finding out the truth (or a reasonable facsimile thereof) is the rational thing to do, I make an effort to distance myself from the "ownership" of my original argument. This is one of the hardest things to do for me, and probably for most people. I probably fail much too often. I can certainly see others fail much too often, and I should not expect to be that much better.

Now, on the surface, your reply should be along the lines of You're Entitled to Arguments, But Not (That Particular) Proof. People generally don't abandon a flawed line of reasoning, they resolve their cognitive dissonance in other ways.

The Bayesian judo example worked mostly because EY is so smart and has thought through the issue countless times before, so the believer on the other side was unprepared and completely outmatched. Plus he made a bad move and was not prepared to cut his losses by abandoning it right away ("e.g. I do not care what you call an AI, if it is not created by God, it's not an intelligence to me", nice and circular).

So, what should you do in a debate like that, provided that you are irrational enough to get sucked into one? First, you should probably recognize by now that the particular argument you mentioned is simply a feint, and so are all other arguments. Your opponent has no intention of changing his or her mind (and neither do you, I suspect), so both of you start with your respective bottom lines already written. How do you know when you are not in an honest argument? The moment you catch yourself thinking "My argument failed, I need to come up with another one", it is a sure sign that you are being irrational.

So, if you honestly want to come to an agreement with someone who does not subscribe to the LW idea of rationality, what do you do?

One much popularized approach (and a negotiation tactics) is to "think like your enemy". If you are unable to accurately model your opponent, you will not be able to affect their thinking. EY once attributed his improbable success in getting a simulated UFAI out of the box to thinking like a UFAI would. How do you know that you do have an adequate model? You will be able to come up with their arguments as fast as they do, and these arguments will make perfect sense within that model. This is very hard, because people generally suck at running virtual machines of other people's minds without letting the VM bleed through to the host and vice versa, corrupting one or both of them.

Assuming you are finally at a point where you can be on the other side of the debate and do as well as your opponent, what's next? Now that you are intimately familiar with your surroundings, you have a better chance of noticing the imperfections and pointing them out, since you are not emotionally invested in keeping this particular VM running. Pick at the imperfections, see if you can fix them from inside of the VM, because your opponent sure would try that first. Once you are sure that there is no adequate fix for a flaw, you can point it out (or, better yet, guide them to discovering it on their own) and let them stew on it... unless your opponent has thought it through and has a counter that you missed. You don't push your point, because it makes people defensive. Minds are not changed overnight.

Pushing this one step further, you should practice this by constructing a VM of yourself first, so you can go through the same process dispassionaly and make sure that your own boat is airtight. For example, the simulation argument has been known to convert an occasional atheist into an agnostic.

Well, this should be enough of preaching for now. Hope some of it makes sense.

I agree with moridinamael, but you also seem to be making a second mistake in your first paragraph:

we're automatically biased for or against something, at times for the wrong things and reasons.

You appear to be conflating social/political biases (ie, for or against something) with cognitive biases which directly influence how we think about things. Cognitive biases, to the extent that they can be described as "for or against", would be "for or against" particular ways of thinking. There is some overlap, which is why I only said "seem to be making a second mistake", from the rest of your post it looks like you are, but a charitable reading says maybe not.

Your third paragraph describes what looks like a version of the "argument from ignorance" fallacy combined with a "default to God" (a version of the God of the Gaps). I don't think there really is a general counter-argument to this. George Smith, in one of his essays in Atheism, Ayn Rand, and other Heresies suggested avoiding getting into arguments like this, unless you are using them to try to convince third party observers.