Comment by krzhang on Understanding Simpson's Paradox · 2013-09-19T08:37:32.645Z · LW · GW

I'm glad it was helpful. =)

Comment by krzhang on Understanding Simpson's Paradox · 2013-09-19T08:36:20.920Z · LW · GW

Hi Vaniver! =D

On the commentary: your eyeballing seems good, but I don't think I ever said anything about relative comparisons between correlation coefficients (namely just overall correlation is positive). As you observed, I could easily make all 3 correlations (blue-only, green only, or blue+green) positive. I don't have any interesting things to say about their relative degrees.

I don't quite see the difference in interpretation from this writing. I agree with basically all the stuff you've written? The fact that the slicing "behaves as a filter", if I interpret it correctly, is exactly the problem here.

I don't know what "have a different origin than Simpson's paradox" means exactly, but here are a few ways they differ and why I say they are "different":

  • a fundamental assumption on Simpson's paradox is that there's some imbalance with the denominators; in your 2x2x2 matrix you can't arbitrarily scale the numbers arbitrarily; all the examples you can construct almost relies on (let's say we are using the familiar batting averages example) the fact that the denominators (row sums) are different.

  • the direct cause of the reversal effect is, as you said, the noise; I don't think Simpson's paradox has anything related to the noise.

Idea: my steel-man version of your argument is that reversal effects arise when you have inhomogenous data, and this is definitely the more general common problem in both situations. In that case I agree. (this is how I teach this class at SPARC, at least).

Comment by krzhang on Welcome to Less Wrong! (July 2012) · 2013-02-19T05:58:03.584Z · LW · GW

Haha hey QC. Remind me sometime to learn the "get ridiculously high points in karma-based communities and learn a lot" metaskill from you... you seem to be off to a good start here too ;)

Comment by krzhang on Welcome to Less Wrong! (July 2012) · 2013-02-19T05:33:23.535Z · LW · GW

I am Yan Zhang, a mathematics grad student specializing in combinatorics at MIT (and soon to work at UC Berkeley after graduation) and co-founder of I was involved with building the first year of SPARC. There, I met many cool people at CFAR, for which I'm now a curicculum consultant.

I don't know much about LW but have liked some of the things I have read here; AnnaSalamon described me as a "street rationalist" because my own rationality principles are home-grown from a mix of other communities and hobbies. In that sense, I'm happy to step foot into this "mainstream dojo" and learn your language.

Recently Anna suggested I may want to cross-post something I wrote to LW and I've always wanted to get to know the community better, so this is the first step, I suppose. I look forward to learning from all of you.