A Gamification Of Education: a modest proposal based on the Universal Decimal Classification and RPG skill trees

Testing is optimized for predicting current and future competence in a subject:

What if this finds that the best predictors for most employers consists of two tests (1) a <10 minute IQ test, and (2) a test showing a student's willingness to submit to authority and learn what he was told, in the manner under which he was told without getting too far ahead or behind his classmates?

Who knows what amazing research teams might be borne of the intellectual equivalent of OK!Cupid.

This sounds like an awesome idea. Does anyone know if such a thing already exists? If not, I would be willing to commit $500 to someone who wants to make it (assuming they already have the appropriate skill set, of course).

I'm reminded of a couple of students at a German university last year who studied all the material by dividing up the classes between them and exchanging notes, took all the exams, and passed in a few months. The university then turned around and sued them for studying too fast.

http://www.thelocal.de/education/20120703-43517.html

It is not in a University's interests to do a good job. It's like any other company: The aim is to extract the most $$$ from you while they give up the least value in return.

In highly competitive markets this leads to fairly marginal profits. But formal education is not a competitive market. Not at all. Very tightly regulated.

The competition for education comes almost entirely around costs for most people. They view education as an expense, not as an investment - which is quite reasonable when you consider the likely quality they're going to get - and aim to minimise that expense. Offer people a better education and the majority of them won't be willing to pay much for it, offer them a cheaper education though, or a faster one - which amounts to more or less the same thing....

That's the financial side of things anyway, and one of the reasons I think your idea's just never going to happen.

#

Assorted

One would determine what to learn based on statistical studies of what elements are, by and large, most desired by employers of/predictors of professional success in a certain field you want to work in.

1. I don't know, but I wouldn't think this data's going to be available or reliable. Google did their own research on this, which suggests to me that the data wasn't available when they looked for it elsewhere. One thing's just that everyone's going to record things slightly differently - there's not an industry standard for measuring this sort of stuff that I'm aware of. The other is that most places are not particularly rational about their HR procedures - there are a set of skills that go into even the basic recording of data that I'd not expect them to have. I'd expect it to be more like they sit there with a sheet of paper that they're marking 1-10 based on their subjective opinion of your answer to a question. That's probably the extent of any data most people will be expected to generate and, perhaps, keep.

In the case of desired by employers I suspect you might find the data was actually contradictory to predictors of success a lot of the time too. I know, when I go for interviews, I tone down a lot of the qualities that let me do things easily - the corollary of don't dress better than the boss is don't act smarter/more skilled than the boss - most people like to hire people they can use as tools for their own success, not colleagues who they have to work with. Especially considering lots of economic behaviour seems to be self-justifying/rent-seeking.

================

That said, I would imagine that what you really want to do is to follow up with people who've left university, both those who won and those who lost, and ask them what they wish they'd known then that they know now. Certainly, a few years out of the gate, I could give my university a great deal of feedback.

================

2 . The incentives in HR seem to be to be risk averse. To find reasons not to take people on. If you hire someone and they do really well then you don't get rewarded, whereas if you hire someone and they're awful then your job is at risk. If that's heavily weighted in the decision maker's consideration, then giving people more granular data will result in dramatically fewer hires, since the probability of a concurrent set of criteria being fulfilled is the multiple of its individual probabilities. Even if the individual probabilities are quite high, HR managers will be able to talk their way into using a great number of individual criteria and then you'll be hung.

3 . It makes the market more competitive, not necessarily in a good way. Students take on a great deal of the financial risk in education these days. Which honestly seems the wrong way around, but there you go. If rich people can afford to grind their way higher up the tech tree, then you're essentially pricing poor people out of the market. The rich person only has to be ahead by one or two points - you're probably going to pick someone who's even slightly better on paper over someone worse. So there's an incentive to have large steps in your pricing.

The main issue in implementing such a system is testing and certification. If you use quizzes then there is nothing to stop the student from looking up the answers in another window. The system can still be made but the tests need to be designed so that students can't just look up the answers. The solution is applied knowledge problems.

For example instead of asking someone about the physical properties of a metal you would ask them what metal they would use for a certain construction and why. Student programmers would be required to write short scripts. Student geographers would interpret maps. Student lawyers could say how a client should behave is different situations.

Basically you would test whether people can use their knowledge instead of whether they can communicate it to avoid students second-hand communication. The downside is that questions are harder to make.

TL;DR- the difficulty of solving this problem is that availability of good data, NOT the lack of decent skill-set models.

So lets say you finish an outline of skills in these trees, as you suggest. Now, we want to make the statistical models that are at the core of your proposal- "One would determine what to learn based on statistical studies of what elements are, by and large, most desired by employers of/predictors of professional success in a certain field you want to work in." Where exactly do you plan to get the data to actually do this? As you "level up" down your skill tree, the number of people who have skill combinations is going to get very, very thin. I might be the only person in the world who has both calculated a Next-to-Next-to-Leading-Order cross-section at Tevatron energy levels AND who has used restricted-Boltzmann-machines to categorize medical billing codes to triage health insurance claims. Backing up a bit, and "quantum field theory" + "health care informatics" + "statistics" might have what, a dozen people, all of whom are going to be outlier-ish for the general field of insurance data analysis.

Lets look at data from physics (because I know it well) here is the AIP's focus on physics bachelor holders. Bachelors outnumber phds by quite a bit, so this is the densest you are going to get for certain types of physics skills. http://www.aip.org/statistics/trends/reports/empinibs0910.pdf. If you subtract out the exclusions (unemployed, people who went back to their pre-college job, and part-time employed), the data is already cut down to 2/3. Of those, the two largest chunks are highschool teachers (about 6% of the total physics bachelors) and engineers (about 7% of the total physics bachelors).

So we have about 350 people who teach highschool, probably highschool science, so you can maybe compare physics-educated general science teachers to other educated general science teachers. A bit more then that are in "engineering" fields, but there are tons of subfields/'skill-trees' in engineering. You'll have <100 per skill-tree at the trees base! The data is already very thin, and its much thinner for all other career paths.

Career organizations that have much larger memberships (i.e. most career organizations) have much less detailed information because collecting it becomes more of a chore. These larger datasets probably won't have the information you'd really need to fill out your skill-trees.

Also, I'm willing to bet that the world moves fast enough in a number of fields that you can't use very deep historical data to do predictions. i.e. two decades ago a physics phds was a decent way to get a quant job on wallstreet, BUT two decades ago a masters of financial engineering degree didn't exist. Maybe in another decade the MFEs squeeze out the phds the way that CS squeezed out physicist programmers in the 70s and 80s.

I'm starting a job in an "adaptive learning" startup soon, and many of the points you make here remind me of the things this company does or plans to do. The basic idea of the company is that it collects data about the student as he or she interacts with an electronic course, then uses this to personalize the course and make recommendations for the student's education path. This isn't quite the same as what you're suggesting, where a student independently finds educational content and then gets certified in those areas. However, there are several similarities.

The tree of learning definitely exists. In your idea, the students find their own paths through this tree, but the adaptive learning way is to recommend paths to each student based on their performance in previous courses.

Both ideas allow each student to have some type of profile that showcases exactly what knowledge they have obtained. Adaptive learning could also add to this profile things that are specific to how the student learns (e.g. is it better to introduce this person this math concept by offering a rigorous proof or by explaining several revealing examples). Something like that could be helpful for the "intellectual OK!Cupid."

You envision a scheme wherein the student acquires access to real or virtual libraries of educational content, with varying degrees of completely independent learning and institutional guided learning. Right now, the company where I will work partners with education content companies (which seem to be mostly textbook publishers), and is only able to offer adaptive learning to students learning through the specific classes created around these textbooks. However, I believe there are plans to make the platform more open, so any course could be built with it.

The main disadvantage of adaptive learning, which your system does better with, is the problem where the teaching and grading is mostly done by the same entity. This seems to be a necessary aspect of adaptive learning, because the data collected about the student as they are taking a test, quiz, or exam is used to learn more about how the student thinks and learns. Still, there is the possibility that an adaptive course can use a test or quiz as a personalized teaching tool, and not only as a way to evaluate performance.

How about starting in a supportive domain that you happen to be interested in where existing taxonomies and skill progressions seem to exist?

• http://intelligence.org/courses/
• http://rationality.org/recommended-reading-on-rationality/
• LW "curriculum"

Tech/skill trees help with the visualization process, and that goes a long way to motivation. Much of my struggle battling akrasia and learning math & beginner rationality is to try to stop imagining forbidding mountains of concepts, nomenclature and future practice, and instead focus on images of cartoonish/animated tech trees that are more appealing, tractable and familiar (for those of us with gaming backgrounds); ultimately, anything less intimidating than an impassible mountain to climb.

Added bonus: You will probably not find a more willing group of participants to enthusiastically input, rate and measure themselves on their cognitive and rationalist skills.

Added bonus 2: I speculate that CFAR may be able to integrate this type of system as additional value-add for post-workshop alumni.

edit: "Data Scientist" seems to be a buzz word used to trigger competence and desirability in job candidates. Here's an example of a visual roadmap: http://nirvacana.com/thoughts/becoming-a-data-scientist/. I think other people are willing to create skill trees, but the missing ingredient is the ability to animate and infuse them with meaning over time (i.e. referring back to your ideas of tests for each node/module with clear achievement and progress indicators).

Going to college and getting a degree by the standard method serves as signalling; it shows that you are the person who is willing to spend a lot of resources (both time and money) on your skills.

Any method of education which doesn't take as many resources for the student will inherently subvert this, and therefore won't be accepted by employers.

Testing and certification is an extremely difficult thing to get right, and indeed I'm not sure anyone really has done this. It takes a lot of effort to write good tests. It is almost impossible to simultaneously reduce both false positives and false negatives. If the test is standardized or repeated, teachers teach to and students train for the test in preference to the subject matter. Ideally you want the test to be such that someone who prepares for the test is also preparing for the skills you wish to measure.

Are there any existing tests (e.g. bar exam, SAT, CCNA, etc.) that you would point to as suitable for this system? If not, what changes would need to be made to make them suitable?

I will admit that the current system of testing and certification is bad, and I remind myself that I need to watch out for status quo bias here.

"Doctor in Physics from Former Soviet Union, current Taxi Driver in New York" is a problem only because "Doctor in Physics from Current United States, current Taxi Driver in New York" is also a problem. No one doubts the quality of physicists from the former Soviet Union as a group. It's just that there aren't enough jobs in physics for all the PhDs we graduate.

the AAA certification of sub-prime junk debt in 2008?

nitpick: it wasn't the debt that was AAA rated, it was certain derivatives based on the debt (which is much less unreasonable. That the AAA rating was undeserved does not follow simply from the fact that the underlying morgages were sub-AAA.)

Employers looking for candidates beyond the entry level tend to be interested in a candidate's experience and work history far more than in any result of a test. Actuaries may be one exception. Off the top of my head it is the only profession I can think of where promotion within the field is gated by tests, and not just admission to the field. However I am not an actuary, so I may not fully understand how this works.

FYI: I regularly recruit, interview, and hire candidates within my field. I would love to have a test I could give to candidates to tell me who to hire, but I know of no such test. At best the written and computer administered tests that exist serve as a bozo filter to eliminate the worst candidates. They do not eliminate all bozos, and do not distinguish the mildly competent from the ridiculously effective, not even at the entry level.

Absolutely love this idea, just one little comment on the watchdogs.

It occurs to me now that, if one wished to be really nitpicky about who watches the watchmen, I suspect that there would be institutions testing the reliability of those meta-institutions, and so on and so forth... When does it stop? How to avoid vested interests and little cheats and manipulations pulling an academic equivalent of the AAA certification of sub-prime junk debt in 2008?

Why do the watchdogs exist? Because business produce demand for accurate measurements of employability, and are willing to pay for a service that offers that information. I'm sure if a significant number of watchdog companies exist a business will need to employ a meta-watchdog to determine which watchdog to employ, but it does eventually terminate at the employer themselves, who has to personally be the watchdog for which meta-watchdog company to employ.

And that information is easily available (albeit after some lag). Which company reliably recommends the tests that actually produce better employees?

How big is the chain of information when you're deciding which restaurant to eat at? That's perfectly analogous scenario, as far as I can tell.