Are there any reliable CAPTCHAs? Competition for CAPTCHA ideas that AIs can’t solve.

One type of question that would be straightforward for humans to answer, but difficult to train a machine learning model to answer reliably, would be to ask "How much money is visible in this picture?" for images like this:

If you have pictures with bills, coins, and non-money objects in random configurations - with many items overlapping and partly occluding each other - it is still fairly easy for humans to pick out what is what from the image.

But to get an AI to do this would be more difficult than a normal image classification problem where you can just fine tune a vision model with a bunch of task-relevant training cases. It would probably require multiple denomination-specific visions models working together, as well as some robust way for the model to determine where one object ends and another begins.

I would also expect such an AI to be more confounded by any adversarial factors - such as the inclusion of non-money arcade tokens or drawings of coins or colored-in circles - added to the image.

Now, maybe to solve this in under one minute some people would need to start the timer when they already have a calculator in hand (or the captcha screen would need to include an on-screen calculator). But in general, as long as there is not a huge number of coins and bills, I don't think this type of captcha would take the average person more than say 3-4 times longer than it takes them to compete the "select all squares with traffic lights" type captchas in use now. (Though some may want to familiarize themselves with the various $1.00 and $0.50 coins that exist and some the variations of the tails sides of quarters if this becomes the new prove-you-are-a-human method.)

You should team up with MIRI and create a captcha based on human values. Either the machines learn to understand human values, in which case we finally get an aligned artificial intelligence; or they won't, and then at least we get a good captcha...

(Narrator: Unfortunately, 90% of internet users also failed to understand human values.)

• At least 90% of internet users could solve it within one minute.

While I understand the reasoning behind this bar, having a bar greater than something like 99.99% of internet users is strongly discriminatory and regressive. Captchas are used to gate parts of the internet that are required for daily life. For instance, almost all free email services require filling out captchas, and many government agencies now require you to have an email address to interact with them. A bar that cuts out a meaningful number of humans means that those humans become unable to interact with society. Moreover, the people who are most likely to fail at solving this are people who already struggle to use computers and the internet, so the uneducated, the poor, and the elderly. Those groups can ill afford yet another barrier to living in our modern society.

I'm not sure this is solvable, but even if it is, I'm not sure its a good problem to work on.

Why, fundamentally, do we care if the user is a bot or a human? Is it just because bots don't buy things they see advertised, so we don't want to waste server cycles and bandwidth on them?

Whatever the reasons for wanting to distinguish bots from humans, perhaps there are better means than CAPTCHA, focused on the reasons rather than bots vs. humans.

For example, if you don't want to serve a web page to bots because you don't make any money from them, a micropayments system could allow a human to pay you $0.001/page or so - enough to cover the marginal cost of serving the page. If a bot is willing to pay that much - let them.

Are there any browser plugins that actually bypass most CAPTCHAs? Honestly, I see them less often recently, but they bug me, and I can't imagine they do much good.  Actually, that's perhaps not true - they may serve the same function as most door locks - not impossible or even all that hard to pick, but a mild deterrent to trying.

Anyway, I predict failure for this quest.  There is no typing or clicking sequence that the second-lowest decile of internet users can easily and reliably do that a human can't train a model to do at least as well.  

The key in this is the assistance of a very smart human to design/train the 'bot.  It'll be a special-purpose ML task, and the hardest part will be connecting the model to the input/output needed to work the CAPTCHA.  

Building a general-purpose browser scraper/assistant that can handle NEW captchas is probably feasible, even, but it's orders of magnitude harder.

Isn't that exactly what the ARC challenge set out to do?

My personal idea would be some kind of 3D shape rotation task. For example, you're presented with an image of a dodecahedron with each face colored differently, and then there are 4 more colored dodecahedrons in the answer section, only one of which is a rotation of the original which you need to identify to pass the test.

The 1 minute time limit is pretty damning though, wordcels might have trouble (sorry, I couldn't resist^^).

The core problem of your request is that with this time limit you are basically forcing the human to shut off System 2 thinking while solving an AI-proof CAPTCHA, when System 2 is all the advantage we have over AI. Kind of like a Turing test where you only get 1 minute and aren't allowed to think too hard.

From ChatGPT:

One idea for a CAPTCHA that meets the criteria you have outlined is to use a combination of text and image recognition. For example, the CAPTCHA could present the user with a series of images and ask them to select all the images that contain a certain object or word. This could be difficult for an AI model to solve, as it would require both image and language processing capabilities.

Another idea is to use a CAPTCHA that involves solving a simple puzzle or task that requires some degree of human cognition and problem-solving. For example, the CAPTCHA could present the user with a set of shapes and ask them to rearrange them in a specific way, or it could ask the user to perform a simple arithmetic calculation. This type of CAPTCHA would be difficult for an AI model to solve as it would require the ability to understand and manipulate abstract concepts.

Another possibility is to use a CAPTCHA that involves asking the user to identify and select certain elements within a larger image, such as a specific object or text within a scene. This type of CAPTCHA would require the ability to understand and interpret the context of the image, which is a challenge for many AI models.

Overall, the key to creating a CAPTCHA that is both easy for humans to solve and difficult for AI models to crack is to design a task that requires some degree of human cognition and problem-solving, rather than relying solely on image or text recognition.

The first jdea is just literally what a CAPTCHA is, but the second sounds interesting, having the user play a game or solve some puzzle (minus the arithmetic), and third is also dumb.

How about somehow utilizing pareidolia? Something like asking how this socket is feeling?

Or who's in this picture?

If we don't have AGI at the level of diamondoid nanotech bacteria, it may be possible to reliably identify humans using some kind of physical smart card system requiring frequent or continuous re-authentication via biometric sensors, similar to breathalyzers / ignition interlock devices installed in the cars of DUI offenders.

Not the most practical or non-invasive method that could be deployed for online services, but it is fairly secure if you're in a lab trying to keep an AGI in a box.

As for online solutions not requiring new hardware, recently I had to take a "video selfie" on my phone matching a scan of my driver's license for id.me as part of my unemployment benefits application. I'm fairly certain this could be fooled, but that's how our government is handling it now.

How does AI do at classifying video these days?

I'm picturing something along the lines of "Pick the odd one out, from these three 10-second video clips", where the clips are two different examples from some broad genre (birthday party, tennis match, wildlife, city street, etc etc) and one from another.

I might be behind the times though, or underestimating the success rate you'd get by classifying based on, say, still images taken from one random frame of the video.

But maybe if you added static noise to make the videos heavily obscured, and rely on a human ability to infer missing details and fill in noisy visual inputs.

How about identifying AI-generated vs real pictures of hands? 

I don't expect this to be possible in like 1.5 years, and expect it's difficult now. Stuff like "only allow users who have a long browser history & have bought things and such" (like reCapcha does) feels like the only way out besides "go to the office in person"