Prodding ChatGPT to solve a basic algebra problem
post by Shmi (shminux) · 2022-12-12T04:09:42.105Z · LW · GW · 6 commentsThis is a link post for https://twitter.com/shminux/status/1602140709204955142
Contents
6 comments
TL;DR: It behaves amazingly like a human would!
It understands the word problem, but struggles to put it in equations, takes multiple tries, screws up again, eventually muddles through to get the right answer. Given a similar problem it does slightly better, but still messes up. On the third problem it messes up again, then realizes that something went wrong... and promptly blames the problem setup, unable to question its own reasoning.
6 comments
Comments sorted by top scores.
comment by Unnamed · 2022-12-12T05:03:07.020Z · LW(p) · GW(p)
Got it in one thinking step by step.
↑ comment by Shmi (shminux) · 2022-12-12T06:22:48.657Z · LW(p) · GW(p)
yeah, looks like "steb by step" is a magic incantation:
https://twitter.com/ESYudkowsky/status/1602177149401989120
... which makes it even more human. I used to almost yell at the students I used to tutor: DON'T SKIP STEPS!
Replies from: Unnamed↑ comment by Unnamed · 2022-12-12T18:07:44.489Z · LW(p) · GW(p)
Incantation rankings, for GPT-3 on math word problems
↑ comment by spkoc · 2022-12-12T13:28:56.148Z · LW(p) · GW(p)
Didn't seem to work for me. It still seems to get confused trying to match similar words together even when they shouldn't be. Again quite, dumb/young human.
↑ comment by Unnamed · 2022-12-12T18:05:42.987Z · LW(p) · GW(p)
It tries to continue its dialogue in ways that fit with its previous replies, so prompts like "think step by step" might fail to get it back on track mid-conversation even when they would've worked in the initial prompt.
Also, there is some randomness.
comment by the gears to ascension (lahwran) · 2022-12-12T04:33:30.473Z · LW(p) · GW(p)
[humor] teenagers, amiright?