I agree. Coming up with the right prompts was not trivial. I almost quit several times. Yet, there is a science to this and I think it’ll become more important to turn out focus away from the spectacle aspects of GPT and more towards reproducibility. More so if the way forward is via interrelated instances of GPT.

As an aside, critique seems much easier than generation. I’m cautiously optimistic about prompting GPT instances to “check” output.

Interesting to think about how this will evolve. Over time, humans will have to do less of the work, and the combined system will be able to do more. (Though the selection pressure that you mention will continue to be there.)

It seems to me that we might not be too far away from "natural language programming". With some combination of the the above approach, plus the program synthesis examples where you just specify a comment, plus some extra tricks, it seems like you could end up just sort of specifying your programs via an algorithm description in English.

You'd want to set it up so that it alerted you when it thought things were ambiguous, and that it auto-generated test cases for different possible interpretations and showed you the results.

I've personally started using TabNine in the last few weeks, and I'd say it's just barely over the edge of being useful. But I can imagine next-gen versions of these things pretty radically transforming the process of programming.

You could prompt with “Q:” + (content) and then “A:”

I use the default settings on the temperature, but I do cut it off after it finishes an answer. However, you likely won’t get my exact results unless you literally copy the instances. Moreover, if you gave up after the first response I think might’ve given up to quickly. You can respond to it and communicate more information, as I did. The above really was what I got on the first try. It’s not perfect, but that’s the point. You can teach it. It’s not “it works” or “it doesn’t work”.

I don’t think there are tutorials, but perhaps in due time someone (maybe me) will get to that. I also feel like ‘trying’ to get it to do something might be a sub-optimal approach. This is a subtle difference, but my intent here was to get it to confirm it understood what I was asking by answering questions.

The approach I've been using (for different things, but I suspect the principle is the same) is

• If you want it to do X, give it about four examples of X in the question-answer format as a prompt (as in, commands from the human plus answers from the AI)
• Repeat for about three times:
  • Give it another such question, reroll until it produces a good answer (might take a lot of rolls)

At that point it is much better than one where you prompted everything to begin with.

I also have replicating difficulties with AI Dungeon. I think it has weaker version of GPT-3 than API.

Thanks! I forgot to do this. Luckily I can go back through the run and put this is in. There is ambiguity whenever it auto-completes, but I hope I did a decent job of noting where this is happening.