Hutter-Prize for Prompts

There are a few things that resemble what I think you might be asking for.

The first thing it sounds like you may be asking is as follows:

There exists a file enwik8, which contains 10^8 bytes of English Wikipedia. The language models everyone has been playing with have almost certainly been fed all or most of that content. Does there exist some prompt which will cause those models to emit either the entirety of enwik8 with high probability?

The answer is a flat "no" for the entirety of enwik8 for any of the main openai models, which can be verified by passing in a random full-context-worth (2048 tokens) of enwik8 and verifying that the completion is not "the rest of enwik8". The models cannot look back before their context, so if any full context in the middle fails to generate the next part, we can conclude that it won't generate the full thing on its own for any initial prompt. Which is not too surprising.

The second thing, and the thing I expect you mean, is

Can we compute how well these language models encode enwik8

to which the answer is "yes". It is possible to get a measure of bits per character for a given model to generate a given sequence of tokens, which equally means it is possible to generate that exact text using the model and the corresponding number of bits. Some examples:

• A model that said all bytes are equally likely to occur would cost 8 bits per byte, or ~8 bits per char (since it's mostly ascii).
• Using the character frequency observed in enwik8, and nothing else, would cost 5.17 bits per char, with a model that weighs a few kB (markov_chars_0 below).
• Tokenizing using the same tokenizer GPT-2 uses, and going by token frequency, gets you down to 3.18, at the cost of tens too hundreds of kB of model (markov_toks_0 below).
• Building a simple markov model that looks at the last 3 chars to get your next char gets you down to 2.48 bits per char. But storing a 4 char sequence + a 3 byte int for each 4 char sequence that occurs now weighs almost 10 MB (markov_chars_3 below).
• Building a markov model of tokens (again using the gpt2 tokenizer) can drop you down to 0.66 bits per char, below Shannon's lower estimate. Except that now your model is recording counts for 16 million distinct token sequences, which will weigh in at something on the order of 180 MB (markov_toks_3 below).
• Likewise, using OpenAI's text-davinci-003 can get you down to 0.56 bits per token. But the model itself weighs several GB (openai:text-davinci-003 below).

Anyway, I threw together a quick script to get approximate bits-per-char to reproduce enwik8, excluding model size. Here are the results, high-to-low (std deviation across 10 trials in parens). Keep in mind that "store the entirety of enwik8 uncompressed" costs 100MB

markov_chars_0:         : 5.17 (0.43)
markov_chars_1:         : 4.02 (0.29)
markov_chars_2:         : 3.18 (0.18)
markov_toks_0:          : 3.18 (0.56)
markov_chars_3:         : 2.48 (0.17)
markov_toks_1:          : 1.79 (0.24)
openai:text-ada-001:    : 1.58 (0.09)
openai:text-babbage-001:  1.06 (0.06)
markov_toks_2:          : 1.04 (0.11)
openai:ada:             : 0.93 (0.09)
openai:text-curie-001:  : 0.91 (0.07)
openai:babbage:         : 0.81 (0.06)
openai:curie:           : 0.72 (0.05)
openai:davinci:         : 0.64 (0.03)
markov_toks_3:          : 0.66 (0.11)
openai:text-davinci-002:  0.57 (0.05)
openai:text-davinci-003:  0.56 (0.05)

If this was what you had in mind, you would probably also be interested in the post Chinchilla's wild implications [LW · GW], which goes a lot deeper into this sort of stuff.

The well-known early experiment in creating a virtual Linux machine inside ChatGPT, linked to a virtual Internet, includes generation of a fictitious web page by viewing it through the lynx browser. 

In my first attempt, using lynx as the second command didn't produce a page. But on a second attempt, I tried pinging a domain, in order to ease it into displaying Internet data, and after that it was willing to display fictitious versions of Wikipedia pages. They don't particularly match the original (not even the September 2021 version, from the time of ChatGPT's knowledge cutoff), but they are on topic. 

Possibly this approach can be refined. 

The machine learning world is doing a lot of damage to society by confusing "is" with "ought" which, within AIXI, is equivalent to confusing its two unified components:  Algorithmic Information Theory (compression) with Sequential Decision Theory (conditional decompression).  This is a primary reason the machine learning world has failed to provide anything remotely approaching the level of funding for The Hutter Prize that would be required to attract talent away from grabbing all of the low hanging fruit in the matrix multiply hardware lottery branches, while failing to water the roots of the AGI tree.  So the failure is in the machine learning world -- not the Hutter Prize criteria.  There is simply no greater potential risk adjusted return on investment to the machine learning world than is increasing the size of the prize purse for the Hutter Prize.  To the extent that clearing up confusion about AGI in politics would benefit society, there is a good argument to be made that the same can be said for the world in general.

This is because 1)  The judging criteria are completely objective (and probably should be automated) and 2) The judging criteria is closely tied to the ideal "loss function" for epistemology:  the science of human knowledge.

The proper funding level would be at least 1% of the technology development investments in machine learning.

This reminds me of an idea : I think it would be great to hold a bi-monthly competition where people try to do something as incredible as possible in just 30 minutes using LLMs or other AIs. The winner being decided by a select few.