Posts
Comments
With the collapse of FTX Future Fund, I'm guessing this prize/contest is no longer funded.
Errors from transposing or misreading numbers, placing the decimal in the wrong location, etc. The machine mind has a "perfect cache" to hold numbers, concepts, and steps involved. Math is just the simple example of their ability. Such machine minds will be able to hold every state and federal law in their mind, and could re-draft legislation that is "clean" while porting case law from the old to the new legal references.
*for an example of current tech getting simple math wrong: https://twitter.com/KordingLab/status/1588625510804119553?s=20&t=lIJcvTaFTLK8ZlgEfT-NfA
*You can't align something that doesn't really work (it not really working is the current danger). A better question is, can you take a working AI and brainwash it? The unhackable (machine) mind? A good system, in the wrong hands, and all that.
**Again, free yourself from the monolithic view of current architectures.
I don't see how mechanistic and modular are different, and it's only clear (upon reading your last point) that your analogy might be flipped. If a car breaks-down, you need to troubleshoot, which means checking that different parts are working correctly. Interpretability (which I think fails as a term because it's very specific to math/xNN's, and not high-level behaviors) happens at that level. Specific to the argument, though, is that these are not two different concepts. Going back to the car analogy, a spark plug and fuel injector are roughly the same size, and we can assume represent an equivalent number of neurons to function. ML fails by being focused on the per-neuron activity; and thus, AGI will be based on the larger structures. We would also ask what is wrong, in terms or elephant psychology, in trying to troubleshoot an elephants mind (how something "lights up" in a region of the brain from a scan is only a hint as to the mechanisms we need to focus on, though maybe the problem is that another area didn't light up, etc).
"Solving" AGI is about knowing enough about the modules of the brain, and then building from scratch. ML research is about sub-module activity, while safety seems to be above module activity. None of these things are compatible with each other.
First, we can all ignore LeCun, because despite his born-again claims, the guy wants to solve all the DL problems he has been pushing (and winning awards for) with more DL (absolutely not neuro-symbolic, despite borrowing related terms).
Second, I made the case that amazing progress in ML is only good for more ML, and that AGI will come from a different direction. Wanting to know that seems to have been the claim of this contest, but the distribution of posts indicates strong confirmation bias toward more, faster, sooner, danger!
Third, I think most people understand your position, but you have to understand the audience. Even if there is no AGI by 2029, on a long enough time scale, we don't just reach AGI, there is a likely outcome that intelligent machines exist for 10's of thousands of years longer than the human race (and they will be the ones to make first contact with another race of intelligent machines from halfway across the galaxy; and yes, it's interesting to consider future AI's contemplating the Fermi Paradox long after we have died-off).
Advances in ML over the next few years as being no different than advances (over the next few years) of any other technology VS the hard leap into something that is right out of science fiction. There is a gap, and a very large one at that. What I have posted for this "prize" (and personally as a regular course of action in calling out the ability gap) is about looking for milestones of development of that sci-fi stuff, while giving less weight to flashy demo's that don't reflect core methods (only incremental advancement of existing methods).
*under current group think, risk from ML is going to happen faster than can be planned for, while AGI risk sneaks-up on you because you were looking in the wrong direction. At least, mitigation policies for AGI risk will target ML methods, and won't even apply to AGI fundamentals.
True enough that n AGI won't have the same "emotional loop" as humans, and that could be grounds for risk, of some kind. Not clear if such "feelings" at that level are actually needed, and no one seems to have concerns about loss of such an ability from mind uploading (so perhaps it's just a bias against machines?).
Also true that current levels of compute are enough for an AGI, and you at least hint at a change in architecture.
However, for the rest of the post, your descriptions are strictly talking about machine learning. It's my continued contention that we don't reach AGI under current paradigms, making such arguments about AGI risk moot.
Computer vision is just scanning for high probability matches between an area of the image and a set of tokenized segments that have an assigned label. No conceptual understanding of objects or actions in an image. No internal representation, and no expectations for what should "be there" a moment later. And no form of attention to drive focus (area of interest).
Canned performances and human control just off camera give the false impression of animal behaviors in what we see today, but there has been little progress since the mid-1980's into behavior-driven research. *learning to play a video game with only 20 hours of real-time play would be a better measure than trying to understand (and match) animal minds (though good research in the direction of human-level will absolutely include that).
This is the boiled-down version of your argument? You clearly state that GOFAI failed and that classic symbolic AI won't help. You also seem to be in favor of DL methods, but in a confusing way, say they don't go far enough (and then refer to others that have said as much). But I don't understand the leap to AGI, where you only seem to say we can't get there, not really connecting the two ideas. Neurosymbolic AI is not enough?
I think you're saying DL captures how the brain works without being a model for cognitive functions, but you don't expand on the link between human cognition and AGI. And while I've thrown some shade on the group dynamics here (moderators can surface the posts they like while letting others sink, aka discovery issues), you have to understand that this 'AGI is near' mentality is based on accelerating progress in ML/DL (and thus, the safety issues are based on the many problems with ML/DL systems). At best, we can shift attention onto more realistic milestones (thou they will call that goalpost moving). If your conclusion is that this AGI stuff is unlikely to ever pan-out, then yes, all of your arguments will be rejected.
Saying "a substantial amount of expertise is required" goes past what general actually implies (I use the example of 8th grade level knowledge, at which point, the system can branch in thousands of directions on it's own).
Saying that a good test of an AGI is for it to build a better AGI, while also saying that such an effort represents an immense risk?
Python syntax is closer to natural language, which plays into what the LLMs do best. I don't think the "symbolic" aspect plays into this in any way, and that kind of misses the argument on symbolic reasoning (that the LLM's are still just just doing correlation, and have no "grounding" of what the symbols mean, nor does any processing happen at that grounding level).
I'm still confused by your position. Saying DL is capturing symbolic values (in this one case), but also that DL is going to fail (because...?).
Anyone talking about reward functions is not talking about AGI. This is the disconnect between brute force results and an actual cognitive architecture. DL + Time is the reward function for the posts doing well, BTW, because it fits the existing model.
Heavy down votes in less than a day, most likely from the title alone. I can't bring myself to vote up or down because I'm not clear on your argument. Most of what you are saying supports AGI as sooner rather than later (that DL just needs that little bit extra, which should have won this crowd over). I don't see any arguments to support the main premise.
*I stated that the AGI clock doesn't even start as long as ML/DL remains the de facto method, but that isn't what they want to hear either.
Interesting, and not far from my take, which is that ML has been wearing AI as a skin (because built-in marketing). Now that it is "advancing," it has to wear AGI as a skin to indicate progress. That AGI was originally an effort to step away from DL's path, and return to something closer to original intent of AI as a field gets lost.
I guess I'm one of those #2's from the fringe, and contributed my 2 cents on Metacalus (the issue of looking for the right kind of milestones is of course related to my post in relation to current challenge). However, I completely reject ML/DL as a path toward AGI, and don't look at anything that has happened in the past few years as being AI research (and have said that AI officially died in 2012). People in the field are not trying to solve cognitive issues, and have rejected the idea of formal definitions of intelligence (or stated that consciousness and being sentient or not isn't an issue). I have to use AGI as a label for "stealth mode" reasons, but would prefer to separate field of study from the implementation side. And while I'm not trying to build better models of the human mind, I have come to understand consciousness in a fundamental way (and working from an embodied framework, it's easier to see just how incapable current efforts are).
I was "archiving" the link to this page and thought I'd see what's been going on. Updates seem to only be on the discord. Anyway, since they allowed me to post longer thoughts there, figured it would be fine for me to drop it here as well. https://sd-marlow.medium.com/slaying-the-ml-dragon-7ce0a2e4e3a6
From your post, you're looking at this in much the same way I was when I attempted to do a short run (to work the bugs out and really understand whats involved). However, "actual thoughts of the DM" is the wrong explanation for what they want. The examples of of what they are accepting look to be nothing more than the "common sense" stuff current ML models fail to capture (thus, explicitly stated in the runs). Also, from comments in the discord, it seems like the info captured is post-process, despite the desire for pre-prompt thoughts. Not trying to discourage; just showing my thinking on the process, and that it wasn't what they wanted.
Using scenes as a marker has some added benefit as I find myself leaving high level comments about some of the next scenes (I had nothing planned beyond the start, but the natural progression leads to speculation about future events or details). This is some of that looking ahead data that this project wanted to capture. Perhaps there should be a FUTURE keyword to wrap these things under? It would basically be a THOUGHT for world building ideas, but not specific to the current part of the story/narrative.
Anything that goes into writing or crafting needs to be captured in "real time" which means dumping it right in the middle of whatever you are doing.
It's the playing chess against yourself problem. I've intentionally done or said "just the right thing" thru the player to get past a section, but I've also tried to resist going with my first choice or replies because the player isn't supposed to know about the world building going on in the DM's mind. One aspect of of this is the DM thinking about how to push the player into doing something, and allowing the player to not follow every planned idea. You could think of it as replay value, where there are branch points not taken, but these are still ideas that need to be captured.
I don't think manually ending at 1,000 steps will be an issue. "Player engagement" is going to be an issue before hitting the 300 step mark. I'd imaging the narrative is going to feel forced and made-up beyond that point.
With no idea what the arc of the run/story will be, it's really hard to plan for 3 acts, so maybe not so useful. But did want to leave another comment about scenes. With 4 scenes being about 50 steps, just as a reference, we can look at the number of scenes in a movie to figure each run could be 500 to 750 steps in total length. I just don't see 1,000 steps as being anything other than an arbitrary dataset requirement. 250-300 steps as a playable run. 500 to 600 steps as a "movie length" representation. And then to double that?
The mental requirement to "film" a Lord of the Rings trilogy while also "filming" the behind the scenes of that filming and also "filming" the real-time documentary required to keep track of everything... while not being clear on how that extra "run time" translates into being better training data.
- Is there going to be a "THIS" post, using sample work that you really like and "demanding" all other entries follow that exact format? How will variations in formatting be addressed? Does it need to be?
- If you get something that checks all the right boxes, with one exception that leads to a rejection, I think we'd all like to know what that one must-have is.
Found a rhythm using PLAT (Prompt. Logic. Action. Thought.) but am only averaging 185 words per step. That would be about 18,000 words for 100 steps, or 54,000 words for 300 (which is the very bottom end of book territory). Agree that 100 steps is no story, but waiting to reach 100 steps before checking-in is waiting to long.
Would recommend anyone near the 20 step or 10 pages mark send that in for feedback before going further. I'm going to ignore my own advice because I'd like to complete the first 3 scenes, which is closer to 10% of the full story.
People are concerned about upfront time commitment, while also being focused on 100 step minimum. In another comment I went over how 250 steps works as a better minimum, but to keep all the numbers aligned, perhaps every story should be in 3 acts of 100 steps each (with at least 300 steps being a requirement; handing-in 299 steps would seem sloppy and rude). That would make each "short" story worth $6k, and each act $2k, which is the same 10% of 1,000 steps. Except, handing in the first act should only reserve your $6k payout, not result in getting $2k at a time (desire for finished products and not having to burden anyone with increased tracking/management). There could also be an $18k cap (for 3 short stories of at least 300 steps each) to both limit the number of short stories submitted and let people know there is no "dominating" of the short story space.
I started with something more "contained" and easier to manage because actual users will go off script every chance they get, and this is basically like playing chess against yourself while reading a book on how to play chess. But, I may have found a kind of working compromise in terms of format and what needs to be captured. Will need a few days to see how it holds up, but right now, this is the basic idea:
Initial PROMPT to get the story started, followed by THOUGHTS that examine them from a gaming perspective, an ACTION, my THOUGHTS, another PROMPT, and.. this is where I was having a tough time because some of the mechanics were not being captured in the THOUGHTS prior. It was only as I wrote the PROMPT that I figured-out certain details or actions that needed to be in play. So when I write a PROMPT that contains these other elements, I write a LOGIC section below them to explain why I "prompted" the way I did.
In crafting the story as you go, the PROMPT is also part of the THOUGHT process! I'm sure anyone giving this a try will be writing and re-writing their prompt as part of the process. Having this extra LOGIC step seems to clean that up, but I don't think any ML algo will ever keep track of story elements, have ideas on where to take the story next, and then backtrack. Perhaps the "prompt" is some adversarial output from the thoughts, but still internal to process, leading to more thoughts (aka the logic), which leads to the actual output.
Just my 2 cents.
Number of steps matters as 1,000 would be (roughly) 12 hours of play. Current ML systems will never last that long, but wondering what the natural play length would be for most. 3 hours? That would be around 250 steps. Without multiple examples of what works and what doesn't, I don't think there should be anyone working toward the full 300,000 word target (yet). $500 for 30k word samples (thru the end of the year)? I still think there is to much focus on having "thoughts" that reflect how current ML systems are trained, so best to see what happens organically?
Edit: Saw that a "best example" of what AI Dungeon can do (story called The Long Lost Queen) was 264 actions, so that fits with my estimate. *have to also note a large number of fans are using them for "non-dungeon" fan fiction of an adult nature, which brings into question how story narratives might have a link to the content (ie, how a DM thinks about a combat scene is going to be different than one crafted for sexual content). Do the samples need to represent different genre?
Agree that it's the 'crafting' part that is what matters, and I don't think we can say a writer/DM is going to explicitly be thinking about all the details of the story at each turn. From the examples.. well as a side effect of doing AI research is that you can't help but read the text of the story and see that the "thoughts" about it are just picking details in a way that even SOTA ML systems have a problem with. They don't read as actual notes about the process. Perhaps there needs to be a request for samples, with a 30k word limit (so no one invests too much time in something that might not be used), and a focus on capturing the process of writing a story as the plot unfolds.
This is a relevant point: An AI that can craft some misdirection into a game or story is showing a deeper level of understanding, but as it's within a context (game/story), that isn't really a lie. The question for MIRI is, does that kind of "knowledge about misdirection" serve as a dual-use technology, where said ability could be used in other circumstances?
I'm just "importing" my twitter thread and adding some additional thoughts.
If some model could spit out 100 of these annotated adventures, then the challenge would have already been solved.
Not sure about that 300,000 word count document idea though... A word-dump focused "result" plays into the strength of LLM's while providing none of the structure that is missing.
The more I work on this, the more I think you want something different. Perhaps use existing choose your own adventure books as a starting point, and work on deconstructing them; expanding on all of the reasoning, mechanics, story elements, etc.
The example given is heavy with exposition, and no real mechanics. That seems to rule-out any desire for explicit replies to a prompt (implication that player goes thru door is enough, not needing "walk thru door").
I get that an algo doesn't care, but example is hard to parse. It fails as an adventure (very on-rails) but also like having director commentary track play over a movie you've never seen, and then get tested on dialog and plot points.
The "thoughts" related to the 4 page sample just look like answers to multiple choice questions about the body of text. This says nothing about the process of crafting the narrative, which is the point, right? Examples of how to craft story structure? Why something was done?
There is a kind of "other minds" problem, in that the story should be constructed with player expectations in mind. Rather than just generating copious amounts of "story text," the adventure is more of a dialog where the DM moves the player thru a story, but also "entertains" with traps and dead-ends. What will happen next feels like ground that is already covered by LLM's, but anticipation of actions is where the dynamic feel comes from (so at the very least, an algo needs to create branching story structure).
30M word dataset's wont do anything to "train creativity" into the system, such as understanding why a small white rabbit isn't a real threat.. until it fly's at your neck.
Edit: Would it not just be easier to craft a framework since all of the questions/considerations required when building a story are going to be the same regardless of player inputs? I'm going to continue-on with the "adventure" track I've already started since the end of act annotations still explain the reasoning, and help point toward future story elements. There is no pre-planned arc, so there is the same level of "real-time" construction as the game progresses. Really not clear how annotating a few copies of War and Peace is useful while also having to write such a story. As stated, after 12k-15k words, you would have discovered a framework that works for the next 15M words.