My ML Scaling bibliography
post by gwern · 2021-10-23T14:41:45.170Z · LW · GW · 9 commentsThis is a link post for https://www.gwern.net/notes/Scaling
Contents
9 comments
9 comments
Comments sorted by top scores.
comment by gwern · 2021-10-23T00:51:31.544Z · LW(p) · GW(p)
Thompson et al is being discussed again (/sigh) and apparently a lot of people discussing don't know what real scaling law research looks like. /r/MLScaling is an excellent resource now, but one might prefer a slightly more organized and compact approach than either an endlessly growing subreddit or my scaling hypothesis essay; thus, this page.
Replies from: orthonormal↑ comment by orthonormal · 2021-10-23T03:42:15.360Z · LW(p) · GW(p)
Is this meant to be a linkpost? I don't see any content except for the comment above.
Replies from: gwern↑ comment by gwern · 2021-10-23T14:43:27.844Z · LW(p) · GW(p)
Hm, it was, I am quite sure I did paste the link into the link field (I'm not that careless). I'm not sure what happened there... I added 2 tags afterwards, I wonder if there is a bug in the tag-adding code which can erase links? (Or if setting the link field doesn't automatically turn a post into a link post, which if true, would be another reason for my old suggestion that link posts should get the little circle link-icon.) Anyway, I've added the link (back?).
Replies from: jimrandomh↑ comment by jimrandomh · 2021-10-23T20:44:07.757Z · LW(p) · GW(p)
I just checked the logs, and I think this is Akismet's fault. It classified your post as spam (because of the empty body?), but we ignored it because you're over a karma threshold, and needless to say that particular code path is not a well tested one.
EDIT: Hmm, I think this is actually not it; I tested in development with a mark-this-as-spam test string and it didn't reproduce the problem.
One possibility that comes to mind: there's a client-side autosave, but it only covers the post body, not the title and link, so if you refreshed the page before submitting, the link would disappear that way?
Replies from: gwern↑ comment by gwern · 2021-10-23T21:52:20.623Z · LW(p) · GW(p)
because of the empty body?
It might make sense to disallow posts with no body and also no link as some sort of malformed post or user error. Are there any cases where a title-only submission was intended and desirable, and the submitter could not have added a short body or link? I can't recall any.
but it only covers the post body, not the title and link, so if you refreshed the page before submitting, the link would disappear that way?
Could the tag dialogue refresh the page? I screwed around with the tag dialogue with a bit (I was surprised there was apparently no 'neural net' tag, at least that I could find, and settled for ML).
comment by Jsevillamol · 2021-10-24T13:57:05.939Z · LW(p) · GW(p)
For people interested in scaling laws, check out "Parameter, Compute and Data Trends in Machine Learning" - the biggest public database of milestone AI systems annotated with data on parameters, compute and data.
A visualization of the current dataset is available here.
(disclaimer: I am the main coordinator of the project)
Replies from: gwern↑ comment by gwern · 2021-10-24T14:33:48.706Z · LW(p) · GW(p)
One new historical example: https://www.reddit.com/r/mlscaling/comments/qd65vu/the_saga_of_highleyman_1961s_data_hardt_recht/
comment by Tom Lieberum (Frederik) · 2021-10-24T11:14:36.388Z · LW(p) · GW(p)
Great stuff, thanks!
Is there are reason you're not including https://deepmind.com/blog/article/generally-capable-agents-emerge-from-open-ended-play? Is it not explicit enough in terms of being a scaling paper?
Replies from: gwern