Do humans really learn from "little" data?
post by Alice Wanderland (alice-wanderland) · 2025-01-14T10:46:09.179Z · LW · GW · 2 commentsThis is a link post for https://aliceandbobinwanderland.substack.com/p/do-humans-really-learn-with-little
Contents
How much data does it take to pretrain a (human) brain? I conducted a (fairer) Fermi estimate. None 2 comments
How much data does it take to pretrain a (human) brain? I conducted a (fairer) Fermi estimate.
The post goes through the following questions:
- How long does it take to grow a human brain?
- How many waking seconds do we have in our life?
- How many “tokens” or “data points” does a human brain process in a second?
- Can we simply count the spikes?
- How many bits (spikes and non-spikes) does it take for the brain to process 1 sensory “piece of information”?
- How do those numbers stack up against LLMs?
To get to this conclusion table:
2 comments
Comments sorted by top scores.
comment by FL33TW00D · 2025-01-14T11:00:27.616Z · LW(p) · GW(p)
I think you're missing a trick here purely focusing on waking seconds.
Demis explains this beautifully in this talk here:
TLDW:
The brain replays memories stochastically during sleep, and replays them OOMs faster than they were experienced. This "multi epoch training" allows the brain to learn much more from the environment, and it can prioritise salient experiences.
Replies from: PeterMcCluskey↑ comment by PeterMcCluskey · 2025-01-14T22:34:31.074Z · LW(p) · GW(p)
"OOMs faster "? Where do you get that idea?
Dreams indicate a need for more processing than what happens when we're awake, but likely less than 2x waking time.