Notifications Received in 30 Minutes of Class
post by tanagrabeast · 2024-05-26T17:02:20.989Z · LW · GW · 16 commentsContents
Introduction Procedure Results Insights from talking to individual students Discussion Conclusion None 16 comments
Introduction
If you are choosing to read this post, you've probably seen the image below depicting all the notifications students received on their phones during one class period. You probably saw it as a retweet of this tweet, or in one of Zvi’s posts. Did you find this data plausible, or did you roll to disbelieve? Did you know that the image dates back to at least 2019? Does that fact make you more or less worried about the truth on the ground as of 2024?
Last month, I performed an enhanced replication of this experiment in my high school classes. This was partly because we had a use for it, partly to model scientific thinking, and partly because I was just really curious. Before you scroll past the image, I want to give you a chance to mentally register your predictions. Did my average class match the roughly 1,084 notifications I counted on Ms. Garza's viral image? What does the distribution look like? Is there a notable gender difference? Do honors classes get more or fewer notifications than regular classes? Which apps dominate? Let's find out!
Before you rush to compare apples and oranges, keep in mind that I don't know anything about Ms. Garza's class -- not the grade, the size, or the duration of her experiment. That would have made it hard for me to do a true replication, and since I saw some obvious ways to improve on her protocol, I went my own way with it.
Procedure
We opened class with a discussion about what we were trying to measure and how we were going to measure it for the next 30 minutes. Students were instructed to have their phones on their desks and turned on. For extra amusement, they were invited (but not required) to turn on audible indicators. They were asked to tally each notification received and log it by app. They were instructed to not engage with any received notifications, and to keep their phone use passive during the experiment, which I monitored.
While they were not to put their names on their tally sheets, they were asked to provide some metadata that included (if comfortable) their gender. (They knew that gender differences in phone use and depression were a topic of public discussion, and were largely happy to provide this.)
To give us a consistent source of undemanding background "instruction" — and to act as our timer — I played the first 30 minutes of Kurzgesagt's groovy 4.5 Billion Years in 1 Hour video. Periodically, I also mingled with students in search of insights, which proved highly productive.
After the 30 minutes, students were charged with summing their own tally marks and writing totals as digits, so as to avoid a common issue where different students bundle and count tally clusters differently.
Results
Below are the two charts from our experiment that I think best capture the data of interest. The first is more straightforward, but I think the second is a little more meaningful.
Ah! So right away we can see a textbook long-tailed distribution. The top 20% of recipients accounted for 75% of all received notifications, and the bottom 20% for basically zero. We can also see that girls are more likely to be in that top tier, but they aren't exactly crushing the boys.
But do students actually notice and get distracted by all of these notifications? This is partly subjective, obviously, but we probably aren't as worried about students who would normally have their phones turned off or tucked away in their backpacks on the floor. So one of my metadata questions asked them about this. The good rapport I enjoy with my students makes me pretty confident that I got honest answers — as does the fact that the data doesn't change all that much when I adjust for this in the chart below.
The most interesting difference in the adjusted chart is that the tail isn't nearly as long; under these rules, nearly half of students "received" no notifications during the experiment. The students most likely to keep their phones from distracting them were the students who weren't getting many notifications in the first place.
Since it mostly didn't matter, I stuck with the unadjusted data for the calculations below, except where indicated.
- Average notifications per student: 20.3 (or 16.16, after the above adjustment)
- Female average: 22 (or 18.3, adjusted), Male average: 17 (or 14.9, adjusted)
- Median notifications per student: 7 (or 2, adjusted)
- Female median: 6 (or 3, adjusted), Male median: 7.5 (or 2, adjusted)
Do my numbers make Ms. Garza's numbers plausible? Yes! If my experiment had run for a 55 minute class period, that would be 37.2 notifications per student. Assuming a larger class of 30 students, that would be 1,116 notifications total, remarkably close to Ms. Garza's 1,084-ish.
Do honors classes get more or fewer notifications than regular classes? While I teach three sections of each, this question is confounded a bit by the fact that honors classes tend to skew female, so let's break down the averages and medians by gender:
- Honors girls: average 29.2, median 6
- Regular girls: average 11.6, median 6
- Honors boys: average 17.2, median 6
- Regular boys: average 16.7, median 9.5
So... it's interesting, but complicated. For girls, the main difference seems to come down to a few of the heaviest female recipients being in honors classes, blowing up their classes' averages despite the medians being identical. For boys, there seems to be a significantly higher median in the non-honors classes, which is less likely to be the result of just a few boys being in one class or the other.
Which apps dominate? Instagram and Snapchat were nearly tied, and together accounted for 46% of all notifications. With vanilla text messages accounting for an additional 35%, we can comfortably say that social communications account for the great bulk of all in-class notifications.
There was little significant gender difference in the app data, with two minor apps accounting for the bulk of the variation: Ring (doorbell and house cameras) and Life 360 (friend/family location tracker), each of which sent several notifications to a few girls. ("Yeah," said girls during our debriefing sessions, "girls are stalkers." Other girls nodded in agreement.)
Notifications from Discord, Twitch, or other gaming-centric services were almost exclusively received by males, but there weren't enough of these to pop out in the data.
Insights from talking to individual students
- The two top recipients, with their rate of 450 notifications per hour (!), or about one every eight seconds, had interesting stories to tell. One of these students had a job after school, and about half their messages (but only half) were work-related. The other was part of a large group chat, and additionally had a friend at home sick who was peltering them with a continuous rant about everything and nothing, three words at time.
- Group chats were a consistent feature of high-scoring tally sheets.
- Friends from other schools that release earlier in the day send a lot of messages to a few students in my afternoon classes.
- Many of the Ring notifications were for girls who had subscribed to a neighborhood watch channel for lost pets.
- Some students who receive very large numbers of notifications use settings to differentiate them by vibration patterns, and tell me that they "notice" some vibrations much more than others.
- I did not collect metadata on GPA or anything, but my impression from mingling was that there was no significant correlation between notification counts and academic achievement, except maybe among boys in the middle chunk of the distribution (as seen in the honors vs. regular stats).
- Official school business is a significant contributor to student notification loads. At least 4% of all notifications were directly attributable to school apps, and I would guess the indirect total (through standard texts, for example) might be closer to 10-15%. For students who get very few notifications, 30-50% of their notifications might be school-related.
- Our school’s gradebook app is the biggest offender, in part because it’s poorly configured and sends way more notifications than anyone wants. Did one of your teachers put in a grade for an assignment? That’s a notification for the added grade, a separate notification for the change in your semester grade, and possibly two notifications from your email app about mailed copies of those two events. There might also be a notification from a separate app that was used to complete the assignment, and/or an email from that other app.
- Much campus coordination is handled through emails, texts, or other dedicated apps, and some of this also creates additional channels for social messaging and group texts.
- I was amused that "Parent Controls" accounted for some notifications on one student's tally sheet.
- Predictably, a few students from my morning classes delighted in trying to bombard their friends in my afternoon classes with messages. I asked students to please not tally any notifications they were pretty sure were trolling of this sort, and I believe they mostly complied.
- Students generally know about settings for disabling undesired notifications by app — and many seemed to use fine-grained controls within apps — but most could benefit from the occasional reminder/invitation to revisit them as they seemed determined to do after this experiment.
Discussion
At the end of our 30 minutes, we had some whole-class follow-up discussions. I will layer some of my own thoughts onto the takeaways:
- Is our school unusually good or bad when it comes to phones? By a vote of 23 to 7, students who had been enrolled in another school during the last four years said our school was better than their previous school at keeping phones suppressed.
- There's still obvious room for improvement, though. I asked my students to imagine that, at the start of the hour, they had sent messages inviting a reply to 5 different friends elsewhere on our campus. How many would they expect to have replied before the end of the hour? The answer I consistently got was 4, and that this almost entirely depended on the phone-strictness of the teacher whose class each friend was in. (I’m on the list of phone-strict teachers, it seems. Phew!)
- Given how the activity of a group chat (under naive assumptions) scales with the number of active participants, phone-strict teachers probably cause outsized reductions in the campus-wide notification rate.
- I asked students if they would want to press a magic button that would permanently delete all social media and messaging apps from the phones of their friend groups if nobody knew it was them. I got only a couple takers. There was more (but far from majority) enthusiasm for deleting all such apps from the whole world.
- I suspect rates would have been higher if I had asked this as an anonymous written question, but probably not much higher.
- I suspect rates would have been higher if I had asked this as an anonymous written question, but probably not much higher.
- I asked if they thought education would be improved on campus if phones were forcibly locked away for the duration of the school day. Only one student gave me so much as an affirmative nod!
- Among students, the consensus was that kids generally tune into school at the level they care to, and that a phone doesn’t change that. A disinterested student without a phone will just tune out in some other way.
- They also put forth the idea that the quality of the teacher has a lot to do with how much they’ll tune in. This take had broad support.
- I will say that, on average, classes these days just seem quieter than the classes I taught 10-15 years ago. I think much of what would have been talking-in-class has been converted to slower conversations that are still happening in class, but silently and with friends who are elsewhere. A phone-strict teacher like me slows or pauses those conversations, but doesn't necessarily cause them to be replaced with verbal in-class chatter even when they provide permitted windows for such chat.
- Among students, the consensus was that kids generally tune into school at the level they care to, and that a phone doesn’t change that. A disinterested student without a phone will just tune out in some other way.
Conclusion
The state of classroom distraction is a complicated one. Students are incredibly varied in their level of digital connectivity in ways that are not obviously correlated with gender or academic achievement. I didn't try to explore any mental health angles, but I also didn't see any obvious trends in the personalities of students who were highly or weakly tied to their social media.
My sense is that the harms of connectivity, like the level of connectivity, are also highly variable from student to student, and not strongly correlated with the notification rate. Most of my students (even among top notification recipients) seem as well-adjusted with regards to their technology as professional adults. Only a few have an obvious problem, and they will often admit to it. What's not clear to me is whether those maladjusted teens would not, in the absence of phones, find some other outlet to distract themselves.
In any event, the landscape of distraction is constantly shifting. I've started to see more students who can skillfully text on a smartwatch, and in just the last two years I've seen an explosion in students who discreetly wear a wireless earbud in one ear and may or may not be listening to music in addition to (or instead of) whatever is happening in class. This is so difficult and awkward to police with girls who have long hair that I wonder if it has actually started to drive hair fashion in an ear-concealing direction. It's also difficult to police earbuds when some students are given special accommodations that allow them to wear one, and I predict that if we ever see an explosion of teenagers with cool hearing aids it will be because this gives them full license to listen to whatever they want whenever they want.
The constant, for at least a little longer, is a human teacher in the classroom. Teachers that students find interesting (or feel they are getting value out of) can continue to command attention. But the bar is going up. This may be especially challenging for new teachers, who are not only in increasingly short supply, but anecdotally seem more likely to bounce off of the profession than in years past.
We must also reflect that distracting technology isn't the only factor driving student apathy, and it never was. Disinterest partly stems from the basic psychology of an age group that struggles to feel like anything after tomorrow will ever be real. But there's also the question of what the day after tomorrow will bring. For as long as kids have been compelled to attend school, we've had students who feel a disconnect between what school is providing and what their adult world will actually expect of them. In this new era where AI can write their essays, solve their math problems, draw their art, compose their music — and yes, even become their "friends" who distract them with notifications during class — I expect this disconnect to grow.
16 comments
Comments sorted by top scores.
comment by gwern · 2024-05-26T17:44:49.250Z · LW(p) · GW(p)
I asked students if they would want to press a magic button that would permanently delete all social media and messaging apps from the phones of their friend groups if nobody knew it was them. I got only a couple takers. There was more (but far from majority) enthusiasm for deleting all such apps from the whole world. I suspect rates would have been higher if I had asked this as an anonymous written question, but probably not much higher.
An anonymous response, or better yet, a list deletion or Bayesian truth serum question, would certainly be worth doing.
It's also striking that you get anywhere close to a majority for that question. (And consistent with Haidt and the other willingness-to-pay experiment about the harms of social media being a systemic problem, where you are unable to optout on an individual, or even very narrow friend-group basis, because that still leaves the rest of society using it.) Imagine asking that question about, say, indoor plumbing or electricity or vaccines? I suspect even much more controversial technologies like cars or video games would still get many fewer votes for deletion at any scale.
Replies from: kabir-kumar-1↑ comment by Kabir Kumar (kabir-kumar-1) · 2024-08-13T21:35:18.075Z · LW(p) · GW(p)
if asked about recommendation algoritms, I think it might be much higher - given a basic understanding of what they are, addictiveness, etc
comment by trevor (TrevorWiesinger) · 2024-05-26T18:57:23.549Z · LW(p) · GW(p)
Strong upvoted, thank you for the serious contribution.
Children spending 300 hours per year learning math, on their own time and via well-designed engaging video-game-like apps (with eg AI tutors, video lectures, collaborating with parents to dispense rewards for performance instead of punishments for visible non-compliance, and results measured via standardized tests), at the fastest possible rate for them (or even one of 5 different paces where fewer than 10% are mistakenly placed into the wrong category) would probably result in vastly superior results among every demographic than the current paradigm of ~30-person classrooms.
in just the last two years I've seen an explosion in students who discreetly wear a wireless earbud in one ear and may or may not be listening to music in addition to (or instead of) whatever is happening in class. This is so difficult and awkward to police with girls who have long hair that I wonder if it has actually started to drive hair fashion in an ear-concealing direction.
This isn't just a problem with the students; the companies themselves end up in equilibria where visibly controversial practices get RLHF'd into being either removed or invisible (or hard for people to put their finger on). For example, hours a day of instant gratification reducing attention spans, except unlike the early 2010s where it became controversial, reducing attention spans in ways too complicated or ambiguous for students and teachers to put their finger on until a random researcher figures it out and makes the tacit explicit. Or another counterintuitive vector could be the democratic process of public opinion turns against schooling, except in a lasting way. Or the results of multiple vectors like these overlapping.
I don't see how the classroom-based system, dominated entirely by bureaucracies and tradition, could possibly compete with that without visibly being turned into swiss cheese. It might have been clinging on to continued good results from a dwindling proportion of students who were raised to be morally/ideologically in favor of respecting the teacher more than the other students, but that proportion will also decline as schooling loses legitimacy.
Regulation could plausibly halt the trend from most or all angles, but it would have to be the historically unprecedented kind of regulation that's managed by regulators with historically unprecedented levels of seriousness and conscientiousness towards complex hard-to-predict/measure outcomes.
Replies from: Richard Horvath↑ comment by Richard Horvath · 2024-05-27T13:01:08.315Z · LW(p) · GW(p)
Children spending 300 hours per year learning math, on their own time and via well-designed engaging video-game-like apps (with eg AI tutors, video lectures, collaborating with parents to dispense rewards for performance instead of punishments for visible non-compliance, and results measured via standardized tests), at the fastest possible rate for them (or even one of 5 different paces where fewer than 10% are mistakenly placed into the wrong category) would probably result in vastly superior results among every demographic than the current paradigm of ~30-person classrooms.
This sounds great.
Googling "math tutor ai" already gives a bunch of competing suggestions. I wonder how well they work though.
comment by Ben Pace (Benito) · 2024-09-10T23:23:27.552Z · LW(p) · GW(p)
Students were instructed to have their phones on their desks and turned on. For extra amusement, they were invited (but not required) to turn on audible indicators. They were asked to tally each notification received and log it by app.
Just checking, were they counting pull notifications as well as push?
- Push notifications: All notifications that appeared on your lock screen / buzz your phone / make a noise.
- Pull notifications: Any in-app notifications that don't have the above effects, but that you see if you choose to open the app.
For instance my FB app has like 5-10 'notifications' for me when I check it once a day, but I only find out when I check it, there are no badges and nothing appears on my lock screen.
I have hundreds of pull notifications per day in my slack app, but I get close to zero push notifications (e.g. yesterday I got ~20).
Replies from: tanagrabeast↑ comment by tanagrabeast · 2024-09-11T04:29:31.826Z · LW(p) · GW(p)
They should not have been counting pull notifications, as they were instructed to not engage with their phones during the experiment except to maybe see what caused a vibration or ding. I don't think students think of pull notifications as real notifications the way we were using the word. They were logging the notifications they could notice while their phone flat was flat on their desk not being touched.
Replies from: Benito↑ comment by Ben Pace (Benito) · 2024-09-11T05:11:06.555Z · LW(p) · GW(p)
Thanks for clarifying, that's what I was expecting.
I just needed to check, because if I'm getting 20/day, and the peak student is getting 450/hour, then that's like 300x difference in terms of notifications, and that really is an extremely different experience of life. Even the mean kid is having a ~30x different experience. I wonder whether I would be able to have time to think if that was my life.
Replies from: tanagrabeast↑ comment by tanagrabeast · 2024-09-11T07:19:12.278Z · LW(p) · GW(p)
From chatting with those peak students during the experiment, I think their experience is more like being in a cafeteria abuzz with the voices of friends and acquaintances. At some point, you're not even trying to follow every conversation, but are just maintaining some vague awareness of the conversations that are taking place and jumping in when you feel like it. People can and do think about other things in a noisy cafeteria. Some even read books! The brain can filter out a constant buzz. It's just wind blowing through the trees.
The upper middle zone where it's still possible to try to follow everything (and maybe even reply) looked like more of an attention trap, and was where I was more likely to find that handful of students I already knew had a problem. The FOMO is probably more distracting than the notifications themselves.
comment by JohnBuridan · 2024-06-02T12:34:09.869Z · LW(p) · GW(p)
At my school all the students have to deposit their cell phones into pouches at the office the beginning of the day.. cell phone strictness alone does not create interpersonal chatter. You just also spend a lot of time and work creating an actual culture of conversation. It's an explicitly taught skill!
Your data on the elasticity of behavior matches my experience well..
comment by snerx · 2024-06-28T02:59:26.506Z · LW(p) · GW(p)
In this new era where AI can write their essays, solve their math problems, draw their art, compose their music — and yes, even become their "friends" who distract them with notifications during class — I expect this disconnect to grow.
Do you have numbers on how many students were using AI friend apps?
Replies from: tanagrabeast↑ comment by tanagrabeast · 2024-06-28T05:57:07.441Z · LW(p) · GW(p)
No. Everyone seemed to know what they were, because they all claimed to know someone who uses them. But I don't recall anyone ever admitting to being such a someone. I sense there's a bit of a stigma around them.
comment by Gus C. (gus-c) · 2024-05-28T14:04:47.206Z · LW(p) · GW(p)
This was really interesting, although it made think differently from you in some points, maybe because of the generation difference.
Let's be honest, at school, we just spent most of our times bored, or learning something we did not want to. Being of the generation that was born with phones, it was not as if I was sucked into a blackhole of information, I WANTED to somehow optimize my time with something I found to be better. This is what justified boredom.
We all have this subconscious idea of time optimisation. And in every class I had back in my mind the same mental dialogue "when I get home, I'll just watch a YouTube video about this theme, and I'll probably learn this 5x faster and better than here". Now with AI, I think you can not only learn faster, but personally tailor how and what you want to be taught much more efficiently. So indeed, the bar for teachers have gone pretty high, and we are just getting started in the AI revolution. Even me, in my fourth year of medschool get caught myself drifting and instead asking for an AI to explain something generally complex, and have a pretty satisfying discussion with it.
My conclusion from this is that if AI at the level of gpt-4o was used as a teaching tool in itself, it might decrease this "wandering mind" that alumni tend to have in normal classes, because of the time optimisation. When you use AI, it's all about input and output, you have to engage with it to receive. Differently from normal human teaching in a class with >≈ 40 students, where the teacher will just keep talking, regardless if you are following or interested.
comment by cata · 2024-05-27T21:07:17.095Z · LW(p) · GW(p)
It seems like the students think that eliminating the distractions wouldn't improve how much they learn in class. That sounds ridiculous to me, but public school classrooms are a weird environment that already aren't really set up well to teach anyone anything, so maybe it could be true. Is it credible?
Replies from: tanagrabeast↑ comment by tanagrabeast · 2024-05-27T21:22:21.233Z · LW(p) · GW(p)
It is credible that eliminating all preventable distractions (phones, earbuds, etc.) wouldn't improve learning much. As a teen, I bet you were distracted during class by all sorts of things contained entirely within your head. I know I was!
There's a somewhat stronger case that video games and social media have given students more things to be preoccupied about even if you make these things inaccessible during class. But I also think that just being a hormonal teen is often distracting enough to fill in any attention vacancies faster than the median lesson can.
comment by Review Bot · 2024-05-27T04:42:11.044Z · LW(p) · GW(p)
The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.
Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?
comment by mesaoptimizer · 2024-05-26T17:36:48.963Z · LW(p) · GW(p)
This is very interesting, thank you for posting this.