Notifications Received in 30 Minutes of Class

post by tanagrabeast · 2024-05-26T17:02:20.989Z · LW · GW · 12 comments

Contents

  Introduction
  Procedure
  Results
  Insights from talking to individual students 
  Discussion
  Conclusion
None
12 comments

Introduction

If you are choosing to read this post, you've probably seen the image below depicting all the notifications students received on their phones during one class period. You probably saw it as a retweet of this tweet, or in one of Zvi’s posts. Did you find this data plausible, or did you roll to disbelieve? Did you know that the image dates back to at least 2019? Does that fact make you more or less worried about the truth on the ground as of 2024?

Last month, I performed an enhanced replication of this experiment in my high school classes. This was partly because we had a use for it, partly to model scientific thinking, and partly because I was just really curious. Before you scroll past the image, I want to give you a chance to mentally register your predictions. Did my average class match the roughly 1,084 notifications I counted on Ms. Garza's viral image? What does the distribution look like? Is there a notable gender difference? Do honors classes get more or fewer notifications than regular classes? Which apps dominate? Let's find out!

Before you rush to compare apples and oranges, keep in mind that I don't know anything about Ms. Garza's class -- not the grade, the size, or the duration of her experiment. That would have made it hard for me to do a true replication, and since I saw some obvious ways to improve on her protocol, I went my own way with it.

Procedure

We opened class with a discussion about what we were trying to measure and how we were going to measure it for the next 30 minutes. Students were instructed to have their phones on their desks and turned on. For extra amusement, they were invited (but not required) to turn on audible indicators. They were asked to tally each notification received and log it by app. They were instructed to not engage with any received notifications, and to keep their phone use passive during the experiment, which I monitored.

While they were not to put their names on their tally sheets, they were asked to provide some metadata that included (if comfortable) their gender. (They knew that gender differences in phone use and depression were a topic of public discussion, and were largely happy to provide this.)

To give us a consistent source of undemanding background "instruction" — and to act as our timer — I played the first 30 minutes of Kurzgesagt's groovy 4.5 Billion Years in 1 Hour video. Periodically, I also mingled with students in search of insights, which proved highly productive.

After the 30 minutes, students were charged with summing their own tally marks and writing totals as digits, so as to avoid a common issue where different students bundle and count tally clusters differently.

Results

Below are the two charts from our experiment that I think best capture the data of interest. The first is more straightforward, but I think the second is a little more meaningful.

Ah! So right away we can see a textbook long-tailed distribution. The top 20% of recipients accounted for 75% of all received notifications, and the bottom 20% for basically zero. We can also see that girls are more likely to be in that top tier, but they aren't exactly crushing the boys.

But do students actually notice and get distracted by all of these notifications? This is partly subjective, obviously, but we probably aren't as worried about students who would normally have their phones turned off or tucked away in their backpacks on the floor. So one of my metadata questions asked them about this. The good rapport I enjoy with my students makes me pretty confident that I got honest answers — as does the fact that the data doesn't change all that much when I adjust for this in the chart below.

The most interesting difference in the adjusted chart is that the tail isn't nearly as long; under these rules, nearly half of students "received" no notifications during the experiment. The students most likely to keep their phones from distracting them were the students who weren't getting many notifications in the first place.

Since it mostly didn't matter, I stuck with the unadjusted data for the calculations below, except where indicated.

• Average notifications per student:  20.3 (or 16.16, after the above adjustment)
  • Female average: 22 (or 18.3, adjusted), Male average: 17 (or 14.9, adjusted)
• Median notifications per student: 7 (or 2, adjusted)
  • Female median: 6 (or 3, adjusted), Male median: 7.5 (or 2, adjusted)

Do my numbers make Ms. Garza's numbers plausible? Yes! If my experiment had run for a 55 minute class period, that would be 37.2 notifications per student. Assuming a larger class of 30 students, that would be 1,116 notifications total, remarkably close to Ms. Garza's 1,084-ish.

Do honors classes get more or fewer notifications than regular classes? While I teach three sections of each, this question is confounded a bit by the fact that honors classes tend to skew female, so let's break down the averages and medians by gender:

• Honors girls: average 29.2, median 6
• Regular girls: average 11.6, median 6
   
• Honors boys: average 17.2, median 6
• Regular boys: average 16.7, median 9.5

So... it's interesting, but complicated. For girls, the main difference seems to come down to a few of the heaviest female recipients being in honors classes, blowing up their classes' averages despite the medians being identical. For boys, there seems to be a significantly higher median in the non-honors classes, which is less likely to be the result of just a few boys being in one class or the other.

Which apps dominate? Instagram and Snapchat were nearly tied, and together accounted for 46% of all notifications. With vanilla text messages accounting for an additional 35%, we can comfortably say that social communications account for the great bulk of all in-class notifications.

There was little significant gender difference in the app data, with two minor apps accounting for the bulk of the variation: Ring (doorbell and house cameras) and Life 360 (friend/family location tracker), each of which sent several notifications to a few girls. ("Yeah," said girls during our debriefing sessions, "girls are stalkers." Other girls nodded in agreement.)

Notifications from Discord, Twitch, or other gaming-centric services were almost exclusively received by males, but there weren't enough of these to pop out in the data.

Insights from talking to individual students 

• The two top recipients, with their rate of 450 notifications per hour (!), or about one every eight seconds, had interesting stories to tell. One of these students had a job after school, and about half their messages (but only half) were work-related. The other was part of a large group chat, and additionally had a friend at home sick who was peltering them with a continuous rant about everything and nothing, three words at time.
• Group chats were a consistent feature of high-scoring tally sheets.
• Friends from other schools that release earlier in the day send a lot of messages to a few students in my afternoon classes.
• Many of the Ring notifications were for girls who had subscribed to a neighborhood watch channel for lost pets.
• Some students who receive very large numbers of notifications use settings to differentiate them by vibration patterns, and tell me that they "notice" some vibrations much more than others.
• I did not collect metadata on GPA or anything, but my impression from mingling was that there was no significant correlation between notification counts and academic achievement, except maybe among boys in the middle chunk of the distribution (as seen in the honors vs. regular stats).
• Official school business is a significant contributor to student notification loads. At least 4% of all notifications were directly attributable to school apps, and I would guess the indirect total (through standard texts, for example) might be closer to 10-15%. For students who get very few notifications, 30-50% of their notifications might be school-related.
  • Our school’s gradebook app is the biggest offender, in part because it’s poorly configured and sends way more notifications than anyone wants. Did one of your teachers put in a grade for an assignment? That’s a notification for the added grade, a separate notification for the change in your semester grade, and possibly two notifications from your email app about mailed copies of those two events. There might also be a notification from a separate app that was used to complete the assignment, and/or an email from that other app.
  • Much campus coordination is handled through emails, texts, or other dedicated apps, and some of this also creates additional channels for social messaging and group texts.
•  I was amused that "Parent Controls" accounted for some notifications on one student's tally sheet.
• Predictably, a few students from my morning classes delighted in trying to bombard their friends in my afternoon classes with messages. I asked students to please not tally any notifications they were pretty sure were trolling of this sort, and I believe they mostly complied.
• Students generally know about settings for disabling undesired notifications by app — and many seemed to use fine-grained controls within apps — but most could benefit from the occasional reminder/invitation to revisit them as they seemed determined to do after this experiment.

Discussion

At the end of our 30 minutes, we had some whole-class follow-up discussions. I will layer some of my own thoughts onto the takeaways:

• Is our school unusually good or bad when it comes to phones? By a vote of 23 to 7, students who had been enrolled in another school during the last four years said our school was better than their previous school at keeping phones suppressed.
  • There's still obvious room for improvement, though. I asked my students to imagine that, at the start of the hour, they had sent messages inviting a reply to 5 different friends elsewhere on our campus. How many would they expect to have replied before the end of the hour? The answer I consistently got was 4, and that this almost entirely depended on the phone-strictness of the teacher whose class each friend was in. (I’m on the list of phone-strict teachers, it seems. Phew!)
  • Given how the activity of a group chat (under naive assumptions) scales with the number of active participants, phone-strict teachers probably cause outsized reductions in the campus-wide notification rate.
     
• I asked students if they would want to press a magic button that would permanently delete all social media and messaging apps from the phones of their friend groups if nobody knew it was them. I got only a couple takers. There was more (but far from majority) enthusiasm for deleting all such apps from the whole world.
  • I suspect rates would have been higher if I had asked this as an anonymous written question, but probably not much higher.
     
• I asked if they thought education would be improved on campus if phones were forcibly locked away for the duration of the school day. Only one student gave me so much as an affirmative nod!
  • Among students, the consensus was that kids generally tune into school at the level they care to, and that a phone doesn’t change that. A disinterested student without a phone will just tune out in some other way.
    • They also put forth the idea that the quality of the teacher has a lot to do with how much they’ll tune in. This take had broad support.
  • I will say that, on average, classes these days just seem quieter than the classes I taught 10-15 years ago. I think much of what would have been talking-in-class has been converted to slower conversations that are still happening in class, but silently and with friends who are elsewhere. A phone-strict teacher like me slows or pauses those conversations, but doesn't necessarily cause them to be replaced with verbal in-class chatter even when they provide permitted windows for such chat.

Conclusion

The state of classroom distraction is a complicated one. Students are incredibly varied in their level of digital connectivity in ways that are not obviously correlated with gender or academic achievement. I didn't try to explore any mental health angles, but I also didn't see any obvious trends in the personalities of students who were highly or weakly tied to their social media.

My sense is that the harms of connectivity, like the level of connectivity, are also highly variable from student to student, and not strongly correlated with the notification rate. Most of my students (even among top notification recipients) seem as well-adjusted with regards to their technology as professional adults. Only a few have an obvious problem, and they will often admit to it. What's not clear to me is whether those maladjusted teens would not, in the absence of phones, find some other outlet to distract themselves.

In any event, the landscape of distraction is constantly shifting. I've started to see more students who can skillfully text on a smartwatch, and in just the last two years I've seen an explosion in students who discreetly wear a wireless earbud in one ear and may or may not be listening to music in addition to (or instead of) whatever is happening in class. This is so difficult and awkward to police with girls who have long hair that I wonder if it has actually started to drive hair fashion in an ear-concealing direction. It's also difficult to police earbuds when some students are given special accommodations that allow them to wear one, and I predict that if we ever see an explosion of teenagers with cool hearing aids it will be because this gives them full license to listen to whatever they want whenever they want.

The constant, for at least a little longer, is a human teacher in the classroom. Teachers that students find interesting (or feel they are getting value out of) can continue to command attention. But the bar is going up. This may be especially challenging for new teachers, who are not only in increasingly short supply, but anecdotally seem more likely to bounce off of the profession than in years past.

We must also reflect that distracting technology isn't the only factor driving student apathy, and it never was. Disinterest partly stems from the basic psychology of an age group that struggles to feel like anything after tomorrow will ever be real. But there's also the question of what the day after tomorrow will bring. For as long as kids have been compelled to attend school, we've had students who feel a disconnect between what school is providing and what their adult world will actually expect of them. In this new era where AI can write their essays, solve their math problems, draw their art, compose their music — and yes, even become their "friends" who distract them with notifications during class — I expect this disconnect to grow.
 

12 comments

Comments sorted by top scores.