Part I: Introduction and Methodology
I've been wrestling with Jonathan Haidt's The Anxious Generation - agreeing with many points while cringing at the shoddy research and inflammatory language. I finally decided to do a series of four blog posts.
This four-part series will examine:
Part I (this post): Methodological flaws in Haidt's research
Part II: Why phone bans ignore digital literacy needs
Part III: The reality of modern schooling vs. free play
Part IV: The patriarchal lens of boys' risk-taking"
The links will be updated in this post as I go. I also should note that I’ll try not to repeat previous (many, valid) critiques of the book, but if you want a strong take-down of the book, listen to this episode of If Books Could Kill. They do a great job laying out counter-arguments (e.g., also happened around 2010: Obamacare and access to mental health support.)
But first, let’s talk a bit about the book. Haidt tackles the reasons for what he describes as a mental ill-health epidemic in Western, digitised societies. Put concisely, he argues that phones make children more anxious. He talks about the fact that children do not play outside in unsupervised, multi-aged, diverse groups like we used to. This is probably true, although the primary study he cites used retrospective reports (adults remembering how much they played when they were younger). Raising two young people in Canada and then in England I have noted several times that play isn’t what it used to be. Growing up (in a remote village in the south of Israel in the 1980s and early 1990s), our parents were working, and no one paid attention to us. It could be portrayed as neglect, but it also gave us the freedom to try out things and learn from our mistakes. As a young mum, I often noticed I was the only one sitting on the bench watching the kids from a distance. Most adults were up on the climbing structure with their charges. Don’t get me wrong, this tendency to “over”-protect our children is a reasonable response to a breakdown of trust in the authorities that are supposed to keep us and our children safe. Nevertheless, it changes the parent-child relationship, and of course, it influences child development. However, this trend didn’t begin in the 2010s; it began in the 1980s.
But Haidt’s claim is more specific. He argues that the rise in phone-based childhood (a childhood characterised by more “screen time” and less unstructured, unsupervised, social outdoor play) is causing an epidemic of anxiety and depression in teenagers. That is a significant theory, and if it is the case, we should definitely, as he suggests, re-examine our leniency in allowing children to spend hours on screens. But is that the case?
Science Methodology 101
This part, I admit, is a bit technical and rather a rant of a scientist about a popular science book. So, if you’re into that sort of thing, by all means, keep reading.
From a scientific point of view, a hypothesis needs two things to be accepted: it must be testable, and it must be supported by evidence. Scientists cannot start making policy recommendations before we establish both of these things.
To be testable, a hypothesis must be refutable. That is, there must be a way to test it scientifically. To give an example of an untestable statement, we can look at Haidt’s book on page 201. He states: “[T]here is spiritual harm, for adults as well as for adolescents, even for those who think their mental health is fine.” Like some of the best Freudian theories, this statement cannot be refuted. According to Haidt, even if you report exceptionally robust mental health, you are actually secretly suffering from spiritual harm caused by digital society.
His main hypothesis that phone-based childhood causes an epidemic of mental ill-health is at least testable, so that’s encouraging. Now, all we have to do is decide whether the evidence supports this hypothesis. We need to show three things: that there is an epidemic of mental ill-health among teenagers; that childhood today is truly ‘phone-based’ (whatever that means); and that there’s a causal link between the two. Haidt fails on all three counts.
First, the epidemic of anxiety and depression. As a social scientist, I have significant issues with Haidt’s graphs. They do not conform to best practices of graph presentation, and there’s a reason for that. Let’s take his first graph, Figure 1.1 (picture taken on my phone). It shows the precipitous incline in teens’ depression symptoms.
I replicated it with my unsophisticated Google Sheets to talk about best practices. Here is my replication:
I’ve put it here so that you can see the data I used, which I took from Haidt’s graph. I estimated the percentages from the graph and probably missed a couple of data points, but they look similar.
Here is what it would look like if Haidt adhered to social sciences best practices:
See the difference? We tell students to always present percentages out of 100 for a reason. When Haidt cuts the y-axis at 30%, we do not see the whole picture. He does this on purpose, of course. My second graph looks much less interesting, much less headline-y.
Another example of presenting data in a misleading way in this book is his presentation of UK self-harm episodes (p. 41 if you have the hardcover, Allen Lane 2024 edition). The very top rate in that graph is 1250 cases of self-harm per 100,000 population. This translates to the rate of 1.25 percent of the population. Even then, it shows incidents per 100,000 population, which means the number of teenagers self-harming is much lower, as many of these are sadly repeated cases. You could call this an epidemic. But you’d be using the word in a very misleading way. I’m not saying these teens don’t suffer. They do, and we should move mountains to ease their suffering. I’m saying there are quite a number of teens (at least 75% of teens who answered the survey as per the first graph above) doing fine. Of course, Haidt won’t take their word for it.
Second, the phone-based childhood. Haidt cites studies saying teens report being on their phones “all the time”. Teens tend to say things like that. You tend to view everything in black and white as a teen. All my friends have phones. Everyone is going to the party. I’ve seen no corroboration of these feelings in hard data (e.g., screen use). That said, it’s entirely possible that teens have a perception of being online all the time, which is a different concern. It’s a concern we should examine and discuss in a nuanced way. I go deeper into this topic in Part II.
Third, a causal link between screen use and mental ill-health. A true causal link is ideally demonstrated with a Random Control Trial. In this hypothetical study, we would take a group of children who never had a smartphone and divide them randomly into two groups: one group would get smartphones and access to social media, and the other group would not. We would measure their mental well-being at the beginning of the trial and then measure their well-being after some time (ideally at least three months, as emotions tend to fluctuate and impact well-being). The idea is that the random division mitigates any other factors. In this context, I want to talk about two studies.
The first study is an experimental study done with university students in Pennsylvania. The researchers measured screen time use and a host of well-being measures. Then they divided the students into two groups. The experimental group limited their social media use to ten minutes per platform per day for Facebook, Instagram, and Snapchat (no reference to TikTok, but the study was published in 2018—before TikTok was popular in the US). The control group kept using their phones as usual. They did this for four weeks, tracking the phone use of both groups using battery screenshots (rather than a self-report, which is a good thing). They report that “limiting social media decreases depression and loneliness”. However, look at the graph below (taken from the article). We can see that for students who started the study with low depressive symptoms, there was no difference between limiting social media use (experimental group) or using social media apps as usual (control group). The difference was for students who started the study with high depressive symptoms. In other words, when young people are already battling depression, it may do them good to lay off social media for a while.
The second study I want to discuss was published a year earlier than this experiment. Przybylski and Weinstein, working at Oxford University and Cardiff University, respectively, surveyed a large sample (about 120,000) of adolescents in the UK. They examined the correlations between digital screen use and well-being. They found that moderate screen use is actually associated with higher levels of well-being, whereas heavy screen use is where we start seeing adolescents’ well-being begin to deteriorate.
Research methodology may sound boring, but it matters. When Haidt truncates his graphs at 30% to make changes look more dramatic or uses phrases like 'spiritual harm' that can't be tested, he's doing more than making presentational choices. He's building a narrative that ignores the complexity of teens' relationship with technology. Most teens in his own data are doing fine - a fact his dramatic graphs obscure. And while some teens absolutely struggle with phones and social media, the solutions aren't as simple as Haidt suggests. In my next post, I'll tackle his first big oversight: you can't just ban phones in a world where digital literacy is becoming as essential as reading.
Comments