When I lived in a dodgy part of Washington, D.C. in the early 90s, I used to get my food from either the pizza joint six blocks down the street, or Dottie’s Liquor on the corner of the dilapidated English basement that I called home. My hours were irregular (hey, I was young and having fun), but I could always count on Dottie’s Liquor to furnish more than a six-pack. I could buy high-fat, high-sodium canned concoctions called “soup” for 99 cents, sugary fruit drinks, and the occasional yellowed roll of toilet paper that the elderly African American cashier would silently pull off the dusty top shelf that hung precariously behind the counter. I didn’t care much about my diet—I was a bike messenger—I could burn off anything. And I never noticed the young Latino and African American families that would crowd the aisles (it was a small store, it only took one family to do that), with kids in tow. It never occurred to me that this was their grocery store because back then, there were no other options within walking distance.

As I got older, I began hearing about “food deserts,” pockets in low-income neighborhoods where a paucity of fresh food and vegetables was the norm. And what little quality food there was cost a fortune. The media coverage would typically feature a few quotes from a researcher and perhaps a food advocate, along with a reasonable-sounding statistic in support.

That framing fit neatly into my personal narrative. I found myself in quick agreement when food activists decried the situation. I never questioned the statistics, either. And when policy makers joined with grassroots campaigns to turn advocacy into policy, I supported it with a sense of satisfaction—in my lifetime, things were changing. Move over Dottie’s Liquor. Farmer’s market produce, come on in. And then, earlier this week, an article on Slate claims that food deserts do not exist—that the claims were made based on inaccurate interpretations of various research studies.

The psychology of data

The idea that by introducing healthy, fresh food one could measurably improve poor health outcomes in low-income populations seemed, not too good to be true, but rather too good to question. So, when Slate published their article questioning claims made about the existence of food deserts, I was surprised and disappointed.

And therein lies the psychology of data. When it proves something you agree with, how likely are you to question it? For a lay person, it’s a question of how well-informed we are. For a policy maker, the burden is much higher.

And the challenge we face, no matter how well informed we attempt to be as members of the general public, is that we are hostage to the facts that trusted messengers—among them, policy makers, journalists and advocates—put in front of us. (For a discussion of the designer’s role, read this previous post.) That’s a big responsibility for them, and the responsibility for us is to question them and hold them to it.

More important than debating the merits of whether or not food deserts truly exist, is examining how the claim of food deserts came to be proven and then disputed. It allows us to walk through the evolution of an idea from the ground up (from advocates, to policy makers, and back to us, the public), and understand the role that data and data literacy plays out across the different actors.

And that’s what this post is about.

Let’s take a quick look at the Slate article and a few of the studies that it references. These studies examine food deserts via the lens of health outcomes, diet and the availability and proximity of healthy food. According to Slate, the increase of healthy food initiatives (those aimed at reducing food deserts and thus, disparities in the health outcomes of low-income populations) has risen sharply in the U.S., due to the largely successful efforts of food activists who lobbied for fresh, affordable food in poor neighborhoods to reduce disparities in health outcomes of low-income people. The charge has even been taken up by Michele Obama.

How did food desert initiatives originate?

In Britain in the mid-90s, there were a few studies (note that Slate describes them as “preliminary”) that suggested that a “a link might exist between distance to a grocery store and the diets of poor people.”  Already you can see how easily a well-intentioned health advocate or policy maker can jump to the conclusion that a correlation exists between poor health outcomes and lack of access to fresh, affordable food available from a local grocery store. And this is exactly what happened. The Slate article traces the history of the food desert movement. In a nutshell—a few studies in Britain in the 90s were followed by a Pennsylvania law in 2004 that funded fresh food programmes, followed in quick succession by adoption of similar programs in 22 U.S. states (to date), according to Slate.

But the data cited by advocates in these studies doesn’t entirely support that correlation. Here is a summary of a few studies that refute this (one of which is written by an author who wrote a study that is often misquoted).

A widely-cited study used to support the existence of food deserts is inconclusive

The Journal of the American Medical Association’s (JAMA) 2011 study, “Fast Food Restaurants and Food Stores: Longitudinal Associations With Diet in Young to Middle-aged Adults: The CARDIA Study,” examined 15 years of longitudinal data (repeated observations over a period of time) from a cohort (group) of 18- to 30-year-olds in the U.S.

Researchers analyzed how often individuals ate fast food, how much of it they ate, the quality of their food diet, and how much they ate of fruits and vegetables, as well as the availability of fast food restaurants and supermarket grocery stores (measured at different distances). You can read the study for yourself—but it concluded that the evidence showing a correlation between bad food resources and poor diet and obesity are mixed, at best.

“Neighborhood supermarket and grocery store availability were generally unrelated to diet quality and adherence to fruit and vegetable recommendations, with similar associations across income levels.”

So as you can see, the conclusions from the JAMA study didn’t quite square with how they were being used by policy makers—other factors were at play. Low-income men were more apt to consume nearby fast food more (and, conversely, did have a better diet when there were supermarkets nearby), but low-income women were not statistically significant. Middle-income individuals showed varied significance (described by the researchers as “weak” and “inconsistent with significant counterintuitive associations in high-income respondents”).

Tensions between the aspirations of social change and the reality of evidence-based research

An essay in the Journal of Epidemiol Community Health, “Good intentions and received wisdom are not enough,” features a powerful (and damning) indictment of the touchy dynamic between the pressures of social change and the research that underscores it. From the authors:

“There is a common view amongst social and public health scientists that there is an evidence-based medicine juggernaut, a powerful, naive, and overweening attempt to impose an inappropriate narrow and medical model of experimentation onto a complex social world.”

The essay pointedly calls out the resistance (“hostility”) of social scientists, health policy makers and advocates to attempts by researchers to use the evidence-based approach traditionally used in medicine, but not public policy (systemic reviews of data or experimental designs, for example). Why? The authors of the essay claim that social change advocates view the real world as too messy and a far cry from the controlled environment of academic and medical research. This applies, the authors note, particularly to what I’ll describe as social issues of the day—issues where good intentions and raw emotions are at the surface as well-intentioned advocates and policy makers attempt to use data to alleviate the very real and valid human suffering that is so visible to all of us. Read it here.

Assertions quickly become facts in the public sphere

The introduction to “‘Food deserts’—evidence and assumption in health policy making,” by Steven Cummings and Sally Macintyre (British Medical Journal) is worth quoting word for word:

“Assertions can be reported so often that they are considered true (“factoids”). They may sometimes even be used to determine health policy when empirical information is lacking.”

It’s telling that this was written in 2002, approximately two years before the elimination of food deserts became a part of American public policy.

The paper attempts to track the rise of the food desert assertion in the UK. It points to three main UK studies that were frequently cited by advocates and policy makers (two are noted above) and systematically dismantles what it characterizes as erroneous assertions by advocates to correlate food deserts with poor health outcomes. How? You can read about it for yourself, but here’s one example.

The study found that, though healthier food costs more than unhealthy food in low-income areas, both actually cost less in low-income areas. Advocates, however, routinely cited a study but claimed simply that good food cost more than bad food. The nuance here is an important one, and the authors point out that it was never made.

The authors also discuss a different study that has been cited by advocates that is also not as conclusive as widely reported—the study shows that small grocery stores have more expensive food and a narrower range of options—but doesn’t compare how this plays out by income distribution (low- versus high-income neighborhoods).

Lastly, the authors refer to a 1992 study (also frequently cited) which compared the cost and availability of a basket of healthy versus unhealthy foods in poor and more affluent neighborhoods. The study (ironically, also published by Macintyre) was simply a pilot study and didn’t use random sampling, significance tests, and other statistical methods that a more robust study would have used. It was, after all, only intended to be a pilot study. Macintyre herself points out that it was widely (and wrongly) cited across the UK and America as evidence of food deserts.

I’ll leave you with another quote by Macintyre:

“If the social climate is right, facts about the social world can be assumed and hence used as the basis for health policy in the absence of much empirical information.”

That pretty much sums it up.

In fairness, these studies also raise many questions. Who are the authors, how are they funded, and how legitimate are the claims they themselves make? But the questions posed by the authors of these studies serve to at least merit a closer examination of the relationship between data and policy.

Implications for social change advocates and public policy

What are the implications for those of us who care about social and public policy?

Not being critical thinkers and examiners of data puts our credibility on the line in the arena of public perception. It arms our opponents with legitimate counter-criticism to our views.

It can distract us from other, more viable paths to social change that truly can be substantiated and measured. And it obscures the broader, but as important, good intentions behind our convictions. In this case, for low-income people who disproportionately suffer from poor health outcomes, what are the contributing factors that have been credibly examined (long hours working several jobs, the stress and worry that accompany poverty, or the lack of education about what constitutes good health habits,)? That’s where public policy can be directed.

Valuing proper research, taking the time to understand it, and respecting its limitations strengthens our arguments

It’s tough for me to write this post. I’m Hispanic and I have spent my entire career in the advocacy and public policy field. This is very much my world and I see every day how hard my friends and professional colleagues toil to right the wrongs that society allows. The passion, integrity and commitment that advocates and policy makers bring to their work can not be underestimated. And that’s why I write this, because valuing proper research, taking the time to understand it, and respecting its limitations only makes our positions stronger.

In an earlier post, I wrote about how lack of data literacy can put social change organizations behind the curve in advancing their goals. In this case, it can do the same to good intentions, and good outcomes.

But let me conclude by saying that just because the data may not support the public narrative of food deserts, that doesn’t mean that it’s okay for poor people to eat bad food. That’s a patently unfair situation for those who live in poverty. There are many benefits to eating fresh, affordable fruits and vegetables. I make that assumption from what I read in  mostly reputable news sources. I further assume that avoiding high-fat, low-nutrition food that delivers scant nutrition for the money is good for other reasons. At least, I want to believe that. But as good as that sounds to me, perhaps I should do a little digging to substantiate my convictions.