04 Jan ESG and the Replicability Crisis
The following commentary/analysis is one I wrote in my capacity as a senior fellow at “the nation’s oldest consumer protection agency,” Consumers Research, where, among other things, I compile a weekly letter for public pension-fund managers. I am sharing it here today because I thought it might be useful to some of you.
Oh.
Long before now-former Harvard President Claudine Gay’s record threatened to expose a plagiarism scandal in higher education research, academia was already plagued by a much larger and much more serious research scandal, the “replicability crisis.” As it turns out, a significant percentage of academic research papers published over the last several decades contain flawed methodologies, producing conclusions that are dubious at best and are, in many cases, irreplicable. Last Spring, Semafor noted a part of this scandal as follows:
Around a third of studies published in neuroscience journals, and about 24% in medical journals, are “made up or plagiarized,” according to a new paper.
The research, referred to as a preprint — meaning it has not yet been peer-reviewed — looked at 5,000 published papers, as first reported by Science….
Everyone in science knows about these problems. But surprisingly few people do anything about it.
The microbiologist Elisabeth Bik has an extraordinary ability to spot duplicated or faked images in scientific journals: She has spotted hundreds over the years. But she told Nature that even five years after she’d reported the fakes to the journals, most of them had not been dealt with.
The Oxford psychologist Dorothy Bishop says this matches her own experience: “If one points out academic malpractice to publishers or institutions, there is often no reply.”
There are plenty of other issues with scientific research and publishing. Journals take scientists’ work for free or even charge to publish it, then charge them again for access.
Editors at one journal walked out recently over “unethical” publishing fees. And the demand for “positive” results incentivizes scientists to hack the data up until they find something. Those are deep systemic problems within science.
But outright fraud, you’d think, should be easy to fix if detected. And yet the scientific community often ignores it….
This is considered a “crisis” or a “scandal” in the hard sciences – in the above case, neuroscience and medicine – because in those fields, research and its results matter. Real-world, sometimes life-and-death decisions are often made based on the research done in the hard sciences. If this research is fraudulent, if it is not replicable and, therefore, not “scientific” at all, then the resulting consequences can, indeed, be quite serious and quite scandalous.
Although the same issues of fraud and replicability exist in social science research – possibly in significantly greater proportions – rarely is this referred to as a crisis or a scandal. This is largely because the results of research in the social sciences often don’t matter. In an academic universe in which “publish or perish” remains defined dogma and which strives relentlessly to “create” “new” knowledge, the results of research are often 100% entirely irrelevant except in the battle for promotion and tenure. This is especially true in newer fields of the social sciences – racial and gender studies, for example – in which research findings are often extremely narrow and non-robust.
This is not to say that social science research is of no value or that it never matters. Most social scientists are conscientious and earnest. And sometimes social science research has significant implications within and even beyond academia. This is usually the case when a researcher uncovers or deftly explains a profound truth, when an industrious social scientist finds a way to discuss a phenomenon in terms that resonate beyond the narrow specifics of academic specialties. Occasionally, however, social science research becomes important and takes on significance far beyond normal bounds because it produces the “right” results, because its findings support all the proper prejudices.
One of the most important studies conducted in the fields of finance and business management in the last couple of decades was undertaken by Robert Eccles, Ioannis Ioannou, and George Serafeim, the results of which were published in 2014 as “The Impact of Corporate Sustainability on Organizational Processes and Performance,” by the journal Management Science. Andrew A. King, the Allen and Kelli Questrom Professor in Strategy and Innovation at the Questrom School of Business at Boston University, recently summarized the basic findings and the impact of this study/paper thusly:
Its authors summarize their analysis as demonstrating that “high sustainability companies significantly outperform their counterparts over the long term, both in terms of stock market and accounting performance” (p. 2835), and this assertion has been widely publicized and used….
Of the articles published after 2013 in the prestigious journal Management Science, it is the most cited. It has been referenced by the head of the Securities and Exchange Commission and by executives in Wall Street firms (Lee, 2020; Blake, 2020). Al Gore (former Vice President of the United States) and David Blood (Founding Partner of Generation Investment Management), claim its findings show that “sustainable companies outperform their unsustainable peers in the long term” (Gore & Blood, 2011). In testimony before the U.S. Senate, it has been used to support the argument that “leaders in sustainability ‘significantly outperform their counterparts over the long-term.’”
In short, this study is one of the foundational documents in ESG-world. It continues to provide much of the justification for the equally foundational claim that ESG is about risk management and long-term value rather than mere politics. Unfortunately, as Professor King explains, it’s also mostly junk:
[A]fter reviewing the report, I conclude that its critical findings are unjustified by its own evidence: its main method appears unworkable, a key finding is miscalculated, important results are uninterpretable, and the sample is biased by survival and selection. After correcting these problems, I replicate the original analysis and conduct additional empirical tests. Despite considering estimates from thousands of models, I find no reliable evidence for the proposed link between sustainability and financial performance.
As a result, King concludes that “neither…[the] original report, nor my analysis, should be used to support the claim that high sustainability companies outperform their low sustainability counterparts with respect to either stock return or accounting performance.”
Needless to say, this is brutal. But it’s hardly unexpected. The claim that ESG is purely about performance and risk assessment has always been dubious, although one of the authors of the paper in question, Robert Eccles, continues to make that claim in print and continues to mock and belittle those who question it.
Part of the problem here is that many in the ESG community suffer from what Hayek called the inability to distinguish science from scientism. They confuse math with science and expect their results to be taken as established fact, even, as in this case, when they’ve done the math wrong. The bigger part of the problem, though, is that many in the social sciences are guided in their research by wishful thinking. They find ways to manipulate their research to produce the results they hoped to find in the first place. Most of the time, no one notices, and no one cares. Every once in a while, however…it matters.
The American university is broken – and the impact of this brokenness can be significant and widespread. ESG is a case in point.
