Over at Slate, Daniel Engber had an interesting and instructive article a while back entitled “Bad Footnotes Can Be Deadly” on how the current opioid crisis has been aggravated by the misunderstanding of a letter to the editor of a medical journal and its misquotation over decades in the medical literature, with the error propagated and compounded by researchers who failed to check the original source.
Engber’s article on the error’s persistence and influence documents a process not unlike the formation of urban legends and faith-promoting rumor. But if you’ve ever done scholarly research, you should also be nodding your head in recognition (and if you aren’t, you’ve never done scholarly research, or not enough of it yet, or at least not the right kind of research). You come across careless footnoting almost inevitably whenever you study the secondary literature of a narrow field deeply enough. Eventually you develop a sense for it. Indirect citation is sometimes necessary but always suspect, and if two different authors use the same indirect citation, I assume there’s something fishy about the original.
Mormon belief is intertwined with history, giving the field of Mormon Studies a rare and enviable intensity and an informed non-scholarly audience, but it also gives church members a stake in scholarship and a need for a basic understanding of how scholarship works. Part of the formation of scholarly instincts involves learning to balance respect for scholarship with appropriate caution, holding things that cannot be personally verified in suspension to a certain extent.
When I’m digging into the literature for my own research or reviewing a book or manuscript, some of the basic questions I have are: can I follow a statement back to secondary literature through those references? can I follow the secondary literature back to a primary source? are there any references at all? A slightly trickier version of this question is: are the things that look like references actually references? And are the things that look like references in the cited secondary literature actually references themselves? Sad experience teaches us that not every superscript numeral actually points to a primary or secondary source that documents an author’s assertion. Some authors regularly use their footnotes or endnotes for many things besides providing sources that can be accessed for verification by any normal human being.
Even if complete scholarly transparency is impossible, it remains a goal worth pursuing. Making secondary literature more easily available remains an important but elusive goal: proprietary research databases are often inaccessible outside of the research universities that can afford them, and open access turns out to be expensive. Digitizing archival sources is even more important.
Bad footnotes are usually not a sign of deception, just of human limitations. In most cases no one is hiding anything, except maybe laziness. Sometimes the requotation sounds sexier than the mundane original, and sometimes the original is in an obscure language or inaccessible in a distant repository. Being cut off from full-text databases or interlibrary loan or just the old issues in the stacks can be a huge problem. So when you’re reading, in Mormon Studies or anything else, keep one eye open for human frailty.
Points well made. I am far from a professional researcher, but on topics I am researching for various purposes, I have often found it difficult to follow a footnote to one source . . . that then sends you to the next source, etc. One on topic, the search actually turned into a circle when the fourth book the footnotes sent me to sent me back to the first one!
Jonathan, could you provide some examples of bad footnoting in the Mormon context to help illustrate your point? Otherwise this just feels more like an underdeveloped idea than a useful concept.
Not quite as on point, but I remember hearing Elder Holland quote Winston Churchill in Oct 2000 conference. I loved the quote so much…only to find that Elder Holland did not cite it. Subsequent searches identified that the quote cannot be authenticated at all. Whoops.
Add to that the story of the three 18 year old boys who gave their lives helping the handcart crossing. President Hinckley cited that story in Oct 1981. Again, whoops. In this case, the stories that actually occurred are just as inspirational; its too bad that an inaccurate account was created to embellish the truth.
And where are we at with the white horse/Constitution’s thread prophecy? Haven’t the citations been debunked or at least viewed with serious doubt now?
Amen. Though you are more generous in the end than I.
I keep human frailty in mind every time I read something, whether it’s a Facebook post or scriptures, especially scriptures.
It’s kind of shocking how many sources get quoted or cited without the author having examined the sources. Leads to all sorts of problems. Things get repeated more like a totem than a source. I can think of one post I’m working on that is an example of this – oft quoted sources that clearly few examined closely.
In these days with the internet and so much on either public repositories or at least sites like JSTOR there’s far less excuse. That said, I do with university libraries would allow access to their digital collections to non-students. (Yes, HBLL I’m looking at you)
I suspect the divide between those at universities and everyone else (despite their taxes heavily paying for research) is why you’re having the rise of services like scihub – although like torrent sites it keeps getting shut down.
I am the editorial director for the oldest Mormon studies journal. I’ve been here for 11 years now. I edit a lot of articles (and a few books) in the field of Mormon studies. Let me just say for the record that in that time I have not seen an article yet with all the footnotes correct. Most of this is just human imperfection or sloppiness, and we are all guilty of this to one degree or another, but one article provides a cautionary tale. It was in a field I call indirect apologetics. We have student interns who try to track down every source. Sometimes the sources are so obscure or hard to find that we just have to trust the author (and our own instincts). But when I got the source-checked copy of this particular article back from our intern, in 18 different places she had written in the margins, “the source does not say this.” Basically, the author was playing fast and loose with both his sources and his interpretations of them. In trying to come up with a tangential defense of the Book of Mormon, he was basically making things up and loosely connecting them to some sources. This is rare. Interpretation is always an issue, and so is the selectivity some authors employ to avoid details that undermine their thesis. This happens quite a bit in Mormon studies, when scholars begin with a conclusion and then try to build a case to support it. I really admire those scholars who are able to analyze the evidence, whatever it is, and then draw conclusions, even when those conclusions do not necessarily support what the scholar may have anticipated going in.
So, yes, footnotes can be full of landmines, or meaningless fill dirt. The best authors are always the careful ones who don’t stretch evidence beyond where it willingly goes and who don’t try to impose their preconceived conclusions on the evidence through tortured interpretation.
Wally, if you’re referring to a quarterly headquartered in Utah Valley, I couldn’t agree with you more. When I was an editorial intern there back in the early 1980s, I was surprised to discover that everyone makes mistakes. Everyone. No exceptions. One simply can’t be too careful.
I don’t think errors in footnotes is the problem. That’s basically unavoidable. Rather the issue is in assuming there aren’t errors or worse in assuming the author correctly represents the source.
To be fair though, the interpretive tradition matters a great deal. So I sometimes object to Nibley’s footnotes. But he’s coming from the structuralist tradition like say Joseph Campbell. That tradition sees the structures as separable from the details. So Joseph Campbell doesn’t think he’s misusing his sources just because he decontextualizes the references. Now I see that as a problem, but I see that as a problem with structuralism in general. Yet structuralism was hugely influential up through the 60’s at least. If not longer. (Campbell’s The Power of Myth came out in the 80’s)
I am sometimes lectured by strangers on the importance of citing sources on my blog, or on a couple of FB pages where I post brief clips from the writings of John A. Widtsoe and Heber J. Grant. Sometimes they say they want to share a clip, but won’t share an unsourced one — good for them; I can understand and support that, if it’s indicating scholarly care. On the other hand, I don’t believe for a moment that a casual reader would check out citations if I gave them.
Too, I get demands — there is no gentler applicable word — from academic readers who want my sources so they can use something I’ve posted as their own work, in their own publications. If I gave them citations, I wonder how many of them would cite to the archival source *as if they had reviewed that source.* There is never — never, ever, ever, ever — any indication that these people recognize that I support myself through my research — no offer to pay me for my work, or request to hire me, merely demands that I turn over my sources for their personal benefit.
Examples; the last one from a BYU professor who *knows* I live by my research work:
The informality of social media, and my use of it as a [mostly failed] advertisement for my skills, excuses me, I think — I hope — from providing my sources for anybody and everybody to adopt as if they had done the work themselves. But you had better believe — and any of my past clients can confirm — that I have complete and scholarly citations for every document in my files, which could be had by scholars who treat me as a colleague rather than as a peon bound to give them my labor without compensation.
Like MH, I wish you would give us an example, Jonathan, to make clear what you are objecting to here. I can think of several possible applications, without quite knowing what you’re thinking about. I just hope that my own refusal to post my sources in the casual world of blogs and FB pages isn’t included in the condemnation.
“I do with university libraries would allow access to their digital collections to non-students. (Yes, HBLL I’m looking at you)”
Do you think the license fees the HBLL pays for these digital collections would be the same if the HBLL made them available to everybody?
N.B. Clerk, as someone outside the university community, I can’t get access to those collections even by paying a proportional share of the presumably increased licensing fee. Even paying to be a “friend of the university” doesn’t get me access to subscription services or to ILL. And as a potentially relevant note, since we’re talking about BYU, I note that for many years before you could get a free (pared down) Ancestry subscription as a member of the Church, the Family History Library provided full access to Ancestry to anyone who came into the Library, regardless of whether or not they had a paid personal subscription.
N. W Clerk, I think the issue of license fees charged to universities for content produced by universities using tax payer dollars is a huge scandal. So don’t get me started there. Open access journals should be required for tax payer funded research publication IMO.
However as Ardis notes, the consequence is basically the extreme difficulty of doing research for anyone not associated with an university. It’s precisely that which is driving sites like scihub and is also why many fields like physics went to open repositories like arxiv decades ago when the implications of the internet became clear. (Arxiv started up at Los Alamos when I was there in the early to mid 90’s. It later moved to Cornell University for various reasons I was never entirely clear upon.)
While I can’t support piracy sites like scihub obviously, it’s pretty hard to shed a tear for the industry scihub is undermining whose entire business model is the classic case of economic rent seeking. (BTW despite the name, scihub mirrors basically everything on JSTOR and not merely science papers)
If, for example, I Google “‘ScienceDirect’ ‘students, faculty, and staff’ ‘off-campus'”, I get about 10000 hits for a wide variety of universities saying off-campus access of ScienceDirect is limited to their students, faculty, and staff. I don’t think BYU invents the terms of these contracts; I think they are terms required by the database owners.
I’m not asking you to shed a tear for academic publishers. I’m telling you that you that the HBLL doesn’t have the power to grant your wish of access to their licensed databases.
Yeah, that’s possible, although I wonder what it means to be a student or faculty. There ought be some way for people doing independent research to do things. But I certainly agree the key culprit isn’t the HBLL but sites like JSTOR. Now JSTOR does have a public offering that they recently offered at $20 a month. So I should limit my complaints somewhat. That’s huge compared to what they used to offer which was ridiculously priced library plans. I think that’s due to pressure from things like scihub, much like back in the 90’s Napster drove the iTunes store as a legitimate alternative.
But of course JSTOR is just one of the databases that the typical university library offers. And it is, from what I can tell, the most progressive in terms of public access. Muse doesn’t have any public offerings that I can see for instance. However according to their FAQ, the Muse license applies to alumni but BYU doesn’t offer it to alumni that I can see.
Years ago I was glancing through an anti-Mormon tract that insisted numerous times, with footnotes, that certain language in the temple was “directly from Satanic worship services.” Intrigued, I thumbed over to the citations, and found at each footnote “Based on personal writings in the author’s possession.” Welp.
Thanks for the comments and discussion. Oddly enough, I don’t feel like denouncing respected, award-winning scholars as horrible footnote writers in a casual blog post. Some other time, perhaps?
And again the problem is not with things (rumor, blog posts, conference addresses) that don’t typically present themselves as academic works with scholarly apparatuses; each makes its own kind of claim and requires its own kind of analysis. The problem is with relatively recent peer reviewed books from reputable presses and articles published in respected journals, formatted according to proper Chicago style and strewn with things that look, at a glance, like footnotes. I’m heartened by Wally’s comment that people do sometimes double-check the footnotes, but sometimes things slip through – both on the apologetic and the antagonistic side, and from established scholars and amateur outsiders.
I’m sure I’ve done the same kind of thing before. It’s difficult and time-consuming to trace every well-known fact back to its source, so you just throw in a reference to a page in a recent article that says the same thing, and no one thinks twice about it. Then later you start poking around in the place where the well-known fact was first stated, and it’s not nearly as clear-cut as everyone has been treating it. You’re more cautious the next time around.
It should be getting easier to track down sources, and in some ways it is. We have unprecedented ease of access…to 19th-century (and earlier) books and journals, thanks to Google Books. Everything later than that is hit and miss, mostly miss, due to copyright laws intended to help Disney being equally applied to Archiv für niederrheinische Kirchengeschichte (3. Reihe) 37 (1921). Open access sounds good, but it turns out that it’s extraordinarily expensive; someone has to pay for those servers, and page fees for good open-access journals run into the hundreds of dollars. So advancements in scholarship end up behind commercial paywalls instead. I do think university libraries should try to accommodate community members, but it’s hard to make the case that the university should pay, say, $10,000 more per year for a license that is not strictly limited to faculty, staff, and students. I wish I had a solution.
Okay, so here’s an anonymized actual case that probably illustrates the problem:
I worked with someone a year or three ago who *did* want to track everything back to the source, rather than merely repeating the citations that appeared in earlier studies of the topic (say, 6 or 8 earlier studies). What we found included: (1) an error in the citation of the first historian to refer to one event … with the erroneous citation repeated in every subsequent study, indicating that every last one of them had merely quoted the first author as if they had actually seen the original, when they had not. (2) An error in the citation of the second historian to refer to another event … with some of the later scholars reproducing the error, while others reproduced the original citation. No way to know if those who repeated the original citation had actually seen it, but certainly the ones repeating the erroneous citation had not. (3) A very important document was correctly cited by the first and all subsequent historians … but when my man went back to the original document, he discovered an unnoticed, not previously reported, highly significant bit of data appearing in another paragraph on the same page, which had been overlooked or unrecognized by the original historian, and which, apparently, nobody else had seen because presumably they hadn’t looked again at the original.
Does that fit what you’re faulting? It’s primary rather than secondary sources, but I hope it still fits.
Footnotes are like banknotes. British bills still carry the written, “I promise to pay the bearer on demand the sum of X pounds,” and originally all paper currency was supposed to be redeemable in precious metal. In modern practice the banknotes themselves carry the value, to the point where we ask the price of gold in dollars instead of asking how much gold a dollar is worth. Gold bullion is a pain to lug around. The convenience of having paper bills themselves be legal tender is too hard to resist—at least as long as everyone agrees to it. It’s getting that way with footnotes. Instead of asking how good your cited sources are, we ask how many citations they have.
In ultimate principle a footnote should tell you where you can get your hands on the physical evidence to see for yourself, but scholars’ lives are so much easier if we all agree to accept the Wikipedia standard, by which anything published counts as authority, no matter how great an idiot the author was. If your subject is literary enough that its physical evidence actually consists of books, then tracking down the raw evidence may be easier than hiking to an archaeological dig, but reading original sources is still tedious. They’re usually way too long, and by the time they finally get around to coughing out the soundbite you want, they’ve probably surrounded it with so much weird old-fashioned context that it doesn’t really mean anything like what it sounds like. It’s much handier to treat the citations themselves as authoritative, regardless of what the cited sources actually say. As long as everyone agrees to play like this, we can all have wonderful scholarly debates with a whole lot less trouble.
[I do not want the gold standard for money. Clark, ArXiv moved from Los Alamos to Cornell with Paul Ginsparg.]
Jonathan: Open access sounds good, but it turns out that it’s extraordinarily expensive; someone has to pay for those servers, and page fees for good open-access journals run into the hundreds of dollars.
NSF grants typically pay for those rather easily. There are public repositories for the social sciences as well. It’s the humanities that are the big standout. The problem is more getting people to use them. Server costs these days are extremely cheap. That’s why there can be multiple pirate full mirrors of all the journals that Muse and JSTOR provide with free access.
Ardis, I was going to say just from your description I had a pretty good idea of the book — a major work on Mormonism in the 90’s where I’d looked up a bunch of the footnotes to find selective quotations that often were undermined by sentences mere inches away. However then I realized that is probably far more common than I want to believe. I suspect in the 90’s the sources were much harder to look up. Now that they are easy to look up a lot of works that still get regularly cited should be used with caution. (Like Jonathan I’m not sure I want to name names – especially not without first spending the time to relook up all the footnotes!)
James, thanks for that bit of history of ArXiv. I also agree with you on the danger of citing papers rather than original sources. Secondary cites are always dangerous.
So the issue rankles people enough to blog about it (as it should), but not enough to cite examples of? While I respect such politeness, I take it as another sign that Mormon studies has a ways to go yet before arriving at it’s long-desired day in the mainstream academic sun.
Well, from the comments here, MH, it seems as though no one here wants to give examples without doing the massive amount of work re-researching the works in question would take. And if they did give examples and were wrong, they might open themselves to liability in tort.
I do remember a post a while ago here or at Wheat and Tares or BCC (or somewhere – see me not citing anything ;) ) that have examples of bad footnotes in the Presidents of the Church series. Not academia, but interesting, and an example.
MH, my problem is that all my books are in storage while my basement gets worked on. But if it’s any consolation I’ve got something I’m writing about a footnote that I might put here or try and submit to the interpreter.
But a good example of questionable footnotes might be Compton’s review of Nibley’s World of the Jaredites
Ardis, yes, that’s exactly what I’m talking about. I once ran into exactly the same kind of thing in grad school, except one of the scholars involved was Jacob Grimm.
Clark, since Mormon-related topics call squarely under the humanities, you can see the problem. Not much NSF funding for Mormon Studies. Humanities research is typically supported with little or no grant funding, and a lot of universities won’t have funding for page fees in the humanities. So the choice often comes down to publishing in a well-known, high-prestige traditional journal, or funding open-access publication in a low-prestige unknown journal out of your own pocket. As for expenses, not long ago the director of a major research library in Europe was showing off their digitalization program and mentioned that their server costs (for many high-resolution scans of primary sources) ran into the hundreds of thousands of dollars per year.
MH, what an odd comment. Taking my a blog post as a guide to the health of Mormon Studies is completely misplaced, since I’m hardly involved in the field. This is a blog post reminding everyone to do their homework – so if you need examples, you’ll need to look through the archives or at the old FARMS reviews. While that’s a style of review I usually don’t care for, they knew how to dissect a footnote.
Jonathan, I’m not sure I buy that. I mean clearly that is the status quo but I think that’s primarily due to the culture itself. The cost is not that high and I think making something like ArXiv would be a much better use of NEH money than a lot it gets spent on. A side effect of the publishing problem is that of course the humanities have become more and more disconnected from regular culture. This in turn leads to more and more esoteric jargon and theories. But by and large, I think the humanities like this despite a cry or two here and there. That is, I’d lay good odds even if there was a general repository like there is for math and physics that there would be tremendous pressure not to use it.
As to scans, having been involved in that industry that’s just nonsense. I’m sure they said that but it’s just not true. I’ll lay good odds that most of that $100,000 was wages. But the technology to automate most of this this is there. Further that’s dealing with old text. I’m talking about new papers where you don’t need to scan at all merely save relatively small pdfs. Look at the typical JSTOR paper. Most are 100-300K in size. A paper with many illustrations is maybe 1MB-2MB. I can buy a 5 terabyte hard drive for $100. The software to distribute loads across servers is free and very mature. And that assumes you handle your own server rather than simply using AWS or a similar service. Again note that scihub for free hosts every JSTOR and Muse paper.
In law school, I spent considerable time double-checking the footnotes of law review articles that other people had written, prior to publication. Not only did I make sure that the formatting was correct, but I pulled up each individual primary source to make sure that it was actually cited correctly. Some of these I could find online; others I could find in the law library. Sometimes I had to make the long trek to the main university library to track down the specific book I needed so that I could verify it was quoted accurately.
Is there a process like this in, say, Mormon Studies?
Yes, misquotation is a huge problem. But a large amount of literature published on matters related to Mormonism end up in journals that cannot be accessed without paying an exorbitant price or having access to a university library and/or university password which allows them access to these journals, so I have a hard time caring if this stuff is referenced well or not. The most important work on Mormon studies is made available online. Wikipedia articles on Mormonism inform people more about Mormonism than academic journals. Blogs and websites that cover Mormonism are far more influential not just on the general discourse but also intellectual discourse about it. The goal should no longer be about getting footnotes right for stuff that hardly anyone reads. The goal should be getting more primary source material up online in its original format so that blogs and websites can correctly reference it and provide a link for people to see. I see the traditional Mormon Studies where scholars publish in peer-reviewed academic journals as in decline. Scholars (meaning a select few people employed at a university) only publish in these journals as proof of academic contribution to keep their jobs. Another thing is that some of these academic articles published in professional journals are so arcane and difficult to understand that they couldn’t possibly have an impact on discourse. So I could care less whether these have the footnotes right. It’s all about the stuff published online.
Is there a tolerance level for citations? It seems most of the problems come when a source turns out to be bad and a lot of work has been done based on that bad source. Maybe a way to help would be to request more diversity in sources, lowering the chance of one point ruining your entire work.
I think this problem (such as it is) will reduce as we get more information digitized. With information online, we should get to the point where the footnotes are links to original sources, so anyone can easily go back and forth at will.
Steve S, which journals are you thinking of? There’s a lot on JSTOR and the main Mormon focused one, Journal of Mormon History, has all the issues through 2014 available for free at Utah State. http://digitalcommons.usu.edu/mormonhistory/vol40/iss4/ And recently the entire journal was put up on JSTOR http://www.jstor.org/journal/jmormhist
BYU Studies appears to be full open access now.
But it seems like more and more Mormon history is being published at the main non-Mormon journals and thus available via JSTOR and Muse.