My PhD dissertation was about bias in cost and ridership forecasts for transit projects. Before getting into any data analysis, I address the question of how we should even be evaluating forecasts in the first place. One response to evidence that forecasts for transit projects have generally proven to be overwhelmingly biased has been an argument that forecast accuracy is unimportant, or less important than other considerations. And it’s true that accuracy isn’t the only possible way to evaluate a forecast.
A 1993 essay on weather forecasting by Allan Murphy (which I came across by way of Nate Silver’s book The Signal and the Noise) defines forecast “goodness” in terms of three characteristics:
(1) Consistency: Is the published forecast consistent with the forecaster’s best judgment? Does the forecaster actually believe her own forecast?
(2) Quality: Does the forecast correspond with what actually occurred? Was it proven to be accurate?
(3) Value: Is the forecast useful to forecast users? Does it help them to make the best decisions?
Individual forecasts might be good in one or more of these ways, without being good in all three. For example, a financial forecaster might try to defraud investors by intentionally inflating her firm’s earnings forecast, but unexpected events occur later that end up making the inflated forecast accurate (thus, the forecast has good quality, but poor consistency). A weather forecaster might intentionally overstate the seriousness of a storm (poor consistency) because she’s knows that people would otherwise under-prepare. Although the storm turns out to be less serious than her published prediction (poor quality), lives were saved because people were more prepared than they otherwise would have been (good value). An economic forecast might be really rigorous and accurate, but so vague and presented with so much technical jargon that it’s ultimately useless to the lay forecast user (good quality; poor value).
In evaluating all of a particular forecaster’s forecasts over time (rather than just one individual forecast), Murphy proposes that the best way to maximize all three types of goodness is to maximize forecast quality. Consistently accurate forecasts will increase the forecaster’s confidence in good methodologies (improving consistency), and the value of a set of forecasts increases with forecast credibility, which correlates with the accuracy of past forecasts.
Murphy goes on to argue that another reason to emphasize forecast quality over other types of goodness is that it’s often the only of the three types of forecast goodness that can be observed at all, since we’re generally no better at reading minds than we are at predicting the future. We don’t know what’s going on in the mind of the forecaster, so it’s hard to really judge how well a published forecast corresponds to the forecaster’s best judgment. Likewise, since we can’t read the minds of forecast users, we can’t really say how the forecast has influenced their decision-making. So we’re left with accuracy.
I think these three types of goodness are a useful way to think about what we might mean when we say, “The Church is True.” We hear and say this a lot in our church, and it strikes me that different people mean and understand different things by it. Applying Murphy’s three types of goodness, we might be referring to:
(1) Consistency: Current and past church leaders and teachers at various levels are sincere in their teachings and truth claims.
(2) Quality: The teachings of the church [fn] accurately reflect reality.
(3) Value: Decisions that follow church teachings can improve a person’s life.
Thinking about which of these three types of goodness should get the greatest emphasis when we’re talking about the truthfulness of the church is a little different than when we’re evaluating forecasts. The quality (accuracy) of most forecasts can be empirically observed, but most church teachings (the nature of God, for instance) don’t lend themselves well to that kind of direct confirmation. We have the same problem with evaluating consistency as we do in the case of forecast evaluation. No one but Joseph Smith (or President Monson) can really know how sincere he was (or is) in his belief in his prophetic calling.
Speaking generally about all people who encounter church teachings, we might run into the same problem with evaluating the value of church teachings that we do with the value of a forecast. We don’t necessarily know how people are using them. However, we can evaluate the value of church teachings on a personal, individual level. For you personally, does the Church work the way you need it to? Are the doctrines helpful to you?
I think scripture verses like Matthew 7:16-20, John 7:17, and Alma 32:27 are an argument that —similar to how a set of forecasts with good quality is most likely to also have good consistency and good value— a set of doctrines that you’ve found to have good value is likely to also have good consistency and good quality. This comes pretty close to how I think about my own testimony and experience with the church. Not that the accuracy of the church’s claims or the sincerity of church leaders are unimportant, but that my own personal experiences with church doctrine and church participation are (first of all) also important, and (second) the closest I can come to any kind of evidence of the other two types of goodness.
[fn] There could be a related, but entirely separate discussion about what “the teachings of the church” include or don’t include. I’m choosing not to get into that here.
Very well thought out! You’ve articulated ideas that have been floating around in my brain for some time, but which hadn’t really crystallized for me.
Suggest you review Sontag’s “Notes on Camp” b/c (Mormon) camp is what you are (perhaps inadvertently) doing here
I’ve used these three ideas for years in my consulting practice as a solution to the typical goal setting mentality in most organizations. Take for example what happens when the sales department exceeds sales and then the production department can’t make their deliveries and the now management is scrambling to find a solution to a problem that they created. The way I apply this trinity of measures to my understanding of what is true in the Church is to apply a psychological perspective. These are the three measures I use.
First I ask myself this question, does this make sense? And in this I’m not taking in a limited sense to just me but to other. For example if I were to stop 100 people and ask them the question I have then what would the majority say?
Next I ask what the Scriptures say about the question in hand. Again I need to expand on my frame of reference beyond my favorite Scriptures or just the one quoted in the manual. The electronic versions of the Scriptures are great for this type of research.
And lastly I ask, what are the behavioral consequences if I and 99 other people believe the same thing? In this I also consider a wide range of others including those who are weak or disadvantaged in some way. A lot of what I hear in Church only works well for those who are above average in intelligence, ability, or circumstances but fails miserably when applied to those whose life is a bed of cherries.
P ~ Care to elaborate?
Carole ~ I very much enjoyed your application of Allan Murphy’s forecasting structure to Mormonism, though I’d like to press you on the case for weighting value so heavily. I’m less concerned with the need for absolute “consistency”–God is admittedly working through mere mortals in a fallen world after all–but I think there is a strong argument that “quality” is right up there with or even more important than “value.” If the Church’s truth claims reflect reality, shouldn’t we, like Job, be willing to endure much hardship, persecution and strife for them? Didn’t Christ say “In the world ye shall have tribulation: but be of good cheer; I have overcome the world.” One could argue that “value” of following the Church’s teachings, at least in this temporal sphere, may not always seem worth it, but that it is the truth claims that change the calculus, both in the long term, spiritual sense (e.g., eternal salvation; eternal families) and in the short term, temporal sense (e.g., the peace of knowing that you are doing what’s right).
Marc: Right. I’m not necessarily saying that value is more important than quality, but rather that when quality is unknowable and value is knowable, value might be a reasonable proxy for (or evidence of) quality.
It’s worth adding that, as you point out, value isn’t always that much more knowable than quality, since some of the consequences to the decisions we make based on Church teachings are anticipated to only be fully realized after death. But I think, taking the Church’s teachings as a whole, that still leaves plenty of value that can be measured in the short-term. In fact, I think that’s a real strength of Mormonism (and I think also a lot of other religions as well) – is that it offers a lot of immediate value rather than relying solely on promises that are only realized after death.
Since some of the church’s truth claims are about value (e.g. the blessings that come from obedience to Church teachings), the line between value and quality is admittedly fuzzy.
A useful way to think about church teachings. (Of course useful true or only.)
I agree that the scriptures cited argue that value is likely correlated with quality. However, driving from value to quality does very little for exclusivity (the “one and only”). However difficult, I think you have to start with quality to make much of a stab at exclusivity positions.
I don’t see the argument for consistency, unless you take the roundabout that leaders and teachers following value better more righteous leaders and teachers more consistency.
Last sentence significantly garbled by my failure at HTML(?) coding. Rather, render as:
I don’t see the argument for consistency, unless you take the roundabout that leaders and teachers following value {may lead to} more righteous leaders and teachers {may lead to} more consistency.
From Marc’s and Christian’s comments – I think I should clarify, I’m not thinking of consistency as necessarily meaning that church leaders and teachers are actually living according to their own teachings, but just that they believe them.
For example, if someone says, “I don’t really know if I believe that God appeared to Joseph Smith in the Sacred Grove, but I do believe that Joseph Smith really believed that he had seen God.” — they’re expressing confidence in the consistency of Joseph Smith’s account(s) of the First Vision, but not in their accuracy.
As another example, if a leader teaches that baptism is necessary for salvation, that teaching has poor consistency if the leader doesn’t actually believe in the importance of baptism, regardless of whether he himself has been baptized.
The question of whether church leaders live according to their own teachings is likely to be externally verifiable, and as Marc points out, probably not terribly relevant to the question of whether those teachings are true. The question of whether they believe in their own teachings is not externally verifiable, but is much more relevant to the question of the truth of those teachings.
The consistency point is one that many have made for figuring out what is or isn’t doctrine. That way every idea that was popular for a few decades in the early Utah period isn’t considered doctrine, or ideas that get pushed by one or two major GAs but no one else.
The equating of “consistency” with sincerity and “quality” with accurate correspondence seems strange to me. Not that I disagree with any of these four values, just that the nomenclature seems strange to me.
Jeff you actually get something similar in the pragmatists. To believe is a kind of habit of behavior. Your strength of belief is proportional to how you act on that belief. So you believe in the germ theory of disease to the degree you act as if there are invisible bacteria and viruses and take appropriate precautions. That means that your sincerity of belief is strongly related to the consistency you act on that belief.
Likewise quality seems similar. You talk about the quality of the lens relative to how accurate the images it produces.
I like this analysis very much, and will have to try to remember it. I think that in some sense, the Quality and Value of church teachings might not be measurable in full until we die, but we can certainly take some interim measurements. (Not too many – on that road lies the insidious Gospel of Prosperity.) I also think that it’s difficult for people to discount Quality and Value without discounting Consistency as well – i.e., denouncing church leaders past and present as frauds and charlatans.
I fall into this trap myself, going back to your thesis, Carol; residing as I do in a metroplex that is eminently unsuited for mass transit yet continues to pour money down the light rail rabbit hole, it’s hard for me to believe that proponents, in the face of demonstrably poor Quality and negative Value, can truly be Consistent. But I should never underestimate the willful blindness of the True Believer.
For example, Marc, this phrase from New Iconoclast post: “I think that in some sense, the Quality and Value of church teachings might not be measurable in full until we die…” Technically this is High Camp, the theological equivalent of a feather boa.