The added benefit is that we now have a presentation of complex data with informative findings that have been verified by a thoroughly independent and objective group of experts using state-of-the-art research tools.
Call to Action Steering Team Report, page 41
The above quote refers to the Towers Watson contribution to the Call to Action Steering Team Report (the “Report”). In this brief (for me) post, I want to look at the contribution from Apex Health Care Group LLC. Does it qualify as the work of a “thoroughly independent and objective group of experts”?
Except where noted, this post draws from the Report’s Appendix 9 (Appendix A: Environment Review and Assessment). All images are “as-is” from the full Report.
Where to start a graph’s vertical axis?
The above graph and accompanying text (from page 187) is included here for two reasons:
- It will provide a good contrast for later graphs. There are visible downward slopes, but they don’t look like they’re steep enough to be dangerous.
- The top trendline looks level (the Southeastern Jurisdiction), and the next trendline (the South Central Jurisdiction) looks like it has a slight decline. The Measure Tool in Photoshop appears to confirm this: the top trendline is about level (same height on the left end and right end), and the next trendline is about 2% lower on the right end than on the left. But this is the opposite of what the accompanying text claims (the Southeastern Jurisdiction supposedly has a -2.9% decline in membership, while the South Central Jurisdiction has remained level).
In the above graph (top of page 187), note the left side. I’m not sure how to interpret the fact that two different lines indicate “2 Million” (for example). But at least the bottom of the left side appears to start at zero. (We can call the left side the vertical axis, or the y-axis.)
In this graph (top of page 186), we have the same problem as in the previous graph. For example, I don’t know which line marks “10 Million”.
More importantly, the vertical axis does not start at zero. In 1954 Darrell Huff wrote about how this as one method of manipulating statistical displays. For discussions of this technique, see here or here.
These two graphs appear on page 191. Each of these graphs displays a decline that takes up about half of the vertical axis. Neither graph actually depicts a decline of 50%. The top graph displays a drop (in total churches and preaching places worldwide) of about 10%, and the bottom graph displays a drop (in number of U. S. churches) of about 6%.
The above two graphs appear on page 200. The vertical axis of each graph could go to zero (in which case, there would be clergy with empty churches).
There’s another problem with these two graphs: so what? Imagine instead that these two graphs displayed “students per teacher.” It would be unusual for parents to complain that this ratio was dropping.
What should the “Attendees per Clergy” ratio be? The Report’s Apex contribution does not say.
This graph is on page 201. Now it’s not realistic to expect the vertical axis of this graph to go all the way down to zero. (By anyone’s definition, a group of newborns would not likely make for effective clergy.) Still, I’m not sure why the vertical axis starts at 44. Maybe this is a minimum age for clergy. Maybe this is a minimum average age for clergy. Maybe using this value ensures that the increase in average age takes up 50% of the vertical axis.
But again, this graph suffers from the same problem as before: so what? This graph tells us that, over the course of ten years (1998 to 2008), the estimated average age of active U.S. clergy increased by about five. This is better than an increase of ten over ten years. Is an increase of five a bad result? I honestly don’t know.
Nominal dollars vs. real dollars
I grew up in the U.S. not too far from the U.S. / Canada border. Someone in high school once said excitedly, “Driving in Canada is awesome! The speed limit up there is 90!” Unlike the posted speed limit of 55 miles per hour in Vermont, the signs in Quebec did read 90. That 90, however, was 90 kilometers per hour, which is basically the same speed as 55 miles per hour. 90 is a larger number than 55, but that doesn’t mean that 90 kilometers is a greater distance than 55 miles. After all, 1 kilometer is smaller than 1 mile.
My point here is that it in comparing two measurements, each measurement has to use units of the same size. Money loses value over time. If we’re comparing prices over time, we need to take into account inflation (Edward R. Tufte, Visual Explanations [1997: Graphics Press, Cheshire, CT], p. 7o).
The above graph comes from page 133, from the Report’s Appendix 7: the Apex Executive Summary. It shows percent change in giving from 1998 to 2008. The change in “Local Chuch Other Benevolence” dominates over the change in “Local Church Benevolences Sent to Annual Conference Treasurer”. Since we’re told that this is in “nominal terms”, there’s no adjustment for inflation.
Now look at this graph (from the bottom of page 203):
This graph shows the actual expenditures, in real dollars (in other words, adjusted for inflation). The top line displays “Connectional Benevolences Sent to Conference Treasurer” (top line) and “Other Benevolences” (next line). This graph tells a different story than the one on page 133: “Connectional Benevolences” did not increase, and “Other Benevolences” did not even double for the same period.
Two more graphs (the second is from the top of page 203):
Each of these graphs shows (a) expenses that are increasing without adjusting for inflation, and (b) a vertical axis that starts well above zero.
What does each of these graphs show? Each vertical axis is a ratio, either expenses per member or expenses per attendee. Decreasing membership or decreasing attendance will increase the respective ratio (these downward trends have been established elsewhere). An increase in nominal expenses (expenses not adjusted for inflation) will also increase these ratios. But so what?
Yes, each graph has the footnote about nominal values. If the graphs don’t show real values, I’m not sure what value they add to the presentation.
One last chart
Finally, a chart from the bottom of page 194:
When I first read this chart I thought, “If the Membership Distribution share for the ‘White’ ethnic demographic went up three percentage points, which ethnic demographic(s) ‘lost out’ in this transition?” Remarkably, this chart shows that every ethnic demographic saw an increase in the share of its Membership Distribution. This result has a simple explanation: the percentages do not total 100%.
1998 | 2008 | |
Asian | 0.7% | 1.1% |
African American / Black | 4.6% | 5.8% |
Hispanic | 0.5% | 0.7% |
Native American | 0.2% | 0.3% |
Pacific Islander | 0.1% | 0.2% |
White | 86.9% | 90.0% |
Multi-Racial | N/A | 0.4% |
TOTAL | 93.0% | 98.5% |
Here’s how Apex interprets the same chart (page 182):
The Church’s ethnic/racial membership demographic profile remains little changed over the Period, with a three percentage point increase in the “White” demographic and a one percentage point increase in the “African American/Black” demographic as a percentage of total church membership – in 2008, the Church membership was predominantly “White” (90%).
Is it unrealistic to expect management consultants to notice when some numbers do not add up?
Concluding remarks
The deep, fundamental question in statistical analysis is Compared with what?
Edward R. Tufte, Visual Explanations, p. 30
A “Call to Action” over declining membership could be a good idea. What’s startling in this entire project is the failure to even ask, “Why is membership declining?” Instead of wrestling with this difficult question, Apex’s contribution consists of many scary graphs showing a decline. Some of these graphs seem more interested in being scary rather than being informative.
It’s easier to draw some graphs rather than ask difficult questions. (Perhaps it’s even supposed to objective; after all, who can possibly argue over a graph?) However this process unfolds in the life of the Church, I hope that it does not continue to hide from difficult questions.