Adverse selection in technical reporting
There are those who can do the technical work and those who can't; those who can't report on it, to dismal results.
Why do so many high-engagement tech news make basic mistakes?
Adverse selection
Adverse selection happens, for example, when incentives make the worst people for a job be those who actually take the job, or the customers who buy a service to be primarily the most costly customers to serve.
A common example is badly-priced insurance, where the premium is calculated using the average cost of a payout given the risk profile of the whole population. In this case, the cautious members of the population won't buy the insurance (because it's too expensive for their risk level) and the daredevils will buy the insurance, leading to a lot more payouts than expected in the premium calculation. Then the insurance companies raise premiums and the problem repeats with fewer insured members but higher average claims, and so on.
Illustration using a uniform distribution, to make things simple:
(Some types of insurance are mandated for the whole population of interest, say drivers, because otherwise it would be very difficult to offer insurance and some of the cost of accidents would be borne by victims of the risk-takers.)
Another example, popular in academic and artistic circles, is that of grant reviewers and art critics: the real researchers and artists are busy researching and creating art, leaving those with no talent to take positions in the institutions that give out grants or to be art critics.1
Of interest to us here today is a less well-known case, but possibly more important, because it’s more widespread and happens more often: adverse selection in reporting (and even popularizing) of technical material.2
Reporting material that requires valuable skills to understand: who gets to do it?
Let's consider what's necessary for someone to understand the business potential of a new invention for synthetic fuel creation, in order to validate information about that potential so that it can be reported on:
A general understanding of markets in energy, and their evolution.
A detailed understanding of fuel markets and their evolution.
Knowledge of the competitive landscape in synthetic fuels.
Ability to estimate how long it'll take and how much it'll cost to go from a prototype to production at scale.
Skill in financial modeling, including sub-modeling of the market evolution, resource costs, and competitive pressures.
A reasonable understanding of the chemistry, chemical engineering, logistics, procurement, operations management, production engineering, and production technology involved in the actual production of the synthetic fuel.
There are people who do have the requisite set of skills: most of them work as specialized analysts for companies that invest in or trade on these markets; and some, who prefer to write for broader audiences, work in the research branches of investment banks, consultancies, or other large institutions.
These are well-paid jobs for which there's always a shortage of competent people.
Which leaves the reporting on business impact of synthetic fuel breakthroughs mostly to the less-employable of the lot, usually because they don't have the skills needed or the focus required to stay current on all the details of an industry to be able to include them in the modeling.3
Note that it's perfectly fine to report facts in the absence of those skills. But many of the reports we see in the business, technology, economics, and even science media aren't just about facts: there's opinion, forecasting, and sermonizing involved.
Note the difference between:
“Stratton-Oakmont’s Jordan Belfort said ‘we have returns in excess of market’”
and
“Stratton-Oakmont uses secret technique to generate returns in excess of market.”
The first reports a fact, that a CEO said something self-serving; the second presents an opinion, one that in order to be accurate would require analyzing returns and studying the S-O trades; this, at best, could only be done by someone with an advanced and highly monetizable skill-set and non-public information.4
The second version, however, hints that the reporter is an expert with that monetizable skill-set and has access to non-public information and is therefore someone that should be taken seriously.
No surprise which positioning is more common in engagement-driven media.
There are some balancing forces at work, of course: reputation-based media take more care in selecting what gets published than engagement-driven media; though the former are a dwindling species mutating into the latter. And media that is associated with a brand that monetizes information to high-paying customers (like a Bloomberg terminal subscription5) needs to consider the effect of the popular side on the high-paying side. But these are exceptions rather than the rule, especially when reporting on the intersection between technology and business.
Okay, but what about people who switch from working in these better-paying jobs to working in media? Wouldn't they mitigate the problem of adverse selection?
Yes, some. For a while.
Times change and we must change with the times
There's a dynamic aspect to adverse selection: as time passes even competent people who choose to become reporters or popularizers lose skills. Those who put in the work to stay up-to-date have some probability of switching back to high-paying jobs, so adverse selection strikes again.
There are two processes that make someone become less and less qualified over time: (a) the field evolves and (b) skills that aren't practiced get lost.
Without practice we lose skills. For example, in college most engineering students know how to take simple derivatives, but ask engineers who graduated ten years ago for the derivative with respect to x of the exponential of x-squared and some, maybe many, will not be able to do it. (It's 2 times x times exponential of x-squared.)
Sadly, foundations — the material on which all the other skills are built, usually indirectly — are among the first to go, and few people even consider taking the time to do refreshers of foundations (this is a problem with some continuing education).
On the other hand, even skills that are retained may become useless as the field changes. For example, in the 80s and 90s electrical engineers learned to design digital state machines using flip-flops or shift registers for state memory and individual gates for next-stage and output decoders; some of us still have that skill. But the field has changed: anyone making a digital state machine will simply use a Raspberry Pi and do the next-state and output mapping in software.6
This, and all this section, happens in the absence of continued learning, of course. But since continued learning and skill practice are effortful and take time, many skills that have no obvious or repeated use are treated as expendable. (Professional associations tend to require some sort of continuing education for continued membership for this precise reason.)
So there are two forces acting against the value of the skills acquired in education, and we can illustrate them (and their combined effect); we'll assume for illustration that skill retention is 90% (an unrealistically high number) per year and the field obsolesces 5% of the skills per year, these two being independent.
In this example, half of the skills learned is lost or outdated after 4 years and 5 months. So, say, while Bob, an economics college graduate, could at graduation analyze the impact of a central bank policy, if Bob becomes a reporter after college, in five years there’s a 50-50 chance he has lost the skills to do that analysis when he has to write an article about such a policy, assuming that Bob didn’t use or update those skills in the meantime.
Interestingly, this suggests that the more popular (and hence less technical) the writings of a reporter or popularizer, the faster that reporter or popularizer's skills decline; the more dynamic a field is (in other words, the faster a skill becomes obsolete, the faster a reporter or popularizer’s skills will be obsolescent; and the longer —and generally more successful— a reporter or popularizer's career, the more likely they no longer have the required skill-set.
A few examples:
Taken globally these parameters are ridiculously optimistic: no one expects people to recall 90% of all they learned in college a year out. These parameters are to be interpreted in narrow skill sets; for example, chemical engineers working in polymers may lose 10% of their usable polymer chemistry skills every year, and the field of polymer chemistry may evolve so that about 2.5% of knowledge in each specific area of polymer chemistry becomes obsolete every year. (Again, these numbers are just for illustration of the phenomena.)
How to tell there’s been adverse selection in the reporting
By no means an extensive list, but some common tells of articles, reports, and books on technical fields for a broad audience whose writers may be lacking in the technical foundations are:
Numbers and other quantitative statements with no context or no explanation of what they mean in the appropriate context.
Charts with no explanation of their relevance, particularly what they would look like in the counterfactual case.
Uncommon (for the audience) units, jargon, or abbreviations used without definition or clarification.7
Unreferenced calculations or calculations missing details or ways to check their correctness. (In many cases that’s because they’re not correct.)
Use of authority and appeal to popularity among a population of experts to sidestep having to explain or illustrate easy-to-explain or easy-to-illustrate technical arguments and calculations that are central to the main argument. (This is easier to notice when we are better at the technical foundations that the writer.)
It grazes the definition of ad-hominem, but: when we see an author blurb that makes it clear the author avoided all technical instruction, made a career reporting on non-technical stories, and eventually landed on the technology desk, it may be a contributing indication. It shouldn’t be a dispositive factor!
Other failures in reasoning, like ad-hominem attacks and non-sequiturs also show lack of technical foundations, but these are more general in that they show lack of capacity to make an argument in all fields, technical or non-technical.
Not to end on a negative note
Two recommendations of good recent-ish books at the intersection of business and technology by reporters:
Edward Niedermeyer's “Ludicrous: The Unvarnished Story of Tesla Motors.”
John Carreyrou: “Bad Blood: Secrets and Lies in a Silicon Valley Startup,” about Theranos.
And, while not about technology, an example of a book by reporters specializing in fields where specialists are very well-paid, that doesn't show any of the effects we'd expect of engagement-chasing, adverse-selected reporting. Possibly because the authors are both Bloomberg reporters who report on commodities and energy and Bloomberg makes its real money off its $25k per year per user information services.
Javier Blas and Jack Farchy’s “The World For Sale,” a book on commodities trading.
So, there's hope.
There are also many books and articles by people whose primary job isn't dependent on engagement: practitioners, professors, researchers, consultants, even serious hobbyists; people whose main occupation isn't getting media engagement but whose love of the field makes them want to bring it to the attention of others.
Note that adverse selection as described above doesn't apply to reporting or popularization of technical material in these cases, quite the contrary. Some of the best popularizers (meaning those who make the fewest mistakes) are those whose career as a popularizer was either a hobby in addition to, a part-time occupation along with, or a follow-on career after, the technical career.8
These competent people are usually derided by the incompetent professionals as being just “amateurs.”
Hence the old joke: what do you do if you like art but have no talent? Become a critic!
In this post we’re talking of those who report or popularize technical material as a primary or only occupation, not those who do it as a secondary occupation or as a hobby. Also, we’re referring to some, perhaps most, but definitely not all of them.
Yes, it's possible someone with the relevant monetizable qualifications prefers mass market appeal to money and actual influence over decision-makers, but we're interested in the majority of technical communicators here.
At best because even specialized consultants and auditors get this kind of thing wrong on occasion, and that’s after weeks of analysis and access to internal information.
Note that there’s a big difference between what gets written and what’s on live video, even for Bloomberg. BloombergTV has produced some big howlers in technology reporting, while Bloomberg written material marked with “before it’s here it’s on The Terminal” tends to have fewer such errors.
In fact, typically the design now happens at a much higher level of abstraction, as most software is higher-level relative to hardware: cycles and conditionals instead of flip-flops and gates.
Also common is the use of mixed units without establishing a baseline of comparison, for example describing last year's California's electrical consumption in TWh and a proposed solar farm capacity in MW, without noting that 1 TWh per [non-leap] year is 114 MW: 1 TW = 1,000,000 MW and 1 [non-leap] year = 8760 hours.
The usual trade-off here is that many of these more knowledgeable people aren’t as good at writing as the professionals, though many of them have the humility to get help in the form of writing mentorship, tutoring, and of course professional-grade editing. (Attentive readers will notice the absence of at least the last in these posts.)