Evidential Notes: This post links to two related editorials by John P.A. Ioannidis, a long prominent figure in the field of medical statistics and in the evidence-based medicine movement. The earlier editorial, from STAT, is from March 17, about a week old at posting time. The later editorial is a fully peer-reviewed pre-print publication in the European Journal of Clinical Investigation, which was published on March 19, five days old at posting time. When the first article was written the ECDC reported about 4,600 cases confirmed in the U.S. Today, at posting time, they report about 46,000 cases confirmed.^[1]

Pandemic Covid-19 is a serious public health threat. Of course it is. One reason it’s serious is biological: it’s a potentially serious disease for anyone who catches it, and a very serious life-threatening disease for some of the people who catch it, and so far it seems to be a fairly contagious disease that’s hard to contain. Another reason it’s serious is institutional: the disease and reactions to the disease have caused overwhelming congestion, resource starvation, temporary breakdowns, and catastrophic failures in some countries’ healthcare systems, which have led to appalling conditions and to deaths, both from the disease itself and from other health emergencies that could not be adequately treated. You should take it seriously, and you absolutely should do what you can to keep yourself healthy and safe to those around you.

There’s also a lot that we do not yet know about Covid-19 as a disease, and a lot that we do not yet know about the effects of the drastic responses that people and institutions have already made in response to the Covid-19 pandemic, or the capabilities and knock-on effects of continuing and escalating measures that are being proposed. There has been an immense, often frenetic attempt to gather more information about the disease, about its spread, and about social and institutional responses to it — not just among experts or hobbyists, but by nearly everyone through news media, scientific journals, situation reports, social media, and every other outlet under the sun. That produces lots of information; it also produces lots and lots of low-quality information and specious info-garbage.

Shared Article from STAT

In the coronavirus pandemic, we're making decisions without reli…

Countermeasures like social distancing may help stop the spread of Covid-19. But how can policymakers tell if they are doing more good than harm? Data…

statnews.com

The current coronavirus disease, Covid-19, has been called a once-in-a-century pandemic. But it may also be a once-in-a-century evidence fiasco.

At a time when everyone needs better information, from disease modelers and governments to people quarantined or just social distancing, we lack reliable evidence on how many people have been infected with SARS-CoV-2 or who continue to become infected. Better information is needed to guide decisions and actions of monumental significance and to monitor their impact.

. . . The data collected so far on how many people are infected and how the epidemic is evolving are utterly unreliable. Given the limited testing to date, some deaths and probably the vast majority of infections due to SARS-CoV-2 are being missed. We don't know if we are failing to capture infections by a factor of three or 300. Three months after the outbreak emerged, most countries, including the U.S., lack the ability to test a large number of people and no countries have reliable data on the prevalence of the virus in a representative random sample of the general population.

This evidence fiasco creates tremendous uncertainty about the risk of dying from Covid-19. Reported case fatality rates, like the official 3.4% rate from the World Health Organization, cause horror — and are meaningless. Patients who have been tested for SARS-CoV-2 are disproportionately those with severe symptoms and bad outcomes. As most health systems have limited testing capacity, selection bias may even worsen in the near future.

The one situation where an entire, closed population was tested was the Diamond Princess cruise ship and its quarantine passengers. The case fatality rate there was 1.0%, but this was a largely elderly population, in which the death rate from Covid-19 is much higher.

Projecting the Diamond Princess mortality rate onto the age structure of the U.S. population, the death rate among people infected with Covid-19 would be 0.125%. But since this estimate is based on extremely thin data — there were just seven deaths among the 700 infected passengers and crew — the real death rate could stretch from five times lower (0.025%) to five times higher (0.625%). It is also possible that some of the passengers who were infected might die later, and that tourists may have different frequencies of chronic diseases — a risk factor for worse outcomes with SARS-CoV-2 infection — than the general population. Adding these extra sources of uncertainty, reasonable estimates for the case fatality ratio in the general U.S. population vary from 0.05% to 1%.

That huge range markedly affects how severe the pandemic is and what should be done. . . .

— John P.A. Ioannidis, A fiasco in the making?
STAT (March 17, 2020). Boldface emphasis added.

Shared Article from Wiley Online Library

Coronavirus disease 2019: the harms of exaggerated information a…

The evolving coronavirus disease 2019 (COVID-19) pandemic1 is certainly cause for concern. Proper communication and optimal decision-making is an ongo…

onlinelibrary.wiley.com

(Archival PDF of the paper as accessed from Wiley.com on March 24, 2020)

The evolving coronavirus disease 2019 (COVID-19) pandemic1is certainly cause forconcern. Proper communication and optimal decision-making is an ongoing challenge, as data evolve. The challenge is compounded, however, by exaggerated information. This can lead to inappropriate actions. It is important to differentiate promptly the true epidemic from an epidemic of false claims and potentially harmful actions. . . .

— John P.A. Ioannidis, Coronavirus disease 2019: the harms of exaggerated information and non-evidence-based measures
European Journal of Clinical Investigation, March 19, 2020. doi:10.1111/eci.13222

What are some likely error bars on early, widely disseminated and decontextualized estimates of the contagiousness of the virus and case fatality rate for the disease? They’re pretty wide.

Exaggerated pandemic estimates: An early speculation that 40-70% of the global population will be infected went viral.^[4] Early estimates of the basic reproduction number (how many people get infected by each infected person) have varied widely, from 1.3 to 6.5.^[5] These estimates translate into many-fold difference in the proportion of the population eventually infected and dramatically different expectations on what containment measures (or even any future vaccine) can achieve. The fact that containment measures do seem to work, means that the basic reproduction number is probably in the lower bound of the 1.3-6.5 range, and can decrease below 1 with proper measures. The originator of the 40-70% of the population estimate tweeted on March 3 a revised estimate of 20-60% of adults, but this is probably still substantially exaggerated. Even after the 40-70% quote was revised downward, it still remained quoted in viral interviews.^[6]

Exaggerated case fatality rate (CFR): Early reported CFR figures also seem exaggerated. The most widely quoted CFR has been 3.4%, reported by WHO dividing the number of deaths by documented cases in early March.^[7] This ignores undetected infections and the strong age-dependence of CFR. The most complete data come from Diamond Princess passengers, with CFR=1% observed in an elderly cohort; thus, CFR may be much lower than 1% in the general population; probably higher than seasonal flu (CFR=0.1%), but not much so. Observed crude CFR in South Korea and in Germany^[8], the countries with most extensive testing, is 0.9% and 0.2%, respectively as of March 14 and crude CFR in Scandinavian countries is about 0.1%. Some deaths of infected, seriously ill people will occur later, and these deaths have not been counted yet. However even in these countries many infections probably remain undiagnosed. Therefore, CFR may be even lower rather than higher than these crude estimates.

— John P.A. Ioannidis, Coronavirus disease 2019: the harms of exaggerated information and non-evidence-based measures
European Journal of Clinical Investigation, March 19, 2020. doi:10.1111/eci.13222

How much of the divergence in estimates is due to errors of analysis? How much to artifacts of measurement (for example, the wildly different availability of testing and different methods of assigning tests in different countries)? How much of it is due to real biological or institutional differences in the different countries involved (differences in health care systems and their capabilities, differences in the age and health and customs of different populations)? In particular, I’m writing from the southeastern U.S., so a lot of the people that I know are wondering whether the situation where they are, or in the U.S. as a whole, is going to be more like the situation in Hubei Province, China, or more like the situation in Korea, or more like the situation in northern Italy. That’s a complicated question, but a lot of people are treating it as if it were a simple one. It’s not immediately obvious. It’s also not at all obvious how the actions being taken in response to the pandemic will affect the biological risk-factors, or how they will affect the institutional risk-factors, from the epidemic in the U.S. How effective are they at their intended goal? Do they have foreseeable side-effects or negative unintended consequences?

Extreme measures: Under alarming circumstances, extreme measures of unknown effectiveness are adopted. . . . Evidence is lacking for the most aggressive measures. A systematic review on measures to prevent the spread of respiratory viruses found insufficient evidence for entry port screening and social distancing in reducing epidemic spreading.^[10] Plain hygienic measures have the strongest evidence.^[10][11] Frequent hand washing and staying at home and avoiding contacts when sick are probably very useful. Their routine endorsement may save many lives. Most lives saved may actually be due to reduced transmission of influenza rather than coronavirus.

Most evidence on protective measures comes from non-randomized studies prone to bias. A systematic review of personal protective measures in reducing pandemic influenza risk found only two randomized trials, one on hand sanitizer and another on facemasks and hand hygiene in household members of people infected with influenza.^[11]

Harms from non-evidence based measures: Given the uncertainties, one may opt for abundant caution and implement the most severe containment measures. By this perspective, no opportunity should be missed to gain any benefit, even in absence of evidence or even with mostly negative evidence.

This reasoning ignores possible harms. Impulsive actions can indeed cause major harm. One clear example is the panic shopping which depleted supplies of face masks, escalation of prices and a shortage for medical personnel. Masks, gloves, and gowns are clearly needed for medical personnel; their lack poses health care workers' lives at risk. Conversely, they are meaningless for the uninfected general population. However, a prominent virologist's comment^[12] that people should stock surgical masks and wear them around the clock to avoid touching their nose went viral.

Misallocation of resources: Policy-makers feel pressure from opponents who lambast inaction. Also adoption of measures in one institution, jurisdiction, or country creates pressure for taking similar measures elsewhere under fear of being accused of negligence. Moreover, many countries pass legislation that allocates major resources and funding to the coronavirus response. This is justified, but the exact allocation priorities can become irrational. . . .

. . . [E]nhanced detection of infections and lower hospitalization thresholds may increase demands for hospital beds. For patients without severe symptoms, hospitalizations offer no benefit and may only infect health workers causing shortage of much-needed personnel. Even for severe cases, effectiveness of intensive supportive care is unknown. Excess admissions may strain health care systems and increase mortality from other serious diseases where hospital care is clearly effective. . . .

Economic and social disruption: The potential consequences onthe global economy are already tangible. February 22-28 was the worst week for global markets since 2008 and the worse may lie ahead. Moreover, some political decisions may be confounded with alternative motives. Lockdowns weaponized by suppressive regimes can create a precedent for easy adoption in the future. Closure of borders may serve policies focused on limiting immigration. Regardless, even in the strongest economies, disruption of social life, travel, work, and school education may have major adverse consequences. The eventual cost of such disruption is notoriously difficult to project. . . .

Learning from COVID-19: . . . If COVID-19 is indeed the pandemic of the century, we need the most accurate evidence to handle it. Open data sharing of scientific information is a minimum requirement. This should include data on the number and demographics of tested individuals per day in each country. Proper prevalence studies and trials are also indispensable.

If COVID-19 is not as grave as it is depicted, high evidence standards are equally relevant. Exaggeration and over-reaction may seriously damage the reputation of science, public health, media, and policy makers.It may foster disbelief that will jeopardize the prospects of an appropriately strong response if and when a more major pandemic strikes in the future.

— John P.A. Ioannidis, Coronavirus disease 2019: the harms of exaggerated information and non-evidence-based measures
European Journal of Clinical Investigation, March 19, 2020. doi:10.1111/eci.13222

Dissenting Views: Ioannidis’s articles have been controversial. For some direct responses, see the comments on the first article, and see also Marc Lipsitch, We know enough to act decisively against Covid-19.

See also.

• GT 2020-03-23: The Infovore’s Dilemma

1. ^[1]Source: ECDC Data on Geographic Distribution of COVID-19 cases worldwide, accessed 24-Mar-2020. Exact totals from the figures in the data table are 4,661 and 46,442, respectively. I’m saying about and sticking to two significant digits here, because different agencies’ reports gather numbers from different sources at different times and people following the data have found their aggregate numbers to be consistently very close to each other, but with persistent gaps due to differences in methodology.↩
2. ^[4]McGinty JC. How many people might one person with coronavirus infect? Wall Street Journal, February 14, 2020, accessed February 27, 2020 at https://www.wsj.com/articles/how-many-people-might-one-person-with-coronavirus-infect-11581676200↩
3. ^[5]Tang B, Bragazzi NL, Li Q, Tang S, Xiao Y, Wu J. An updated estimation of the risk of transmission of the novel coronavirus (2019-nCov). Infect Dis Model. 2020;5:248-255.↩
4. ^[6]Axelrod J, CBS News, March 2, 2020: Coronavirus may infect up to 70% of world’s population, expert warns. Accessed in https://www.cbsnews.com/news/coronavirus-infection-outbreak-worldwide-virus-expert-warning-today-2020-03-02/ on March 3, 2020.↩
5. ^[7]https://www.who.int/dg/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19—3-march-2020↩
6. ^[8]Frank Jordans. Experts: Rapid testing helps explain few German virus deaths. Associated press, https://apnews.com/ad9a6af47c3b55fd83080c9168afaaf4, accessed March 10, 2020.↩
7. ^[10]Jefferson T, Del Mar CB, Dooley L, Ferroni E, Al-Ansary LA, Bawazeer GA, et al. Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database Syst Rev. 2011;(7):CD006207.↩
8. ^[11]Saunders-Hastings P, Crispo JAG, Sikora L, Krewski D. Effectiveness of personal protective measures in reducing pandemic influenza transmission: A systematic review and meta-analysis. Epidemics. 2017;20:1-20.↩
9. ^[12]https://www.thetomahawk.com/featured-news/what-i-am-doing-to-minimize-corona-virus-infection-from-james-robb-m-d/, accessed March 5, 20202.↩

The basic predicament for intelligent action in a crisis is that information is laborious to sort, measurements are costly to get and costly to vet, and analysis takes time. Peer review and consideration takes longer. People act under uncertainty; when they are urged to act rapidly and drastically, they necessarily act — at best — on the best information they have at hand, and they are going to seek out and to produce lots more information — tentative results, estimates, conjectures and seemingly reasonable assumptions — that are very rough, because they also happen to be ready. The more rapidly, the more urgently, and the more drastically decision-makers start to act, or try to react, the sharper the predicament becomes: the supply of information available becomes necessarily more rushed and more tentative at the same time that the demand for certainty and unanimity becomes higher; the range of possible effects grows wider and deeper; the decisions and reactions themselves change the very regularities in other people’s choices, other people’s knowledge, other people’s circumstances, and the natural world that you are trying to observe or to assume. The more drastically and rapidly you act, the more they change, both in ways that you may be able to foresee and in ways that you cannot foresee, do not intend, and cannot control.

The result is that crises often produce a vast glut of information; but a lot of that information, and often the information most critical to making urgent decisions or taking drastic measures, is relatively low-quality information at best, information which has been produced rapidly, not vetted carefully, made on multiple simplifying assumptions, with huge error bars and wild, systematic skews that may be understandable in the pragmatic context of making a decision. (The extreme worse-case scenario might be highly salient even if it’s unlikely; data sets that aren’t entirely comparable may be the best you have for two things that you really need to compare; you might need to piously hope that some things go as planned even if you can’t be sure.) The more or less necessary predicament — you need time and effort to understand intelligently, but you need speed and freedom to take action — is often made worse by a number of extremely tempting, but extremely misleading, errors. A real need bold conjectures and decisive action is often conflated with unrealistic demands for dogmatic certainty; the real benefits of coordinated action are often conflated with a punitive demand for unanimity in belief and deference or conformity to appointed authorities. The deep epistemic problem with understanding the situation intelligently becomes not only the fact that high-quality information becomes so hard to find, but that low-quality information, or misinformation, crowds around all the watering holes in the cognitive ecosystem. Anecdotes are presented as data, toy models are presented as charts, tentative results are presented as What We Know Now, large scale syntheses of poorly comparable data from disparate sources are put forward as observed facts, third-hand sloganeering reports of experts’ tentative conclusions are put forward as conclusive arguments, simplifying assumptions are put forward as obvious and incontestable dogmatic principles. Actively seeking out information and absorbing it doesn’t necessarily serve to better inform or to improve your cognitive position; it often ends up being an exhausting means to skew your own judgment towards the prevailing trends and groupthink of the info-garbage that is most readily available to you.

None of this is any reason not to rely on imperfect information if you have to make a decision — what else are you going to rely on? It is a reason to act with the awareness that you’re taking a certain number of shots in the dark. It is a reason to prominently state simplifying assumptions used in arguments or models, and to acknowledge them as assumptions, not as oracular revelations, wherever possible. It is a reason to actively seek out, and publicize, the parts of what you’re saying that you’re least certain of, or that you know will be most contested by others, and to acknowledge what would follow or what might follow if those underlying premises turned out to be false. It’s a reason to be ready for and to do whatever you can to hedge against the risks of unintended consequences. It’s a reason to state numbers with error bars and to try to figure out lowball and highball constraints on what the real figure might be, if you’re wrong.

In circumstances that lead to a high risk of groupthink and overreach, it’s a reason to explicitly employ evidential markers when reporting claims; it’s a reason to cite and link to specific sources for specific claims rather than simply repeating them or presenting them as What Experts Are Saying, and it’s a reason for readers to spend some time following links and footnotes where they have been made available, or to significantly discount stories that don’t bother to provide them. It’s also a reason to actively seek out and cultivate second guesses, minority reports and dissenting opinions, rather than ignoring, scolding or punishing them.

In a high info-garbage environment, it is often worthwhile to deliberately limit, compartmentalize or substitute the consumption of certain kinds of low-quality or risky information. In particular, to restrict your intake of information where the persuasive power of the presentation is especially likely to outrun its real evidential import. You may be better off glancing at boring charts a few times over a few days than you are looking at infographics in a newspaper article; you are almost certainly better off reading the abstract and a paragraph or two of one scientific paper than you are reading through an explainer article attempting to gloss the conclusion of that paper while weaving it together narratively with interviews from two or three other pronouncements by experts in the field. Commentary is prone to be less valuable than reporting, and reporting less valuable than sources or data. In a high info-garbage environment it’s also especially important to be sensitive to the likelihood of mistakes, to record claims in a testable and falsifiable form and to go back and check on them over time, to prepare for imperfect or piecemeal implementation of plans, and actively try to gather information on potential or actual unintended consequences and perverse incentives.

The problem here is not that people will draw conclusions that are wrong, or to make decisions that turn out to be mistakes. Of course they will. If that wasn’t a real danger, then it wouldn’t be a crisis in the first place. The problem here is that if you want to draw conclusions that are less wrong, more often, — if you want to do less damage and realize more quickly when you make the wrong decision, — if you want to lower the chance of being misled — then that may mean being more selective rather than more completist in the sources of information that you pursue. And the sources to be most selective about will often be the ones that seem the most appealing from the standpoint of your own social and ideological starting-points. Consume thoughtful discussion and information, not too much, mostly data.

Amidst the turbulence and disorder of faction, a certain spirit of system is apt to mix itself with that public spirit which is founded upon the love of humanity, upon a real fellow-feeling with the inconveniencies and distresses to which some of our fellow-citizens may be exposed. This spirit of system commonly takes the direction of that more gentle public spirit, always animates it, and often inflames it, even to the madness of fanaticism. . . . The man of system . . . ^[4] is apt to be very wise in his own conceit, and is often so enamoured with the supposed beauty of his own ideal plan of government, that he cannot suffer the smallest deviation from any part of it. He goes on to establish it completely and in all its parts, without any regard either to the great interests or to the strong prejudices which may oppose it: he seems to imagine that he can arrange the different members of a great society with as much ease as the hand arranges the different pieces upon a chess-board; he does not consider that the pieces upon the chess-board have no other principle of motion besides that which the hand impresses upon them; but that, in the great chess-board of human society, every single piece has a principle of motion of its own, altogether different from that which the legislature might choose to impress upon it. If those two principles coincide and act in the same direction, the game of human society will go on easily and harmoniously, and is very likely to be happy and successful. If they are opposite or different, the game will go on miserably, and the society must be at all times in the highest degree of disorder.

— Adam Smith (1759) The Theory of Moral Sentiments, Part Sixth, Section II, Chapter 2, ¶Â¶ 15-17
Of the order in which Societies are by nature recommended to our Beneficence

1. ^[4]Smith: on the contrary, i.e., as contrasted with the man whose public spirit is prompted altogether by humanity and benevolence, who (therefore) will respect the established powers and privileges even of individuals, and still more those of the great orders and societies into which the state is divided. –C.J.↩

Listening to: 1A (25-Jan-2020), Get In Line: What It Takes To Legally Immigrate To The United States

… But every day, millions of people contend with the U.S.’ legal immigration system. Many have been living and working in America for years, stuck in residency limbo as they contend with an alphabet soup of visas and green cards and a system congested with red tape and long wait times.

An immigrant [from India] who applies for a green card today can expect to wait in line for 50 years.

— NPR, Get In Line: What It Takes To Legally Immigrate To The United States
1A, 25 January 2020

Shared Article from NPR.org

'Get In Line:' What It Takes To Legally Immigrate To The United …

"Over the past few years, [USCIS] has gotten far more difficult to navigate, it is much more difficult to speak to a human being, to make an appointme…

npr.org

The half-century queue for permanent legal status is, of course, the direct result of the predictable, runaway overwhelming of an insanely restrictive system of immigration caps and national quotas. In the early 1920s, the U.S. created a madly restrictive system of immigration limits organized around a nativist and racist system of national-origins quotas.^[4] This was widely understood to have been a mistake by 1965 — so in order to fix the system, they kept the insanely low caps on the total levels of authorized immigration, but they (thankfully finally) permitted Asiatic nations like India and China finally to claim their shares^[5]; then they also (for the first time) imposed the same system of insanely restrictive caps on Mexico and the rest of Latin America.^[6] Then, just to make things fair, they reallocated the national quotas so that every country, from Liechenstein to Honduras to the entire Republic of India, gets an equal sliver of the total. The completely predictable result has been that high-emigration countries have been accumulating runaway backlogs of applications. In theory there is a queue; in practice, the queue has become so mind-breakingly long that many 25 year olds seeking legal status now cannot reasonably expect to ever get a shot at naturalization unless they survive years beyond the average human life expectancy.

There is some discussion in the show about proposed ways to fix the system; mostly in the form of debates about how to reallocate the shares of the insanely low limit on total immigration so that countries with more demand for permanent-residency visas can get access to more permanent-residency visas per year. But the problem is not the allocation of shares among different nationalities of immigrants. The problem is that immigration is deliberately kept to a minuscule level that is wildly, completely out of touch with the realities of global migration, the number of people who are looking to come to the U.S. and with the level of demand and number of opportunities for immigrant workers, students and families within the U.S. The solution is not to reallocate the quotas again, according to some other rule, but to get rid of the caps and quotas entirely. There is no good or sensible way to triage something that people can and ought to have by right. There’s no fair method for rationing access to something that shouldn’t even be scarce in the first place.

1. ^[4]1921: The national-origins system was debuted in the Emergency Quota Act; 1924: The system was regularized and made even more restrictive under the Immigration Act of 1924. The idea behind the act was, explicitly, to reduce the rate of demographic change in the U.S. and to keep undesirables out of the country, notably Asians and Jewish refugees from Eastern Europe and the Weimar Republic.↩
2. ^[5]Nearly all immigrants from every Asian country had been barred entirely under a pair of racist provisions of the old system — one of them forbidding immigration from China or from anywhere in the Asiatic Barred Zone, and another forbidding immigration by any alien ineligible to citizenship, which under the racial prerequisites embedded in U.S. Naturalization laws amounted to forbidding Asians on racial grounds.↩
3. ^[6]Immigrants from Western Hemisphere countries used to be non-quota immigrants under the old system. Although they were subjected to a number of other costs and restrictions, which allowed for frequent large-scale harassment, round-ups, and deportations during times of anti-immigrant backlash, they were not subject to any hard cap on numbers, in the way that Eastern Hemisphere countries were, until after the 1965 reform.↩