Opening the black box of digital healthcare: Makes sense with “evidence”

The ongoing lawsuit against Elizabeth Holmes – the person allegedly responsible for the $ 9 billion Theranos scam – has uncovered an even more fundamental challenge for healthcare technology: information asymmetry. To the average patient, doctor or investor, these technologies are “black boxes”, their intricacies becoming more and more opaque and uncertain. And while it is tempting to blame this regarding the trend of technical complexity, we must note one more insidious factor at play: the degraded quality and availability of “evidence”.

Holmes took advantage of this trend brilliantly: With venture capital increasingly accustomed to investing in black boxes, Theranos was able to amass billions without providing concrete scientific evidence. In the health care system, fortunately, most are not as obvious as Theranos was seen in hindsight. More often, uncertainty arises in the form of a nagging question: What is “evidence” that something is beneficial?

We have all seen the gleaming pharmaceutical TV commercials announcing cures, be it magic blue pills that counteract impotence, or harmless tablets that ward off heart disease. We largely trust drugs because they undergo a rigorous review by the Food and Drug Administration, even though this process is fraught with flaws. Yet, even with such elaborate and time-tested processes, opacity continues.

Recently, this lack of clarity about evidence and results has become much deeper and darker with the advent of Digital Therapeutics (DTx), software interventions aimed at preventing, managing or treating medical illness. Livongo, Omada and Pear Therapeutics are just a few of the companies in this area. The scope of digital therapy is enormous, and existing products address indications ranging from metabolic diseases to musculoskeletal disorders to mental disorders. DTx providers often try to distance themselves from the many mobile “wellness” apps (such as MyFitnessPal or mindfulness apps) by making claims about “clinical validation.” But how convincing is this basis for “evidence”?

Randomized control trials (RCTs) are the gold standard for assessing whether a medical product is actually beneficial, but these are rare in digital health. Some of the best validated products have only performed a single RCT, while the rest do not bother to do it themselves. Most often, the “evidence” comes from less rigorous studies without control groups (for example, those comparing to usual care). Many DTx companies provide no proof on their website. And while some DTx solutions provide clinically significant benefits, others stand on shaky ground. For example, earlier this year, Hinge Health, a DTx company focusing on musculoskeletal care, acquired a technology that uses transcutaneous electrical nerve stimulation (TENS) for alleged pain relief. What they do not disclose is that evidence regarding TENS effectiveness is contradictory at best; many medical guidelines recommend avoiding it altogether. In general, DTx claims of “proven clinical benefit” are rarely linked to a credible evidence base.

The most enterprising and discerning patients can ask their doctors about the evidence base of these products. However, many, if not most, will simply instruct their doctor to give a recommendation on whether or not to use the product. But doctors are overworked and constantly lagging behind, and they are unable to keep up with literally hundreds of thousands of medical articles published every year. And unfortunately, even enterprising patients and well-meaning doctors who seek to decipher the “evidence” are often ill-suited to do so. Just as a sushi chef may be ill-equipped to run a bakery, a cardiologist may encounter greater difficulties than a psychiatrist when evaluating experiments with digital therapy for mental health. Domain expertise is essential to make sense in these studies. Each black box has a separate key.

In today’s information overloaded world, shortcuts – any opportunity to give up fighting with more options, data, statistics – are a rare and welcome respite. For example, Consumer Reports provides a reliable and well-designed platform with fast, action-oriented insight; we no longer need to search the internet for customer reviews and expert opinions. The analogue of medicine is Choosing Wisely, a set of readily available, jargon-free guidelines that promote best practice toward effective, cost-effective treatment.

Digital therapy needs something similar to Choosing Wisely. It may be that venture capital firms (VCs) are best positioned to help bring this to life. It makes intuitive sense that DTx products rely on provider support because patients rely on their doctors’ expertise. But it is in the best interest of medical technology companies – and more, the VC funds that advocate them – to demonstrate reliable and honest clinical efficacy.

The consequences of false advertising, of secretly downplaying negative effects or unethical enhancing effectiveness are potentially serious. Investors – motivated by long-term reputation, profits and publicity – are motivated to only support and join startups of the highest quality. The case of Theranos shows why this is true. Furthermore, by establishing credibility as an investor in only clinically sound, thoroughly researched medical technologies, an investor could differentiate himself and burn his reputation. Medical professional companies are just as important: they would be wise to conduct independent reviews, apply methodological standards to assess the credibility of claims, ease the decision-making burden for physicians, and speak for their respective areas with one voice. Finally, payers are equally well positioned. As forms increasingly provide coverage for digital therapies, payers should move towards transparency by publishing their internal analyzes of different technologies. Aggregating information will improve the need for redundant review processes and accelerate the coverage and introduction of high quality digital therapies.

Health technology has the potential for tremendous good. Where available, rigorous studies have shown that DTx can exceed or increase the effectiveness of standard treatment regimens. But there is no free lunch: These therapies have the potential to increase healthcare costs, can cause delays in seeking alternative treatment, and are particularly susceptible to digital fatigue and improper compliance. To avoid another Theranos, it is imperative that we – doctors, patients, entrepreneurs, investors – open the black box and appropriately weigh the risks and benefits of digital therapy.

The authors’ note

Dominick L. Frosch and Robert M. Kaplan are both principals at Health Science Diligence Advisors, LLC.

Leave a Comment