AHRQ Labors Mightily to Produce a Mouse

Richard A. Lawhern, Ph.D.
December 4, 2017

Largely unknown to many Americans, many medical “best practices” are strongly influenced by the Agency for Healthcare Research and Quality (AHRQ), an organization of the US Government. AHRQ provides grants and monitors research intended to improve the safety and quality of US healthcare. Sometimes they make real contributions – and sometimes not so much.

Over a year ago, AHRQ issued a public call for comment on a series of questions intended as a basis for analyzing available medical evidence pertaining to “noninvasive, non-pharmacological treatment” of five types of pain in adults. I commented on this data call at length, pointing out a number of serious weaknesses in the logic and definitions proposed. On January 15, 2017, the Journal of Medicine of the US National College of Physicians published my comments under the title “The CDC’s Fictitious Opioid Epidemic, Part I” [see https://www.ncnp.org/journal-of-medicine/1929-doctors-fleeing-pain-management-dumping-patients.html]

It is now almost a year later, and AHRQ has circulated a for-comment draft of their systematic review report. I read this 300+ page monster and sighed. The following is an extract from comments I returned to AHRQ


As noted in the report abstract, the conditions studied and the key questions were as follows:

Scope and Key Questions

This comparative effectiveness review focused on noninvasive, nonpharmacological therapy, with a Key Question (KQ) for each of five common chronic pain conditions:

KQ 1: Chronic low back pain
KQ 2: Chronic neck pain
KQ 3: Osteoarthritis (knee, hip, hand)
KQ 4: Fibromyalgia
KQ 5: Chronic tension headache

For each condition, [AHRQ] addressed the following subquestions:

a. What are the benefits and harms of noninvasive nonpharmacological therapies compared with sham treatment, no treatment, waitlist, attention control, or usual care?

b. What are the benefits and harms of noninvasive nonpharmacological therapies compared with pharmacological therapy (e.g., opioids, NSAIDs, acetaminophen, antiseizure medications, antidepressants)?

c. What are the benefits and harms of noninvasive nonpharmacological therapies compared with exercise or (for a headache) biofeedback?


As noted in my comments to AHRQ, a key failing of their effort is that the published studies reviewed by their panels frequently lacked explanations for what “usual care” actually comprised of the alternatives studied.  From page 268-269 of the draft report, we learn:

“Some of the limitations described in the review process reflect limitations of the evidence base, including those related to heterogeneity within and across interventions and heterogeneity within a given condition. Details of concurrent interventions and components of usual care were generally not reported or poorly reported. Additionally, it is assumed that most patients with chronic pain likely continued medications and other therapies or practices during the trials. These factors likely resulted in the substantial mixing of effects, so that the effects of the intervention studied could not be separated out based on information presented in the trials. These factors possibly attenuated observed effects.”

My personal reading of the AHRQ draft report is less optimistic. The so-called “observed effects” of the therapies reviewed in this effort seem to me at best marginal in all of five of the classes of disorder studied.  Compounding weak evidence and protocols, the review encountered most of the problems which I pointed out before nearly a million dollars of taxpayer money was spent.

The following were “Key Messages” of the draft report:


  • A number of nonpharmacological interventions may improve function or pain outcomes 1 month to 1 year after the completion of therapy; evidence for some treatments and conditions is limited.
  • Exercise, acupuncture, multidisciplinary rehabilitation, mind-body and mindfulness practices and psychological therapies such as cognitive-behavioral therapy may improve function or pain outcomes for specific chronic pain conditions.
  • There was no evidence suggesting serious harms from any of the interventions studied, although data on harms were limited.
  • Additional comparative evidence with data on the sustainability of effects on function and pain outcomes is particularly needed for chronic tension headache, as well as for osteoarthritis, chronic neck pain, and fibromyalgia.


Unfortunately, however, I read the outcomes of this review quite differently than its authors seem to. The term “usual care” was particularly suspect. “Usual care was defined as care that might be provided or recommended by a primary care provider” [Draft Report, Page3]. However, many of the therapies that were studied would more often be employed by specialists than by primary care providers. Moreover, the term “Quality of Evidence – Low” appears over 100 times in the draft report.

In my view, the conclusions of the systematic review might read quite differently than they appear above. I paraphrase from my comments as submitted to AHRQ:

     “Database searches conducted for this comprehensive review identified 4,470 potentially relevant articles. After dual review of abstracts and titles, 1,091 articles were selected for full-text dual review and 205 publications (192 trials) met inclusion criteria. We included 65 trials (69 publication) on chronic low back pain, 23 trials on chronic neck pain, 51 trials (54 publications) on osteoarthritis, 44 trials (50 publications) on fibromyalgia, and 9 trials on chronic tension headache. One-fourth of the trials excluded at full text did not meet our criteria for follow-up duration (i.e., a minimum of 1 month of follow-up after termination of the intervention).

“After a detailed review of the 205 publications, AHRQ essentially found that the quality of documentation for therapy outcomes was severely compromised by investigators’ failure to identify, explain and assess the baseline effects of “usual care” therapies that were likely combined with the techniques studied.  It is plausible and within the range of uncertainty for outcomes that none of the trialed techniques actually generated a positive effect beyond placebo when added to “usual care.” 

“At the present state of medical knowledge, published research has failed to demonstrate a measurable and conclusive positive effect for non-invasive, non-pharmacological treatments for chronic or intractable pain.  To correct this deficiency, most of the research will need to be repeated with much more rigorous attention paid to fully documenting trials protocols and applying methods of factors analysis to assess relative contributions when multiple therapies are applied concurrently.”


I would add parenthetically, that I am not surprised by the outcomes reported in this systematic study.  In the 20 years I have supported chronic pain patients and families, I’ve talked with thousands of people about the outcomes of so-called “alternative” therapies.  While a few have reported improvements in their pain or the quality of their lives, the great majority have not. 


I also suggest that outcomes of this systematic review seem directly pertinent to best practices in chronic and intractable pain management — and are contradictory to conclusions drawn in the 2016 CDC opioid prescription guidelines.  This review runs seriously counter to the naïve trust placed by the CDC guideline writers in non-invasive, non-pharmacological therapies for severe chronic pain. The AHRQ review can reasonably be said to demonstrate that medical science now lacks any standard of care for chronic pain which reliably improves on the record of opioid analgesics — or which offers realistic hope of replacing opioid therapy in patients who experience severe chronic and intractable pain.