The Lack of Clinical Quality Guidelines Causes Underestimation of Efficacy in Sham Controlled Acupuncture Trials

A serious flaw in the design of acupuncture research studies has been contaminating “sham controlled” clinical trials for decades. Unfortunately, this problem has been largely unacknowledged having only recently been questioned by researchers in China (1) even though it should be readily seen as rendering such trials incapable of answering the fundamental question they are designed to answer: How much of acupuncture’s clinical effects are due to placebo effects or the real but “nonspecific” effects of needling anywhere?

The flaw I am referring to is not the one that has been widely (and rightly) criticized – that the sham controls are not inactive and may in fact provide a therapeutic effect similar to the traditional acupuncture points being studied(2-5). Nor am I referring to the more recently raised question of trials being “underpowered” by small sample sizes although this may also be a valid concern (6). What I am referring to is a serious oversight in the design of many trials’ “real/verum/active” arms. Simply put – the lack of vetted clinical quality guidelines to guide the active arms’ clinical protocols means there is no way of knowing if the real acupuncture is being afforded the chance to be as effective as it could be. To further illustrate this problem, it is helpful to consider a term used in some legal circles: Maximum Therapeutic Benefit (MTB).

MTB is a term used in the medical insurance industry and in systems such as the U.S. Social Security Administration. Its meaning is just as its name implies – a state where a patient/client has achieved the maximum benefit that can be reasonably expected for a given therapy. Determining if MTB has been achieved is an important factor as most medical insurance plans are obligated by law to cover medical services until MTB is reached and establishing monetary awards for permanent disability is typically delayed until MTB in achieved in the treatment of the disability.

Unless reasonable steps are taken to insure MTB can be achieved in the active arm of a sham controlled trial it is impossible to estimate the real/verum acupuncture’s efficacy even if one could insure the sham control is inactive and the sample size is adequate. The lack of clinical quality guidelines can lead to the real acupuncture being administered in a suboptimal manner lowering its effectiveness rates below its true potential in a manner similar to using a suboptimal dosage of a drug in a placebo controlled drug trial. There is good reason to believe that some, perhaps many, of the trials that found real acupuncture to be only as effective as sham may well be invalid because the real acupuncture was carried out in a suboptimal clinical manner that did not to allow it to achieve MTB.

Why do I say there is “good reason” to believe this is a serious flaw in these trials? Because of two unresolved, glaring discrepancies seen across different trials; the first involving effectiveness rates and the second involving the number and frequency of treatments given. These two marked discrepancies are especially (but not exclusively) seen in trials conducted in East Asia vs. those conducted in the West.

Trials conducted in countries such as China, Korea, Japan, and Taiwan consistently show real acupuncture having high effectiveness rates in the 75%-90+% range and to clearly outperform sham controls (7). Many acupuncturists claim similar effectiveness rates within their practices and a recently published patient satisfaction survey taken of 89,000 U.S. acupuncture patients found similar rates of success in the treatment of their primary complaint (8). Sham controlled trails done in the West, however, tend to show a mixed set of results. Some trials find higher effectiveness rates similar to trials conducted in East Asia with real acupuncture outperforming sham while others report effectiveness rates in the 40%-50% range with real acupuncture not outperforming sham. The higher the effectiveness rates, the more likely the real acupuncture will outperform the sham as sham rates are rarely found to go beyond the 50%-60% range and are usually in the 40%-50% range. The critical question to ponder is what factors are responsible for the much higher effectiveness rates or, conversely, what factors are responsible for the comparatively low effectiveness rates seen in a significant portion of Western trials?

A common refrain from acupuncture critics is that “higher quality” sham controlled RCTs tend to find real acupuncture to not outperform sham while trials that show real acupuncture to outperform sham are of lower quality. Some have gone so far as to say that the consistently high positive outcomes (effectiveness rates) of Chinese trials mean those trials cannot be trusted (9-10). Yet the trial “quality” these critics are referring to is limited to the methodological quality of the trial design/implementation not the clinical quality. Of course, few, if any of these critics have the expertise to judge the clinical quality of the active arms of acupuncture trials.

When it comes to clinical quality for acupuncture therapy, while there can be many factors involved, experienced acupuncturists agree that effectiveness rates will be impacted by the frequency and numbers of treatments given over time, otherwise, why bother to ever do more than just one treatment? It is also common knowledge within the acupuncture community that in China, acupuncture treatments are typically done on a daily or every other day basis in the first stage of the treatment process and that dozens of treatments may be utilized in total, especially in the treatment of chronic conditions. Yet the vast majority of acupuncture trials conducted in the West use far fewer treatments in total and at a substantially less frequent interval. It seems quite reasonable to wonder if this might be a factor in the relatively lower effectiveness rates that plague Western trials.

Trials in the West often begin treatment at once a week and may only undertake 4 to 10 treatments in total. Some trials only use 1-3 treatments! Rarely will Western trials employ more than 2 treatments a week or beyond 20 treatments in total. The higher treatment frequencies/total numbers employed in East Asian trials are utilized by highly experienced acupuncturists to help achieve the highest effectiveness rates (MTB). Yet, despite this great disparity in treatment frequency/total numbers, no clinical rational is offered to explain why Western researchers have confidence that their treatment frequency/total numbers are clinically justified. Researchers often go to great lengths to explain why their trial design is adequate – blinding methods, sample sizes, control protocols, etc., but the rationale behind clinical protocols is given far less consideration. It would be hard to imagine such a great disparity in drug dosages being accepted in drug trials without researchers offering some clinical rational for using those different dosages.

There are some rare exceptions. Two trials on osteoarthritis of the knee, one done in the U.S.A. and the other in Israel (11-12), both utilized a higher treatment frequency (twice a week for 8 weeks) and total treatment numbers than is usually the case in Western acupuncture trials . Both of these trials found that the effectiveness rates for the real acupuncture did not start to significantly surpass the sham until 12 weeks – a longer timeframe than most Western studies allow for. Other studies have found similar trends with higher numbers of treatments leading to higher effectiveness rates (13-14).One has to wonder how many trials that did not find real acupuncture to outperform sham may have done so if they had utilized more frequent treatments carried-out over longer periods of time.


Until it becomes the norm to place as much emphasis on clinical quality as is placed on trial methodology quality, research on acupuncture will not give an accurate picture of that therapy’s efficacy and true clinical potential.


Some researchers may feel justified in the use of these lower treatment frequency/total numbers in their trials because many acupuncturists in the West treat their patients with similar frequency/total numbers. This type of research design, it could be reasoned, is mimicking actual practice. That may (or may not) be true. However, sham controlled acupuncture trials are not being promoted as a means to test acupuncture’s efficacy based on how it tends to be practiced by some acupuncturists in the West where cost concerns often constrain optimal treatment numbers. They are being promoted as a test of acupuncture’s actual efficacy and the stigma of trials showing real acupuncture not outperforming sham has tainted acupuncture’s legitimacy and slowed its acceptance in the West.

I understand that some involved with the acupuncture research community may take the view that the focus for acupuncture research moving forward should be on directly comparing acupuncture to other therapies (comparative effectiveness) rather than sham controlled studies. I agree. In fact, I believe the focus should have been on comparative effectiveness all along. Still others would argue that even with the poor clinical quality trials having been done, acupuncture’s efficacy has been shown in several conditions to be at least equal to and sometimes better than other widely used therapies and that acupuncture is being unfairly held to a higher standard than other therapies. I agree with that also. However, there is a large body of research in the record, both sham controlled and otherwise, that has been conducted with poor clinical quality standards that very likely lowered true acupuncture’s actual effectiveness profile. This flaw needs to be acknowledged, implications debated, and corrections made to both help interested parties set the existing record straight and guide acupuncture research moving forward whether involving sham controlled trails or not.

Considering that sham controlled acupuncture trials have caused a significant negative impact on acupuncture’s acceptance, it is incumbent upon the acupuncture community – both researchers and clinicians – to address the issue the absence of clinical quality guidelines is having on such trials. Until it becomes the norm to place as much emphasis on clinical quality as is placed on trial methodology quality, research on acupuncture will not give an accurate picture of that therapy’s efficacy and true clinical potential. The following steps will help address this issue and the Acupuncture Now Foundation is very interested in working with those who wish to pursue dialog and action on these steps or similar steps.

1. Acupuncture researchers need to acknowledge that in order for sham controlled trails to legitimately gauge true acupuncture’s potential efficacy, steps must be taken to insure the verum arm of the trial allows the true acupuncture a chance to achieve MTB and justification of these steps needs to be described.
2. If a sham controlled trial is not intended to gauge true acupuncture’s potential efficacy but rather to test the potential efficacy of some limited application of acupuncture techniques, this needs to be clearly stated in the title and description of the research when published.
3. Experienced acupuncture clinicians need to work at developing best practice clinical quality guidelines to guide both researchers and clinicians to reach, at very least, an understanding of how many treatments over what period of time acupuncture therapy may need to be applied to achieve MTB.
4. Until such time that best practice clinical quality guidelines can be established to guide both future acupuncture research and systematic reviews of acupuncture trials, panels of clinical experts should be formed and used to guide researchers in understanding how to distinguish optimal from suboptimal clinical quality protocols to reduce the number of poor clinical quality RCTs and foster more accurate analysis and sub-analysis in systematic reviews.
5. To help guide future research and clinical practice, more systematic reviews of acupuncture trials should be undertaken contrasting effectiveness and/or efficacy findings relative to the use of different factors influencing clinical quality including (but not limited to) treatment frequency/total numbers and the training and experience of involved acupuncturists. The largest review that attempted this to date had only 21% of included trials that utilized two treatments per week, no trials utilizing more than two per week, and was dominated by trials using acupuncturists with quite limited training (14).

References:

1. Analysis and Thoughts about the Negative Results of International Clinical Trials on Acupuncture
Wei-hongLiu,YangHao,Yan-jingHan,Xiao-hongWang,ChenLi,andWan-ningLiu Evidence-Based Complementary and Alternative Medicine Article ID 671242
2. Birch S. A review and analysis of placebo treatments, placebo effects, and placebo controls in trials of medical procedures when sham is not inert. J Altern Complement Med. 2006 Apr;12(3):303-10.
3. Lund I, Lundeberg T. Are minimal, superficial or sham acupuncture procedures acceptable as inert placebo controls? Acupunct Med. 2006 Mar;24(1):13-5.
4. Lund I, Naslund J, Lundeberg T. Minimal acupuncture is not a valid placebo control in randomised controlled trials of acupuncture: a physiologist’s perspective. Chin Med. 2009;4:1.
5. Lundeberg T, Lund I, Sing A, Naslund J. Is placebo acupuncture what it is intended to be? Evid Based Complement Alternat Med. 2011;2011:932407.
6. Influence of Control Group on Effect Size in Trials of Acupuncture for Chronic Pain: A Secondary Analysis of an Individual Patient Data Meta-Analysis Hugh MacPherson , Emily Vertosick, George Lewith, Klaus Linde, Karen J. Sherman, Claudia M. Witt, Andrew J. Vickers, on behalf of the Acupuncture Trialists’ Collaboration Published: April 4, 014https://doi.org/10.1371/journal.pone.0093739
7. Do certain countries produce only positive results? A systematic review of controlled trials. Vickers A, Goyal N, Harland R, Rees R. Control Clin Trials. 1998 Apr;19(2):159-66.
8. ACUPUNCTURE Does Acupuncture Provided Within a Managed Care Setting Meet Patient Expectations and Quality Outcomes? A 2-Year Retroactive Study of 89,000 Managed Network Patients SOURCE: American Specialty Health Incorporated Health Services Department http://files.clickdimensions.com/ashcompaniescom-a7oce/files/acupuncturecahps.pdf
9. Positive Results in Randomized Controlled Trials on Acupuncture Published in Chinese Journals: A Systematic Literature Review Yuyi Wang, Liqiong Wang , Qianyun Chai, Jianping Liu http://online.liebertpub.com
10. “And this is why we might as well forget about Chinese acupuncture trials” Published Wednesday 21 May 2014 http://edzardernst.com
11. Effectiveness of Acupuncture as Adjunctive Therapy in Osteoarthritis of the Knee A Randomized, Controlled Trial. Brian M. Berman, MD; Lixing Lao, PhD; Patricia Langenberg, PhD; Wen Lin Lee, PhD; Adele M.K. Gilpin, PhD; and Marc C. Hochberg, MD Annals of Internal Medicine 2004;141:901-910.
12. Delayed Effect of Acupuncture Treatment in OA of the Knee: A Blinded, Randomized, Controlled Trial. Ehud Miller, Yair Maimon, Yishai Rosenblatt, Anat Mendler, Avi Hasner, Adi Barad, Hagay Amir, Shmuel Dekel, and ShaharLev-Ari Hindawi Publishing Corporation Evidence-Based Complementary and Alternative Medicine Volume 2011, Article ID 792975, doi:10.1093/ecam/nen080
13. Acupuncture for the prevention of episodic migraine. Cochrane Database Syst Rev. 2016 Jun 28;(6):CD001218. doi: 10.1002/14651858.CD001218.pub3.Linde K1, Allais G, Brinkhaus B, Fei Y, Mehring M, Vertosick EA, Vickers A, White AR.
14. Characteristics of Acupuncture Treatment Associated with Outcome: An Individual Patient Meta-Analysis of 17,922 Patients with Chronic Pain in Randomised Controlled Trials MacPherson H, Maschino AC, Lewith G, Foster NE, Witt CM, et al. (2013) PLoS ONE 8(10): e77438. doi:10.1371/journal.pone.0077438

Matthew Bauer began his full-time practice of acupuncture and Chinese Medicine in 1986 after several years of studying Taoist history and philosophy with a 74th generation Taoist Master. In addition to working with several acupuncture/Chinese Medical organizations, Matthew founded the Acupuncture Now Foundation in 2014. Serving as a consultant in the managed care industry since 1998, Matthew helped to create the first managed care acupuncture-based credentialing and utilization management guidelines and now serves on the Board of Directors of American Specialty Health Group, Inc. As a managed care consultant, Matthew took part in a consensus-building think-tank with a dozen experienced acupuncturists from the U.S., Mainland China, Taiwan, and Korea that developed clinical quality guidelines. That experience convinced Matthew of the need for experienced acupuncturists to work at building consensus on clinical issues to guide clinicians and researchers and further our understanding of this ancient healing system. Matthew has authored dozens of articles and two books and he has a passion for educating people about acupuncture believing it to be a responsibility of those who understand how it works to share this knowledge with those who don’t.

Posted in Uncategorized.

One Comment

  1. Well said Matthew, I couldn’t agree more. Maybe it would also be useful – to help with this issue – if acupuncture trials described themselves as testing ‘an acupuncture approach/protocol/intervention’ rather than acupuncture per se. After all, we don’t see trials of ‘conventional medicine for migraine’ we see a specific drug or intervention under study.

    I recently attended the Acupuncture Resource Research Council conference in London, and part of Dr Hugh MacPherson’s presentation showed average placebo response rates across a variety of trials, from conventional medicine to acupuncture. I believe the average was 40% (but don’t quote me!) I thought this begged the question: if we already have LOADS of data that tells us what an average placebo response is, why do we bother having a placebo (or even worse ‘sham’) arm at all?? Why don’t all researchers just save themselves a load of money and trouble and simply only do pragmatic trials now, comparing to usual care or best current treatment, taking into account that 40% of everyone’s response is due to non-specific effects.

Leave a Reply

Your email address will not be published. Required fields are marked *