The use of medicinal plants is an important part of traditional medicine in many cultures and is the basis of ongoing efforts to develop new drugs in conventional medicine. A common criticism of herbal medicine is the lack of properly designed clinical trials. Although many trials exist for Western and Chinese herbs, reviewers and meta-analysts have repeatedly commented on the poor methodological quality of available studies and the possible presence of publication bias.

The objective of this study was to compare the effects observed in placebo-controlled trials of Western herbal medicine with trials of conventional pharmaceutical medicine and to assess the quality of trials and the probability of such publication and related biases.

The authors searched 19 electronic databases and the Cochrane CENTRAL trials register. A trial was included if it (1) was a controlled trial of treatments or preventative measures with clinical outcomes, (2) had a parallel group design with placebo control, (3) used random or quasi-random assignment to groups,

(4) had sufficient data to allow the calculation of odds ratios, and (5) was a trial using an herb listed in the monograph of the European Scientific Cooperative on Phytotherapy (ESCOP) or approved by the Swiss licensing authority. Studies of Chinese herbs, herbal dietary supplements, Ayurveda and Bach flower therapy, aromatherapy, and anthroposophical medicines (i.e., those based on the work of author, philosopher Rudolph Steiner) were excluded, as were trials in healthy volunteers and those with a crossover design.

Prespecified criteria for identifying the outcome for inclusion in the analysis were used. The first choice was the main outcome measure, which was defined as the outcome used for sample size calculations. If a main outcome was not specified, alternative outcomes were selected in the following order: (1) patients’ global assessment of improvement, (2) physicians’ global assessment of improvement, or (3) the other most relevant clinical outcome measure.

The authors analyzed 89 placebo-controlled trials of herbal medicine and 89 matched conventional medicine trials. The results demonstrated that trials involving mental or vascular disorders and neurological/cognitive complaints in older age were the most common conditions studied in the paired trials. Furthermore, preparations of ginkgo (Ginkgo biloba, Ginkgoaceae), St. John’s wort (Hypericum perforatum, Clusiaceae), evening primrose (Oenothera biennis, Onagraceae), ginger (Zingiber officinale, Zingiberaceae), hawthorn (Crataegus laevigata, Rosaceae), and saw palmetto (Serenoa repens, Arecaceae) were examined in 56 (63%) of the trials. Among trials of herbal medicine, more beneficial effects were observed when preparations were standardized and were for indications listed in the corresponding ESCOP monograph.

In general, the methodological and reporting quality of the included trials of Western phytotherapy was on average superior to trials of conventional medicine; although, in both groups a clear majority of studies were of inadequate or uncertain quality. Only 5% of the conventional medicine trials were considered of higher quality—a striking contrast to the 21% of herbal medicine trials. That is, the herbal trials were rated higher than conventional medicine trials by a factor of over 4 to 1.

However, the inclusion criteria for this review allowed only studies that used sophisticated research strategies, ensuring more high quality phytotherapy research. In addition, the exclusion criteria for this study eliminated not only non-Western practices such as Traditional Chinese Medicine and Ayuvedic medicine, but also studies of “traditional Western herbalism,” reflecting a selection bias that is not broadly representative. Studies of herbal medicine were smaller, less likely to be published in English, and less likely to be indexed in MEDLINE than their counterparts from conventional medicine. In both groups, smaller trials showed more beneficial treatment effects than larger trials.

Edzard Ernst, MD, PhD, professor at the department of Complementary Medicine, Peninsula Medical School, Universities of Exeter & Plymouth, suggests caution when interpreting the results of this study: “We know that smaller studies tend to report misleadingly large effect sizes” (E. Ernst e-mail to M. Blumenthal, August 13, 2007). Dr. Ernst goes on to admit this is a simple explanation of the phenomenon, because randomized controlled trials are costly and large ones are significantly expensive. Thus, herbal and phytomedicine companies, as compared with pharmaceutical companies, have smaller research budgets and are thus less likely to fund the larger trials associated with conventional pharmaceuticals.

According to Dr. Ernst, “Phytotherapy researchers are doing their very best to conduct rigorous research—so much so, that they publish better quality than their conventional colleagues who have the benefit of being able to rely on huge funds, while the phytotherapy sector has to make ends meet on a shoe string. One does not need to be a rocket scientist to find the solution to the problem: as phytotherapy research has no big industry backing, it requires much more official support from governments and other non-commercial sources to carry on its good and important work.”

Although the authors of this study were unable to confirm the widely held belief that the quality of the evidence on the effectiveness of herbs and phytomedicines is inferior to the evidence that is available on conventional medicine for comparable conditions and outcomes, they opened the door for more discussion and research on assessing methodological and reporting quality of trials.

—Jennifer Minigh, PhD