Drinking Water Information Home

Close this window to return to the previous page.

Drinking water information and resources, contaminants, health effects, treatment methods

 The Problem With Testimonials or Anecdotes

Testimonials and case studies are worthless as evidence,
but not as inspiration
-
Cedar Riener
Sincerity and passion can never validate a scientific theory - R. Johnson

The Bottom Line:
Testimonials about the effectiveness of a health treatment are important, but ONLY as a source of questions and ideas for further rigorous testing, because there is no way to determine whether the reported treatment outcomes were actually caused by the treatment or by one or more of these alternative explanations::
  The Placebo Effect
  Spontaneous Improvement
  Confusion between Causation and Correlation
  A Misdiagnosis of the health problem
  Undocumented Treatments
  Subjective Symptoms & Outcomes
  A Sample Size of One
Cognitive Biases
Unscrupulous Marketing Practices
Only Positive Outcomes Are Reported
  A Desire to Bypass Regulations
Scientists can actually use what amount to testimonials to collect data on subjects who participate in controlled experiments.
If a placebo product seems to make me feel better what's the harm in using it, and what's the harm if the company promoting it using pseudoscientific claims?
We are constantly bombarded by marketing programs (whether they are media ads or in-person demonstrations) that try to provide compelling evidence that will convince us that their product is effective.  There are two basic types of evidence that can be used to provide validation that specific claims about any product are true:
  • Testimonials or Anecdotes - people's subjective observations, opinions and descriptions about how they believe a product has helped or harmed them.
  • Experimental Evidence - specific, measurable (objective) information about a product's performance and effectiveness that has been collected in a controlled manner to minimize errors and bias.

This discussion will examine Testimonial Evidence and my contention that: In the absence of valid, reliable experimental evidence to support health claims made for alkaline water, homeopathy, other enhanced water products and hundreds of other 'health' products and services, promoters of the products or services must rely on what I call Uncontrolled Testimonials - stories and experiences that are collected specifically (and selectively) to support product claims that cannot be validated by any sort of controlled process like a scientific experiment, or even a well-designed survey.

Because uncontrolled testimonial evidence is completely subjective, different people might well come to very different conclusions about whether a product worked for them, so there is no way to actually assess a product's real effectiveness.  Think about it --- if you talk to two people at a sales meeting; one relates a positive experience with a product and the other describes a negative experience after using the same product, how can you determine which testimonial is an accurate description of the product's effectiveness?

Testimonials are important, but ONLY as a source of questions and ideas for further rigorous testing (blinded experiments where practical). One person’s (or even many people’s) uncontrolled reaction to a treatment that is claimed to produce health benefits is completely unreliable as a proof the treatment actually has any effect.

The problem with uncontrolled testimonials is that there are many alternate, and often more plausible, explanations for the observation of increased health that have nothing to do with the 'health' product or service being promoted.  The points discussed below can cover any of the water products or treatments that claim to produce a health benefit including, but not limited to, water that has been: ionized, clustered, structured, exposed to magnets, energized, oxygenated, vortexed, hydrogenated, intentionalized, catalyzed, m-activated, infused with noble gasses, exposed to pi ceramics, or serially diluted with succussion (shaking). 

This warning to be skeptical of uncontrolled testimonials applies to any product or service that does not have valid experimental evidence to support claims of effectiveness. 

11 reasons to be skeptical of uncontrolled testimonials about health benefits of any product or service:

  1. Placebo Effect:  Observed results can be caused by the placebo effect - a perceived health improvement caused by the expectation of a beneficial outcome.  How does one determine whether perceived health benefits of drinking alkaline water, drinking oxygenated water or sipping a homeopathic remedy are due to actual biochemical effects of the product and not a result of the placebo effect?  The placebo effect and the “power of suggestion” are remarkably compelling phenomena that can lead to the perception of greater energy, reduced pain, and a number of other health benefits that have been well documented over the years.  The placebo effect can even lead to mind-generated biochemical changes in the body that mimic the effect of some drugs.  There is no way that reported health benefits of alkaline water, homeopathy or other enhanced water products can be shown to be different from manifestations of the placebo effect without rigorous, unbiased experimentation.  If you take the time to locate and critically examine any alleged scientific evidence provided to support some of the altered &/or enhanced water claims, you will find either that no health benefits have been demonstrated or that, in carefully controlled experiments, any benefits can be shown to result entirely from the placebo effect.  An interesting question to consider: If a treatment causes either a perceived benefit to someone or actually results in biochemical changes that reduce pain or anxiety, is that a bad thing - even though the claimed process has no scientific validity and the treatment has no actual, direct, measurable biochemical effect on the body?
     
  2. Spontaneous Improvement:  Spontaneous, natural improvement in symptoms is attributed to a treatment.  In these cases, people will often try a treatment when their symptoms are the worst – which often just happens to be when the symptom severity will normally begin to improve.
    • Many diseases, like colds, flu, cough, allergies, etc., spontaneous improve over a week or two in most people regardless of treatment.
    • Symptom severity in patients with chronic ailments like anxiety, back pain, migraines, etc., may naturally fluctuate over time.
    If perceived healing can be caused by timing of the treatment, what observations or measurements could one person make to demonstrate that a given healing episode was due to the treatment and not to ordinary healing processes?  It has be said that if you have a cold you will normally get over it in a week, but if you take a special treatment like alkaline water or a homeopathic remedy, you will be cured in only seven days. 

    Spontaneous improvement of symptoms can lead to confusing causation with correlation - in other words, two events might be linked in time but there may be no cause and effect relationship between them.  This is an important specific example of point number three that needed its own discussion.  This is an example of confusing causation with correlation, but in this discussion it is common enough to list separately.
     
  3. Confusing Causation with Correlation is one of the many logical fallacies employed by those who promote pseudomedicine and pseudoscience and can be illustrated by the example, "Many people who have strokes have been taking blood pressure medications. Those blood pressure pills must be causing strokes, therefore to be safe use our completely natural, μαγεία drink to lower your blood pressure instead of those dangerous medicines!"

    Attributing the cause of an observed health effect (or any event for that matter) to some incident that precedes it, as described above, is just one example of confusing Correlation (a relationship in time) with Causation (a real cause and effect relationship).  In other words, what may appear true on the surface because events are related in time, may lead to a completely false, though apparently reasonable, conclusions.  This rather extreme illustration demonstrates why eating pickles is dangerous and should be avoided at all costs.  While this is a parody, it clearly illustrates how events that are correlated in time (eating pickles followed by serious illness and death) can be misconstrued and used to try and prove causation.  Another interesting report links the installation of each additional cell-phone tower to the birth of 18 additional babies.  The confusion between correlation and causation is evident when marketers use uncontrolled testimonials to try and demonstrate their product actually caused a positive health outcome when, at best, there may only be a correlation.


    The correct interpretation of the first example is that one of the risk factors for stroke is high blood pressure, so those who take blood pressure medication will already have an increased risk of strokes, and the blood pressure medications may actually reduce the stroke risk.  To discover whether or not specific medications or other treatment options used to reduce blood pressure will also lower the risk of strokes, experiments must be performed to control for the many potential factors that can complicate the interpretation of simple observations.

  4. A Misdiagnosis:  The symptoms or illness could be misdiagnosed.  For example, a person is informed by a care giver they have cancer (or other specific disease).  They are told (or believe) the condition is incurable or would require a drastic intervention to cure.  They begin to drink alkaline water, take a homeopathic remedy or try some other treatment because a friend tells them it can help cure cancer.  Two months later they have another checkup and are told there is no cancer.  Is it more likely the cure was a result of the treatment, a miracle or a misdiagnosis?

    A personal example: In June 2013 my 90 year old mother was told she had untreatable lung cancer and perhaps three months to live by her primary physician based on an x-ray. In this case, we did not believe it and asked for another diagnosis from another doctor based on a MRI scan. That scan found no evidence of cancer.  It would have been easy, in desperation, to try some alkaline water or other alternative treatment in the weeks before the MRI was scheduled and then attribute the absence of cancer to to the treatment's effectiveness.

    It is also quite possible, in some cases, that people simply decide they have a specific disease (based on some symptom and something they hear on TV, a friend’s comment, an Internet article, etc.), embark on a treatment protocol like drinking alkaline water, feel better after a few days/weeks, and believe the treatment cured them of their self-diagnosed, nonexistent illness.

  5.  
  6. Undocumented Treatments:  Switching or adding treatments or changing behavior may cause a real improvement in symptoms, but the ‘cure’ is attributed to the original treatment.  An example might be if someone who has been working hard lifting heavy objects develops lower back pain and begins taking a homeopathic pill that’s ‘guaranteed to reduce pain’.  They would probably also stop lifting things and they might go to bed and use ice and heat treatments.  After a few days on the homeopathic remedy their back feels better.  It’s likely that the homeopathic treatment will receive credit for the cure (because they had to actively pursue and purchase the treatment) and the contribution of the other life-style changes will be overlooked or minimized.  With an uncontrolled testimonial there is no way to know what else the person might have done to try and improve their health. 

  7. Subjective Symptoms & Outcomes:  Subjective evidence is the testimony of how a product performed based on the perception of the subject.  The quality of subjective evidence depends upon the honesty of the subjects, and their ability to accurately perceive reality.  These testimonials almost almost never include measurable reference points or targets for the symptoms or treatment goals.  In other words what does “increased energy”, “increased mental acuity”, or "increased heart health" or other common claims like "better hydration", or "increased detoxification" really mean?  What measurable characteristics can be documented before and after drinking alkaline water?  In the absence of specific measurements and documentation, the mind can create any story it likes based on a desire to become healthier.  Even if an individual created and documented measurable reference points and recorded some health benefits after drinking alkaline water or trying another enhanced water product, that would only suggests a correlation, not a cause and effect relationship - and could not rule out the placebo effect or some of the other reasons discussed here. 
     
  8. A Sample Size of One:  How does anyone know what would have happened if they had not tried alkaline water, a homeopathic remedy, another enhanced water product, or some other treatment?  I submit that it is impossible to know “what might have been if only I had done something differently”.  A testimonial I received about alkaline water, for example, claimed that “Except for the roughest allergy season I've experienced in my 45 year memory, I haven't been sick in almost 5 years now. Does that mean alkaline water is not effective against allergies - perhaps alkaline water causes severe allergic reactions every five years?  Might some of the allergic symptoms actually have been a cold?  Might there be other reasons to account for her lack of colds?  Can anyone really know what their health would have been over five years if they had not been drinking alkaline water?

    Testimonials can be hard to ignore, particularly when they come from a family member, trusted friend, someone you accept as an authority, or within the context of a meeting of excited individuals who have similar testimonials.  Humans are 'hard-wired' to pay attention to stories from others - that is how everyone learns virtually every thing they know about the world (besides their own personal experiences, of course).  Information is communicated from a person who has knowledge and experience to someone who does not. The problem is that a testimonial, even from a trusted source, may not be accurate because, even though the person may be sincere about their experience, they may be completely wrong about the cause and effect relationships for any of the reasons discussed here.  
     
  9. Cognitive Biases:  There are a number of unconscious psychological phenomena that conspire to influence people's perception of reality and can lead to the reporting of an effective treatment (or otherwise skew reality) when there was no actual physical change.  Examples include:
    • Confirmation Bias (people naturally seek information that supports their beliefs and disregard disconfirming evidence - self-fulfilling information gathering)
    • Bandwagon or Herd Effect (people tend to follow the crowd and interpret their experiences as others in their circle do)
    • Social Desirability (the tendency of people to answer questions in a way that will be viewed favorably by others - related to the bandwagon effect)
    • Selective Perception (the tendency for expectations to affect perception - a component of the placebo effect where there is no actual physical change)
    • It is informative to go through this list and see what other biases might fit.
       
  10. To put it bluntly, Some People Use Unscrupulous Marketing Practices – particularly when there is a profit to be made.  There is usually no way to determine whether testimonials used in marketing materials are sincere reports of people’s experiences (or whether those experiences were influenced by the points described above), paid endorsements, or complete fabrications by those selling the product/process.
     
  11. To illustrate this point, I went to an Amazon sales site and copied a several anti-testimonials for the Enagic Kangen Water Leveluk SD501.  These testimonials will obviously not make it into the sales brochure or be promoted at distributor meetings:

    I was fooled and duped into buying one of these machines, and two years later my health problems are exactly the same. . I did everything the company said to do, like giving up soda and all "acidic" beverages like certain juices and other drinks. I thought at first I was seeing real results, so I bought a machine, but all I did was saddle myself with debt for the machine. After visiting the doctor, it was clear that my health had not improved. The person who sold me the machine told me to be patient, that it would probably take me a full year for my body to become "alkaline". So I kept it up for two entire years, but nothing ever improved (and some things got worse).  Sera

    I've had a Kagen water machine for two years. Before that my family and dogs were healthy. In the last 6 months my husband was diagnosed with Melanoma and my 5 year old Maltese who gets Kagen water daily with Mast Cell Cancer. I've seen no benefit what so ever from this product. Do not waste your money.  MichelleY

    I would LOVE to start a Class Action Suit, and get my hard earned money back, for believing this machine and the water it produces, have healing affects
    Earthie Earth
  12. Only Positive Outcomes are Reported (Cherry Picking):  Testimonials (at least as used by those marketing placebo products) only document and report positive outcomes. There is no way to know how many people used the product and experienced no positive benefits or got worse because their condition was not effectively treated.  Marketing for FDA approved medications must include all reported side effects – at least there is some transparency in traditional medicine.

    On the other hand, there is no requirement to demonstrate the effectiveness of alternative medical treatments or report the number of patients who do not improve, who experienced side effects or who got worse after treatment - or who died.  Also note the interesting reference to 'Undocumented Treatments' in the first anti-testimonials to the right that clearly shows how the company representatives suggested making additional life-style changes that, by themselves, could contribute to perceived changes in health.
     
  13. Used to Bypass Regulations:  Testimonials are regularly used by placebo marketers to bypass FDA and FTC regulations against using untested, unsupported claims about specific disease treatments. A company seldom directly states that drinking alkaline water, for example, can cure cancer, diabetes or other specific disease, but many distributors list testimonials from consumers who claim to have been cured or alternative practitioners who claim to have treated or cured a specific disease (search Google for "kangen testimonials").

    The only requirement necessary to market alternative products and treatments that claim to provide such vague health benefits as: Greater energy and vitality, Increased hydration, Enhanced oxygen delivery, Better overall health, Decreased stress levels, A more positive attitude, Boosted metabolism, Detoxification of the body, etc., etc. is to add the Get Out of Jail Free Disclaimer everyone has seen, "This statement has not been evaluated by the FDA.  This product is not intended to diagnose, treat, cure or prevent any disease"  to their product. 

    Check out the websites of most large manufactures of enhanced/altered products.  Their actual claims - the ones for which they can be held legally responsible - are all vague and meaningless.  It is left to the distributors and the testimonials they employ, which are 'outside the control of the parent company' to make specific claims about how the product really does cure arthritis, diabetes, cancer, etc.

The points above illustrate why I am skeptical about relying on uncontrolled testimonials to any support health claims for any product.

This table briefly outlines how blinded experiments (described in more detail here) can produce controlled testimonials (and other information) that minimize the problems associated with uncontrolled testimonials - Specifically:
1.  The Placebo Effect - Blinded study participants do not know whether they are receiving the real treatment (or product) or a placebo, so any placebo effects will be similar in both groups and cancel each other out.
2.  Spontaneous Improvement - Blinded study participants are similar in age, health status, etc., so health conditions and spontaneous improvement rates would be expected to be similar in both groups and cancel each other out.  Improvements can be demonstrated to be caused by the treatment.
3. Confusing Causation with Correlation - Blinding does not directly help the process of sorting out cause and effect relationships, but it allows the process to take place with minimal bias and expectations that can cause either the researchers or the subjects to incorrectly conclude a correlated event is actually a cause.
4.  A Misdiagnosis - Always a possibility, but since the subjects are followed over the course of the experiment and have as close to the same health status as possible (or the same symptoms & diagnosis) the chance for significant misdiagnosis is minimized.
5.  Undocumented Treatments - Part of a well designed blinded study is to instruct all participants in both groups not to try other treatment methods during the experiment and to follow the same routines when taking the treatment.
6.  Subjective Symptoms & Outcomes - Part of a well designed blinded study is to design objective outcomes that can be clearly identified and measured (or rated) and instruct participants in both groups how to identify and record the outcomes of interest.
7.  A Sample Size of One - A well designed blinded experiment has a number subjects in each group - the more the better - so data can be collected (and results compared) from many people.  The results are analyzed in a way that best demonstrates if there are any real differences between outcomes in the two groups or if any differences observed are probably due to chance.
8. Cognitive Biases - All of the biases described above are minimized because neither the subjects nor the experimenters know which group is taking the real, active product or treatment and which is taking the placebo, so any biases in reporting or analysis should be random and cancel out between the experimental groups.
9. Unscrupulous Marketing Practices - Sometimes scientists will falsify their evidence and produce fraudulent findings.  Usually, however, other scientists will identify and expose the fraud.  Science is designed to be self-correcting.
10. Only Positive Outcomes Reported - A well designed and conducted blinded study will collect all data, both positive and negative,  from all subjects.  Since neither the participants nor those conducting the experiment know which group is taking the active product until after the analysis has been completed, there is no benefit to withholding evidence.
11.  A Desire to Bypass Regulations - Obviously, where there is lots of money to be made, there is a desire by some (inside and outside of the scientific community) to break the rules.  At least there are checks and balances for those selling FDA regulated products that may catch those who try and circumvent the regulations. There are no equivalent checks and balances for those selling products and services for which health claims are made that are not FDA regulated.

As noted, however, testimonials are actually used to collect data from participants in valid, scientific experiments.  These controlled testimonials (or data) collected from experimental subjects are entirely different from uncontrolled testimonials with the limitations described above.  They are records of the experiences (and can include specific biological measurements) of individuals who participate in experiments, they are not selectively collected only from those who have experienced a positive outcome.  A good experiment is designed so that the eleven limitations described above are minimized, and the testimonials can actually be used to help establish a cause and effect relationship (if one exists) between treatment and a health outcome.  The most effective experiments are blinded - they compare the effectiveness of a product or service as a treatment for some health issue with a placebo (a product or service that is as similar as possible to the product or service that is being tested, except that it has no real biological action - the famous 'sugar pill').

You might ask, What is the harm in all the different alternative health treatments?  If the products don't really do anything except perhaps trigger the placebo response (and actually make someone feel better), or perhaps help someone feel like they have some control of their health situation, why criticize the products?  If the only consequences of using products with no real biological effect (beyond the placebo) were either that there was no health effect or that someone actually felt better, the main criticism would be the use of unproven claims and deceptive marketed practices.  Unfortunately, there are more serious consequences.  Most people are wise enough to not use these products in life threatening situations, and they will seek real medical treatment if their health situation continues to deteriorate.  However, some individuals fall completely under the spell of these illusionists and wind up seriously injured or dead.  Tim Farley has developed a site, What's The Harm?, that documents what you will never hear in the testimonial-driven marketing of alternative treatment products.

One common strategy of nearly all marketing programs for products that are based on pseudoscience is the proclamation that mainstream science is not to be trusted, followed by a litany of alleged evils attributed to science, the medical industry, government regulations, big pharma, etc.  I am the first to admit that science, traditional medicine, government, big pharma, etc. are not perfect (individuals and groups have taken advantage of and corrupted the system), but they at least have checks & balances and structures in place to detect and minimize bad science and fraud.  Most of the evil examples cited by the pseudoscientists were, in fact, discovered by and exposed by other scientists using standard scientific practices.

By discrediting mainstream science, the goal is to elevate their platform (alternative/pseudoscience) to legitimacy.  Unfortunately, that argument does not work - even if mainstream science and traditional medicine were demonstrated to be completely untrustworthy and evil, that would not automatically validate the processes or conclusions of the alternative/pseudoscience community. 

Pseudoscience has no system of checks and balances, there is no plan or process within the community of those promoting enhanced/altered water products - or any of the thousands of alternative treatments for which science cannot provide evidence of effectiveness - to detect, expose and correct misleading advertisements or fraud.

How can anyone believe that a community that promotes marketing which uses uncontrolled testimonials, that has no self monitoring system or regulations to ensure safety and effectiveness of treatments and that dismisses controlled experiments as worthless is superior to the mainstream scientific community.  Mainstream science, for all its faults, is self monitoring, self correcting, and has ultimately created the world (both the good and bad elements) that we inhabit today.  Pseudoscience has never produced anything of real and lasting value.

It is interesting to observe that most experiments carried out by mainstream scientists that have been reviewed and published can be successfully duplicated by anyone else in the world - whether they believe the results or not.  The published methods and analyses are transparent.  On the other hand, most experiments carried out by those who promote alternative treatments seem to only be reproducible by others who already believe in the outcomes that are 'supposed to occur'.

Close this screen to return to the previous page.

Copyright © 2005 Randy Johnson. All rights reserved.

Updated November 2011