We are constantly bombarded by marketing programs (whether they are media ads or
in-person demonstrations) that try to provide compelling evidence that will
convince us that their product is effective. There are two basic types of
evidence that can be used to provide validation that specific claims about any
product are true:
|The Bottom Line:
Testimonials about the effectiveness of a health treatment are important, but ONLY as a source of
questions and ideas for further rigorous testing, because there is no
way to determine whether the reported treatment outcomes were
actually caused by the treatment or by one or more of these
The Placebo Effect
Confusion between Causation and
A Misdiagnosis of the health problem
Subjective Symptoms & Outcomes
A Sample Size of One
Unscrupulous Marketing Practices
Only Positive Outcomes Are Reported
A Desire to Bypass Regulations
Scientists can actually use what amount to
testimonials to collect data on subjects who participate in
If a placebo product seems to make me feel better
what's the harm in using it, and what's the harm if the
company promoting it using pseudoscientific claims?
- Testimonials or Anecdotes - people's subjective observations,
opinions and descriptions about how they believe
a product has helped or harmed them.
- Experimental Evidence -
specific, measurable (objective) information about a product's performance
and effectiveness that has been collected in a controlled manner to minimize
errors and bias.
This discussion will examine Testimonial Evidence and my contention
that: In the absence of
valid, reliable experimental evidence to support health claims made for alkaline water,
homeopathy, other enhanced water products and hundreds of other 'health'
products and services, promoters of the products or services must rely on what I
call Uncontrolled Testimonials - stories and experiences that are
collected specifically (and selectively) to support product claims that cannot
be validated by any sort of controlled process like a scientific experiment, or
even a well-designed survey.
Because uncontrolled testimonial evidence is completely subjective,
different people might well come to very different conclusions about whether a
product worked for them, so there is no way to actually assess a product's real
effectiveness. Think about it --- if you talk to two people at a sales
meeting; one relates a positive experience with a product and the other
describes a negative experience after using the same product, how can you
determine which testimonial is an accurate description of the product's
Testimonials are important, but ONLY as a source of
questions and ideas for further rigorous testing (blinded experiments where
practical). One person’s (or even many people’s) uncontrolled reaction to a treatment
that is claimed to produce health benefits is completely unreliable as a
proof the treatment actually has any effect.
The problem with uncontrolled testimonials is that there are many alternate, and often more plausible, explanations for the
observation of increased health that have nothing to do with the 'health'
product or service being promoted. The points discussed below can
cover any of the water products or treatments that claim to produce a health
benefit including, but not limited to, water that has been: ionized, clustered, structured, exposed
to magnets, energized, oxygenated, vortexed, hydrogenated, intentionalized,
catalyzed, m-activated, infused with noble gasses, exposed to pi ceramics,
or serially diluted with succussion (shaking).
This warning to be skeptical of uncontrolled testimonials applies to any
product or service that does not have valid experimental evidence to support
claims of effectiveness.
11 reasons to be skeptical of uncontrolled testimonials about
health benefits of any product or service:
- Placebo Effect:
can be caused by the
- a perceived health improvement caused by the expectation of a
beneficial outcome. How does one determine
whether perceived health benefits of drinking alkaline water, drinking
oxygenated water or sipping a homeopathic remedy are due to
actual biochemical effects of the product and not a result of the placebo
effect? The placebo effect and the “power of suggestion” are
remarkably compelling phenomena that can lead to the perception
of greater energy, reduced pain, and a number of other health benefits
that have been well documented over the years. The placebo effect
can even lead to mind-generated biochemical changes in the body that
mimic the effect of some drugs. There is no way that reported
health benefits of alkaline water, homeopathy or other enhanced water
products can be shown to be different from manifestations of the placebo
effect without rigorous, unbiased experimentation. If you take the
time to locate and critically examine any alleged scientific evidence
provided to support some of the altered &/or enhanced water claims, you
will find either that no health benefits have been demonstrated or that,
in carefully controlled experiments, any benefits can be shown to result
entirely from the placebo effect. An interesting
question to consider: If a treatment causes either a perceived benefit to
someone or actually results in biochemical changes that reduce pain or
anxiety, is that a bad thing - even though the claimed process has no
scientific validity and the treatment has no actual, direct, measurable
biochemical effect on the body?
- Spontaneous Improvement: Spontaneous, natural improvement in symptoms
is attributed to a treatment. In these cases, people will often try a treatment when their symptoms are the worst – which often just happens to be when the symptom severity will normally begin to improve.
If perceived healing can be caused by
timing of the treatment, what observations or measurements could one person make to demonstrate that a given healing episode was due
to the treatment and not to ordinary healing processes? It
has be said that if you have a cold you will normally get over it in a
week, but if you take a special treatment like alkaline water or a
homeopathic remedy, you will be cured in only seven days.
- Many diseases, like colds, flu, cough, allergies, etc., spontaneous
improve over a week or two in most people regardless of treatment.
- Symptom severity in patients with chronic
ailments like anxiety, back pain, migraines, etc., may naturally
fluctuate over time.
Spontaneous improvement of symptoms can lead to confusing causation with
correlation - in other words, two events might be linked in time but
there may be no cause and effect relationship between them.
This is an important specific example of point number three that needed
its own discussion. This is an example of confusing causation with
correlation, but in this discussion it is common enough to list
Correlation is one of the many
logical fallacies employed by those who promote pseudomedicine and
pseudoscience and can be illustrated by the example, "Many people who have strokes
have been taking blood pressure medications. Those blood pressure pills must be causing strokes,
therefore to be safe use our completely natural, μαγεία drink to lower your blood pressure
instead of those dangerous medicines!"
the cause of an observed health effect (or any event for that matter) to
some incident that precedes it, as
described above, is just one example of confusing Correlation (a
relationship in time) with
(a real cause and effect relationship). In other words, what may
appear true on the surface because events are related in time, may lead
to a completely false, though apparently reasonable, conclusions. This rather extreme
demonstrates why eating pickles is dangerous and should be avoided at
all costs. While this is a parody, it clearly illustrates how
events that are correlated in time (eating pickles followed by serious
illness and death) can be
misconstrued and used to try and prove causation. Another
report links the installation of each additional cell-phone tower to the birth of 18
additional babies. The
confusion between correlation and causation is evident when marketers
use uncontrolled testimonials to try and demonstrate their product
actually caused a positive health outcome when, at best, there may only
be a correlation.
The correct interpretation of the first example is that one of the risk
factors for stroke is high blood pressure, so those who take blood
pressure medication will already have an increased risk of strokes, and
the blood pressure medications
may actually reduce the stroke risk. To discover whether or
not specific medications or other treatment options used to reduce blood pressure will also lower the risk of
must be performed to control for the many potential factors that can
complicate the interpretation of simple observations.
- A Misdiagnosis: The symptoms or illness could be misdiagnosed. For example, a person is informed by a care giver they have cancer (or other specific disease). They are told (or believe) the condition is incurable or would require a drastic intervention to cure. They begin to drink alkaline water,
take a homeopathic remedy or try some other treatment because a friend tells them it can help cure cancer. Two months later they have another checkup and are told there is no cancer. Is it more likely the
cure was a result of the treatment, a miracle or a misdiagnosis?
A personal example: In June 2013 my 90 year old mother was told she had
untreatable lung cancer and perhaps three months to live by her primary physician based on an x-ray. In this case, we did not believe it and asked for another diagnosis from another doctor based on a MRI scan. That scan found no evidence of cancer.
It would have been easy, in desperation, to try some alkaline water or other
alternative treatment in the weeks before the
MRI was scheduled and then attribute the absence of cancer to to the
It is also quite possible, in some cases, that people simply decide they have a specific disease (based on
some symptom and something they hear on TV, a friend’s comment, an Internet article, etc.), embark on a treatment protocol like drinking alkaline water, feel better after a few days/weeks, and believe the treatment cured them of their self-diagnosed,
- Undocumented Treatments: Switching or adding treatments or changing behavior may cause a real improvement in symptoms, but the ‘cure’ is attributed to the original treatment. An example might be if someone who has been working hard lifting heavy objects develops lower back pain and begins taking a homeopathic pill that’s ‘guaranteed to reduce pain’. They would probably also stop lifting things and they might go to bed and use ice and heat treatments. After a few days on the homeopathic remedy their back feels better. It’s likely that the homeopathic
treatment will receive credit for the cure (because they had to actively
pursue and purchase the treatment) and the contribution of the other life-style changes will be
overlooked or minimized. With an uncontrolled testimonial there is no way to know what else the person might have done
to try and improve their health.
- Subjective Symptoms & Outcomes: Subjective evidence is the testimony of
how a product performed based on the perception of the subject. The quality of subjective evidence depends upon the honesty of the
subjects, and their ability to accurately perceive reality. These
testimonials almost almost
never include measurable reference points or targets for the symptoms or
treatment goals. In other words what does “increased energy”, “increased mental acuity”,
or "increased heart health" or other common claims like
"better hydration", or "increased detoxification" really mean? What measurable characteristics
can be documented before and after drinking alkaline water? In the absence
of specific measurements and documentation, the mind can create any story it
likes based on a desire to become healthier. Even if an individual created and documented measurable
reference points and recorded some health benefits after drinking alkaline
water or trying another enhanced water product, that would only suggests a correlation, not a cause and effect
relationship - and could not rule out the placebo effect or some of the
other reasons discussed here.
- A Sample Size of One: How does
anyone know what would have happened if
they had not tried alkaline water, a homeopathic remedy, another enhanced water product,
or some other treatment? I submit that it is
impossible to know “what might have been if only I had done something
differently”. A testimonial I received about alkaline water, for
example, claimed that “Except for the roughest allergy season I've
experienced in my 45 year memory, I haven't been sick in almost 5 years
now.” Does that mean alkaline water is not effective against allergies
- perhaps alkaline water causes severe allergic reactions every five
Might some of the allergic symptoms actually have been a cold?
Might there be other reasons to account for her lack of colds? Can anyone really know
what their health would have been over five years if they had not been drinking alkaline
Testimonials can be hard to ignore, particularly when they come from a
family member, trusted friend, someone you accept as an authority, or
within the context of a meeting of excited individuals who have similar
testimonials. Humans are 'hard-wired' to pay attention to stories
from others - that is how everyone learns virtually every thing they
know about the world (besides their own personal experiences, of
course). Information is communicated from a person who has
knowledge and experience to someone who does not. The problem is that a
testimonial, even from a trusted source, may not be accurate because,
even though the person may be sincere about their experience, they may
be completely wrong about the cause and effect relationships for any of
the reasons discussed here.
- Cognitive Biases: There are a number
of unconscious psychological phenomena that conspire to influence
people's perception of reality and can lead to the reporting of an
effective treatment (or otherwise skew reality) when there was no actual
physical change. Examples include:
- Confirmation Bias (people
naturally seek information that supports their beliefs and disregard
disconfirming evidence - self-fulfilling information gathering)
- Bandwagon or Herd Effect (people tend to follow the crowd and interpret
their experiences as others in their circle do)
Desirability (the tendency of people to answer questions in a way that will be viewed favorably by others
- related to the bandwagon
- Selective Perception (the tendency for expectations to affect perception -
a component of the placebo effect where there is no actual physical change)
- It is informative to go through
this list and see what other biases might fit.
- To put it bluntly, Some People Use Unscrupulous Marketing Practices – particularly when there is a profit to be made. There is usually no way to determine whether testimonials used in marketing materials are sincere reports of people’s experiences
(or whether those experiences were influenced by the points described above), paid
endorsements, or complete fabrications by those selling the
|← To illustrate this point, I went to an Amazon sales site and
copied a several anti-testimonials for the Enagic Kangen Water Leveluk SD501.
These testimonials will obviously not make it into the sales brochure or
be promoted at
I was fooled and duped into buying one of these machines, and
two years later my health problems are exactly the same. . I did
everything the company said to do, like giving up soda and all
"acidic" beverages like certain juices and other drinks. I
thought at first I was seeing real results, so I bought a
machine, but all I did was saddle myself with debt for the
machine. After visiting the doctor, it was clear that my health
had not improved. The person who sold me the machine told me to
be patient, that it would probably take me a full year for my
body to become "alkaline". So I kept it up for two entire years,
but nothing ever improved (and some things got worse). Sera
I've had a Kagen water machine for two years. Before that my family and dogs were healthy. In the last 6 months my husband was diagnosed with Melanoma and my 5 year old Maltese who gets Kagen water daily with Mast Cell Cancer. I've seen no benefit what so ever from this product. Do not waste your money. MichelleY
I would LOVE to start a Class Action Suit, and get my hard earned money
back, for believing this machine and the water it produces, have
- Only Positive Outcomes are Reported
(Cherry Picking): Testimonials (at least as used by those marketing placebo products) only document
and report positive outcomes. There is no way to know how many people used the product and experienced no positive benefits or got worse because their condition was not effectively treated.
Marketing for FDA approved
medications must include all reported side effects – at least there is some transparency in traditional medicine.
On the other hand, there is no requirement to demonstrate the effectiveness of alternative medical
treatments or report the number of patients who do not improve, who experienced side effects or who got worse after treatment - or who died.
Also note the interesting reference to 'Undocumented Treatments' in the
first anti-testimonials to the right that clearly shows how the company
representatives suggested making additional life-style changes that, by
themselves, could contribute to perceived changes in health.
- Used to
Testimonials are regularly used by placebo
marketers to bypass FDA and FTC regulations against using untested,
unsupported claims about specific disease treatments. A company seldom
directly states that drinking alkaline water, for example, can cure
cancer, diabetes or other specific disease, but many distributors list testimonials from consumers who claim to have been cured or
alternative practitioners who claim to have treated or cured a specific disease
(search Google for "kangen testimonials").
The only requirement necessary to market alternative products and treatments that claim
to provide such vague health benefits as: Greater energy and vitality,
Increased hydration, Enhanced oxygen delivery, Better overall health,
Decreased stress levels, A more positive attitude, Boosted metabolism,
Detoxification of the body,
etc., etc. is to
add the Get Out of Jail Free Disclaimer everyone has seen, "This
statement has not been evaluated by the FDA. This product is not
intended to diagnose, treat, cure or prevent any disease"
to their product.
Check out the websites of most large
manufactures of enhanced/altered products. Their actual claims - the ones
they can be held legally responsible - are all vague and meaningless.
It is left to the distributors and the testimonials they employ, which are
'outside the control of the parent company' to make specific claims
about how the product really does cure arthritis, diabetes, cancer, etc.
The points above illustrate why I am skeptical about relying
on uncontrolled testimonials to any support health claims for any product.
This table briefly outlines how blinded experiments
(described in more detail here) can produce controlled testimonials
(and other information) that minimize the problems associated with
uncontrolled testimonials - Specifically:
The Placebo Effect - Blinded study participants
do not know whether they are receiving the real treatment (or product) or a placebo, so
any placebo effects will be similar in both groups and cancel each other
Spontaneous Improvement - Blinded study participants are
similar in age, health status, etc., so health conditions and spontaneous improvement rates would be
expected to be similar in both groups and cancel each other out.
Improvements can be demonstrated to be caused by the treatment.
Confusing Causation with Correlation - Blinding does not directly help the process
of sorting out cause and effect relationships, but it allows the process to
take place with minimal bias and expectations that can cause either the
researchers or the subjects to incorrectly conclude a correlated event is
A Misdiagnosis - Always a possibility, but since the
subjects are followed over the course of the experiment and have as close to
the same health status as possible (or the same symptoms & diagnosis) the
chance for significant misdiagnosis is minimized.
Undocumented Treatments - Part of a well designed
blinded study is to instruct all participants in both groups not to try other treatment
methods during the experiment and to follow the same routines when taking
Subjective Symptoms & Outcomes - Part of a well
designed blinded study is to design objective outcomes that can be clearly identified and
measured (or rated) and instruct participants in both groups how to identify
and record the outcomes of interest.
A Sample Size of One - A well designed
blinded experiment has a number subjects in each group - the more the better
- so data can be collected (and results compared) from many people. The results are analyzed
in a way that best demonstrates if there are any real differences between
outcomes in the two groups or if any differences observed are probably due
Cognitive Biases - All of the biases described above
are minimized because neither the subjects nor the experimenters know which
group is taking the real, active product or treatment and which is taking the placebo, so any
biases in reporting or analysis should be random and cancel out between the
Unscrupulous Marketing Practices - Sometimes
scientists will falsify their evidence and produce fraudulent findings.
Usually, however, other scientists will identify and expose the fraud.
Science is designed to be self-correcting.
Only Positive Outcomes Reported - A well designed and
conducted blinded study will collect all data, both positive and
from all subjects. Since neither the participants nor those conducting the
experiment know which group is taking the active product until after the analysis
has been completed, there is no benefit to withholding evidence.
A Desire to Bypass Regulations -
Obviously, where there is lots of money to be made, there is
a desire by some (inside and outside of the scientific
community) to break the rules. At least there are
checks and balances for those selling FDA regulated products
that may catch those who try and circumvent the regulations.
There are no equivalent checks and balances for those
selling products and services for which health claims are
made that are not FDA regulated.
As noted, however, testimonials are actually used to collect
data from participants in valid, scientific experiments. These controlled
testimonials (or data) collected from
experimental subjects are entirely different from uncontrolled testimonials with
the limitations described above. They are records of the experiences
(and can include specific biological measurements) of individuals who
participate in experiments, they are not selectively collected only from those
who have experienced a positive outcome. A good experiment is designed so that the
eleven limitations described above are minimized, and the testimonials can actually
be used to help establish a cause and effect relationship (if one exists) between treatment and a
health outcome. The most effective experiments are blinded - they compare the effectiveness of a product or service as a treatment for
some health issue with a placebo (a product or service that is as similar as possible to
the product or service that is being tested, except that it has no real
biological action - the famous 'sugar pill').
You might ask, What is the harm
in all the different alternative health treatments? If the products
don't really do anything except perhaps trigger the placebo response
(and actually make someone feel better), or perhaps help someone feel
like they have some control of their health situation, why criticize the
products? If the only consequences of using products with no
real biological effect (beyond the placebo) were either that there was
no health effect or that someone actually felt better, the main
criticism would be the use of unproven claims and deceptive marketed
practices. Unfortunately, there are more serious consequences.
Most people are wise enough to not use these products in life
threatening situations, and they will seek real medical treatment if their health
situation continues to deteriorate. However, some individuals fall
completely under the spell of these illusionists and wind up seriously
injured or dead. Tim Farley has developed a site,
What's The Harm?,
that documents what you will never hear in the testimonial-driven marketing of
alternative treatment products.
One common strategy of nearly all marketing programs for
products that are based on pseudoscience is the proclamation that mainstream science is
not to be trusted, followed by a litany of alleged evils attributed to science,
the medical industry, government regulations, big pharma, etc. I am the first to admit that science,
traditional medicine, government, big pharma, etc. are not perfect (individuals
and groups have taken advantage of and corrupted the system), but they at least have checks & balances and structures in place to
detect and minimize bad science and fraud. Most of the evil examples cited
by the pseudoscientists were, in fact, discovered by and exposed by other scientists using
By discrediting mainstream science, the goal is to elevate
their platform (alternative/pseudoscience) to legitimacy. Unfortunately,
that argument does not work - even if mainstream science and traditional medicine were
demonstrated to be completely untrustworthy and evil, that would not automatically
validate the processes or conclusions of the alternative/pseudoscience
Pseudoscience has no system of checks
and balances, there is no plan or process within the community of those
promoting enhanced/altered water products - or any of the thousands of
alternative treatments for which science cannot provide evidence of
effectiveness - to detect, expose and correct misleading advertisements or
How can anyone believe that a community that promotes marketing
uncontrolled testimonials, that has no self monitoring system or regulations to
ensure safety and effectiveness of treatments and that dismisses controlled
experiments as worthless is
superior to the mainstream scientific community. Mainstream science, for all its faults, is self
monitoring, self correcting, and has ultimately created the world (both the good and
bad elements) that we inhabit today. Pseudoscience has never produced
real and lasting value.
It is interesting to observe that most experiments carried
out by mainstream scientists that have been reviewed and published can be
successfully duplicated by anyone else in the world - whether they believe the
results or not. The published methods and analyses are transparent. On
the other hand, most experiments carried out by those who promote alternative
treatments seem to only be reproducible by others who already believe in the
outcomes that are 'supposed to occur'.
Close this screen to return to the previous page.
|Copyright © 2005 Randy Johnson. All rights reserved.
Updated November 2011