Drinking Water Information Home

Close this window to return to the previous page.

Drinking water information and resources, contaminants, health effects, treatment methods

 Evidence, Experiments and Blinded Studies
- How to turn testimonials into scientific evidence -


Science is like an inoculation against charlatans who would have you
believe whatever it is they tell you
. Neil deGrasse Tyson

The Bottom Line:
It is possible to collect reliable evidence that can validate (or disprove) health claims about the effectiveness of a product.  The evidence must be collected in a manner that minimizes the potential for bias, expectations or beliefs to influence data collection, the analysis or conclusions of the experiment.
An introduction to the methods of science and some terminology.
Quantitative and Qualitative Data
Control Groups, Experimental Groups and Placebos


Experiments - Description:
_ Open Experiment
_ Single Blind Experiment
_ Double Blind Experiment
Surveys - Description:


Experiments - Examples:
Double Blind Experiment
 
_ Product Effectiveness
_ Product Comparison
Surveys - Examples:


Disclaimer
Observational Evidence - Another type of testimonial in which natural processes are carefully observed and documented to determine patterns and deduce cause and effect relationships.
We are constantly bombarded by marketing programs (whether they are media ads or in-person demonstrations) that try to provide compelling evidence that will convince us that their product is effective.  There are two basic types of evidence that can be used to provide validation that specific claims about any product are true:
  • Experimental Evidence - specific, measurable (objective) information (data) about a product's performance and effectiveness that has been collected in a controlled manner to minimize errors and bias.
  • Testimonials or Anecdotes - people's subjective observations, opinions and descriptions about how they believe a product has helped them.

This discussion will examine Experimental Evidence (collected using scientific research techniques) and my contention that: To provide reliable evidence that supports health claims about any product or service (particularly altered/enhanced water products that are allegedly energized, vortexted, structured, clustered, ionized, succussed, magnetized, oxygenated, etc.), it is necessary to control the way the evidence is collected, analyzed and presented.  This requires establishing and recording objective outcomes that can be clearly defined, tested and measured.  Two important characteristics of collecting controlled outcomes are that: (a) all outcomes (positive and negative) are collected and clearly documented - not just outcomes that support the claim, and (b) a clear description of how the data was collected and analyzed is recorded, so anyone can evaluate the experimental process and try to reproduce and verify the results.

I discuss elsewhere how uncontrolled testimonials, which are generally solicited to help market a product or support pseudoscientific claims, cannot be trusted to provide accurate, unbiased information about product effectiveness. 

If we can't necessarily trust what people tell us about their experiences with a product, though, how can we discover useful information about a product and determine whether a claim made about some product or service is accurate and trustworthy or whether it is pure marketing fantasy?  Testimonials are not all worthless, and they can actually provide useful information but only if collection of the testimonials is controlled in a way that minimizes the potential problems outlined in the uncontrolled testimonials discussion.  The scientific experiment is one process specifically designed to collect information in a controlled manner and provide reliable information that allows an accurate evaluation of the product's effectiveness.

A brief introduction to the methods of science and some terminology:
The fundamental goal of science is to understand the natural universe in which we live.  Much of this understanding involves the discovery and description of causality (cause and effect relationships) in nature.  When natural cause and effect relationships are understood it is often possible to manipulate the causes and thus modify the effects.  If the causes of a disease are understood, for example, processes can be developed to remove or reduce exposure to those causes and/or modify the body's response (with vaccines or treatments) and minimize the normal disease effects. 

The products of interest in this discussion (altered/enhanced water products) are marketed with claims that they can have some real, significant effects on health that are caused by REAL biological effects on the body and not just placebo responses.   If those claims are valid, there must be a way to collect valid supporting evidence using the methods outlined here.

Over the last several centuries scientists have developed a methodology (often referred to as the Scientific Method) that formalized guidelines for discovering possible cause and effect relationships (hypotheses) and then testing them to see if they can be validated.  There is lots of interesting (to me anyway) information available about about how science works.  However in the context of this very specific discussion, I will mostly focus on how to use the methods of science to collect reliable information about whether a specific product has a real effect on a person's health.  If you would like additional general information, you can read brief discussions on the goals of science and the scientific methodWikipedia and Berkeley also have good descriptions of the scientific method.

The key difference between the collection of uncontrolled testimonials to support a product's effectiveness and the scientific collection of information to determine whether a product is effective, is that scientists do not set out to collect only data that supports their hopes, expectations or beliefs about how a product should work.  Scientific data collection is (or should be) designed so all relevant data are collected, recorded and analyzed in a manner that minimizes ANY biases, expectations or beliefs scientists have which might influence how experiments are designed and how the data are collected, analyzed and interpreted.  Similarly the effects of the biases, expectations or beliefs of those who will use the product (the experimental subjects) should also be minimized.

It is important to realize that both the scientific researchers and the experimental subjects DO HAVE biases, expectations and beliefs about any experiment in which they are involved.  If the researchers were not interested in something about the product, and if they didn't have some expectations about the product's effects, they would not bother to conduct an experiment - if they were hired by a company to test a product, they might have a bias to produce favorable results.  If the subjects didn't have some expectations and beliefs about some claim made for a product they might not participate in the experiment.
The examples below will illustrate how several scientific data collection methods can minimize the effect of all the biases everyone has and let the data 'speak for itself'.  

Other scientists judge whether an experiment is valid and has successfully demonstrated a cause and effect relationship in part by how well an experimental design, the methods and data analysis have successfully minimized all potential sources of bias.  High quality scientific journals strive only to publish results of studies where effects of bias, and expectation have been successfully controlled (a good starting list).  Research results submitted to these journals are reviewed by other researchers who are respected in their field, and papers are only approved for publication if all aspects of the experiment are high quality and meet stringent requirements to minimize bias and maximize objectivity, accuracy and reproducibility.  This process of Peer Review is employed by all high quality scientific journals. 

In recent years, however, a number of 'journals' have been established that may have impressive names but almost no quality control measures, and their peer review processes are non-existent or poorly controlled.  These 'journals' may be business ventures that require payment from the authors to be published &/or they may collect and publish 'research' that supports very biased positions that are not accepted by the scientific community as a whole.  Those who publish these suspect 'journals', and those who depend on them to become published, will claim there are conspiracies by the all the standard, high quality journals to suppress the findings of those researchers who are outside mainstream science.  They also claim traditional journals are completely biased toward traditional theories, only publish papers that support traditional scientific and medical beliefs and discriminate against anyone with novel ideas by ignoring their research.  An interesting example played out in late 2012 and early 2013 when a researcher from the Sasquatch Genome Project claimed to have sequenced Big Foot's DNA and was unable to get her paper published in a reputable journal.  She apparently purchased rights to an online journal, DeNovo, and published the paper.  The saga can be followed, here: a, b, c and d, from the Dallas Observer, the Huffington Post, io9 and the author herself.

Quantitative and Qualitative Data:  There are two basic types of information (or data) about experimental subjects that can be collected and analyzed in scientific studies and used to form a conclusion about the product's effectiveness:

  • Quantitative Data - Specific, objective information that can be directly measured and collected as numbers.  Examples include; height, weight, age, temperature, the pH, O2 saturation and glucose levels of blood, the number and size of tomatoes picked from the garden, etc.  This information is fairly straight forward, and as long as it is collected accurately, the data will be trustworthy and can be used to compare the attributes of multiple subjects in a study or from the same subjects over time as they use a product.
  • Qualitative Data - Descriptive, subjective information that cannot typically be directly measured or recorded as numbers.  Examples include; a person's reported status for: gender, race, nationality, energy, stress, hydration, health, mobility, emotional state or pain level. Reports people's religious, political, economic, persuasions as well as the taste, color and odor of tomatoes picked from the garden are also qualitative data. 

    When qualitative data is collected during a scientific experiment or survey, the possible responses are often limited to a coded scale - in other words, instead of asking 'what is your energy level after drinking OmyGod Oxygen Water?' (and receiving a virtually unlimited number of possible responses to try and analyze), a limited amount of control is imposed by presenting a response scale - for example a 10-level energy scale where the subject is asked, 'How would you rate your energy level five minutes after you drink a bottle of OmyGod Oxygen Water?  Please answer with a number from 1 to 10, where 1 = I  actually had significantly less energy and wanted to go to bed, 5 = I noticed no change in my energy level, and 10 = My energy level was significantly better and I felt like running around the block'. 

    Other coded scales could be: Gender; 1=Female, 2=Male, 3=Other or Political Affiliation; 1=Democrat, 2=Republican, 3=Libertarian, 4=Green, 5=Other.  Probably no group has taken qualitative data coding to greater and more creative lengths than the wine industry that has catalogued well over 100 descriptors for the flavor, nose, finish, structure, appearance & texture of different wines.  Coded scales provide some ability (depending on how well the scale is constructed and understood by the participants and the researchers) to compare responses from many people in a study and to compare responses from the same individuals over time.  Qualitative Data collected in scientific studies are essentially controlled testimonials.

The scientific community has developed two research processes (Experiments and Surveys) that enable controlled, interactive data collection by researchers from subjects on whom products are tested.  There are several important characteristics of experiments (and to some extent, surveys) that help minimize and control for possible sources of bias.  These include:

  • Use of Control Groups and Experimental Groups:  The goal of an experiment is to isolate and identify those effects in a group of subjects that are actually caused by the product that's being tested.  To help identify experimental observations that are the result of the action of the product, another group of subjects that have similar traits is recruited.  Subjects in the Control Group undergo exactly the same routines as subjects in the Experimental Group, and they use a product as nearly identical to the test product as possible - except that it has no biological action (the Placebo).  Results are collected from both the experimental and control groups.  Similar observations from both groups are assumed to be the result of placebo activity, biases or chance and not the result any action caused by the product under investigation.
  • Blinding:  This technique prevents the researchers and/or subjects from knowing whether a specific subject is using the real product that's being tested or an inactive placebo.  In the case of product comparisons the identity of all products that are being compared are hidden from the researchers and/or subjects.  Blinding reduces bias introduced by expectations of how a product 'should' work.
  • Understanding and Managing Experimental Subjects: 
    • Collect more data than is strictly necessary to evaluate the product.  In other words, collect data like age, gender, race, health status, activity levels, political preferences and any other attributes that might have an impact on how the product that's being tested functions or is perceived.  This is important because during the analysis of the data the results are adjusted to account for any effects that might be caused by these other attributes instead of (or in addition to) the specific contribution of the product.  A product, for example, might work only on children and not adults.  That important detail would be missed and the results might be interpreted incorrectly if age was not collected.
    • Select subjects that have similar traits.  Whether a study is testing the effects of a product on the subject's health, or a group of subjects is comparing different products, the analysis is often easier if the subject attributes are similar - there are fewer possible cause and effect relationship to try and untangle.  If a study population ranges from 10 to 90 years old and half of them have serious health issues, it would be very difficult to determine whether any observed results were cause by the product or by the extreme differences in the subjects.
    • Randomize subjects into the study groups.  This is another way to minimize bias.  If someone is allowed to decide which individuals are in the experimental group and which are assigned to the control group they may consciously or unconsciously assign subjects with specific characteristics (they appear sicker, perhaps) into one or the other group.

So, finally a brief description of the different types of experiments and surveys that are used to collect data about a product.

  • Experiment - Description:  An experiment is an orderly procedure carried out with the goal of verifying, refuting, or establishing the validity of a hypothesis, an expectation about how a particular process or phenomenon works. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs when a particular factor is manipulated.  An experiment usually tests a hypothesis. However, an experiment may also aim to answer a "what-if" question, without a specific expectation about what the experiment will reveal, or to confirm prior results. If an experiment is carefully conducted, the results usually either support or disprove the hypothesis. (Wikipedia)

    Scientific experiments can be extremely complex and costly. However, it is fairly easy to illustrate important characteristics of experiments that allow collection of controlled data and can provide useful information about whether a product is effective at treating a health issue - or whether one type of wine, water, beer, soft drink, coffee, etc. really tastes better than another.

    Experiments can be designed to collect two kinds of information about products:

    Product Comparison - Attributes (taste and aroma for example) of two or more products are compared.
    Product Effectiveness - Claims that a product has a real, beneficial effect on a subject's health are tested.

    There are three recognized types of experiments that I will describe below in increasing order of complexity and with an increasing opportunity to collect reliable data.
    1. Open Experiment:  No attempt is made to hide the identity of the product(s) or the expected effects.
      • Product Comparison - All products to be compared are fully identified to those conducting the experiment and those making the comparison.  Limitations: There is no way to minimize researcher or subject biases.
      • Product Effectiveness - Subjects are randomly assigned to control and experimental groups.  Both the researchers and the experimental subjects know which group everyone is in and who will be taking the placebo or the test product.  Limitations: There is no attempt to minimize researcher or subject biases - little better than collecting uncontrolled testimonials, except that negative data is not excluded.
    2. Single Blinded Experiment: A better way than the open experiment to collect information about the effectiveness of a product or how different products compare to each other is to conduct a blinded study where the researchers know the identity of the product(s), but the subjects do not. 
      • Product Comparison - The researchers conducting the comparison know the identity of the products that are being compared.  The subjects who are comparing the products and providing information about the product are blinded - they are unable to see the labels, and in some cases, where appearance might influence taste perception, they might not even be able to see the product.  Limitations: If the researchers can be seen by, or are interacting with, the subjects they may unconsciously pass along biases that could influence how the subjects evaluate the products.  Researcher bias might also influence how the data are analyzed.
      • Product Effectiveness - Subjects are randomly assigned to control and experimental groups by the researchers.  The subjects do not know which group they are in or whether they will take the test product or the placebo.  The researchers know which are the experimental and the control groups and thus which subjects will be taking the placebo and the test product.  Limitations: There is no way to minimize researcher bias in the collection and analysis of data, and it is possible the researchers might unconsciously communicate information that would enable the subjects to figure out whether they were in the experimental or control group.
    3. Double Blinded Experiment:  The most effective way to collect unbiased information about the effectiveness of a product or how different products compare to each other is to conduct a double blinded study where neither the researchers nor the subjects know whether a given individual is in the experimental group and using the product or in the control group and using a placebo.  When products are being compared, neither the researchers nor the subjects know the identity of the product(s). 
      • Product Comparison - The researchers in charge of the comparison do not know the identity of the products that are being compared (they are blinded) - someone who will not interact with the researchers or the subjects sets up the products for comparison, de-identifies them and labels them with a code.  The subjects who are making the comparisons and providing information about the product are also blinded - they are unable to see the labels, and in some cases, where appearance might influence taste perception, they might not even be able to see the product.  
      • Product Effectiveness - Researchers are blinded.  Subjects are randomly assigned to control and experimental groups by assistants.  The assistants will also create and blind doses of the product or the placebo for distribution to the appropriate study groups.  The assistants will not communicate information about which was the experimental group and which was the control to either the researchers, the analysts or the subjects until after the results have been analyzed and evaluated. 
  • Surveys - Information is obtained by asking questions of many people about their experiences, beliefs, preferences, health, etc.  Survey questions can be asked of many people at roughly the same time (cross-sectional survey) or of the same group of participants over some period of time (longitudinal survey).  Surveys provide somewhat more reliable information than uncontrolled testimonials because the same, specific information is actively requested from many people.  In a well designed survey an attempt is made to control and understand the population that is being questioned so the results can be interpreted in some context.

    I tend to be fairly skeptical about many survey results because, in the end, each person's response to a survey is a mostly uncontrolled testimonial - the only control is asking the same set of questions (and collecting all answers - both positive and negative) from many people and having some control over the population you are surveying.  Consequently survey answers have many of the same potential problems outlined in my uncontrolled testimonial discussion.  People with specific biases might also be more or less prone to participate in surveys.  It is also nearly impossible conduct a blinded survey, so questions might be developed and asked (and answers interpreted, recorded and analyzed) by someone with a bias.  This blog contains a good, brief outline of the advantages and disadvantages of surveys, and this article describes a checklist of good practice in the conduct and reporting of survey research.

    Cross-sectional surveys can collect information quickly from a lot of different people fairly inexpensively, so they are good for discovering possible patterns and possible cause and effect relationships in large groups of people.  To confirm causality, however, more controlled
    studies would need to be conducted.

Specific Examples of Surveys and Double Blind Experiments will be described below.  The intent of providing the examples is that you can use the outlines, and substitute other products that have some measurable claims and actually test the claims you might be interested in evaluating.  These examples will test the claims of bottled OmyGod Oxygen Water that is marketed as containing eight times the normal amount of dissolved oxygen and has just become the rage on college campuses.  Two testable claims are made for OmyGod Oxygen Water:
  1)
 Better taste
  2)  Extra energy and vitality

Participants in a typical research experiment or survey might include:

  • The Researcher designs and manages the experiment/survey.
  • Assistants help conduct the experiment or administer the survey.
  • Subjects are the active participants who sample the experimental product and record their experiences or provide responses to survey questions.
  • Analysts analyze the data to see if the results are statistically significant or whether they might have been This is just an example of those who might participate in many research projects. 
  • The Researcher might actually conduct the experiment and analyze the data in an open trial. Researchers can analyze the data in a double blind trial if they are competent statisticians.  There may also be more than one researcher.  There may be groups that provide funding and administration, and so on.

  1. Experiment - Examples:  

    These fairly detailed examples will describe steps of of double blinded experiments for a product comparison and product effectiveness.  Hopefully it will be obvious how each of the steps helps preserve the integrity and reliability of the data and reduce possible biases that can influence the data collection and ultimately the reported outcomes of the experiment. 


    Experimental details described below are just examples of basic experimental designs: These experiments have not been evaluated by the FDA. These experiments are not intended to diagnose, treat, cure or prevent any disease.
    1. Double Blind, Product Effectiveness Experiment:
      This experiment will test the claim that drinking OmyGod Oxygen Bottled Water (OmGOBW) causes extra energy and vitality over drinking regular water.  Extra energy is a subjective claim, but that extra energy and vitality would have to be caused by something, and the only cause based on the claims would be the extra dissolved oxygen.  The only way the dissolved oxygen could possibly actually increase energy and vitality would be if it was absorbed into the blood stream and transported to the brain, muscles and other organs.  Increased oxygen in the bloodstream is a real biological change that will produce objective data that can be measured with a pulse oximeter.  The hypothesis that will be tested is "there will be no difference in reported energy or vitality or in oxygen saturation levels between the group that is drinking OmGOBW or filtered tap water" - if subjects who drink oxygenated water actually have higher recorded energy and blood oxygen levels than the controls, the hypothesis will be disproved, and scientific evidence for the effectiveness of the oxygenated water will have been obtained.  The process is as follows:
      1. The Researcher will recruit one or more Assistants who will then recruit 16 test subjects who have similar traits - 8 men and 8 women, all enrolled in the same college biology class; ages from 19 to 20; moderately active; healthy with no history of chronic diseases and no active illnesses.  None of the subjects will have used Oxygenated water before.  The researcher will not participate in the process below and will have no knowledge of which group is which or which subjects have been assigned to each group.
      2. Subjects will be assigned to one of two groups the 'A' group or the 'B' group.  Each group of test subjects will consist of 4 men and 4 women.  The men and women will draw straws or have some other random assignment into their groups.  The subjects will not be told what the types of water the experiment will be testing, so there can be no product-based expectations.
      3. All subjects in each group will be introduced to the "Energy/Vitality Scale" - a self-rating of how much energy the person is experiencing at the time the measurement is made.  The Energy Scale will have ten levels - from 1 = extremely low energy and minimal vitality (I'm barely able to move) to level 10 = I have so much energy and vitality that I can't sit still (I'm all wound up and feel like I have to "run it off).  Level 5 would indicate moderate levels of energy & vitality.  Subjects will also be instructed not to talk about the experiment to others to avoid potential bias.  Subjects cannot wear fingernail polish during the experiment since that can interfere with the Pulse Oximeter readings.
      4. All subjects will be instructed to spend three days recording baseline information.  They will drink a 12 oz. bottle of normal filtered water at 8:00 am, at noon, at 4:00 pm, and at 8:00 pm.  Five minutes before they drink the water and 20 minutes after drinking the water they will record their energy levels.  Five minutes after drinking the water their blood oxygen levels will be measured and recorded with a pulse oximeter.  The subjects will also record activities they were participating in when they drank the water, record the amount and time they drank any other water during the day, and record any other unusual occurrences that might effect their oxygen or energy levels.
      5. The assistants will manage the blinding and distribution of the water samples:
        • They will obtain 64, 12 oz. bottles of OmGOBW for each day of the experiment (each of the 16 subjects will drink four bottles of water per day).
        • The bottle labels will be removed, then 32 of the bottles will be re-labeled 'A' and 32 will be re-labeled 'B'.
        • The water in all 'A' bottles will be dumped out and replaced with filtered tap water - the controls.
        •  If the OmGOBW has obvious bubbles on opening, and it is easy to tell which bottles contain the oxygen, a small quantity of pressurized nitrogen or argon (odorless, tasteless gases without any biological activity) would need to be added to the control bottles to create bubbles when opened.
        • The lid seals of all 'B' bottles will be cracked and lids retightened so it will not be possible to
        • determine by sealed lids which bottle contents were altered.
        • Water bottles will be prepared any time before early morning of the day they will be used.
      6. At 6:45 am on each day of the experiment the bottle managers will set all 64 labeled bottles for the day on a table and leave the room without communicating with the subjects.
      7. At 7:00 am the subjects will enter the room and each take four bottles of water according to their group and leave for their normal daily activities. 
        This is one key to a blinded experiment - the subjects do not (and cannot) know whether they are drinking the treated water or the untreated water so their expectations and beliefs can't bias their behaviors or observations.  The other key is that the researchers and analysts do not have knowledge of which group is which until after the analysis step.
      8. The subjects will drink one 12 oz. bottle of their 'A' or 'B'  water at 8:00 am, noon, 4:00 pm, and 8:00 pm.  Five minutes before they drink the water and 20 minutes after drinking the water they will record their energy levels.  Five minutes after drinking the water their blood oxygen levels will be taken and recorded with a pulse oximeter.  The subjects will also record activities they were participating in when they drank the water, record the amount and time they drank any other water during the day, and record any other unusual occurrences that might effect their oxygen or energy levels.
      9. Steps 6 through 9 will be repeated for each day of the experiment - perhaps a week - to accumulate sufficient data to draw some conclusions.
      10. Crossover double blinded study: There is an enhancement to the double blind experiment that allows even better collection of reliable, unbiased data.  After a period of time where the subjects in group 'A' drink the filtered tap water and those in group 'B' drink the OmGOBW as described above, the type of water assigned to each group is switched, so group 'B' drinks filtered water and group 'A' drinks OmGOBW.  The subjects are unaware of the switch.  Steps 6 through 9 are repeated for the remainder of the experiment. 

        Crossover studies provide the opportunity for the experiences of each person to be recorded for both the experimental product and the control placebo.  That has the tremendous opportunity to reduce biases introduced into a study by differences in how subjects might respond to the experimental product or to the experimental process. 
      11. After the data collection period has ended an Analyst who has no knowledge of which group drank the oxygenated water and which group drank the regular, filtered water (in a double blinded experiment, the researcher could analyze the data) will compare all results from all subjects in both groups and determine if there were differences in the reported energy levels or the blood oxygen levels after drinking oxygenated water, and if those who drank the control water.  Statistical tests would be used to explore the possibility that some small observed differences could be caused by chance rather than the experimental product.
      12. After the results have been analyzed the groups will be unblinded and the results presented and explained.
    2. Double Blind, Product Comparison Experiment:

    3. Since it would be fairly boring to only compare the taste of one other kind of water to the OmyGod Oxygen Bottled Water (OmGOBW), four other kinds of water will be used in the comparison - tap water, tap water filtered with a high-end activated carbon filter and two commercial bottled water brands, Fiji and Evian.  The hypothesis that would be tested by this experiment is that there will be no clear 'winner' in the taste test.  The process is as follows: 
      1. The Researcher will recruit an Assistant.
      2. The assistant will recruit 16 Subjects, 8 men and 8 women in this example, ages between 19 and 20 who are all enrolled in the same college chemistry class.  None of the subjects will have tried Oxygenated water before.
      3. The assistant will manage the blinding and distribution of the water samples and instruction of the subjects. 
      4. The researcher will not have any communication with the assistant or the subjects.
      5. The assistant will obtain 64 unlabeled plastic bottles that are identical to the OmGOBW bottles.  16 of the bottles will be labeled 'A', 16 will be labeled 'B', 16 will be labeled 'C' and 16 will be labeled 'E'.  The labels of 16 bottles of OmGOBW will be removed and the bottles relabeled 'D'.
      6. 16 stations will be set up on tables with five bottles (one each 'A', 'B', 'C, 'D' and 'E') at each station.
      7. The assistant will fill and cap all 'A' bottles with the Evian spring water, all 'B' bottles with tap water, all 'C' bottles with filtered tap water and all 'E' bottles with Fiji artesian water.  The cap seals for all 'D' bottles will be broken but the caps will be retightened so the extra oxygen will not escape.  As in the double blinded effectiveness experiment described above, if the OmGOBW has obvious bubbles on opening, and it is easy to tell which bottles contain the oxygen, a small quantity of pressurized nitrogen or argon (odorless, tasteless gases without any biological activity) would need to be added to at least one of the the control bottles (in this case 'C') to create bubbles when opened.

        All bottles should look identical.  In most beverage taste tests there is no need to go to the fairly elaborate precautions of keeping the liquid in bottles, the different samples could just be poured out into identical glasses.  In this experiment, however, the beverage being tested for taste contains pressurized oxygen.  If the oxygen or the bubbles actually contribute to the perception of taste, it would be unfair to pour the water into a glass where the bubbles might be gone before the water is sampled.  Blinding an experiment can be complicated, and sometimes it is impossible.
      8. The subjects will only know that they will be tasting up to five different water samples, but not what specific types of water they will be testing.  The subjects will not observe the bottle filling process.  Subjects will be instructed to go to one of the 16 tasting stations when the samples are ready, taste each of the five types of water in whichever order they wish and as many times as they wish.  As they sample the water they will be asked to make notes (numeric scales could be devised to provide more precise data) about taste, odor, feel of the water in their mouth, etc. for each water sample and complete the test by ranking the five water samples from the best (=1) tasting to the worst (=5) tasting.  Subjects will also be instructed not to make any comments as they taste the samples or otherwise indicate how they feel about any of the water samples.
      9. After all samples have been tasted, the researcher or analyst would compare all results, and among other analyses, would average the ranks for each of the five water samples to determine which, if any, were consistently ranked best (or worst).
      10. After analysis, the samples 'A' - 'E' would be unblinded and the results presented and explained.
  2. Survey - Examples:
    Cross-Sectional Survey example on the energy benefits of O
    myGod Oxygen Water (OmGOW):
    • A caveat here - in order for this specific survey to work, a sizable portion of the population contacted actually would have had to tried the oxygenated water - probably an unrealistic assumption, unless the survey population was restricted to members of health clubs or college campuses where fads can develop - but the process is the same whether the survey is about experiences with oxygenated water or is sampling residents of Colorado about experiences with medical marijuana use.
    1. The researcher will develop a set of questions to ask the subject - in this survey, there will be eight questions, (1) have you ever tried OmGOW; If the answer is yes, continue with: (2) how long have you been drinking OmGOW, and how often do you drink it?  (3) what is your gender?; (4) what is your age?; (5) on a scale of 1-10, how active are you (1=not active)?; (6) on a scale of 1-10 how would you rate your overall health (1=poor)?; (7) does OmGOW taste better than tap water or worse?; (8) after you drink OmGOW do you believe your energy and vitality levels increased or decreased?
    2. The researcher or assistants will call or contact in person as many individuals as is feasible within the constraints of available resources, ask people the eight questions and record the answers.
    3. After enough people have been contacted to provide usable information - perhaps several hundred people who have experienced OmGOW - the results will be analyzed to see what percent of the respondents thought their energy and vitality levels increased after drinking the oxygenated water.  Other results could be reported as well like the percent of users who thought OmGOW tasted better than tap water, whether females experienced the product differently than males, whether the length of time using the oxygenated water might have influenced their answers, etc.
    Longitudinal Survey example on the energy benefits of OmyGod Oxygen Water (OmGOW) over time:
    1. The researcher will develop a set of questions to ask the subjects - in this survey, there will be six questions that will be asked at the beginning of the survey to collect basic data on the subjects, (1) have you ever tried OmGOW; If the answer is yes, continue with: (2) how long have you been drinking OmGOW, and how often do you drink it?  (3) what is your gender?; (4) what is your age?; (5) on a scale of 1-10, how active are you (1=not active)?; (6) on a scale of 1-10 how would you rate your overall health (1=poor)?
      Three questions will be asked during the course of the survey, (1) does OmGOW taste better than tap water or worse (who knows, the extra oxygen might fry the subject's taste buds)?; (2) after you drink OmGOW do you believe your energy and vitality levels increased or decreased? (3) has there been any significant change in your daily routine that might have affected the way you experienced OmGOW?
    2. The researcher or assistant will call or contact in person as many individuals as it takes to find 20 individuals who regularly drink OmGOW daily and who are willing to provide information about their experiences.  Answers to the initial six questions plus the three ongoing questions will be recorded.
    3. The researcher or assistant will call each subject daily  between noon and 1:00 pm and record the answers to the questions about taste and energy/vitallity and routine changes.
    4. After a week or two of collecting data on OmGOW - the results will be analyzed to see if the subjects experienced any changes in how they experienced OmGOW during the experimental period.

These examples are fairly simple and could be adapted for a high school science fair project, but they provide a basic outline of the experimental process and processes that are similar whether the experiment is small and trivial or a multi-million dollar, multi-year experimental study.

I hope it is obvious how evidence collected in a controlled environment is more trustworthy than the uncontrolled testimonials used by those trying to peddle a product for which no objective controlled supporting evidence can be obtained because their product only works in the imagination and not in the real world.  It should be obvious too that surveys can't provide the experimental controls that are possible with blinded experiments.

Additional Resources:
  Experimental Design for Advanced Science Projects
  The Basics of Experimental Design

Disclaimer: The above discussion is not an exhaustive description of how scientific experiments work nor an outline all of the specific details, controls and statistical analyses that must be managed in order to design and execute a high-quality, reproducible experiment that stands a chance at publication in a high-quality journal.  Nor are all of the potential pitfalls that can sabotage even the best-intentioned experiment mentioned.  The discussion is provided to highlight some of the more important characteristics of scientific experiments, illustrate how they can provide useful information about whether or not claims for a product (or process or procedure) are true, and provide instructions so you can conduct your own blinded tests.

Hopefully too, this information provides a clear contrast between how high quality scientific experiments can provide evidence about whether product claims are valid and how uncontrolled testimonials only provide the information those marketing the product wish you to have.  To keep the discussion as simple as possible, I have not attempted to use or describe many of the scientific terms (empirical falsifiable, null hypothesis, a posteriori, theories, axioms, deduction, etc.) that are used to precisely define/describe some of the scientific methodology.

In the context of this discussion, another method in which valid scientific evidence can be obtained needs to be mentioned - Scientific Observation.  Those who try to justify the use of uncontrolled testimonials to validate product claims might point out that science has always depended on observing the world and the universe to understand how the natural universe behaves - and that is just what their testimonials are, observations of what happened when someone tried their product.  What they apparently do not understand is that Observational Science is typically used when it is impossible to conduct experiments - much of astronomy, evolutionary biology and geology are founded on observations.  More important, predictions based on theories that were developed using careful observations are often validated experimentally when science progresses to the point where evolving technology finally enables experiments to be conducted.  The observations of pseudo-scientists and those collecting uncontrolled testimonials do not lead to the development of any testable theories or useful predictions - their only goal is to market products.

The Theory of Evolution, for example, was formulated by Charles Darwin in the mid 1800s and was based largely on his observations.  Since that time a number of predictions about evolution and natural selection formulated on observations have been experimentally validated.  Similarly, observations of geological features by several scientists in the early 1900s lead to the development of the Theory of Plate Tectonics.  The process of Observational Science and the validity of theories based on the observations are completely different from information collected by and conclusions based on Uncontrolled Testimonials.

Close this screen to return to the previous page page.

Copyright 2013 Randy Johnson. All rights reserved.

Updated May 2014