New website to help scientists design better experiments
More than 50 million laboratory animals are estimated to be used in experiments each year1. According to an editorial in Nature Genetics “Animal experimentation in scientific research is a good thing: important, increasing and often irreplaceable.”, but It goes on to say that “Careful experimental design and reporting are at least as important as attention to welfare in ensuring that the knowledge we gain justifies using live animals as experimental tools.” It suggests that many experiments are under powered, but rather than simply increasing sample size scientists should look closely at the various sources of variation. Controlling these and improving experimental designs to include variables such as gender will increase both power and the external validity of the results.
Problems with the quality of animal experiments
Twenty years have passed since the publication of the first paper to suggest that there is room for improvement in the design and statistical analysis of experiments using laboratory animals2. Many more papers have been published since then, but progress has been slow. A 2009 survey of 271 papers involving rats, mice or non-human primates3 found basic principles of experimental design were widely ignored. For example, 87% failed to mention randomisation, 86% did not report “blinding” in studies where it would normally have been appropriate, none of the studies justified the sample size that they used, 26% failed to state the sex of the animals, 24% reported neither age nor weight of the animals and 4% didn’t even state the number of animals which they had used. The omission of many experimental details was clearly a problem. The ARRIVE guidelines4, based on the CONSORT guidelines were subsequently developed to help eliminate such omissions. These have been accepted by many journals.
Poor experimental design may mean that the experiments can’t be repeated. In a commentary article in Nature the authors5 claimed to have attempted to repeat fifty-three “landmark” studies in cancer research, sometimes with the help of the authors. They “…acknowledged from the outset that some of the data might not hold up, because papers were deliberately selected that described something completely new, such as fresh approaches to targeting cancers or alterna¬tive clinical uses for existing therapeutics.”, but only six (11%) could be repeated. They considered this a “shocking” result. In a following commentary article the author6 claimed that “research is riddled with systematic errors” which could erode public trust if it is not put right. Interestingly, neither of these papers attributed these problems primarily to poor experimental design. They both suggest that the fault lies in a culture which strongly favours positive results, extreme competition so that investigators are more likely to make selective use of their data to obtain positive results (which amounts to scientific fraud) and excessive emphasis on publication in high impact journals by funding organisations.
Better training of research scientists
Whatever the cause of these problems, the majority of scientists want to design their experiments well, but in many cases their formal training in experimental design and statistics is rudimentary or non-existent. Those few scientists who are already proficient in statistics are usually too busy doing their own research to teach others, and in any case there is no incentive for them to do so. Nor are there many statisticians, with the required biological background, willing and able to take on this role. The scale of the problem is immense as there are many thousands of post-graduate students starting their research in the life sciences each year. It is logistically impossible to provide even short introductory courses for so many people scattered in so many universities world-wide. But given the right tools scientists should be able to teach themselves the basic principles, which are not too difficult to understand. The World Wide Web provides a flexible and economical way of helping them.
The new website www.3Rs-reduction.co.uk
This is a free, non- commercial site which can be used on desk-top, lap-top or tablet computers. It starts with the ethical background based on the “3Rs” (Replacement, Refinement and Reduction) introduced by Russell and Burch in 19597. Pages on research strategy, the” experimental unit”, the characteristics of a “good” experiment, avoiding bias, power and sample size follow. There are further pages on controlling variability and strains of mice and rats. The page on experimental designs explains how randomisation can be done using an EXCEL spread-sheet for both completely randomised and randomised block designs. There are also pages on factorial designs, regression and correlation, and statistical analysis, although the emphasis is on design not analysis. Scientists who have access to one of the large commercial packages such as SPSS or MINITAB are advised to use it, but alternatively they can down load R and use the R Commander front end to carry out most of their analyses. Raw data for example analyses is given. There are pages on presentation of results, literature and guidelines. A pdf of the ARRIVE and Gold Standard Publication Checklist can be down-loaded. Each section has an associated “Test yourself” page with true/false questions. Anyone who works through the site can down load a page where they can self-certify that they have done so. The hope is that Ph.D. supervisors and ethical review committees will get all new Ph.D. students and scientists new to animal research to work through the site before they start work with animals.
Although the site is designed for individual study it could also be used as a teaching aid. Statisticians who teach experimental design and statistics to biologists using animals will find that it covers much of the biological background for such a course. The cost of developing the site has been minimal. Two or three month’s work by a retired scientist/statistician, some inexpensive software and a small web hosting fee. In due course it will need to be taken over by an organisation, but in the meantime it remains flexible so that improvements can easily be made.
Publicity
Distribution is a problem. Establishing direct contact with the many thousands of biomedical scientists who might find the site useful is not easy. However, in countries where most research is carried out the animals are protected by animal welfare legislation. Animal houses will usually be under the control of a professional scientist, often with veterinary qualifications, and in many cases there will be an ethical review committee (Institutional Animal Care and Use Committee in the USA). Most countries also have laboratory animal science associations, and there are a number of list-servers. Publicity has therefore been directed towards the laboratory animal science community in the hope that they will be able to guide research scientists planning to work with animals to the website.
Results
The site was first publicised at the beginning of April 2012, although it was used in an eight-hour course taught in Madrid in March. Since then it has had about 4000 visitors, of which 70% were new and 30% were returns. The “bounce rate” (people who left immediately) was 46%. On average, visitors who stayed looked at five pages and spent four minutes on the site. Most visits were from Spain followed by Australia, UK, USA, Ireland, Germany, Norway, India and Canada with hits from a total of 100 countries.
It is too early to tell whether people are finding it useful. The short duration of the visits might be because many are from heads of animal houses or members of ethical committees checking it out for future use by others, rather than the end users. People do seem to be testing themselves in the true/false pages. One of the most visited pages is “No, your answer is wrong”!
Conclusion
There is certainly a need for better training of research scientists in experimental design who use laboratory animals. This website offers one cost effective approach. Whether it will have any impact remains to be seen.
Comment on this article
Skip to Main Site Navigation / Login
