Systematic reviews and project licences in animal research - a contribution from RDS

Recent reviews have exposed the methodological failings of many studies using animals. For example an article examining US animal studies of amyotrophic lateral sclerosis suggested that most of the studies that were published, including some in top-rank journals, had been done with small sample sizes, with no randomisation of treatment and control groups, and without blinded evaluations of outcomes (Nature 2008). If the analysis is correct, then this is clearly unsatisfactory.

There is increasing recognition of the need to improve both the design and analysis of studies using animals. RDS has consistently supported the use of systematic reviews as one such way to improve research. This is highlighted on our website, on our blog, and has been featured in our newsletter. Such reviews could also help further the 3Rs. SABRE does a good job of pointing out where systematic review are appropriate.

The SABRE petition proposes that each licence application includes references to systematic reviews of existing relevant studies. Depending on how exactly this is interpreted, such a proposal could cause difficulties should the referencing of systematic reviews become a ‘requirement’ for the award of a project licence.

The UK project licence is given under the Animals (Scientific Procedures) Act 1986 (ASPA) is a legal document based on a cost-benefit assessment of whether it is permissible to cause harm to an animal. It is a contract between the UK government and an individual researcher. Its award is dependent on a legally-defined process of review designed to ensure that animals are used in research only when there are no other means of obtaining the results and only when the anticipated outcome warrants any harm inflicted on the animals.

Systematic reviews and meta-analyses of animal studies are rare. Only 103 such reviews, comprising 57 systematic reviews, 29 systematic reviews accompanied by a meta-analysis and 17 meta-analyses were identified up to 2005 (Peters et al, 2006). For the vast majority of proposed animal studies, there will be no systematic reviews to refer to when applying for a project licence.

If systematic reviews do not exist in some areas, and cannot be forced into existence, they cannot be made a legal requirement for the award of a project licence. This is particularly the case when novel, groundbreaking research into basic biological mechanisms or structures is proposed, rather than evaluation of an existing drug, compound or procedure. Even where reviews can be carried out, there remains the difficulty of deciding whether the requirement would be for review of the procedures utilised or the substances undergoing test.

Basic animal studies may be designed to study biological mechanisms (rather than outcomes), or species, strain and methodological differences. Animal studies may open completely new fields of research where there is little prior knowledge (for example the creation for the first time of a genetically modified animal model of the human disease). Yet for a systematic review to draw a meaningful conclusion there must be sufficient available primary studies. But repetition in animal studies is strongly discouraged both by regulators and by the publication process. For much basic research, the literature may be neither sufficiently extensive nor sufficiently consistent to permit systematic review.

Additional questions remain about the applicability of systematic reviews to all types of animal research (Lemon and Dunnett, 2005). Some examples may help to illustrate the problems.

Much basic research is shallow and broad in its cover – that is, only one or two research groups world-wide may be studying an area. This is commonly the case in environmental/conservation research. Even flicking a dart into some deer to monitor their populations in the wild, or taking a biopsy or blood sample from a live seal washed up on the shore, may need a project licence, as would the use of guinea pigs as hosts for a study of tick biology.

Furthermore, a project licence is not always awarded for a single experiment or research project, but more often for a series of procedures. Some of the procedures to be carried out may depend on results obtained during an earlier project licence, which have not yet been published. Researchers do not always know in advance which procedures will be carried out within the remit of their project licence. So, for instance, to require application for renewal of a licence to be delayed until a systematic review had been conducted of the results of the current licence, would be totally unworkable.

In the case of toxicology tests, laboratories use well-characterised and relatively standardised techniques. However, it is not known in advance which substances will be tested over the course of the project licence. So it is unclear what would be the nature of the systematic review required. About 10% of all procedures are for the production of biological materials such as tissues, blood products, infectious agents, vectors, neoplasms and antibodies. And in the assay and quality control of biologicals, the models are generally well validated and the studies are already carried out in methodologically consistent ways.

When animals are used as models of humans, systematic reviews can be used in two ways: (i) to study the results of a particular method or intervention in an animal model to give a clear conclusion or (ii) to determine whether the animal model is a good predictor of results in humans. No systematic review can be used for the latter purpose unless the response in humans is known. However, if animal studies indicate that a potential treatment does not work, then that treatment would not normally proceed into human trials. In such cases, it is not possible to determine whether the animal models are predictive of humans, as no human data is available. In the case of nimodipine (Evans et al 2006), a systematic review of animal studies at the pre-clincial stage would not have led to human trials, and would not allow us to know whether an animal model is or is not predictive of the human situation. On the other hand, a comparison of systematic reviews of animal studies against the results of clinical trials can help validate the animal models, but obviously cannot further inform the decision taken to proceed to human trials. These two distinct purposes of carrying out systematic reviews are different, and must be recognised as such.

We are not arguing against systematic reviews here, nor suggesting that the current mechanisms for undertaking animal research are perfect (or even adequate). Animal studies need to be improved, just as improvements are needed for all types of research. Even in clinical trials, where systematic reviews are accepted as the gold standard, there remain improvements to be made.

RDS acknowledges that systematic reviews are at the top of the hierarchy of evidence, and are superseding narrative reviews because of their improved quality. It is said that the point of conducting systematic reviews is to reduce bias as much as possible in the evaluation of animal research (SABRE 2006). If the use of systematic reviews amounts to no more than the use of explicit and reproducible methods to reduce bias, then all reviews should be systematic.

However, most descriptions of systematic reviews (which are usually about assessing the effect of a therapeutic intervention) emphasise the importance of having a clearly defined topic and a specific question to answer, as well as having rigorous inclusion and exclusion criteria. Both present practical difficulties when basic research using animals is founded on a very diverse range of prior research. Sometimes results from in vitro and in silico studies are relevant, as well as information from other fields of animal research, or even knowledge from human epidemiological and clinical studies. This is a concern because it seems the most common reason for discrepancies between systematic reviews is differences in inclusion criteria (Linde 2003). Even proponents of systematic reviews seem to recognise this issue by calling for ‘initial research to discover the best methodological approaches for systematically reviewing animal studies’ (SABRE Charter, May 2008). We support this proposal.

Narrative (critical) reviews are currently common in basic research in which animal-based studies play a part. By definition even narrative reviews require a substantial body of research to have been undertaken, but at least they can overcome diversity and a lack of consistency in approach, or patchy coverage of the initial studies in a field.

A valid criticism of traditional narrative reviews is that they are ‘rarely explicit about how studies are selected, assessed and integrated. Thus the reader is generally unable to assess the likelihood of prior beliefs or other biases clouding the review process’ (Davies). There is no reason for this to continue. RDS considers that good methodology which leads to an objective account of the available data can be applied to both the systematic and narrative review process, and that the most useful analysis of the literature is obtained though use of the most appropriate review method.

Many of the protocols associated with systematic reviews can also be applied to narrative reviews, so that the data is assessed in a reproducible, transparent manner and bias is reduced. This requires that search criteria are broad, inclusion criteria are specified, and methodologies of included studies are evaluated critically. This approach would allow readers to reach more consistent conclusions from narrative reviews.

There are nonetheless many instances when systematic reviews as currently practised could be of great value. For example, they are desirable when a number of papers studying the same intervention have been published without any clear conclusion, or when a candidate medicine is considered suitable to proceed to human clinical trials. RDS continues to support systematic reviews in animal research.

References:
Davies HTO, Crombie, K (2001) What are systematic reviews? http://www.jr2.ox.ac.uk/bandolier/painres/download/whatis/Syst-review.pdf
Evans I, Thornton H, Chalmers I (2006) Testing Treatments: Better Research for Better Healthcare.
Linde K and Willich SN, J R Soc Med (2003) How objective are systematic reviews? Differences between reviews on complementary medicine.96(1) January.
SABRE Charter (2008) http://www.sabre.org.uk/ May.
SABRE personal communication (2006).
Lemon R and Dunnett SB (2005) Surveying the literature from animal experiments. BMJ 330: 977-978.
Nature News feature. Standard model. (2008) 454/7.
Nature Medicine editorial: when less is not more. (2008) 14, Number 8.
Perel P, Roberts I, Sena E, Wheble P, Briscoe C, Sandercock P, Macleod M, Mignini LE, Jayaram P, and Khan KS (2006) Comparison of treatment effects between animal experiments and clinical trials: systematic review. BMJ 15 December.
Peters, J L, Sutton, AJ, Jones, DR, Rushton, L, and Abrams, KR (2006) A systematic review of systematic reviews and meta-analyses of animal experiments with guidelines for reporting. J Environ Sci Health B 41: 1245-1258.

’It is imperative that European institutions reflect the views of citizens by enshrining in law the principle that research using live animals not longer has a place in a civilized 21st century Europe,’ states Michelle.

There is simply no evidence that EU citizens hold those views. If they did, one would expect them to be coming out of the woodwork to endorse Michelle’s article with their comments. But they don’t. That’s fair enough - maybe EU citizens don’t all read the Guardian. 

What about the UK ones though? Out of the first 20 comments the article received, seven supported (either clearly or vaguely) Michelle’s article (with one comment directly from the BUAV) and 12 supported the use of animals in research and the role this work plays in the development of treatments for diseases. That’s 35% agreeing with Michelle and 60% disagreeing with her (one comment was neutral).

This analysis is, of course, no more reliable or scientific than the claim by Michelle to represent the views of EU citizens, based as her’s is on a self-selected sample of antivivisectionists! 

One such example is that of the work of Dr Siouxsie Wiles, who won the NC3Rs prize in 2005 for her work in improving design of bacterial infection experiments, leading to a reduction in the number of mice needed for this research. Dr Wiles found infection was more efficient when mice were infected naturally by putting an infected mouse in a cage with others, compared with artificial infection routes.

Dr Wiles has been awarded funding for a new project, aiming to develop ways of making human disease agents ‘light up’. Traditional disease models can involve animal suffering, and bacteria numbers are determined by killing infected animals at defined points. If the bacteria are genetically engineered to produce light, they can be measured using imaging techniques in real time, in living animals. So this project should reduce the numbers of animals used and the suffering of individual mice.

The government’s commitment to funding this research was announced last year. The innovative projects announced today are a welcome addition to the extensive scientific work already being done - by public bodies and by commercial concerns - to reduce, replace and refine animal use in research.

Not surprisingly, we don’t support the decision. It is baffling because there is no clear distinction between basic and applied research, except at the ends of the spectrum. It is widely appreciated that much basic research can lead to medical advances, and that much applied research can shed light on basic biological functions.

Whilst the research institute is appealing the decision, the final outcome will also be baffling, since its impact more widely is not known. Switzerland is an exceptional case. It is renowned for having held a number of referenda on animal experiments, all of which have gone in favour of continued research using animals, but which have led to pressure to tighten regulations.

Ultimately, there is no reason that the European Union, which is reviewing its own laws on animal experiments, should be unduly swayed by this decision. It is up to the Swiss to decide how their own laws work. 

The total UK science spending stands at £5.4 billion. This is, admittedly, a large figure. However, this is total science spending - encompassing research into new technology, chemistry, physics, education programmes.... The list is endless. If we look at the proportion of government money spent doing animal research in the past decade (as opposed to developing alternatives to animal research), it has fallen year on year, despite a massive increase in Government bioscience and medical research (reported by RDS website here).

One particular figure caught my interest - that quoted as the amount spent by the Ministry of Defence on non-animal replacements. By the Trust’s own admission, this figure is made up (see reference 16 of the report). Following a FOI request for exact figures, the MOD said it was ‘actively seeking replacements’. This led the Trust to insert ‘the minimal amount ... that could feasibly constitute a claim’, ie a round figure of £70k each year, which is a tad suspect.

Developing alternatives to animal research and testing is no easy task. Whilst research can be conducted using computer modelling, tissue engineering, and the use of stem cells, these methods are really complementary to animal tests rather than alternatives. They offer limited information about what happens in a whole, living animal – and this is where the difficulty lies.

So while we simply do not have sufficiently advanced technology to develop large-scale replacements, funding is directed at the other 2Rs, refinement and reduction. The government estimates the total spend to be £10 million – see their FAQs on the matter.

Reduction is clearly happening - even the Dr Hadwen Trust report acknowledges this, and refinement is a key concern for all, highlighted in the recent ASPI report (we blogged about it here). The New Statesman has a recent article, written by the Communications Director of the Dr Hadwen Trust. Unsurprisingly, this is as critical of the government as the report is. Hopefully the public will make up their own minds on this score.

For the ‘surplus’ they use old estimates produced by Great Britain (80%) and Norway (38%). The authors themselves admit ‘These data are very incomplete and variable, and may not justify a formal extrapolation to worldwide figures’ and that they ‘do not command such high levels of confidence’. Nevertheless, they use this arguably invalid extrapolation to add 59% to the total.

Another assumption which may not be valid is that most countries do not include breeding of GM animals in their official estimates. The authors allude to two countries that do include them; they comprise a significant minority (33%) and a very tiny minority (0.7%) of procedures in Britain and the Netherlands respectively. Based on these wildly disparate and limited figures, they produce an average of 17% and apply this to other countries, ie they add 17% to official estimates.

While examining the ATLA paper, I came across an interesting statistic that relates to rodents bred for research regarded as ‘surplus’. While many are used as breeding stock or humanely killed so that their tissues and organs may be used in research, some are killed and sold as pet food – for raptors and reptiles. According to the Wall Street Journal in 1999 (reproduced by the UK’s Animal Procedures Committee in 2003), some 180 million rodents are killed for this purpose every year in the USA. At least 10 times greater than the number of rodents used in research in that country.

The antivivisectionists must feel that overall numbers are important, as they have clearly spent a long time collecting a mass of data. But without context they are rather meaningless. That context may be ‘social’ as above, or related to the bigger medical research picture. For instance, spending on UK biomedical research has increased by at least 50% since 1995 while animal procedures have increased by only 18%. 

The strict regulatory regime in the UK does provide, on an annual basis, a mass of statistics about animal research. We know that in 2007, 3.2 million procedures were conducted using just over 3 million animals, and that 83% of these procedures used rodents. We also know the numbers of GM mice used in breeding (850,000 in 2007). Of course, this is not news – we covered the latest UK statistics in a recent blog entry.  Such bald figures may not be of much public interest either.

While it is unacceptable that any infringements occur, it is good to know that this level of accountability exists. In all instances, the people involved were admonished, given extra training where required, and practices were modified to ensure they did not happen again. In 2007, ASPI inspected the work of 14,438 personal licensees, and the report states there were 30 cases of infringements – none of which required a licence to be revoked.

Figures out this week from the RSPCA show a trend of increasing cruelty to animals – 2007 saw an increase of 24% in the number of convictions for animal cruelty. There were 1,149 people convicted for crimes against animals, with 54 people being given prison sentences, mainly for cruelty to pets. When you have figures such as these, the 30 infringements in research – half which were self-reported, and the majority involved minor non-compliance – pale in comparison.

For each one of the tortured dogs, kicked cats, and abused horses in the RSPCA report, you can bet the owners didn’t bring themselves forward, take on responsibility – they had to be thoroughly investigated to get the convictions. The animal research community is clearly much more open and accountable; it should be recognised that the welfare of animals involved in research is of paramount importance to all concerned. 

One scientist quoted by Arron cautioned that the risks of nanoparticles could outweigh the benefits. But any debate about the use of mice in medical research was clearly not considered relevant to the piece.

Of course the antivivisectionists are quick to latch onto statistics such as only 1 in 5000 drugs getting approved for human use. Last week our young friends at Speaking of Research (the US counterpart of Pro-Test) were busy debunking the oft-quoted ‘92% of drugs that pass animal tests fail human trials’ statistic. We have addressed such nonsense several times on this blog.

Tom at Speaking of Research points out that you can use statistics allied with assumptions to ‘prove’ anything. He comes up with his own (hypothetical) case that 90.5% of dangerous drugs have been kept out of human trials thanks to animal safety tests.