Search strategies to identify information on adverse effects: a systematic review

Search strategy

The reviewers anticipated that much of the literature in this newly developing area of search filters for information on adverse effects would be identified by searching beyond MEDLINE and EMBASE and that much of the relevant research would not be published as peer-reviewed journal articles. For example, a previous systematic review on a similar methodological topic only identified thirteen of thirty of the included papers through searching MEDLINE and EMBASE [21]. A range of bibliographic databases was therefore searched in the current study (Table 1). These databases were carefully selected to allow the identification of reports, dissertations, and gray literature in addition to journal articles. Hand-searching of key journals in librarianship, drug safety, and research methodology was carried out to identify articles either not indexed in electronic databases or not easily identifiable in electronic databases. These journals were selected by consultation with experts and from the authors' own knowledge of relevant literature.

Unpublished material was also sought by hand-searches of conference proceedings, by scans of evidence-based websites, and through discussion with experts involved in the Cochrane Adverse Effects Methods Group. Conference proceedings and web sources were selected on the basis of their coverage of systematic review methodology In addition, the bibliographies of any eligible articles identified were checked for additional references. Citation searches were carried out for all eligible articles using ISI Web of Knowledge.

Searching for methodological papers can prove very difficult in databases other than those in which methods papers are specifically labeled as methodology papers, such as the Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE). To develop search strategies in databases such as MEDLINE and EMBASE, a pragmatic approach was used to keep the results set manageable (Appendix, online). Terms were often limited to the title field only, and some potentially relevant text-words and indexing terms that retrieved thousands or tens of thousands of irrelevant records were omitted. All the search strategies were checked by a second experienced information scientist.

No date or language restrictions were applied to the searches. Although logistical constraints meant that non-English publications were not included in the data extraction or quality assessment, the reviewers thought an estimation of the size of the non-English literature was useful.

Inclusion criteria

A research study located with the above strategy was considered eligible for inclusion in the data extraction and quality assessment portion of this review if one of its main objectives was the development and/or evaluation of a search filter or filters that could generally be used for retrieving articles with data on the adverse effects of any health care intervention (resulting from drugs, surgery, etc.) from an electronic database. Eligible research studies were required to give at least one measure of performance, such as the sensitivity and recall or precision and specificity of the filters. Studies were excluded if they:

1. did not include a comparative evaluation of the different search strategies used to identify studies on adverse effects

2. were developed to identify only animal or laboratory studies

3. studied strategies to identify causation or etiology (for example, studies on environmental factors, such as pesticides and pollution, and studies on risk factors, such as cigarette smoking and drug abuse)

4. did not include any health care intervention

5. were not written in English and had no translation available at the British Library

Data extraction

Information was extracted about the database and interface for which the search filter was devised, the type of interventions, type of adverse effects, and methods used to create or test the search filter, such as the source and size of the reference set of relevant records or the validation set of records. Primary outcomes of interest were measures of sensitivity and recall (proportion of relevant articles retrieved by the filter), precision (number of relevant articles divided by the total number of studies retrieved with that filter), accuracy (proportion of all articles that are correctly classified by the filter), numbers needed to screen and read (inverse of precision), or specificity (proportion of irrelevant articles that were not identified by the filter). Any search strategies recommended by the authors were also noted.

Assessment of methodological quality

The methodological quality of the included studies was assessed using published criteria adapted specifically for this review [22,23]. The included studies were assessed using the following questions:

1. Were the tested and recommended search strategies and search filters described in sufficient detail to allow reproducibility (i.e., were the exact search terms with relevant truncation, field limits, and combinations presented and the search interface stated)?

2. Were the tested search terms objectively derived (e.g., by statistical analysis using word frequency counts comparing relevant and nonrelevant records)?

3. Was an adequate reference set of relevant records obtained (i.e., was the set of records on which the search strategies were tested obtained from a range of resources, such as databases and hand-searching, or a broad enough search strategy, for example, without any adverse effects terms or with lots of synonyms, to capture a relatively comprehensive set of relevant records)?

4. Did two or more researchers screen the retrieved records for relevant studies?

5. Were clear inclusion criteria given for the reference set (i.e., were details presented on the types of outcomes [adverse effects] and on any exclusion criteria that might have implications for the search strategy, such as study design)?

6. Were confidence intervals calculated for the performance estimates (confidence intervals enable the reader to assess how precise the performance estimates [such as sensitivity and precision] were)?

7. Were the results tested on a validation set of relevant records (i.e., was the performance of the developed search strategy or filter tested on a different set of relevant records from those used to derive the search filter)?

Included studies

Three studies met the inclusion criteria for this review [24–29]: two were published as full papers, and one was a conference presentation [24,25] (Table 2). Although studies were not excluded from this review according to the type of health care intervention, all three of the studies that met the inclusion criteria aimed to maximize the sensitivity of search strategies to identify papers on the adverse effects of drug interventions.

One study evaluated search strategies for a named specific adverse effect (breast cancer with oral contraceptives) [27–29], whereas the other two studies aimed to develop search strategies to capture all or all serious adverse effects for a particular group of drugs [24–26]. All three studies looked at search strategies for MEDLINE, and one study included search strategies in EMBASE [26].

Excluded studies

Seventeen studies were excluded from this review: eight contained no evaluation of the search strategies for adverse effects data that they proposed [30–37]; three were designed to identify sensitive search strategies for causation or etiological studies [38–40]; two did not suggest any filters but undertook co-word analysis [41,42]; three were in not in English [43–45]; and one evaluated search strategies to retrieve systematic reviews of adverse effects, rather than primary data [46].

Summary of methodological quality

One study was only published as a conference abstract [24,25], and although the slides of this presentation were available, the level of detail of reporting of the methods was not comparable to published research papers.

The number of relevant records in the reference sets varied considerably. The largest study was based on several hundred records and had a validation set of records. The authors stated that precision of the searches could be measured with their particular study design [24,25]. However, this study was not published in a journal, and full details of the methods were not presented. The other two studies tested their search strategies on smaller numbers of relevant records (fifty-eight and eighty-four records) [26–29] and did not test the search strategies on a validation set of records (another set of relevant records) (Table 3) [26–29]. None of the studies reported the confidence intervals for the point estimates of sensitivity.

Although each study used a number of sources to identify their reference set of records, it was possible that the original search strategies (despite searching a range of sources) failed to retrieve a substantial number of relevant records. The original search strategies might then have biased the results obtained from evaluating the search filters. For instance, if the reference set is obtained using the term “adverse” (among others), then this term is more likely to retrieve articles in the reference set and so is more likely to have a higher sensitivity when tested on that reference set. In this way, evaluation of search filters is often in danger of becoming self-fulfilling [47]. Each study was also limited to a particular class of drugs, limiting the generalizability of the results. The derivation of the search terms was not described for one study, whilst the other two studies used either terms derived from relevant records from a systematic review or used in previous studies.

Comparison of recommended search strategies

All 3 studies were able to create highly sensitive search strategies that had between 97.0% and 100.0% sensitivity (Table 4). However, the results of the 2 studies that also measured precision indicated that this high sensitivity was achieved with very poor precision (between 0.9% to 2.8%) [26–29]. This precision rate means that to retrieve one additional article on adverse effects, between 36 and 125 records retrieved with this strategy will need to be screened, which may be potentially unmanageable, given that full-text checking is often necessary.

The search strategy with the highest sensitivity in Wieland et al. [27–29] did not contain any text-words for the intervention (oral contraceptives), as searching specifically for the intervention would have missed nine of the relevant citations. However, as the authors acknowledge, any search strategy that excludes terms for the interventions is likely to lead to unmanageably large results.

The studies by Badgett et al. [24,25] and Golder et al. [26] both indicate the value of using floating subheadings (subheadings not attached to any indexing terms) for highly sensitive searches in MEDLINE [27]. Badgett et al. [24,25] suggested using the subheadings “adverse effects,” “complications,” “poisoning,” and “drug effects,” whereas Golder et al. [26] recommended using “adverse effects,” “complications,” and “drug effects.” Golder et al. [26] was the only study to attempt to develop a search filter for EMBASE, and the suggested search strategy for EMBASE in that study did not differ substantially from the suggested search strategy in MEDLINE other than in the use of subheadings. While the MEDLINE search strategy indicated the value of floating subheadings, the results of the EMBASE search strategy suggested that using subheadings attached to the named drug intervention (for example, vigabatrin/adverse drug reaction or vigabatrin/drug toxicity) performed better.

Badgett et al. [24,25] and Wieland et al. [27–29] both included study designs in their filter, and Golder et al. [26] and Wieland et al. [27–29] both included specified known adverse effects. Text-words such as “adverse effects,” “side effect,” and “adverse reaction” were only included in the filter by Golder et al. [26].

Use of adverse effects terms

Two of the studies included in this review recommended search strategies for adverse effects using not only adverse effects terms, but also study design terms. Adverse effects terms alone, therefore, might not be sufficient to identify papers with information on adverse effects. A study by Derry et al. [48] has identified particular problems in using terms for adverse effects alone to create highly sensitive search strategies. They studied 107 trials that reported adverse effects data and assessed how many papers were indexed with relevant terms for adverse effects in MEDLINE and EMBASE and how many titles or abstracts contained “adverse effects” or related terms. They found that a combined search covering the 2 databases using both index and text-word terms for adverse effects would have retrieved only 82 of 107 (77%) trials [48]. Other studies have also indicated the problems of searching on terms for adverse effects in the title and abstract [49–51]. One study found that of the adverse effects literature from one database, 64% (of 3,040 studies) contained adverse effects terms in the title [49], whilst 2 more recent studies found that adverse effects were mentioned in only 53% (130/243) and 63% (328/521) of abstracts of journal articles [50,51].

Use of intervention terms

It is often assumed that search strategies should contain terms for the intervention under investigation. While this is probably true for clinical trials, the situation is different for observational studies that are focused on identifying the etiology or multiple risk factors behind a particular adverse outcome (e.g., the risk factors for breast cancer). Here, Wieland et al. [27–29] found that not all studies of adverse effects contained terms for one of the suspected drugs (oral contraceptives) in the bibliographic details. However, other studies have indicated that searching on drug terms in the title might be an effective method for searching for adverse effects, identifying 99% of papers [49]. It should be noted, however, that the study in question [51] is now over 30 years old.

Use of subheadings

Some guidance on searching for adverse effects is currently available [30–37], although how evidence-based this guidance is, is difficult to ascertain. Much of this guidance, however, has tended to emphasize the usefulness of subheadings (such as “adverse effects” or “drug toxicity”) in MEDLINE and EMBASE [30, 31, 33–37]. The results from Golder et al. [26] and Badgett et al. [24,25] do suggest that subheadings are useful.

Use in a clinical setting

Although all three of the included studies aimed to create search filters that maximized sensitivity, the suggested terms and approaches may be adapted (with appropriate limits, such as specific adverse effects terms or subheadings) for searches for adverse effects in clinical settings where the busy clinician would prefer greater precision. However, the problems of poor indexing and inconsistent terminology will also be an issue for all searchers, whether they aim to maximize sensitivity or precision.

Limitations of the study

This systematic review has a number of limitations. First, the difficulty in searching electronic databases for methodology papers might mean that some relevant studies were missed. However, hand-searching, reference checking, and citation searching were included to overcome this limitation to some extent. Second, there was a potential for bias, particularly as the author of one of the included studies was also an author of this systematic review. To reduce any potential bias, data extraction and analysis were performed in duplicate, with the second reviewer working independently. Full consensus was reached between the two reviewers.

Conflict of interest

No conflicts of interest have been declared.

Source of funding

This research was undertaken by Su Golder as part of a Medical Research Council (MRC) fellowship. The views expressed in this presentation are those of the authors and not necessarily those of the MRC.