Systematic Review

Record management

Record management is an important process when undertaking a systematic review and needs to be clearly documented so the process is transparent and can be reproduced.

The search history can be saved within each of the databases a search is undertaken in. Researchers will need to set up a free personal account to be able to save the search history (it is recommended that this be done prior to undertaking any search).

Saved search results can be exported to the chosen reference management system by using the direct export option.

Some of the reference management systems available are:

  • EndNote (available to CQUniversity students and staff).
  • RevMan 5 (Used for Cochrane Systematic reviews)
  • Covidence (Collaborative database for managing the screening and data extraction. First systematic review process is free)

It is recommended that for each database searched, the results be recorded using the PRISMA flow diagram .

*Alerts can be set up once the final search has been undertaken in each database. This will enable researchers to receive notifications of any new published literature related to their search terms.

A reference management system will assist in calculating the total number of records identified and assist in the removal of duplicates. This will enable to reviewers to move onto the screening process without having to manage excess articles.

Screening

Screening Abstract and Title

Two or more reviewers should undertake the screening process. Reviewers will screen the same list of titles and abstracts.

During the screening process, reviewers will apply the inclusion criteria set out in the protocol, screening the title and abstracts to identify any relevant studies. If reviewers come across an abstract where they are unsure it meets all of the inclusion criteria, the recommendation is the article be included.

Whilst undertaking the screening of title and abstract, reviewers are not to discuss the articles they have included or excluded. This is to reduce the risk of bias.

Once this initial screening process is completed, reviewers are to meet and discuss the number of articles they have agreed and disagreed on. Where there is a disagreement reviewers either can employ an independent third party to evaluate the title and abstracts to the inclusion criteria or undertake a through discussion.

To ensure there is a high level of inter-rater reliability between reviewers the use of a measurement such as Cohen’s Kappa, which measures the “degree of agreement” (Fleiss, J.L. 1971. p. 378) between two reviewers, should be utilised

*This is a time consuming process and can take days or even weeks depending on the number of records identified in the search.

 

Screening Full Text

Once screening and abstracts is completed, reviewers will need to record the number of records excluded and then obtain the full text of included articles.

To enable reviewers to follow the protocol, the creation of a table (example below) or checklist for each reviewer will assist in this process. If reviewers are not using the Covidence software, an excel spreadsheet can be set up to assist keeping data in one place.

Author/s

Date

Publication

Study Aim

Hypothesis

Research question/s

Location

Study Design

Analysis Methods

Results

Findings

Relevancy to research

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

When all the full text articles have been obtained, the screening process is undertaken again. Reviewers again will need to complete this process independently and employ the assistance of a third reviewer to screen any disagreements.

Reviewers will need to record the number of records to be included into the data extraction process and the number of exclusions along with the reasons as to why the article is not eligible for inclusion.

*Again, this is a time consuming process and a finer screening lens needs to be applied.

 

Remember that reference management software can be used in the storing of citations and assist in screening such as:

Endnote

Covidence

Once you have established the number of studies to be included in your review, assessing the quality of the studies needs to be undertaken.

 

Reference:

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. http://dx.doi.org/10.1037/h0031619

Assessing the quality of papers

There are many checklists available to assist researchers with assessing the quality of the studies included.

The following list is just an example of the checklists available. The types of studies included in the review will dictate which checklist could be used for the assessment process.

CASP Checklists enable researchers to assess systematically the trustworthiness, relevance and results of published papers.

JBI Checklist. JBI provides a list of 13 critical appraisal tools

CONSORT Checklist with focuses on reporting trial design, analysis, and interpretation.

ROBINS-I tool Risk Of Bias In Non-randomized Studies – of Interventions.

QUADAS-2 in systematic reviews can be used to evaluate the risk of bias and applicability of primary diagnostic accuracy studies.

Cochrane Handbook – Chapter 8: Assessing risk of bias in included studies.

 

After assessing the quality of the included articles, it is time for data extraction.

Data extraction/management

During the undertaking of the quality assessment, researchers will have become familiar with the data within the included studies and start to identify data for extraction. The data extraction process requires meticulous planning and management.

 

For the data extraction process researchers can create their own data extraction template (example of the Data Extraction Form from the Cochrane Review Group below) or if using a tool such as Covidence, the data extraction form can be customized to the researchers needs. (The data extraction form and database should align with the suggestions provided by a statistician)

Key steps for researchers to consider when undertaking data extraction are:

  1. Identify what data you are wanting to extract to assist in answering your review question
  2. Develop and test your data extraction form and database designed for recording your data
  3. Develop a clear plan of who will be undertaking the data extraction and the time frame in which it is to be completed
  4. Identify where the extracted data is going to be stored, for example will this be cloud based verses hard drive; and who will have access to the data
  5. Complete the data extraction table. Completing the tables in to a format for presenting evidence enables researchers to refer to the completed data when writing and inserting tables into a thesis or article for publication.

 

To assist analysing the extracted data there are several tools available such as:

For further information on these software applications, you will need to contact the Research Division at research-training@cqu.edu.au.

 

For information on how to manage your data during this process, you can find information in the Library guide – Research Data Management.

Data Synthesis

Before you undertake data synthesis, it is important for researchers to meet with a statistician, especially if this is their first undertaking of a meta-analysis.

Ideally, researchers would have already met with a statistician during writing their protocol and at key points though out the review process.

A statistician will guide a researcher on the exact methods for carrying out a meta-analysis using the appropriate software and assist researchers in understanding the results.

To discuss data synthesis researchers should contact the Research Division.

The Cochrane Handbook Chapter 9: Analysing data and undertaking meta-analyses can provide researchers with further information on data synthesis.