The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching

Haddaway NR, Collins AM, Coughlin D, Kirk S
The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching.
PLoS ONE 10(9): e0138237
doi:10.1371/journal.pone.0138237

Objectives

This paper analyses the use of Google Scholar as a source of research literature to help answer the following questions:
1. What proportion of Google Scholar search results is academic literature and what proportion grey literature, and how does this vary between different topics?
2. How much overlap is there between the results obtained from Google Scholar and those obtained from Web of Science?
3. What proportion of Google Scholar and Web of Science search results are duplicates and what causes this duplication?
4. Are articles included in previous environmental systematic reviews identifiable by using Google Scholar alone?
5. Is Google Scholar an effective means of finding grey literature relative to that identified from hand searches of organisational websites?

Methods

Using systematic review case studies from environmental science (seven), this paper analyses the utility of Google Scholar in systematic reviews and in searches for grey literature.  The search strings used herein were either taken directly from the string used in Google Scholar in each systematic review’s methods or were based on the review’s academic search string where Google Scholar was not originally searched. Search results in Google Scholar were performed both at “full text” (i.e. the entire full text of each document was searched for the specified terms) and “title” (i.e. only the title of each document was searched for the specified terms) level using the advanced search facility. 
Since Google Scholar displays a maximum of 1,000 search results this was the maximum number of citations that could be analyzed.

Results

- Between 8 and 39% of full text search results from Google Scholar were classed as grey literature (mean ± SD: 19% ± 11), and between 8 and 64% of title search results (40% ± 17).
-  Google Scholar's search results show a greater percentage of grey literature than academic literature in title search results (43.0%) than full text results (18.9%).
- Most of the grey literature documents were usually displayed around page 80 (±15 (SD)) for full text results, whilst it occurred at page 35 (± 25 (SD)) for title results.
- Google Scholar demonstrated modest overlap with Web of Science title searches: this overlap ranged from 10 to 67% of the total results in Web of Science
- The percentage of total results that are duplicate records for Google Scholar range from 0.56 to 2.93% and for Web of Science from 0.03 to 0.05.
- Many of the included articles from the six published systematic review case studies were identified when searching for those articles specifically in Google Scholar (94.3 to 100% of
studies). However, a significant proportion of studies in one review [31] were not found at all using Google Scholar (31.5%)
- When searching specifically for individual articles, Google Scholar catalogued a larger proportion of articles than Web of Science (% of total in Google Scholar / % of total in Web of Science: SR1, 98.3/96.7; SR4, 94.3/83.9; SR6, 99.4/89.7).
- None of the 84 grey literature articles identified by (systematic review 5)  were found within the exported Google Scholar search results (68 total records from title searches and 1,000 of a total 49,700 records from full text searches). However, when searched for specifically 61 of the 84 articles were identified by Google Scholar

Available
 http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0138237