On Low Overlap Among Search Results of Academic Search Engines: Google Scholar, Semantic Scholar Microsoft Academic, Scopus

Mitra, A., & Awekar, A. (2017)
On Low Overlap Among Search Results of Academic Search Engines. 

https://arxiv.org/pdf/1701.02617

Introduction:  To tackle this information overload, researchers are increasingly depending on niche academic search engines. Recent works have shown that two major general web search

engines: Google and Bing, have high level of agreement in their top search results.Various works have tried to predict coverage of academic search engines. However to the best of our knowledge, there is no existing work that systematically studies overlap in the search results of academic search engines.

Methods: We collected over 2300 query terms from 2012 ACM Computing Classification System. We sent these 2500 queries to all four selected academic search engines. For each academic search engines, we looked at top eight results as each academic search engines returns at least eight results on the first page. We computed similarity between any two sets using the Jaccard similarity. 

Results: For all 2500 queries, intersection set of all four academic search engines considered together was always empty. In other words, for each query, no research article appears in the top results list of all four academic search engines. This shows strong diagreement among academic search engines. 

Therefore we observe that overlap in search result sets of any pair of academic search engines is significantly low and in most of the cases the search result sets are mutually exclusive.