Enrique Orduña-Malea & Emilio Delgado López-Cózar
Performance Behavior Patterns in Author-Level Metrics: A Disciplinary Comparison of Google Scholar Citations, ResearchGate, and ImpactStory
Frontiers in Research Metrics and Analytics, 22 December 2017
https://doi.org/10.3389/frma.2017.00014
Performance Behavior Patterns in Author-Level Metrics: A Disciplinary Comparison of Google Scholar Citations, ResearchGate, and ImpactStory
Frontiers in Research Metrics and Analytics, 22 December 2017
https://doi.org/10.3389/frma.2017.00014
INTRODUCTION
The main goal of this work is to verify the existence of diverse behavior patterns in academic production and impact, both among members of the same scientific community (inter-author variability) and for a single author (intra-author variability), as well as to find out whether this fact affects the correlation among author-level metrics (AutLMs) in disciplinary studies. in order to answer the main goal mentioned above, we pose the following research questions:
RQ1: Is it possible to detect different academic behavior patterns among authors working in the same discipline (inter-author variability) using the AutLMs available in online academic profile services?
RQ2: Is it possible to detect different academic behavior patterns by the same author (intra-author variability) using the AutLMs available in online academic profile services?
RQ3: If the two previous questions can be answered affirmatively, do these behaviors affect the correlations between AutLMs?
METHODOLOGY
Ttwo samples are examined: a general sample (members of a discipline, in this case Bibliometrics; n= 315 authors), and a specific sample (only one author; n = 119 publications). Four AutLMs (Total Citations, Recent Citations, Reads, and Online mentions) were extracted from three platforms (Google Scholar Citations, ResearchGate, and ImpactStory).
RESULTS
The analysis of the general sample reveals the existence of different performance patterns, in the sense that there are groups of authors who perform prominently in some platforms, but exhibit a low impact in the others. The case study shows that the high performance in certain metrics and platforms is due to the coverage of document typologies, which is different in each platform (for example, Reads in working papers).
As regards AutLMs, a non-linear distribution in the data extracted from the three platforms (Google Scholar Citations, ResearchGate, and ImpactStory) has been found. There are few authors with a high performance, and a long tail with moderate, low, or null performance. Moreover, the high performance authors are not the same across the three studied dimensions of impact (Citations, Reads, and Online mentions). The lack of correlation might be explained by the fact that each platform offers different documents, targeted to different audiences.
This fact has facilitated the identification of different patterns of online academic behavior in the studied platforms (RQ1). Some authors present a markedly formal performance (Citations, mainly to journal articles) while other authors stand out in Reads (both to articles and to other document typologies), or in Online mentions (mainly Articles). Combined patterns have also been found (high performance in Citations and Reads, and low in Online mentions). This issue evidences that the analysis of a single platform, not even considering of the demographic aspects related to the population of a discipline that is reflected, can mask the performance of an author who has particularly high or low values in any given platform.
Lastly, the ego-analysis has allowed us to confirm the existence of authors with different patterns of online academic behavior depending on the types of documents that they publish (RQ2). In this case, we could observe the existence of working papers with a high amount of Reads and Online mentions, as well as the existence of a large group of articles with a lower number of citations. That is, certain typologies are generating an impact (Reads in ResearchGate) that cannot be observed in other platforms. Again, the different nature of the research activity (article: generating knowledge; report: application of knowledge to solve a problem; educational materials: knowledge transfer, etc.) determines everything. The people who cite are scientists, the same ones that produce scientific knowledge, whereas practitioners read but do not cite as much, so it is less likely that they would cite other studies.
This fact again brings us to the need not only of considering different online academic profile platforms (in order to capture different impact profiles) but also to categorize the type of impact according to the document typologies, because a general analysis of authors might mask their actual impact.
All this makes us question the usefulness and precision of the correlation analyses of AutLMs within a discipline that have not taken into account inter-author or intra-author variability to model the multidimensional impact of authors. This is one of the aspects in which Altmetrics studies should focus their attention from now on.
The main goal of this work is to verify the existence of diverse behavior patterns in academic production and impact, both among members of the same scientific community (inter-author variability) and for a single author (intra-author variability), as well as to find out whether this fact affects the correlation among author-level metrics (AutLMs) in disciplinary studies. in order to answer the main goal mentioned above, we pose the following research questions:
RQ1: Is it possible to detect different academic behavior patterns among authors working in the same discipline (inter-author variability) using the AutLMs available in online academic profile services?
RQ2: Is it possible to detect different academic behavior patterns by the same author (intra-author variability) using the AutLMs available in online academic profile services?
RQ3: If the two previous questions can be answered affirmatively, do these behaviors affect the correlations between AutLMs?
Ttwo samples are examined: a general sample (members of a discipline, in this case Bibliometrics; n= 315 authors), and a specific sample (only one author; n = 119 publications). Four AutLMs (Total Citations, Recent Citations, Reads, and Online mentions) were extracted from three platforms (Google Scholar Citations, ResearchGate, and ImpactStory).
RESULTS
The analysis of the general sample reveals the existence of different performance patterns, in the sense that there are groups of authors who perform prominently in some platforms, but exhibit a low impact in the others. The case study shows that the high performance in certain metrics and platforms is due to the coverage of document typologies, which is different in each platform (for example, Reads in working papers).
The correlations (Spearman; α < 0.1) between the different metrics (Total Citations, Recent Citations, Reads, and Online mentions)
As regards AutLMs, a non-linear distribution in the data extracted from the three platforms (Google Scholar Citations, ResearchGate, and ImpactStory) has been found. There are few authors with a high performance, and a long tail with moderate, low, or null performance. Moreover, the high performance authors are not the same across the three studied dimensions of impact (Citations, Reads, and Online mentions). The lack of correlation might be explained by the fact that each platform offers different documents, targeted to different audiences.
This fact has facilitated the identification of different patterns of online academic behavior in the studied platforms (RQ1). Some authors present a markedly formal performance (Citations, mainly to journal articles) while other authors stand out in Reads (both to articles and to other document typologies), or in Online mentions (mainly Articles). Combined patterns have also been found (high performance in Citations and Reads, and low in Online mentions). This issue evidences that the analysis of a single platform, not even considering of the demographic aspects related to the population of a discipline that is reflected, can mask the performance of an author who has particularly high or low values in any given platform.
Lastly, the ego-analysis has allowed us to confirm the existence of authors with different patterns of online academic behavior depending on the types of documents that they publish (RQ2). In this case, we could observe the existence of working papers with a high amount of Reads and Online mentions, as well as the existence of a large group of articles with a lower number of citations. That is, certain typologies are generating an impact (Reads in ResearchGate) that cannot be observed in other platforms. Again, the different nature of the research activity (article: generating knowledge; report: application of knowledge to solve a problem; educational materials: knowledge transfer, etc.) determines everything. The people who cite are scientists, the same ones that produce scientific knowledge, whereas practitioners read but do not cite as much, so it is less likely that they would cite other studies.
This fact again brings us to the need not only of considering different online academic profile platforms (in order to capture different impact profiles) but also to categorize the type of impact according to the document typologies, because a general analysis of authors might mask their actual impact.
All this makes us question the usefulness and precision of the correlation analyses of AutLMs within a discipline that have not taken into account inter-author or intra-author variability to model the multidimensional impact of authors. This is one of the aspects in which Altmetrics studies should focus their attention from now on.
No hay comentarios:
Publicar un comentario