Thursday, January 13, 2011

The impact of the impact factor on the scientific publication process

The impact factor is a measure of how often articles published in a given scientific journal get cited by other research articles. Scientists tend to use the impact factor as the main method for evaluating the quality of a given journal publication. Because the impact factor is the main source of respectability for a given journal publication among scientists, journal editors are naturally biased towards achieving as high an impact factor as possible for their journal. This bias leads to a connected bias: the rejection of studies for which the journal editor qualitatively believes that the study is not likely to generate citations from other scientific research in the near future.

Normally the impact factor is measured over a period of about 2 years. Thus the impact factor is not really a measure of the impact of a given collection of articles, but a measure of the short-term impact of a collection of articles. Some impact factors operate over 5 years, but this is still a relatively short period of time when compared with large-scale long-term scientific progress. Since the impact factor is based on a short period, it contributes to biasing science towards the publication of articles in 'hot' fields with lots of active scientists and the process biases publication away from articles with substantial long term but limited short-term potential benefit.

A great example from the medical literature involves the analysis of longitudinal studies. A longitudinal study is one where you monitor a population for a long period of time. Longitudinal studies are time consuming, expensive and extremely important in medical research. An amazing new technology will often have to clear many testing hurdles in a research environment before proving itself so well that large amounts of funds can be raised for a longitudinal trial. If a researcher has an analysis on methods for evaluating the results of longitudinal trials, then the analysis could be a major scientific contribution. However, longitudinal trials are very time consuming and expensive, thus they are relatively infrequent. The total number of longitudinal trials that may cite a new analysis on how to evaluate the studies may be somewhat small - especially in the 2 years following publication of the new method for analyzing the results. However, the long term potential of such an analysis may be quite huge.

Studies with much long-term potential for a solid scientific contribution may get ignored because of its lack of short term potential (and associated ability to help the journal with its impact factor). This phenomenon makes the scientific publication process unusually short-sighted.

Jacob Levman