Til projektoversigt

When Scholarship is Measured: Consequences of the Danish Bibliometric Research Indicator

Postdoc-stipendium i Danmark | 10/06/2016

A large number of countries have implemented performance-based university research funding systems (PRFSs) that make the government’s distribution of resources to universities dependent on ex post evaluations of research output. This is also the case in Denmark where the so-called Bibliometric Research Indicator (BRI) took effect from the fiscal year 2010. Some evidence of this type of systems having an impact on the quantity and quality of publications produced does exist. However, only very limited empirical research has been undertaken to investigate the effects of PRFSs on the targeted researchers’ motivation, behaviour, and performance. This postdoc-project will contribute substantially to our knowledge of the consequences of PRFSs at the level of the individual researcher by addressing the research question: What are the consequences of the BRI for Danish researchers’ motivation, behaviour, and performance (qualitatively and quantitatively)? With inspiration from Motivation Crowding Theory (MCT), the project’s main hypothesis is that the consequences of the BRI for Danish researchers’ motivation, behaviour, and performance will be contingent on to which degree the individual researcher perceives the system as supportive or controlling. The analysis is based on quasi-experimental one-group pretest-posttest design.

Performance-Based University Research Funding Systems

Performance indicators and extrinsic incentives have become popular tools within almost all areas of the public sector in the attempt to increase the accountability, efficiency, and performance of public organisations (Pollitt 2013). Contemporary reforms within the university sector in many ways mirror these general reform trends (Christensen 2011; Rabovsky 2014). A large number of countries have  implemented performance-based university research funding systems (PRFSs) which make the government’s distribution of resources to universities dependent on ex post evaluations of research output (Hicks 2012). While these systems vary in their unit of analysis, method of measurement, frequency, and census period, the official aim is uniformly to foster research “excellence” (ibid.).

The government’s 2006 globalisation strategy had as a stated objective that Denmark in order to improve its competitiveness, economic growth, and development shall have “world-class” universities. A large majority in the Danish Parliament decided to introduce the so-called Bibliometric Research Indicator (BRI) which factor in the universities research productivity in the formula for allocation of new basic funds to the universities (Regeringen 2009; Regeringen 2006). The BRI took effect from the fiscal year 2010. Summarised, the BRI calculates the research productivity at Danish universities by assigning a point score to publications according to type (monograph, journal article etc.) and quality (“normal quality” (level 1) and “high quality” (level 2)).  Table 1 presents the point-system. BRI-points have accounted for 25% of new basic funding to the universities. The old and new funding formula is presented in table 2.

Table 1. BRI point-system

Type

Level 1

Level 2

Monographs

5

8

Chapters in anthologies

0.5

2

Journal articles

1

3

Doctoral theses

5

Patents

1

   Source: Sivertsen & Schneider 2012

Table 2. Funding formula for allocating new basic funds to Danish universities. Percent.

Indicator

Old system

New system

Earning from educational programs (STÅ)

50

45

External funding

40

20

BRI1

-

25

Number of PhD degrees

10

10

1 The BRI was introduced gradually into the funding formula with a weight of 10% in 2010 and 15% in 2011.

Source: Regeringen 2009

Within the research community (in Denmark and internationally), PRFSs has been met with numerous concerns about whether the whole enterprise is wrongheaded and damaging, whether such systems in fact can be used to make judgments about what constitutes good research, and whether it is even possible to use (financial) incentives to encourage this (Lewis 2014). Critics accuse PRFSs of impairing the intrinsic motivation of researchers by substituting “a taste for science” with “a taste for publications” (Osterloh 2010) and causing a range of undesirable behavioural reactions such as “slicing strategies” where researchers increase their publication counts by dividing their articles to a least publishable unit (Weingart 2005), the “prostitution of ideas” in order to getting published (Frey 2003), and a homogenisation of knowledge production which discourages creative, unorthodox, and idiosyncratic research (Gillies 2008; Hazelkorn 2009). However, only very limited empirical research has been undertaken to investigate the effects of PRFSs on the targeted researchers’ motivation, behaviour, and performance (Butler 2010). Some evidence of this type of systems having an impact at aggregated levels does exist, however. Most famously, Butler (2003) illustrated how the introduction of a performance-based funding formula based on publication counts in Australia made the quantity of journal publications go up but their quality or impact (measured by citations) go down. The postdoc-project applied for will contribute substantially to our knowledge of the consequences of PRFSs at the level of the individual researcher by addressing the research question:

What are the consequences of the Bibliometric Research Indicator for Danish researchers’ motivation, behaviour, and performance (qualitatively and quantitatively)?

High quality research is commonly accepted a key lever for a country’s competitiveness, growth and development. Thus, whether the BRI leads to quantitatively more and qualitatively better research at Danish universities or whether it, as critics argue, will be more harmful than helpful are likely to have an important impact on the future economic and societal development in Denmark. By investigating the consequences of the BRI for Danish researchers’ motivation, behavior, and performance, this project will advance our knowledge of the system’s intended and unintended effects. The results will be highly policy-relevant since they can be used with regard to future changes in the funding system for universities - not just in Denmark but also in other countries.

Theory

The project’s main theoretical focus is on Motivation Crowding Theory (MCT) (Frey & Jegen 2001; Frey 1997).Shortly descripted, MCT argue that external management interventions not only affect the targeted employees’ motivation and performance (directly) through a relative price effect (i.e. by imposing higher marginal cost on shirking or increasing the marginal benefits of performing),but they also affect it (indirectly) through a so-called crowding effect. If the external intervention is perceived as controlling (i.e. impairing self-determination and ignoring professional competences), it is expected to “crowd-out” intrinsic motivation. If it, on the other hand, is perceived as supportive (i.e. increasing self-determination and acknowledging professional competences), intrinsic motivation is expected to be “crowded-in”. Whereas a crowding-in effect further strengthens the external intervention’s positive effect on employees’ motivation and performance, a crowding-out effect may - depending on the relative strength of the price effect and the crowding-out effect - reduce, remove, or even reverse the system’s positive effect.

A number of empirical studies supports that external management interventions can crowd-out public employees’ intrinsic motivation. The existing studies, however, either only focus on the intervention’s effect on performance and attribute a decrease in performance to a crowding-out effect but do not actually include measures of (changes in) the employees’ motivation (e.g. Jacobsen & Andersen 2014; Andersen & Pallesen 2008). Otherwise, they only focus on changes in the employees’ intrinsic motivation without linking it to measures of performance (Stazyk 2013; Jacobsen, Hvitved & Andersen 2013; Bertelli 2006). This project’s data design makes it possible, however, to link changes in the researchers’ intrinsic motivation to changes in their self-reported research behaviour and their objective performance (measured by individual BRI-points and citations).

Research Design and Methods

Universities pose a critical case for MCT since researchers typically find their work interesting and intrinsically rewarding, consequently, making crowding-out effects more likely and acute (Weibel, Rost & Osterloh 2010). As Moynihan (2010) has highlighted, longitudinal approaches are valuable when one is interested in understanding crowding-out effects. The project, accordingly, applies a quasi-experimental one-group pretest-posttest design (Cook & Campbell 1979) in order to investigate the consequences of the BRI for Danish researchers’ motivation, behaviour and performance. This type of design helps to deal with the problems of endogeneity that often haunt cross-sectional studies of management interventions (Meier & O’Toole 2013; Hicklin 2010). Based on the two surveys a balanced panel is constructed. In addition, an objective publication and citation counts are collected for all researchers at the sampled departments for each year in the period 2009-2015 and merged with survey responses. The multiple data sources secure that common source bias is not a threat to the results (Favero & Bullock 2014). 

The Carlsberg Foundation’s Postdoctoral Fellowship in Denmark enables me to future specialise in performance management systems at universities. It also allows me to spend a semester in the US at Center for Organizational Research and Design (CORD), Arizona State University.

References

Andersen, L. B. & T. Pallesen (2008): ”Not Just for the Money? How Financial Incentives Affect the Number of Publications at Danish Research Institutions.” International Public Management Journal 11(1): 28-47.

Bertelli, A.M. (2006): “Motivation Crowding and the Federal Civil Servant: Evidence from the U.S. Internal Revenue Service”. International Journal of Public Management 9(1): 3-23.

Butler, L. (2003): “Explaining Australia’s increased share of ISI publications – the effects of a funding formula based on publication counts”. Research Policy 32(1): 143-155.

Butler, L. (2010): Impacts of Performance-Based Research Funding Systems: A review of the concerns and the evidence, in: OECD Workshop Proceedings: Performance-based funding systems for Public Research in Tertiary Education Institutions. OECD Publishing, Paris, pp. 127-165.

Christensen, T. (2011): “University governance reforms: potential problems of more autonomy?” Higher Education 62: 503-511.

Cook, T.D. & D.T. Campbell (1979): Quasi-Experimentation. Design & Analysis Issues for Field Settings. Boston: Houghton Mifflin.

Favero, N. & J. Bullock (2014): “How (Not) to Solve the Problem: An Evaluation of Scholarly Responses to Common Source Bias” Journal of Public Administration Research and Theory forthcoming: doi: 10.1093/jopart/muu020.

Frey, B. S. (1997): Not Just for the Money. An Economic Theory of Personal Motivation. Northampton: Edward Elgar Publishing Inc.

Frey, B.S. (2003): “Publishing as prostitution?  - Choosing between one’s own ideas and academic success”. Public Choice 116, 205-223.

Frey, B.S. & R. Jegen (2001): “Motivation Crowding Theory.” Journal of Economic Surveys 15(5): 589-610.

Gillies, D. (2008): How Should Research be Organised?  London: College Publications.

Hazelkorn, E. (2009): “Pros and Cons of Research Assessment: Lessons from Rankings”. Dublin: Reports Dublin Institute of Technology, Directorate of Research and Enterprise.

Hicklin, A. (2010): “Methods”. In Walker, R.M., G.A. Boyne & G.A. Brewer (eds) Public Management and Performance. Research Directions. Cambridge: Cambridge University Press, pp. 253-267.

Jacobsen, C.B., J. Hvitved & L.B. Andersen (2013): “Command and Motivation: How the Perception of External Interventions Relates to Intrinsic Motivation and Public Service Motivation”. Public Administration published online: 25 MAR 2013.

Jacobsen, C.B. & L.B. Andersen (2014): “Performance Management for Academic Researchers: How Publication Command Systems Affect Individual Behaviour”. Review of Public Personnel Administration 34(2), 84-107.

Lewis, J.M.  (2014): Academic Governance. Disciplines and Policy. Routledge Research in Higher Education. London: Routledge.

Meier, K.J. & L.J. O'Toole (2013): “Subjective Organizational Performance and Measurement Error: Common Source Bias and Spurious Relationships”. Journal of Public Administration Research and Theory 23(2): 429-456.

Moynihan, D. (2010): “A Workforce of Cynics. The Effects of Contemporary Reform on Public Service Motivation”. International Public Management Journal 13(1): 24-34.

Osterloh, M. (2010): “Governance by Numbers. Does It Really Work i-n Research?”. Analyse & Kritik 02/2010: 267-283.

Pollitt, C. (2013): “The Logics of Performance Management”. Evaluation 19(4): 346-363.

Rabovsky, T. (2014): “Using Data to Manage for Performance at Public Universities.” Public Administration Review 74(2): 260-272.

Regeringen (2006): AFTALE mellem regeringen (Venstre og Det Konservative Folkeparti), Socialdemokraterne, Dansk Folkeparti og Det Radikale Venstre om ny model for fordelingen af basismidler til universiteterne 30. juni 2009. http://ufm.dk/uddannelse-og-institutioner/videregaende-uddannelse/universiteter/okonomi/basismidler-efter-kvalitet/aftale-om-basismidler-efter-resultat.pdf (retrieved October 28, 2014).

Regeringen (2006): Fremgang, fornyelse og trykhed. Strategi for Danmark i den globale økonomi – de vigtigste initiativer. http://www.stm.dk/multimedia/Fremgang_fornyelse_og_tryghed.pdf (retrieved October 28, 2014).

Stazyk, E. C. (2013): “Crowding Out Public Service Motivation? Comparing Theoretical Expectations with Empirical Findings on the Influence of Performance-Related Pay”. Review of Public Personnel Administration 33(3): 252-274.

Weingart, P. (2005): “Impact of bibliometrics upon the science system: Inadvertent consequences?” Scientometrics 62(1): 117-131.