Copyright© International Federation of Scholarly Associations of Management, All Rights Reserved.
The first IFSAM conversation letter (ICL) is devoted to the insights of our first webinar on Jan 29 2021 on management research
This first conversation is between one article kindly written by Andrew Jack and another by the co-chairs of the webinar, Patricia Genoe McLaren and Xavier Castañer. It was released prior to our next webinar in this series on February 24: https://www.ifsam.org/
An ICL will follow each webinar with the core insights of the session.
IFSAM Conversation Letter 1 – February 15 2021
On the First Webinar, January 29, 2021
of IFSAM Webinar Series on Management Research Evaluation
Co-chairs:
Xavier Castañer, President Elect, International Federation of Scholarly Associations of Management (IFSAM)
Patricia Genoe McLaren, President, Administrative Sciences Association of Canada
Panelists:
Herman Aguinis, Professor at George Washington U; President Elect, Academy of Management
Jerry Davis, Professor at U. of Michigan, former ASQ EiC; Founding Member, Responsible Research in Business and Management Network
Gerardine Doyle, Associate Director, UCD Business School; President, European Institute for Advanced Studies in Management
Henrich R. Greve, Editor in Chief, Administrative Science Quarterly, 2016-2020
Andrew Jack, Global Education Editor, Financial Times
Silke Machold, Dean of Research, University of Wolverhampton, UK
Zhen Tang, Associate Dean of Research, Business School, Hohai University, China
Alessandro Zattoni, Director of Management Department, Luiss University; President, International Corporate Governance Society; former co-editor of CGIR
In December 2020, the Financial Times (FT) launched a survey to solicit feedback on the list of 50 journals it uses to determine the research rank of business schools (known as the FT50 journal list) and more broadly asking for input on different ways of measuring the quality and impact of management research from both individual academics and deans in management schools. This triggered an asymmetric response across the world, in particular, as regards to scholarly associations of management as well as the editors of their journals.
Academics around the world subsequently received emails from learned societies’ presidents and journal editors asking them to complete the survey, with some encouraging them to identify their journals as critical to the FT50 list.
The IFSAM executive committee was concerned with aspects of the survey, particularly with the section on identifying the valuable journals, and felt strongly that this procedure was asymmetrically reaching the field. The launching of the survey, and the various responses it generated, gave fresh impetus to the thorny issue of how best to judge the quality of management research and publications.
As part of its response to addressing the issue, IFSAM decided to organize a series of webinars dedicated to examining various aspect of the evaluation of management research, the first of which was held on January 29, 2021. Andrew Jack, the Global Education Editor for the Financial Times, was joined by a select group of presidents of scholarly associations of management, management journal editors, and research-focused academic administrators.
In this letter we provide our summary of the key insights from this first webinar. Designed to open the conversation, it may also serve as a platform on which to build and channel the debate. The months ahead will see IFSAM hosting a series of additional webinars involving other key stakeholders with particular involvement in management research evaluation including university presidents and deans, publishing houses, government officials and management practitioners.
Concern about the global diffusion of a tenure system based solely on journal publications, and, even more narrowly, on an ‘elite’ set of journals, is growing outside of the mainstream US management schools. When only a select few journals are considered ‘top tier’ and the process for selecting those journals is dominated by editors and scholars working within the mainstream schools, a whole raft of scholars are excluded and their scholarly contributions ignored. This also reduces diversity in research and publication genres or types, in particular the disregarding of books, which though allowing for a more holistic and detailed treatment of an issue, are almost completely ignored (or considered worthless) in most tenure processes.
As a result, management knowledge has become homogenized as scholars carefully craft their work to conform to a particular format (journal article) from the prestigious journals (which promote again a specific form) so as to be acceptable to their reviewers.
To the extent that this faculty and attendant research model is adopted across the world, as we have seen in the last 20 years in Europe and in Asia (notably in China), as well as in Latin America, the focus on this particular research genre and format entails a dangerous reduction in knowledge inquiry generation and diversity worldwide. The individual and organizational incentive to conduct valuable research that doesn’t fit into the Western scientific paradigm vanishes if it cannot be published in the journals that are considered ‘worthy’ of publishing in for hiring (for PhD students) and for retention and promotion. Scholars from non-Western countries, scholars focusing on gender, race, labour, sustainability, power, and critical management studies are to a large extent ignored and excluded. (Case in point being the recent University of Leicester decision to ‘disinvest from research and scholarship in Critical Management Studies and Political Economy.’)
Many panelists and attendees acknowledged during the webinar that the FT50 journal list is only one symptom of a much more insidious problem. The FT is not responsible for creating the current culture and incentive system in management schools that places overwhelming value on articles published in a select set of journals. The FT50 journal list was created to support the FT ranking of business schools, which was designed to help professionals select MBA programs. It is, of course, not as simple as saying that the FT50 journal list itself has had no impact. As the journal ranks MBA programs, and as MBA programs are a source of significant financial resources for business schools, pressure has been increased on faculty members to publish in the FT50 journals, which business schools can then use in recruitment materials as a strength of their school.
Acknowledging the FT50 journal as a symptom, rather than a cause, led to the discussion of the reasons, consequences, and potential fixes for the over-valuing of published research in “top tier” journals. As US business schools rooted themselves firmly in “the management as a science paradigm” in the 1950s, management theory, research, and scholars became constrained by the scientific production of knowledge – focused firmly on positivist, quantitative research and the idea that management theory was constantly evolving into something “better.” The ideology of science became so deeply embedded in the way that management scholars think, it became the obvious solution for how management research should be evaluated i.e. through ranking and counting.
Some panelists advocated for a scalable, global research evaluation system, one that evaluates in a more nuanced manner than journal ranking lists. Herman Aguinis proposed design thinking as a way to develop a better system, while Andrew Jack and Jerry Davis talked about adding an assessment of impact for journals that are included on the FT50 list. The FT survey asked respondents for ideas on how to measure impact on teaching and society, how measurement might be accomplished at the individual article level, for alternative metrics, and additional forms of academic output that should be considered.
The session left us with far more questions than answers. Why do we need a measurement or evaluation system at all? Why can’t individual business schools evaluate their faculty members’ work by reading it? Why are all of the measures of quality based at the journal level, when scholarship takes many more engaging and interactive forms than just academic articles? Why limit the number of journals we view as valuable to 50? Why not an FT100 or an FT200? Why rely on a global ranking, rather than local or regional rankings? Why are we closing the tent and actively excluding many scholars who are working in a whole range of different ways to strengthen our understanding of management beyond just the publication of academic articles in particular outlets?
These are important questions that deserve thoughtful consideration and determined action. IFSAM is committed to continuing this conversation and supporting resulting changes. Join us on February 24 for our 2nd webinar on management research evaluation. We will be gathering the views of a group of university presidents and management school deans from around the world.
Xavier Castañer, President Elect, International Federation of Scholarly Associations of Management (IFSAM) – xavier.castaner@unil.ch
Patricia Genoe McLaren, President, Administrative Sciences Association of Canada – pmclaren@wlu.ca
When the Financial Times launched a questionnaire late last year on how it assesses academic research, it unleashed a fierce reaction.
Editors and the professional bodies behind some publications got in touch to plead the case for inclusion of their particular journals in the FT50 list, and circulated petitions via email, online and via social media to urge their contributors and supporters to lobby on their behalf.
Professors wrote in to express concerns about manipulation. One blog argued: “Journal mafias are rallying to make their journals remain and/or join the FT journal ranking. They’ve been sending unashamed e-mail messages to their affiliates asking to favourably answer an on-line survey.”
IFSAM itself announced a webinar with a statement saying it was “concerned” about the process, which it characterised as a “vote” on scientific and practitioner journals. It criticised an “apparently democratic procedure which is asymmetrically reaching the field”.
I was happy to engage in that discussion, as part of an extensive broader range of consultations that we have launched with business schools and those who interact with them – from students and recruiters, to external individuals and organisations that speak at, train with or seek the best thought leadership from academia.
The FT recognises and takes very seriously its relationship with business schools, and the rigour with which it attempts to assess them through both quantifiable measures such as our rankings and qualitatively through extensive wider reporting. We are mindful that the FT50 is widely used and has an influence in publishing, recruitment, promotion and funding.
Academic output – as currently captured by the FT50 – is an important part of our rankings, as an indicator of research and its translation into teaching. As society and academia evolves, specialisms and interests change and reputations shift, it seems reasonable to periodically re-examine the list – especially since the last just revisions took place six years ago.
But our consultation is not simply limited to scrutinising the composition of the journals currently listed in the FT50. Those who have completed the questionnaire will see that it ranges far more widely. It includes a request for assessments of the journals included, whether some should be removed and others added, whether the list should be made longer and what themes should be a focus.
More fundamentally, it asks whether using journals at all is the best form of assessment of output; whether individual articles and citations should be taken into account alongside or instead; how to balance credit between multiple authors; and how to attribute between the institution where the authors were based at the time of publication versus where they are currently employed.
It seeks to explore ways to capture other, broader forms of academic work such as books, chapters and case studies; and to canvass views not just on the significance of academic rigour but also the relevance of topics covered and the wider resonance that insights have with practitioners and society beyond academia.
In other words, our questionnaire is part of a much broader consultation about our rankings, assessments and coverage of business education to make them as useful as possible, taking into account factors such as societal impact and reflecting the thinking of important initiatives including the Responsible Research in Business & Management network and the UN PRME initiative, as well as DORA and Open Science.
That is reflected in editorial activities we have launched in recent months to capture best practices. There is little consensus around what and how to measure output. Our contributions so far include an FT series on responsible business education with a mixture of self-assessments and judgements by third parties. We plan further efforts to explore these areas more deeply.
As to the process, our questionnaire is not designed as a simplistic vote at the end of which the loudest or largest number of voices will “win” admission for journals to a new FT50 list. It distinguishes between individual opinions and institutional responses from business school leaders.
But surely it is reasonable to encourage a broad, transparent consultation and debate rather than simply to discuss in private with a select few? Analysis of the responses of the questionnaire from many thousands of academics is still underway. We have, and will continue, to explore different approaches including through articles and invitations to groups to share alternative approaches. Any changes will be carefully analysed and modelled. We are mindful that reforms could bring discomfort and disruption.
There are many critiques of the current approach – far more than there are well defined and better alternatives. We certainly will not introduce changes simply based on the view of the majority. Whatever we ultimately decide needs to take into account multiple different perspectives from existing and new users, to identify approaches that are consistent and measurable, and are feasible to collect and analyse.
We welcome further input as we proceed: notably of tangible alternatives.
Do send ideas to respbus@ft.com
Andrew Jack, Global Education Editor, Financial Times – respbus@ft.com