The REF: Results of Pilots and Future Developments

I recently attended a one day programme organised by Kings College London on The REF: Results of Pilots and Future Developments. Supported by HEFCE, the day gave those involved, and not involved, with the pilot alike the opportunity to learn more about the planned direction of the REF. The day consisted of a number of presentations to all delegates in the morning including two speakers from HEFCE, with parallel sessions on various related topics running in the afternoon.

A large emphasis within the REF Pilot had been on citation analysis, with debate circulating as to the use of such analyses to replace the peer review process used previously within the RAE. What papers would be considered within the REF was also the subject of debate with the pilot considering three possible methods of identifying academic’s papers for citation analysis (as explained to the WRN at this month’s Gregynog Colloquium Repository Stream by Lyndsey Savage, Bangor University): all academic papers identified by author’s name; all academic papers by author’s institutions; selection of academic papers for authors.

From attending the London event, it seems that the REF will bear a resemblance to the previous RAE, with a selection of an author’s best papers being put forward for peer- review assessment, and the citation analysis for each paper being provided to inform the panel. What still remains unclear is which sources will be used to inform the citation analysis. Web of Science and SCOPUS were both used within the pilot but concern was expressed by a number of delegates within one of the parallel sessions I attended that their institution either subscribed to one service or the other, and not both. The use of Open Access sources such as repositories was not going to be used to gather citation analyses as far as I could tell from discussions.

The assessment of ‘impact’ of research will also form part of the REF, a factor that none of the discussions I'd heard previously had even considered. How institutions will construct, record and store such information along with the author data, the research ouput data and the citation data is a new challenge to be met.

The use of repositories for the management of such information to construct an REF return was touched upon. Two JISC projects presented in one of the afternoon sessions were R4R (Readiness for REF) and a project at the University of Reading, with both looking specifically at how a repository can be developed for this purpose.

The second afternoon session I attended ‘ICT implications for the REF,’ discussed research conducted on behalf of JISC that surveyed REF pilot institutions as to their use of ICT in preparing information for the pilot. The study found that the use of ICT was varied between each institution, and that institutions would have to develop much better processes for capturing research outputs, the crux point for all being trying to successfully link research output data to staff data.

A number of systems were being used within the institutions of the attendant delegates, with both research management systems and repositories in place. A second JISC study, ‘Repositories and Research Management Systems,’ found minimal integration between the two systems or their processes when in place within an institution. Some of the newly funded JISC projects, including Cardiff University’s I-WIRE project, which are looking as to how such systems can be integrated were presented on the day.

There were around 300 delegates in attendance on the day and it was interesting to see the variety of job titles; with a large majority either affiliated with the research office, the library or the repository. This highlighted to me the number of stakeholders involved in an institution with an RAE/ REF return and brought home further the need for not only integration between disparate systems but collaboration between disparate departments.