Development of a diagnostic framework and its application to open access journal publishing in Korea

To enhance open access (OA) journal publishing environments, this study proposed and applied a diagnostic framework for OA journal publishing. The framework includes three dimensions: OA policy establishment and disclosure, OA sustainability and journal openness quality. By applying this framework to nine OA journals in the field of science and technology in Korea, challenges faced by OA publishing were uncovered. For the dimension of OA policy establishment and disclosure, it was found that participants had lower perceptions of the importance of OA‐related financial disclosure and archiving policies compared with copyright policies. The dimension of OA sustainability was recognized as critical among participants. Finance was recognized as the most critical component of OA sustainability. Financial sources were mostly internal funding and article processing charges (APC), and participants perceived the current financial status of their journal positively. However, they were concerned about long‐term financial security. For the journal openness quality dimension, the participants were hesitant in disclosing reviews and reviewers, which can facilitate the prevention of fake peer review of predatory journals. Based on the findings, the article discusses how the challenging issues identified could be addressed. The results of this study may provide a guide for OA policy and OA system development.


INTRODUCTION
. Over the past 20 years, OA publishing has steadily increased. Bjork and Korkeamaki's (2020) study on OA publishing market shares in 18 Scopus-indexed disciplines reported that 18.4% of journals were OA journals overall and that figure varied widely by academic field (from 27.2% in agriculture and biological sciences to 7.0% in business).
Despite the increased number of OA journals and consensus on the OA movement, the level of compliance with OA components vary (SPARC, 2022). Brainard (2019) stated that it is not easy to move to OA due to concerns about financial loss, particularly for small scholarly societies relying on journal subscription revenues. In addition to the transition to OA journals, the sustainability of OA journals is difficult. Matthias et al. (2019) reported that 152 active OA journals were converted back to subscription journals between 2005 and 2019, and 95 (62%) of them had reverse flipped (from subscription to OA to subscription journals again). They argued that reverse flipping presents a unique perspective on OA journals that requires further attention. Laakso et al. (2021) analysed OA journals that disappeared from the web between 2000 and 2019 across all disciplines and geographical regions in their follow-up study. They reported that 174 OA journals were no longer available and urged efforts for preventing the loss of scholarly knowledge. It is therefore important to prepare publication environments so that OA journal publication is possible and to sustain the environment in which they operate.
This research project was commissioned by the Korea Institute of Science and Technology Information (KISTI) 1 with the aim of proposing a practical framework to determine OA journal publishing practices in science and technology fields that pursue OA principles. The developed framework was applied to Korean OA journal publishing practices from which challenges and recommendations were identified to help improve OA journal publishing environments in Korea. It is a characteristic of Korean scholarly communication that scholarly societies are the main publishers of scholarly journals. Therefore, a limitation was that the developed framework would be appropriate for OA journals published by scholarly societies; however, this framework was deemed valuable for diagnosing challenges and strengths of OA journal publishing practices, identifying the areas that require efforts for enhancement, and providing a roadmap to progress in improving OA journal publishing practices.

CONCEPTUAL FRAMEWORK
To construct a framework for diagnostic assessment of OA journal publication environments and practices, this study reviewed two types of studies: OA initiatives (COPE, 2019;Open ROADS Community, 2020;Plan S, 2022;SPARC, 2022) and assessment frameworks for digital transformation (Digital Maturity Model: Gill & VanBoskirk, 2016;Van Boskirk, 2017;Anderson & Ellerby, 2018: SMB Digital Maturity Index: Cisco, 2019, 2020Open Digital Maturity Model: 2020). 2 By synthesizing components extracted from previous studies, this study constructed an initial conceptual framework including 32 questions across three dimensions (OA policy establishment and disclosure, OA sustainability, and journal openness quality). Table 1 presents how these 32 questions are hierarchically structured in terms of 3 dimensions, 12 items and 25 sub-items.
The first dimension, OA policy establishment and disclosure, emphasizes the presence of specific policies dealing with the establishment and disclosure of OA policy and includes two items: OA policy establishment and OA policy disclosure and registration. The first item, OA policy establishment, focuses on documenting OA policies in copyright, archiving and finances related to OA publication. The Principles of Transparency and Best Practice in Scholarly Publishing (COPE, 2019) states that copyright policy must be stated explicitly and preservation policy must be made clearly specified. The sub-item of OA finance disclosure policy is clearly stated in the Principles of Transparency and Best Practice in Scholarly Publishing (COPE, 2019) and the Guidance on the Implementation of Plan S (2022).
The second dimension relates to the sustainability of OA and contains four items: finance, people, organization and culture, and collaboration. The finance item relates to two aspects: financial sustainability and financial disclosure for OA. As the Open ROADS Community (2020) points out, financial sustainability for OA is a crucial element that sustains OA journal publication.
Financial disclosure for OA is significant, creating a clear distinction from predatory journals, as the Guidance on the Implementation of Plan S (2022) and Best Practice in Scholarly Publishing (COPE, 2019) suggest. The second item relates to people, particularly training and education of staff regarding OA principles and practices (Open ROADS Community, 2020). Similarly, the organization and culture item is concerned with awareness of OA

Key points
• A practical framework to diagnose open access (OA) journal publishing environments was proposed, comprising three dimensions: OA policy establishment and disclosure, OA sustainability and journal openness quality.
• In terms of OA policy establishment and disclosure, while copyright policy is well-recognized, perceptions on OArelated finance disclosure policies are low.
• Although finance is regarded as an important item for OA sustainability, the main financial sources are internal funding and article processing charges.
• There is a reluctance to disclose reviews and reviewers. 1 KISTI is the Korean National Research Institute in Science and Technology Information. KISTI intends to use this developed framework to diagnose the OA publishing environment and needs of scholarly societies publishing OA journals, as well as to identify areas that need support and resources to implement OA publications. 2 The preliminary version of the framework was presented at a conference (Chung et al., 2022). principles and practices among members of the scholarly community (Open ROADS Community, 2020). The collaboration item contains two sub-items: collaboration for OA publication and collaboration for OA dissemination (Berghaus & Back, 2016).
The third dimension mainly relates to the extent of openness of a journal, which is defined by six items: submission and review, author rights, reader rights, findability, accessibility and monitoring.
The first item, submission and review, concerns the submission and review of article manuscripts and comprises three sub-items: reviewer disclosure, review disclosure and author cost disclosure. Of these three sub-items, reviewer disclosure and review disclosure are stated to be OA dimensions that broaden and deepen OA through open peer review (Bosman, Frantsvåg, et al., 2021). In addition, author cost disclosure is suggested in both the Principles of Transparency and Best Practice in Scholarly Publishing (COPE, 2019) and the Guidance on the Implementation of Plan S (2022). Author rights comprises the two sub-items of author copyright and author posting rights. As recommended in SPARC (2022) and the Guidance on the Implementation of Plan S (2022), authors have copyright with no restriction and the right to post in various places for the dissemination of the article. Readers' rights comprises two sub-items, as pointed out by SPARC (2022), which are free readership rights and generous reuse of articles. The findability item comprises four subitems: postings, dissemination optimization for search engines, unique identifier for authors and articles, and machine readable metadata for articles. The sub-item, postings, is related to the availability of trusted repositories immediately upon publication (SPARC, 2022). Search engine optimization for dissemination is concerned with whether articles are optimized for search engines in terms of improving the findability of an article on the web. The unique identifier for authors and articles sub-item is concerned with providing a unique identifier for authors via ORCID and ISNI, for example, and for articles using DOI and URI. Unique identifiers are highlighted by SPARC (2022). Machine-readable metadata for articles is considered a significant element for improving the findability of articles (SPARC, 2022). The accessibility item is related to being able to access the full text of an article and comprises two subitems: machine readability and mobile-responsive design. Machine readability refers to accessing article full text via API (SPARC, 2022).
Mobile-responsive design refers to whether the full text of an article can be presented on a device in a neutral manner. The monitoring sub-item is concerned with monitoring article use (European Data Portal, 2020).

RESEARCH DESIGN
This study comprised two parts: (1) developing a practically applicable framework for diagnosing OA publishing environments, and (2) testing the developed framework by diagnosing OA journal publishing practices in the science and technology fields in Korea.
For both parts of the study, data was collected through in-depth interviews using the following process. First, chief editors of OA journals in the fields of science and technology were selected to take part in the in-depth interviews.
Of the journals using the Article Contribution Management System (ACOMS)-a manuscript management system operated by KISTI-53 scholarly journals in science and technology were identified. Invitations were sent to 28 chief editors, of whom seven agreed to participate in in-depth interviews. The interviews were conducted in July and August 2022 (first round), and the initial conceptual framework was revised following the interview results. Then, to review the completeness of the revised framework, chief editors of two selected journals were contacted for second-round interviews in October 2022. Table 2 presents the nine interview participants from both rounds of interviews.
The aim of the interviews was to obtain data on (1) participants' comments on the framework and (2) their OA journal publishing practices. This was achieved by answering the questions of the framework. For the first round of interviews (n = 7), the initial conceptual framework was sent to the interviewees in advance, and interviewees were asked to identify their journal practices using the initial conceptual framework before the interview session took place. During the interview, participants were asked to give feedback on the appropriateness and clarity of the dimensions, items, sub-items and questions, and to describe their journals' practices in accordance with the questions of the framework. Because similar comments were observed by the research team repeatedly without additional comments (Glaser & Strauss, 1967), further recruiting was not attempted. For the second round of interviews (n = 2), the revised framework was sent to the participants in advance, and they were asked to identify their OA journal publishing practices. During the interview sessions, the clarity of the revised framework was assessed and additional explanations on their OA journal publishing practices were gathered. All interviews were conducted remotely via Zoom and each interview lasted approximately 2 h. Monetary compensation was provided for interviewees' time and efforts. The interview sessions were recorded with participants' agreement and transcribed by the research team. Then, data were analysed for both parts of this study. For the first part, comments of first-round interview participants on the initial conceptual framework were analysed, and the initial framework was revised and refined. Then, after the second-round interviews, the revised framework was finalized. For the second part of this study, OA publishing environments of nine journals were assessed with the finalized framework. The assessment was performed by the research team based on journals' websites, policy documents, copyright documents, author guideline documents and data collected during the in-depth interviews (Table A1).

Developing a diagnostic framework for OA journal publishing practices
To develop a practically applicable framework, comments from indepth interviews were reflected on the initial conceptual framework, which was constructed based on previous studies on OA initiatives and assessment frameworks for digital transformation. The initial framework comprised 3 dimensions, 12 items, 25 sub-items and 32 questions, while the final framework comprised 3 dimensions, 12 items, 23 sub-items and 28 questions (see Table 3). Three types of revisions were made. First, two sub-items and four questions were eliminated due to redundancy and a lack of direct relevance to OA publishing. General questions about written policies on OA publishing were deemed redundant due to written policies about copyright, archiving, and OA finance disclosure, and disclosure of article processing charge (APC) waiver policies was also deemed to be part Among the seven journals having one written policy or more, one participant said they did not publicize their policies on their websites, while the other journals publicized all or part of their policies on their websites.
…To create a written OA policy, we need a template or guidelines for it… it is hard to just create… It can be quite ambiguous, and the situation can be different, some scholarly societies are currently conducting OA, but some are just planning. (F Journal in Artificial Intelligence, Editor-in-Chief,

years' experience)
Although two participants answered that they did not have written policies on copyright, it was stated that all journals registered their copyright policies in the Korea Journal Copyright Information (KJCI) system run by the National Library of Korea. This is mainly because the items provided on KISTI's ACOMS journal management system are linked with KJCI. However, none of them registered their copyright policy in Sherpa/Romeo. All nine journal editors answered that they were not aware that there is a place that registers an OA copyright policy in other countries. Participants generally did not understand how to Author cost disclosure Q14. Is the APC information clearly disclosed on the journal homepage or announced to the authors?
Q15. Are OA publication fees waived or discounted for independent researchers and/or students?
Author rights Copyright Q16. What is the type of author copyright?
(1) The copyright is retained by authors w/o restrictions (2) The copyright is co-owned by both authors and the publisher, or the authors own the reuse right with restrictions (3) The copyright is owned by the publisher and only fair use is allowed Posting rights Q17. How does the journal allow the author(s) to retain posting rights?
(1) Authors can choose to post any version of the manuscript to any repository immediately upon publication (2) Authors are allowed to post a specified version of the manuscript to any repository immediately upon publication (3) Authors are allowed to post a specified version of the manuscript to a specified repository (4) Authors are not allowed to post any version of the manuscript to any repository or websites Reader and re-use rights Reader rights Q18. What are the readers' rights in accessing the full text of article?
(1) Readers can freely access the full text of the article from repositories and websites, immediately upon publication (2) Some repositories or websites provide the full text of the article with a fee or with embargo period ( Unique identifier for author and article Q22. Does the journal adopt an article unique identification system such as DOI and URN?
Q23. Does the journal adopt an author unique identification system such as ORCID or ISNI?
Machine-readable metadata for article Q24. Does the journal provide the metadata for each article in a machine-readable format (e.g., JSON, BibTeX, RIS, etc.)?
Full-text accessibility Machine readability Q25. Does the journal provide the full article in an XML format? Q26. Does the journal provide the full article in a PDF or html format?
Mobile-responsive design Q27. Are journal articles provided with mobile responsive design?
Monitoring Use monitoring Q28. Does the journal collect and disclose usage statistics for each article? register the OA policy of a journal in Sherpa/Romeo, which may enhance article usages.
(I didn't know about the Sherpa/Romeo) we need to have information for registering in Sherpa/Romeo, KJCI, etc.
(A Journal in Biology, Editor-in-Chief, 2.5 years' experience)

Open access sustainability
The dimension of OA sustainability was measured by finance, people, organization and culture, and collaboration. Regarding financial sustainability, the main financial sources of OA publications were described, with six journals being mainly funded from society budgets, and three from APC. It was noticeable that external funding was not reported among nine journals. Financial sustainability was described as very secure for six journals, and somewhat secure for three journals. It seems the types of financial sources did not influence the perceptions of financial sustainability. However, for some journals that do not charge an APC for authors (Diamond model of OA), doubts about its continued financial position were expressed by participants.
…The financial burden may increase depending on the circumstances of individual scholarly societies. In order to promote OA publication, appropriate national financial support is required for scholarly journals in underrepresented fields… (F Journal in Artificial Intelligence, Editor-in-Chief, 2.5 years' experience) With regard to the training and education opportunities for staff at OA publications, two journal editors reported that their internal staff never participated in training or education and seven journals reported that their staff participate one or two times per year. It was found that they do not have their own formal or systematic OA publication-related training or education programmes: …Maybe, Korean Council of Science Editor 4 s, they do workshops, we tend to attend those workshops once or twice a year… (B Journal in Information Technology, Editor-in-Chief, 2 years' experience) …We do have a mother scholarly society and several sub scholarly societies. We have seminar for our members and do have a little bit of mentioning OA practices of our societies, but not much of it… (G Journal in Interdisciplinary, Editorin-Chief, 7 years' experience) With regard to awareness and promotion of OA journals within scholarly societies, it was reported by five journal editors that most society members were aware that their journals were OA, while one participant reported that about half of the society members were aware that their journals were OA.
…Our society has been in OA for quite some time. So there is no need to inform the members of our society. Because everyone knows… (C Journal in Chemistry, Editor-in-Chief, 5 years' experience) However, two participants reported that only a few members were aware that their journals were OA and one said that none of their members were aware. In addition, some negative perceptions of OA was found to be a hindrance to the publicity of OA.
When we say OA, I think the perception is a predatory journal. That has been ingrained in people's minds more recently… Is it good to announce this (OA) or… Is it better to quietly and systematically supplement it? I guess I didn't really feel the need to do this (publicity). (F Journal in Artificial Intelligence, Editor-in-Chief, 2.5 years' experience) With regard to collaborative efforts in publishing OA journals, five participants reported that there are collaborative efforts for OA publishing and dissemination of OA journals. It is difficult to see that either intra-or inter-disciplinary collaboration is a universal activity in all fields.
The question was difficult. And in the case of editorial communities by field, it is a bit unclear whether they have internal and/or external communication channels and practices… I

Journal openness quality
The dimension of journal openness included reviewer and review disclosure, author cost disclosure, author's copyright and posting rights, user and reuse rights, findability, accessibility and monitoring.
With regard to reviewers and review disclosure, questions arose about the effectiveness of disclosing reviews and reviewers. It was found that only one journal releases a list of reviewers, and only two journals disclose reviews and opinions on journal articles.

years' experience)
It is doubtful whether disclosing reviews would be meaningful in Korean society… (I Journal in Biology, Editor-in-Chief, 20 years' experience) APC information was clearly disclosed in all nine journals, but only four journals were found to offer APC waivers or discounts for independent researchers or students.
It's a bit vague, but our journal does not charge an APC. We With regard to author rights, it was reported that two journals allow authors to retain copyright without restrictions, six journals have co-owned copyright between authors and the publisher (hence, their scholarly societies) or the authors own the reuse right with some restrictions. One journal reported that the publisher holds copyright. Regarding posting rights, it was reported that seven journals allow authors to post any version of their manuscript to any repository immediately upon publication. One journal allows authors to post only a specified version of their manuscript to any repository immediately upon publication, and one journal did not allow authors to post any version anywhere.
With regard to readers' rights, the level of availability of fulltext articles was explored. Two participants reported that readers can freely access the full text of articles from repositories and websites immediately upon publication, and seven journals reported that some repositories and websites provide the full text of articles with either a fee or embargo periods. Reuse rights were reported as follows. Two journals give liberal CC BY licences, six journals give CC BY with some conditions, and one journal gives reuse rights at the level of fair use only. The degree of findability was examined through posting, search engine optimization, use of unique identifiers, and machine-readable metadata. Eight of the nine journals were found to post articles to as many trusted repositories as possible.
If you are using KISTI's journal management system, ACOMS, your papers will be automatically deposited in ScienceOn and KoreaScience, and such automatic deposit system seems necessary. (D Journal in Earth Science, Editor-in-Chief, 2.5 years' experience) All nine journals were found to provide search engine optimization. It was also found that participants understood search engine optimization to be the extent to which metadata information can be accurately and sufficiently inputted when depositing their articles.
Even if we don't do this in particular, I think we put meta information on manuscripts properly so that they can be searched. (F Journal in Artificial Intelligence, Editor-in-Chief, 2.5 years' experience) With regard to the use of unique identifier systems, all nine journals use DOI as an article unique identifier, but only three journals use an author identifier system (ORCID or ISNI). All nine journals provide metadata for each article in machine-readable format.
These concepts (metadata, machine-readable, etc.) were difficult. It was unfamiliar to me, so I didn't know what it meant. (G Journal in Interdisciplinary, Editor-in-Chief, 7 years' experience) To understand the degree of full-text accessibility, it was asked whether the journals provide full text in XML and PDF formats. This was confirmed to be the case for all nine journals for PDF format but only three journals provide their articles in XML format. It was found that participants had little understanding of the aggregator service or reuse of more aggressive OA articles as the full text of the article is provided in a similar structure to XML.
I'm curious, I am not sure that doing it in XML format is more advanced and has more advantages. (G Journal in Interdisciplinary, Editor-in-Chief, 7 years' experience) I'm just saying I don't know. What does machine readability mean? (C Journal in Chemistry, Editor-in-Chief, 5 years' experience) With regard to monitoring status, all nine journals collect and disclose usage statistics for each article.

DISCUSSION AND CONCLUSION
This study proposed a practical framework to identify OA journal publishing practices in science and technology fields in Korea. Indepth interviews were conducted initially to support the development of the framework, which was then tested with nine Korean OA journals' publishing practices in the science and technology field. Issues and challenges in publishing OA journals in Korea were examined. The following discussion describes how the identified challenging issues can be improved.
In the dimension of OA policy establishment and disclosure, the existence of written policy documents of copyright, finance disclosure and archiving establishment was examined. Among these policies, a lack of awareness of OA-related financial disclosure policy was revealed. In a scoping review of predatory journal characteristics, Cobey et al. (2018) demonstrated that no specifications on APC, hidden APC, and hidden information on APC are characteristics of predatory journals. Some participants of the current study were concerned about negative perceptions related to OA journals due to confusion with predatory journals. OArelated financial disclosure is critical because an important demarcation between these predatory journals and OA journals is made in terms of whether or not they aim for unusual profit through OA publication (Beall, 2015). Some awareness of the archiving policy of OA journals was identified, although cases were also identified that were not aware of the need for an archiving policy at all. The importance of archiving efforts to prevent the loss of scholarly works were raised by Laakso et al. (2021) who examined the disappearance of OA journals. In addition, it was found to be uncommon to disclose all established policies on journal homepages or to register them on international OA policy disclosure sites (Sherpa/Romeo, etc.). Journal policy disclosure should be encouraged to enhance the transparency of journal operations as well as global visibility.
Sustainability is a vital factor in OA journal environments.
Recent studies reported the issue of disappearing OA journals on the web (Laakso et al., 2021) and reverse flips to the subscription-based journal model (Matthias et al., 2019).
According to a survey with diamond OA journals (Bosman, Frantsvåg, et al., 2021), approximately 20% of diamond OA journals either considered or decided to move away from the current OA publishing mode, mainly due to feasibility and economic viability. The current study examined four items (financial, people, education/training and collaboration), which were identified in the sustainability dimension. Participants commonly considered journal finance to be a very critical issue for journal sustainability and reported that their journals were operated by stable OA publishing finances mostly with internal funding or APC. However, even if the current financial situation for OA publication was stable, they were also concerned about the security of future financial sources. For the long-term financial security of OA publishing, it is worth considering more stable financial models.
Various funding models exist, such as collaboration with libraries, government agencies and national funding (Lee & Joung, 2020), and preferred funding models may be chosen by their OA  Huh et al. (2015) reported that finance is one of the challenges that editors in the science and technology field in Korea face when publishing open access journals. As national funding and government agencies are a major financial source for diamond OA journals (Bosman, Frantsvåg, et al., 2021), it is desirable to extend their financial support to OA publishing.
With regard to the journal openness quality dimension, the most characteristic result in the submission and review item is that it is difficult to accept the disclosure of reviews and reviewers. Participants raised questions about the need for disclosure of reviews and reviewers, such as open peer reviews, and pointed out that there are cultural obstacles to implementation, especially in the Korean scholarly community. There was little recognition of open review and reviewers, although open peer review might be a tool that prevents fake peer review among predatory journals (Al-Khatib, 2016;Dalton et al., 2020). In terms of copyright ownership and readers' rights, fairly conservative practices were identified. Copyright was mostly found to be jointly owned by authors and publishers (scholarly societies). In addition, in terms of reader and reading rights, some limited access (membership registration, embargo) or restricted CC licences were shown. This result can be compared to a previous study that reported that approximately half of OA journals registered in DOAJ were found to be copyright without restrictions (Bosman, Frantsvåg, et al., 2021). These results are probably derived from the practice that most Korean OA journals are published at scholarly societies, and authors entrust the distribution of their articles to the societies as members of the societies. In terms of findability, unique identification numbers for authors (such as ORCiD or ISNI) appeared to be unfamiliar to participants. In terms of improving findability, it is desirable to support the XML conversion process in connection with the contribution/publishing system for XML format publication. In addition, it is necessary to expand awareness of the need for author ID submission.
In conclusion, for improving OA journal publishing environments, a diagnostic framework was proposed and applied with three dimensions: OA policy establishment and disclosure, OA sustainability, and journal openness quality. For the OA policy establishment and disclosure dimension, the participants' perception of the importance of financial disclosure and archiving policies was lower than that of copyright policies. The OA sustainability dimension was considered critical, with finance being the most critical component. Participants reported using internal funding and APCs as their primary financial sources, and they viewed their journal's current financial status positively. However, they expressed concern about its long-term financial security. The participants were also hesitant about disclosing reviews and reviewers in the journal openness quality dimension, which could prevent predatory journals. . Are articles posted to as many trusted repositories as possible? More than two More than two More than two None More than two More than two More than two More than two More than two