Who guarantees the quality of the quality assurance agencies? The exploration of the establishment and growth of the Asia-Pacific Quality Register (APQR)

Abstract

Purpose

After the “quantity era,” today higher education has entered into the “quality era” and as “the gate keepers of quality,” quality assurance agencies (QAAs) are playing more and more irreplaceable important roles and their social status are becoming more and more prominent. However, how to guarantee the quality of the QAAs? Who can review the QAAs? The purpose of this paper is based exploration of these questions.

Design/methodology/approach

Following the founding of the European Quality Assurance Register (EQAR) for Higher Education, the Asia Pacific Quality Register (APQR) became the second in the international quality assurance (QA) networks to implement QA register, in 2015 with initiative of Asia-Pacific Quality Network.

Findings

This paper first retrospects the history and process of APQR, and subsequently the implementation of APQR is described in detail from the two aspects of the criteria and the procedure, and at the end, the paper concludes with a summary of the three characteristics of this first formal implement of APQR: APQR is an international register open to all the QAAs; APQR emphasizes characteristics evaluation of diversity; and APQR highlights the combination of quantitative assessment and qualitative assessment.

Originality/value

Today on the international stage of QA, APQR has emerged as “the watchman of quality” in the Asia-Pacific region as counterpart of EQAR in Europe. How far away does such newly emerging form of guaranteeing the QAAs’ quality go forward, what is its future prospects and other concerning issues, are some of the question that need enthusiastic attention and contribution.

Citation

Zhang, J. and Patil, J. (2017), "Who guarantees the quality of the quality assurance agencies? The exploration of the establishment and growth of the Asia-Pacific Quality Register (APQR)", Higher Education Evaluation and Development, Vol. 11 No. 2, pp. 58-67. https://doi.org/10.1108/HEED-07-2017-0001

1. Preface

The keystone of a quality assurance (QA) in education is to define “who has to evaluate,” namely, what is the evaluation subject? The profiles of four QA agencies in China, Indonesia, Malaysia and Thailand (Zhang, 2010) show that some characteristics of four QA agencies in Asia, i.e. from the establishment time, the establishment of the National Accreditation Board for Higher Education (BAN-PT) in Indonesia is the longest, which was established in 1994 and the other three institutions were established after 2000; from the nature of institutional evaluation, the four institutions are essentially obligatory; from the perspective of independence, they are under the jurisdiction of the Ministry of Education; they cannot act independently, except BAN-PT in Indonesia; and in terms of ownership, all agencies are owned by the governments of the four countries, and work under the direct leadership of their governments. The so-called third-party QA organizations are “decorated” in the educational evaluation, and they have no real participation in education evaluation. Then, in the context of the QAAs being “both athletes and referees in a game,” who can guarantee the quality of higher education (HE) QA institutions?

In the field of QA of HE in the world, the “register” is a new and attractive creative activity. Following the footsteps of the European Quality Assurance Register (EQAR) for higher education, the Asia Pacific Quality Register (APQR) also made a bold attempt: having carried on a review process with a specific set of standards and criteria to review the vision, mission, objectives, evaluation and accreditation process and other 11 criteria of a quality assurance agency (QAA), in order to review and verify the QAA’s capability and qualification, as well as regulate the behavior of its QA. APQR guarantees and improves the legitimacy, effectiveness and openness of the QAAs in the Asia-Pacific region. It is convenient for the governments, HEIs and all the stakeholders to identify the credibility and professionalism of all kinds of countless QAAs, and thus will have a significant impact on the development of HE.

On the basis of the existing research results of the QAA mechanism of the international, national, HEIs, professional and the third-party QAAs, the initiative of this paper, by opening a new road for itself, focuses on the registration system of regional and international QAAs, aiming to further enrich the accreditation and registration theory of the QA of HE.

2. The development background of APQR

The development trend of international HE is the same. After the “quantity era,” today HE has entered into the “quality era.” The new social demands to HE have been put forward, which urgently requires HE to get out of the “ivory tower.” As “the gate keepers of quality,” QAAs are playing more and more irreplaceable important roles, and their social status are becoming more and more prominent. However, how to guarantee the quality of the QAAs? Who can review the QAAs?

2.1 Learning from European experience: EQAR for higher education

QA is the cornerstone of “the Bologna Process” in Europe. According to “Standards and Guidelines for Quality Assurance in the European Higher Education Area” in 2005, the European Association of Quality Assurance proposed the establishment of the EQAR for higher education. In March 2008, EQAR officially began its operation. EQAR promotes the long-range goals and objectives of European HE. EQAR’s vision is a coherent QA framework for the European Higher Education Area in which HE institutions have the freedom to turn to any EQAR-registered agency for their external QA reviews, and in which qualifications are thus universally recognized. EQAR has four main objectives which it has been devoted since its founding: promote student mobility by providing a basis for the increase of trust among HE institutions; reduce opportunities for “accreditation mills” to gain credibility; provide a basis for governments to authorize HE institutions to choose any agency from the Register, if that is compatible with national arrangements; provide a means for HE institutions to choose between different agencies, if that is compatible with national arrangements; and serve as an instrument to improve the quality of agencies and to promote mutual trust among them (EQAR, 2017).

Its registered members cover public, private and professional QAAs in the whole Europe. EQAR’s mission is to further the development of the European HE area by increasing the transparency of QA, and thus enhancing trust and confidence in European HE. EQAR is to facilitate the identification of the credibility and professional quality of both international and national QAAs, as well as the QA field that they receive. From its establishment to 2015, EQAR has registered 40 QAAs, among which eight QAAs in German, three in Spain, seven in Holland, separately two in Belgium, France, Switzerland and the UK, one in Austria, Finland and other 14 European countries (European Quality Assurance Register for Higher Education (EQAR), 2015).

In global scope, we can see that some countries have established an accreditation system of the QAAs, one of the representatives is Council for Higher Education Accreditation (CHEA) in the USA. CHEA is a national advocate and institutional voice for promoting academic quality through accreditation, CHEA is an association of 3,000 degree-granting colleges and universities and recognizes 60 institutional and programmatic accrediting organizations. (CHEA, 2015). From the international perspective, the role of the EQAR is equivalent to a national accreditation system to an external QAA, helping the governments, HEIs, students, parents and other stakeholders know the right choice for their educational needs. EQAR has greatly promoted student mobility by providing a basis for the increase of trust among HE institutions, reduced opportunities for “accreditation mills” to gain credibility (EQAR, 2015) and what is more it has brought the experience of the pioneer of the registration system to the world.

2.2 The genesis of register in the Asia-Pacific region: APQR

After learning good experience from EQAR, in April of 2013, the annual general meeting (AGM) of the Asia Pacific Quality Network (APQN) has endorsed the proposal of the establishment of APQR as part of its Decennial agenda. APQN is a non-profit, non-governmental international organization in the domain of QA of HE in the Asia-Pacific region, while APQR is a register which recognizes the QAAs of APQN’s members under the umbrella of APQN, listing those agencies that have demonstrated their substantial compliance with a common set of principles for QA in the Asia-Pacific region. On June 25-27 of the same year, APQN organized a three-day pilot review to Quality Assurance and Accreditation Council in Sri Lanka, APQN President Patil Jagannath as the Coordinator, Dr Pillai Latha as the Review Chair. On the basis of the pilot review, the APQR project went on developing under the leadership of APQN president.

After two years of consultation with various stakeholders, the document of “APQR” was officially issued in special meeting at Macao during January 22‐23, 2015. Thus APQR comes into effect from January 2015 (APQN, 2014).

In April 17-19, 2015, APQN Conference and AGM was held in the Yunnan University in Kunming in China, the attendance of 192 participants from 28 countries in the world added to the credibility of APQR. APQN president excitedly said in the open ceremony: “Now […], the dream project of APQR is finally a reality […] the work done by us at APQN has been globally admired and recognized” (Patil, 2015). It is fair to say that after three-year hard work with perseverance, APQR finally came into existence having achieved a breakthrough in the registration system of education QAAs in the Asia-Pacific region.

3. The implementation of the APQR

APQR is an international, non-governmental, regulatory and constrained activity to the QAAs. Its registered members for inclusion will cover public, private and professional QAAs in the Asia-Pacific region (in the future it might be expanded to institution members made up of HEIs).

At the Kunming Conference in April 2015, the APQR Council (APQRC) was formed which is responsible for reviewing and approving the expression of interest (EOI) of the applicant QAA. Very soon, APQRC decided to carry out the “substantive review” of the first formal register according to the EOI from the Fiji Higher Education Commission (FHEC).

Under the leadership of APQN president and the APQRC, a review panel of three leading experts in the Asia-Pacific region is formed. The first formal review for APQR has been conducted on the FHEC in Suva, Fiji from June 24 to 26, 2015. “This exercise would be counted as milestone in history of APQN and APQR,” said the APQN president on July 2 (APQN Secretariat, 2015). From May 28 to 30, 2017, APQR organized a review panel of three well-known international experts from France, Kazakhstan and China to conduct on the Certification Association “Russian Register” Saint Petersburg, Russia and the APQRC has considered and decided to include Russian Register on APQR. Below, this paper will discuss the standards and the review procedure of the first formal implementation of APQR, and summarizes three characteristics of the first APQR review.

3.1 The review criteria of APQR

All evaluation, assessment, accreditation, review and other QA activities are bound to have standards, guidance or criteria. Good standards and guidance can guarantee QA activities achieving the desired objectives in accordance with the established QA development mission. APQR Standards and guidance for inclusion is especially crucial because APQR is a meta-evaluation to QAAs.

APQR is one kind of QA mechanisms based on self-evaluation and peer review. It is also one of the important means of self-management for QAAs. APQR criteria for inclusion are as follows.

The above 11 criteria are based on the eight criteria from “APQN Constitution” which form the basis to determine the types of APQN membership, as well as the principles from “Higher Education Quality Assurance Principles for the Asia Pacific Region (known as “Chiba Principle”). As we know, the “Chiba Principles” declared by APQN in February 2008 emphasize a generic approach that has relevance for all HEIs, QAAs and quality assessment practices in the region regardless of the level of development, size and national context, which are helpful to potential students, employers, parents, governments, HE institutions and professional bodies, both nationally and internationally. (APQN, 2008) Among the 11 criteria, seven are from “APQN Constitution” and four are from “Chiba Principle” (see Table I).

The structure of the above criteria shows that APQR has a tightly close relationship with APQN, and have the potential of inheritance and sustainable development. APQR’s review criteria still adhere to the “APQN Constitution” as well as the “Chiba Principle,” the registered members of APQR will still adhere to the APQN’s mission, objectives and strategies (Table II).

3.2 The review process and principles of APQR

According to the APQR mission, criteria, procedures and other regulations, the APQRC made “the Guidance and Templates for the Asia Pacific Quality Register” in June, 2015. The entire process of the first APQR review was carried out in accordance with it. The first formal review process is followed by the seven steps see Figure 1.

Compared with the process of the acceptance of APQN members, one change of APQR is that the review panel should be on-site. The review panelists themselves carry on an on-site visit to QAA, through the verification of the QAA’s self-evaluation report (SER) and multiple evaluation methods, “to strengthen the connotation construction of the quality assurance agencies in higher education, encourage its characteristics project for guaranteeing educational quality, stimulate potential of the sustainable development, enhance credibility of the QAAs, and better service higher education” (The APQR Council, 2015). Both the review panel and the reviewed QAA try to be in an equal communication and use the methodology of mutual problem-finding, mutual problem-analyzing and mutual problem-solving (The APQR Council, 2015). The main purpose of the on-site review panel is to analyze the QAA’s situation, diagnosis the “hidden issues” and promote its development. Therefore, in the entire review process, the following three principles are obeyed by the review panel.

3.2.1 Establish the “negotiating register” mechanism advocated in “the fourth-generation-evaluation” by combining the review panel and the QAA

That is, based on SER and situation of the QAA, the review panel tries to find the QAA’s commendations and negotiatively give the recommendations. Early before the review, FHEC submitted “Self-Evaluation Report of Fiji Higher Education Commission” (up to 45 pages), “Good Guidelines for FHEC – Booklet,” “Quality Assurance – Policy,” “FHEC Annual Business Plan 2015” and other 15 relevant supporting materials. Before the arrival to FHEC, the panelists have carefully read the SER and formed a preliminary review conclusion. During the on-site visit, according to the problems found in the desk review, the panelists aimed to the key analysis and special judgment combined the on-site situation.

3.2.2 Combining the comprehensive whole background with crucial review elements

For example, the first review activity is FHEC “Presentation on the FHEC and Self-Evaluation Report” from the macro perspective, and then the second activity is the individual introduction from QA section, professional services unit, information technology and other five sections. Besides, except document reading and in-depth interviews, the panelists held a focus group made up of assessor, council member, EER evaluator, CBT assessor, program evaluator and other 15 persons. The panel drove to the campus of three stakeholders for in-depth interviews and discussion: University of the South Pacific (a public university), ServicePro Institute of Tourism and Hospitality (a private institute) and Corpus Christi Teachers College.

3.2.3 Combing review and guidance together

It suggests that the review must make judgments to the basic situation of the QAA, but also supply the guidance in details for the future development of the QAA. According to the SER and on-site review, the panel describe the basic situation of the 11 criteria against the “the Guidance and Templates for the Asia Pacific Quality Register.” The last part of the review is to sum up 11 commendations of the FHEC, at the same time put forward ten recommendations for FHEC’s improvements. For example, the panel commends “the FHEC for its multifaceted methods for staff professional development, which include staff exchanges, job rotation and other learning activities;” and also recommends “QA will separate from FHEC” (APQR, 2015).

The specific review methods of the panel are including the following: to listen to the SER presentation from the QAA; to investigate the QAA’s infrastructure, facilities and working environment; to access and read the supporting information and documents of the QAA; to have interviews and focus group with the stakeholders of the QAA; and to conduct in-depth interviews with the leaders and important members of the QAA. According to the actual situation of the QAA, the panel can flexibly adapt the review methodology, e.g. based on quantitative statistics; qualitative review focused on in-depth interviews can be mainly used.

4. The characteristics of APQR

APQR will register the qualified QAA, whose primary purpose is to guarantee the qualified QAA; to provide an inspirational target for QAAs; to serve as a quality hallmark; to establish the basis for mutual recognition and to supply the basis for cross-border operation of quality agencies/institutions as well as reference to global stakeholders on trustworthy EQAAs in the Asia-Pacific region (APQR, 2015). Since 2012, APQR has gone through the development stage of “survey – research – design – pilot – programs – formal review;” its basic features can be summarized as follows.

4.1 APQR is an important measurement of education globalization for the Asia Pacific region

In the era of globalization, all the criteria, standards and guidelines in the QA must be measured from the global perspective. As a professional, international organization consisting of 159 members from 40 countries and regions, the diversity of APQN itself is obviously self-evident. Its regional and international features are more prominent (Zhang and Patil, 2015). Only by reading the name of “Asia-Pacific Quality Network,” the “Network,” can we realize that the nature of APQN is non-mandatory, national, regional and international. “The Asia-Pacific Region” includes much of East Asia, South Asia, Southeast Asia and Oceania, spanning as far west as Afghanistan, as far north as Russia, as far east as Fiji and as far south as New Zealand according to the UNESCO model APQN, 2005. The theme of 2015 APQN Conference held in Kunming is “Globalization and Diversification of Quality Assurance of Higher Education” which emphasizes the importance of the HE globalization in the era of globalization.

“APQR” (in Macao) clearly stipulates that the establishment of a registration system has seven main purposes, among which two highlights the international characteristics: to supply the basis for cross-border operation of quality agencies/institutions (the fifth purpose) and reference to global stakeholders on trustworthy EQAAs in Asia Pacific (the seventh purpose). Thus, the ultimate goal of APQR is to promote the international registration in the whole globe.

From the composition of the first formal review panel, one can also see its international feature, three experts are separately from QAA Council of Sri Lanka, Yunnan Higher Education Evaluation Center of China and the Department of Higher Education, Research, Science and Technology of Papua New Guinea. They are well-known international experts in QA in the Asia-Pacific region.

4.2 APQR emphasizes the characteristics review of diversity

In today’s “fourth-generation-evaluation” era, the new idea is to promote the development, to change from the “facts judgment” or “value judgment” to the developing evaluation of “value co-building,” and lead to an equal, credible and constructive negotiated dialogue (Zhang Jianxin and Liu Kangning, 2013). Although “APQR” (in Macao) provides 11 criteria and descriptions in detail, but there is no specific statistic requirements, whose purpose is to reach the basic criteria according to the diversity characteristics of the QAA.

“The Guidance and Templates for APQR” advocates taking full account of the diversified characteristics of diversified regions, diversified countries, diversified QAAs. According to their diversified mission, objectives and service clients, APQR review diversified QAAs to form their own characteristics. This registration performance lies in the QAA’s situation, its performance and set aside enough space for registration, which will be conducive to mobilize the enthusiasm of the QAA, promote the QAA’s characteristics and personality and to achieve the sustainable development of the QAA.

In addition, the first formal review emphasis the dynamic development of the QAA. “The Guidance and Templates for APQR” requires “not to compare with your own QAA.” The regulation of “no horizontal comparison” makes the review mechanism base on its own diversified characteristics; choose its own development path, reflecting a dynamic development of the evaluation concept (The APQR Council, 2015). Inclusion to APQR will be valid for a period of five years. The governing body of the register has the right to cancel the membership if there are circumstances that question the substantial adherence of the agency to the review criteria.

4.3 APQR highlights the combination of quantitative and qualitative review

Acceptance onto APQR is based on “substantial compliance” with these criteria. Each criterion will be judged “fully, substantially, partially or non-compliant;” and substantial compliance with the whole set needs full or substantial compliance with each criterion.

To make decision on the above four categories, APQR review uses the qualitative evaluation method, pay very much attention to the “evidence,” any judgment should be evidence-based. These “evidence” is not only data, but more focus on the relevant facts and information from all the process. In the first formal review, the standard is “evidence-based” combing the quantitative and qualitative evaluation methods together, made up of three components such as: the basis of the review, the criteria statement and the core body. For example, in the review of the first criteria “Organization,” the first programmed is “Review Panel Observations are based on 1) the Self Evaluation Report; 2) the legislative status of the FHEC; and 3) interview evidence ” (The Review Panel, 2015). Then the objective statement of the criteria and the core body are followed.

5. Concluding words

To answer the question of “who is to guarantee the quality of QAA” at the beginning of this paper, according to the relationship among the QAA, the government, the HEIs, the enterprises and other professional associations, we can generally divide into three types: the national education administrative departments (including its affiliated institutions) directly register the QAA; the quasi-governmental agencies established in “legislative and independent administrative agencies”; and the third-party organizations such as professional associations (Dong Youzhi, 2013). Today, in addition to the above three categories, we can add a new the fourth one – that is, the international registration system.

How to ensure the quality of QAAs? Who can review the QAAs? This is where this research begins. Following the founding of the EQAR for Higher Education, APQR became the second in the international QA networks to implement QA register. APQR has the following three characteristics: APQR is an important measure to promote the internationalization of HE in the Asia Pacific region; APQR individual characteristics evaluation; and the APQR method of quantitative evaluation and qualitative evaluation outstanding combination.

From April 2012 when APQN launched the initiative of APQR, so far, five years passed, and substantial progresses of APQR have been made in theory as well as in practice. However, the exploration of APQR is still not optimistic, and we still have a long way to go. The following questions are to be answered:

1. From the practice level, The EQAR for HE has been rapidly developed in full swing (as of June 2017, EQAR has recognized 47 QAAs), why is APQR has not got the active participation from the QAAs in the Asia-Pacific region?

2. From the theoretical level, how can the international registration system, such as APQR, to ensure the quality of the QAAs in the Asia-Pacific region, together the three types mentioned above (national education administrative departments, the quasi-governmental agencies and the third-party organizations)?

3. From the strategic level, how can APQN take effective measures to promote the QA of the QAAs in this region? How does APQN help the QAAs who have been accepted on to APQR play more effective roles in the trends of international HE and cross-border education?

Look around, today on the international stage of QA, APQR has emerged as “the watchman of quality” in the Asia-Pacific region as counterpart of EQAR in Europe. How far away does such newly emerging form of guaranteeing the QAAs’ quality go forward, what is its future prospects and other concerning issues, are some of the question that need enthusiastic attention and contribution.

Figures

Figure 1

The flowchart of the review process

Table I

Criterion for the APQR

Criterion	Description
(1) Organization	The QAA is a full member of APQN or is a QA body which is valid entity recognized by the appropriate authority in the relevant country/territory/region, and is accountable to stakeholders
(2) Operations	The quality assurance agency undertakes quality assurance activities (at institutional and/or program level) on a cyclical basis
(3) Mission and objectives	The mission statement and objectives of the agency are understood consistently by its stakeholders
(4) Staff and reviewers	The profile of the agency staff and the profile of the reviewers the agency uses are consistent with the mission statement
(5) Independence	The quality assurance agency is independent and has autonomous responsibility for its QA operations. The judgments and recommendations of the agency’s reports cannot be changed by third parties
(6) Resources	The agency has sufficient resources to run its operations in accordance with its mission statement and objectives
(7) Process and criteria	The description of the processes and criteria applied by the agency are transparent and publicly available and normally include: self-evaluation, site visit, public report and follow-up measure. The published standards and criteria are applied consistently and rigorously
(8) Appeals	An appeals mechanism is available for the institutions
(9) Quality assurance	The agency has effective quality assurance measures in place to monitor itself and is subject to occasional review
(10) Monitoring and evaluation	The agency undertakes research on internal and external quality assurance and provides information and advice to the higher education institutions
(11) Agency linkages	The agency cooperates and collaborates with other agencies and key players across national borders

Table II

The list of the sources of APQR criteria

S/No.	APQR criteria	Source
(1)	Organization	From “Chiba Principle”
(2)	Operations	From “APQN Constitution” No. 1
(3)	Mission and objectives	From “APQN Constitution” No. 2
(4)	Staff and reviewers	From “APQN Constitution” No. 3 and 4
(5)	Independence	From “APQN Constitution” No. 5
(6)	Resources	From “APQN Constitution” No. 6
(7)	Process and criteria	From “APQN Constitution” No. 7
(8)	Appeals	From “Chiba Principle”
(9)	Quality assurance	From “APQN Constitution” No. 8
(10)	Monitoring and evaluation	From “Chiba Principle”
(11)	Agency linkages	From “Chiba Principle”