The Discipline-Based Rating System (D-SETARA) is a rating system developed and implemented to assess the quality of teaching and learning in Higher Education Institutions (HEIs) in Malaysia. The system aims to rate the institutional level quality of teaching and learning of specific (clusters of) disciplines at level six (undergraduate level) of the Malaysian Qualifications Framework (MQF) in universities and university colleges. The system, the first of its kind, is a complement to the existing institutional rating through Sistem Penarafan Institusi Pengajian Tinggi Malaysia (SETARA) which has been successfully carried out for several years. Whilst SETARA focuses on the institution, D-SETARA focuses on the schools or faculties where the respective discipline is offered. The spirit is one of benchmarking against a set of quality standards for the purpose of improvement of the institution and discipline but not for competition between the institutions.
The first D-SETARA exercise focuses on four clusters of disciplines: Engineering; Medicine, Dentistry and Pharmacy; Health Sciences; and Hospitality and Tourism. Participation in D-SETARA is voluntary; invitations were made to all HEIs offering undergraduate programmes and have had the first batch of graduates in 2011. The Engineering discipline however, involved only HEIs which offer four-year degree programmes. The number of institutions rated in each discipline were 25 (Engineering), 15 (Medicine, Dentistry and Pharmacy), 14 (Health Sciences) and seven (Hospitality and Tourism).
The management of D-SETARA is through a two tier committee: comprising of a Steering Committee and Sub-Committees for each of the discipline. The terms of reference of the Steering Committee are to:
- provide an overall framework for the exercise;
- coordinate the development of the instrument;
- undertake overall analysis and overall reporting; and
- provide technical and logistical support to Discipline subcommittees
The terms of reference of the sub-committees are:
- Categorization of the various programmes;
- Identification of all HEIs offering the programmes;
- Modification of the SETARA '09 instrument (include specialized indicators) or develop new ones; and
- Engagement of stakeholders (e.g., HEIs, Employers)
The modus operandi of the exercise is similar for all the four clusters which generally include: buying-in process from MOHE, team meetings, stakeholders consultation, data collection and verification, analysis and report writing. The various stakeholders were involved through meetings carried out at various stages of the exercise. Three meetings were carried out with different purposes. The first stakeholders meeting was on the 19 July 2011, with the purpose of briefing the whole exercise, including the principles to be used, framework of the instrument and the data collection procedures. This was followed by a second meeting on the 17 January 2012 where consultation was focussed on matters relating to the final D-SETARA instruments, sources of data, data collection procedures, updating on the Pilot Study, updating on responses received from HEIs since the first meeting, and agreement on Employer's Survey methodology. The final meeting was carried out on 6 March 2012 where briefing on the data based on the Tracer Study was discussed.
The ultimate aim of the exercise is a tier rating (based on a six tier rating system) of HEIs in each of the discipline. The basis for rating is the total scores and cut-off values determined for each of the tiers. The scope of evaluation of teaching and learning covers the main domains of Input-Process-Output, and under each of the domains there are criteria followed by indicators for each of the criteria. An instrument was developed by each sub-committee based on the SETARA '09 and SETARA '11 instruments, this mainly done to ensure comparability with the institutional rating. The weight according to the domains: Input- Process-Output is similar to SETARA '11, i.e., 20:40:40 for the Engineering and Medicine, Dentistry and Pharmacy discipline; 30:40:30 for Health Sciences, and 30:35:35 for Hospitability and Tourism. The weights of the criteria and the indicators also varies between disciplines. The justifications for the varying weighting system in domains, criteria and indicators are due to the specific needs of the disciplines and explained in the respective report.
Data for the various indicators are collected from various sources, which include Institutional Data (through self-assessment with documentary evidence), Employer Surveys, Tracer Study and also accreditation data. Various steps have been taken to ensure data integrity from each of the sources. The instrument was verified by a pilot study on ten selected HEIs for the disciplines. For the purpose of data collection, a template was prepared for the HEIs. Data submitted by the HEIs are compiled and then verified by the sub-committees. Subsequently, data analysis were undertaken which ultimately results in the tiering of HEIs in each discipline. Verification of the exercise was done through a Verification Committee appointed by the MQA. Two verification meetings were carried out on 15 May 2012 and 24 November 2012. The verification exercises cover the process of instrument development; process of data collection, entry and analysis; missing data analysis and treatment; and the results.
The results show that for Engineering discipline, out of the 25 universities and university colleges rated, 11 institutions achieved a Tier 5 category, 13 institutions achieved Tier 4, and the remaining one, achieved Tier 3. For Health Sciences discipline, out of the 14 universities and university colleges rated, five institutions achieved a Tier 5 category, six institutions achieved Tier 4, and two institutions are in Tier 3. One institution was not rated because of insufficient data. As for the Hospitality and Tourism discipline, the result shows that out of the seven universities and university colleges rated, one institution achieved the Tier 6 (Outstanding) category, three institutions in Tier 5 and one institution each in Tier 4 and Tier 3. One institution in this discipline was not rated because of insufficient data. For Medicine, Dentistry and Pharmacy discipline, out of the 15 universities and university colleges rated, only one institution achieved Tier 5 category, 11 institutions in Tier 4, and two institutions in Tier 3. One institution was not rated because of insufficient data. In all the ratings, the results show that none of the universities and university colleges is in Tier 1 or Tier 2.