DOI:
10.18413/2313-8971-2020-6-2-0-3
Quality assessment of university students training
Abstract
Focus on the quality of education is a priority of the state program of the Russian Federation “Development of education” (approved by the RF government decree of December 26, 2017 № 1642). The main activities of the direction “Improving the management of the education system” of this program include measures to implement mechanisms for evaluating and ensuring the quality of education in accordance with state educational standards. Assessment of the quality of education in the universities is determined by the necessity of the competence orientation of the educational process and its content, technology implementation, re-evaluation procedures, technologies and tools to assess the quality of training students within the framework of the competence-oriented educational standards. The article deals with the problem of evaluating the quality of training of students of higher educational institutions. The definition of the concept of quality of education from the point of view of stakeholders is given. The article presents a method for evaluating the formation of learning outcomes based on a two-criteria approach (competence and disciplinary). The main criteria for evaluating the formation of learning outcomes are highlighted. The results of testing the methodology in one of the areas of training are considered. To study the level of competence formation, a scale of assessment of competence formation (the stage of competence formation) is proposed. The results of testing the methodology for evaluating the formed learning results presented in the article can be useful for interested persons who are directly involved in the process of designing, developing and implementing educational programs, as well as in making methodological and managerial decisions aimed at improving the quality of education.
Keywords: quality of education, quality of training, level of competence, competence approach, evaluation of residual knowledge, the system of internal evaluation of the quality of education, quality management system, educational organization
Introduction. Currently, the Ministry of Science and Higher Education of the Russian Federation pays special attention to the independent assessment of the quality of education. Obvious on this basis is the expediency of forming a system of independent assessment of the quality of education in higher education institutions of the Russian Federation. The basis for the development is the need for a comprehensive independent assessment of educational activities and training of students, expressed in the degree of their compliance with The Federal State Educational Standards, the requirements of specialized organizations and enterprises, as well as the requirements and needs of an individual or legal entity in whose interests educational activities are carried out, including the degree of achievement of the planned results of the educational program.
One of the components of an internal independent assessment of the quality of training of students is the implementation of measures to control the availability of students ' formed learning results in previously studied disciplines (modules)[1].
In this article, the quality of education is considered as a complex characteristic of educational activities and training of students, expressing the degree of their compliance with The Federal State Educational Standards, The Federal State Requirements and (or) the needs of an individual or legal entity in whose interests educational activities are carried out, including the degree of achievement of the planned results of the educational program.
The procedure for evaluating the formation of learning outcomes in higher education institutions should be based on the following legislative and legal documents:
– Federal law “On education in the Russian Federation” dated 29.12.2012 No. 273-FL (ed. from 06.04.2020) (article 95 “Independent assessment of the quality of education”);
– Federal State Educational Standards of higher education;
– Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG);
– Methodological recommendations for the organization and conduct of internal independent assessment of the quality of education in higher education institutions for higher education programs – bachelor's programs, specialty programs and master's programs (letter of the Ministry of Education and Science of the Russian Federation dated 15.02.2018 № 05-436 “On methodological recommendations”);
– University regulations: Regulations on the system of independent assessment of the quality of education, Regulations on the main educational program of higher education, Regulations on the main professional educational program of higher education, Regulations on the Fund of evaluation funds, Regulations on the point-rating system for evaluating the quality of development of basic educational programs, Regulations on the electronic educational and methodological complex of the discipline for the e-learning system, etc.
Main Part. Research analysis. The problem of the quality of education has been the subject of research of many scientists for a long period of time, and yet, a common opinion on the criteria for evaluating the quality of education has not yet been formed among researchers (Tjurikov A.G., Borovinskih O.S., Golubeva K.A., Kunizheva D.A., 2018; Zhuravleva M.M., Nakonechnyh V.N., 2019; Ledovskaja T.V., Solynin N.Je., 2019; Razov P.V., Amiraslanova A.N., 2018; Abdallah K.M., 2019).
Analysis of research results of Russian scientists (V.P. Bespalko, V.P. Simonov, N.V. Pakharenko, I.N. Zolnikov, M.M. Zhuravleva, V.N. Nakonechnyh, T.V. Ledovskaja, N.Je. Solynin, etc.) indicate the presence of different approaches to the evaluation of completeness of learning outcomes, lack of common view on the assessment of competence, lack of scientific and methodological basis of the study of the level of formation of competences, differences in understanding of requirements of The Federal State Educational Standards of Higher Education in terms of evaluation of competences and the insufficient development of technologies for the creation of assessment materials (Aguado, M.L., 2018; Goos, M., Salomons, A., 2017;Karin J. Gerritsen-van Leeuwenkamp, Desirée Joosten-ten Brinke, Liesbeth Kester, 2017; Wang, Dongfang & Sun, Yuting & Jiang, Ting, 2018).
Analysis of foreign studies on the methodology of evaluating learning outcomes (B. Bloom, D. Bock, J. Guilford, S. Thorpe, J. Clifford) indicates the possibility of using multi-level taxonomies (Gerritsen-van Leeuwenkamp, Karin & Brinke, Desirée & Kester, Liesbeth, 2019); Cechova, Ivana & Neubauer, Jiří & Sedlačík, Marek, 2019; Deng, S, Que, X., 2019).
A common understanding of quality assurance in all countries and among all stakeholders is provided by The Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG – The European Standards and Guidelines) developed by the European Network for Quality Assurance on direct instructions of the Ministers Conference of European countries education that signed The Bologna Declaration. Russia also signed The Bologna Declaration, which means that, along with other European countries, it has committed itself to building a unified educational space (Flores, M.A., Brown, G., Pereira, D. et al., 2020).
In this regard, these standards and recommendations should be used in higher education institutions as a single reference guide for the formation of systems of external and internal quality assurance in higher education.
Methodology for evaluating the formation of learning outcomes. Assessment of control the presence of students formed learning outcomes of previously studied disciplines (modules) in the part of formation of competences is carried out in the framework of formation in the BSU system of internal evaluation of the quality of education (the basis of methodical recommendations of the Ministry of Science and Higher Education of the Russian Federation).
The purpose of assessing the quality of training of students is to achieve a positive dynamics in the quality of training of students by forming an objective assessment of the results of the development of educational programs implemented by an educational organization. The main principles of internal independent quality assessment should include: objectivity, collegiality, independence, and frequency. These principles determine the following requirements for the preparation and implementation of measures to control the availability of students' formed learning results in previously studied disciplines (modules):
a two-criteria approach (competence-based and interdisciplinary);
conducting the assessment procedure no earlier than 6 months after the completion of the discipline;
the form of conducting-computer testing, based on evaluation materials for disciplines that form a certain competence;
when forming the test, the classical model[2] is used; the test is formed from 30 tasks, each of which corresponds to the chosen competence for the study; the test is given 45 minutes to complete;
the quality of evaluation materials is confirmed by an expert opinion (review) of a representative of the relevant organization or enterprise;
disciplines, that form the competence are selected for research at random, taking into account the above requirements. Formation of competence depending on the complexity and duration of skills development may include several stages, if the competence has not yet been formed, the assessment is made at the stage of competence formation;
the procedure is organized by an independent Department responsible for managing the quality of educational activities at the University;
to form objective results, in terms of the completeness of coverage of the elements of the studied population, continuous observation is used (actual participation of at least 80 % of students in the studied field of training / specialty, taking into account the valid reasons for absence);
representatives of student self-government bodies, representatives of specialized enterprises and organizations are invited as independent observers of the course of computer testing.
To study the level of competence formation, it is proposed to use the following assessment scale (table 1).
The threshold value (the minimum acceptable percentage) when evaluating whether students have formed learning outcomes in previously studied disciplines (modules) in terms of competence formation is defined
as 50 %.
Results of the method testing. The summary results of the diagnostic study of the formation of learning outcomes in the framework of determining the level of competence formation based on the proposed methodology are reflected in the analytical table 2, presented below.
This fragment of the study presents the results of assessing the level of formation of General Cultural Competence-2 (further GCC-2) in the field of training 38.03.01 Economics. 14 students of the 3rd year of the training direction 38.03.01 Economics took part in the diagnostic testing. The results of the study sample are checked for reliability, validity, representativeness, and reliability requirements and can be applied to the entire general population.
The obtained results of diagnostic testing allow to conclude that most of the students who took part in the testing have an advanced level of competence formation. The percentage of students who showed the level of competence formation below the threshold value (0-50) was 0 %, which demonstrates the formation of competence in students of the training direction 38.03.01 Economics.
Conclusions. Testing of the method allows to make a number of generalizing conclusions and recommendations. Assessment of the level of formation of learning outcomes makes it possible to determine the level of implementation of educational programs by the University. Assessment of the trajectory of competence formation allows to conclude that the main principle of phasing in the formation of competencies (increasing complexity from basic knowledge to advanced level) is observed. The analysis of the structure and content of educational programs in terms of curricula in the field of training/ specialties and the matrix of competence formation in the disciplines of the curriculum allows to determine the areas of possible improvements.
The matrix of competence formation is a key element of the main professional educational program, which allows to clearly reflect the formation of competencies within the disciplines of the curriculum. Based on it, lining up the passport of competencies, competence maps disciplines and evaluation funds, as well as the evaluation of competence and as a consequence the quality and the level of development of the basic professional educational program. During the design trajectory of competence formation should note: the basic approach of the pedagogical design in the formation of educational outcomes (the result is a tool – the content); developed evaluation funds (evaluation materials) in the discipline should be structured (to correlate with the theme of sections of discipline code controlled competencies and indicators). This will help to more correctly prepare tasks for assessing the formation of competencies.
An important condition that precedes the procedure for assessing the formation of competencies is the quality control of test tasks in the e-learning system used. However, during testing, it is advisable to record the facts of students' complaints about the presence of errors/typos in the proposed tasks: technical errors (incorrect display of test tasks, drawings, formulas, etc.); incorrect wording of questions, suggesting ambiguous answers; the absence of specific rules for filling in answers in the open form before the task; the absence of all possible variants of correct answers in questions such as “Short answer” / “Missing word”. As part of the determination of opportunities to improve assessment tools created by University teachers and placed in the e-learning system, it is recommended to cross-check test tasks by teachers at the Department level, form a database of test tasks from tasks of different types, varying degrees of complexity, use competence-oriented tasks (cases, creative tasks, project tasks).
In terms of strengthening the interaction of the Institute with specialized enterprises and organizations on improving the educational process for the formation of the most objective assessment of training results, it is recommended to conduct an examination of evaluation tools with the involvement of representatives of specialized enterprises and organizations. The form of confirmation of passing the examination can be the receipt of a review.
To increase the motivation of students to successfully master educational programs, it is recommended to use the developed evaluation materials as part of the current and intermediate control, as an element of the point-rating system, as well as self-control of students during the semester.
Testing of the methodology for assessing the formation of competence based on a disciplinary approach has shown that it can be used to assess the formation of learning results in determining the level of competence formation. Nevertheless, it is necessary to move from the classical model of forming evaluation materials used in the research to the use of more advanced models, for example, to adaptive testing. Since the testing revealed the shortcomings of the classical model of forming the final test, namely, due to a random selection of questions, it is impossible to determine in advance what tasks of complexity will be formed for the student. When using the classic model, a combination of easy tasks for one student and difficult tasks for another student is possible. Therefore, the result depends only on the number of correct answers and does not take into account the complexity of tasks, which has low reliability, since the lack of consideration of task parameters often makes it impossible to objectively assess the student's knowledge.
At the moment, the proposed method for evaluating the formation of learning outcomes has been finalized by the areas of improvement identified during the study and is at the stage of re-testing.
[1]Guidelines for conducting an independent assessment of the quality of educational activities of organizations engaged in educational activities: letter of the Ministry of Education and Science of the Russian Federation dated 03.04.2015 no. AP-512/02. (In Russian)
[2] The classical model is characterized by the following criteria. There are n jobs in a particular field of knowledge, several knowledge areas or parts of area of expertise (section, topic, etc.). From this set of tasks is randomly chosen k jobs (k<n) which are available to the learner. The trainee selects or enters the correct answer, in their opinion. The result of the response to each task is evaluated as “correct” or “incorrect”. The result of testing is the percentage of correct responses of the trainee. The rating depends only on the number of correct answers and does not take into account the complexity of tasks.
Reference lists