Digital Examinations: Less Effort for More Quality

Digital Examinations for Journeymen in the Motor Vehicle Industry in North Rhine-Westphalia 

With the digitalisation of examinations through the development of an online examination platform, we have succeeded in turning several screws at the same time: less organisational effort, better and faster evaluability and thus an increase in quality. 100% future-oriented!

Category: eTesting

Project partner:

Verband des Kfz-Gewerbes Nordrhein-Westfalen e.V. &

BPS Bildungsportal Sachsen GmbH

The central final examinations are very often associated with a great deal of preparation and follow-up work. In order to reduce the effort on the one hand and improve the quality of the examinations on the other, the Verband des Kfz-Gewerbes Nordrhein-Westfalen has set itself the task of creating an online format for the journeyman's examinations. The association supports its 46 regional organisations (guilds) with around 8,000 member companies and organises the semi-annual journeyman's examinations throughout the state. Since spring 2020, the association used the ONYX examination platform from BPS (Bildungsportal Sachsen GmbH).

An online examination platform for all

Motivation

The aim of the project is to convert the previously paper-based examination to an online format. The sectoral organisation benefits from the fact that vocational colleges are now almost completely equipped with PCs and laptops.

With the online examination platform, the examinations in the federal state are to be carried out decentrally at the same time. Approximately 4,000 trainees are to be supervised per examination throughout the state. For this purpose, a powerful examination interface is to be used, which is expandable and can be operated by those responsible on site.

Four motives were in the foreground for the branch organisation:

1. quality of the examination: the visualisation of the increasingly complex tasks was to be improved by zoomable graphics, 3-D images and animated material, and the combination of different task formats was to be ensured.
2. contemporary appearance: the young apprentices should be able to take the examination in a technical environment that seems contemporary and familiar to them. The examination format is at the same time a statement for the future viability of the chosen profession and does not represent a break with the working environment.
3. relief of the volunteer examiners: easier supervision through individually randomised task sequences. Answers can be evaluated completely or to a large extent automatically. Assessment of performance should take place in an objective procedure. Paperless archiving of individual examination results with a secure data deletion concept. 4.
4. sustainable quality optimisation: examination results and statistical data must be retrievable in a timely manner. Examination tasks can be evaluated individually. Each examination task can be archived, optimised and reused.

With the new examination format, these four goals could be realised: The examination results are immediately available and the statistical processing for those responsible for the examination can be called up directly. This enables a simple and direct announcement of the results to the examinees to be trained. A variety of new task formats can be mapped, which is ideal for determining the training level of the examinees. In particular, the action-oriented task of a workshop assignment as a fixed task block is suitable for realistically determining the level of performance. The statistical evaluation tools help to increase the quality of the final examination and thus of the training.

In addition to the trainees in the motor vehicle trade who use the online examination platform every six months to take the final examination, there are the following other target groups who are effectively involved in the use of the platform. These stakeholders are mapped as follows:

• The exam authoring team. This is made up of the vocational school teachers and the instructors in the companies, who create new examination content every six months and therefore need to be able to use the platform in depth.
• Guild administrators. They should be able to assign examiners and examiners to the respective examinations.
• Guild examiners. These are usually vocational school teachers and company trainers. They must be able to activate trainees for examinations on site at the PC pools, supervise them, evaluate examination results and finally publish them.
• System administrators of the respective PC pools. These are to ensure the correct and performance-compliant IT infrastructure. This includes the installation of an examination browser, which will be used as a basis for the examination board.

Not only intended for queries

Methods:

• Digital examinations

Project progress

After the installation of the examination platform in spring 2020 at seven selected guilds in North Rhine-Westphalia, the first demo examination content was made available and personal user accounts were set up for all stakeholders. This served to familiarise themselves with the examination platform. This was followed by individual training sessions for the main stakeholders. The training content for authors was the creation of digital exam questions and the configuration of the flow control (navigation through the exam, setting of incident parameters, use of integrated tools such as scientific calculators or attached files). For administrators, the focus was on importing trainee and examiner user accounts and assigning the user accounts to the respective exam items. The examiners learned how to register the students in the PC pool on the exam day and how to control the digital exams. This also included training on how to solve any problems that might occur.

In order to optimise and improve the examination portal, several test runs were carried out at the respective locations. As a result, the first examination run was carried out in the selected guilds in September 2020. BPS provided support both on site and in the form of remote support.

After the run, the promotion of the first successfully conducted examination of the project was carried out in the entire motor vehicle association of North Rhine-Westphalia. In addition, the improvement requests noted by the first run were implemented to optimise the platform. Since spring 2020, more and more guilds joined the online examination concept with each bi-annual final examination. With the growing number of participating guilds, the number of online examinees also increased enormously and the online examination platform was further optimised with each run. In May 2021, the final examination took place with 14 guilds and a total of almost 1,000 trainees.

Michael Dittmar, motor vehicle entrepreneur and head master of the Bochum guild, commented after the examination in May 2021: "We in Bochum have been participating in the online examination since the beginning of the project and have had nothing but good experiences right from the start: a stable IT system and really forward-looking evaluations. The intuitive operation of the platform means that the trainees have no fear of contact."

Project result

The goal of successively expanding the examination platform to other guilds has been achieved. It was important to be able to use the existing technology of the PC pools of the guilds without having to buy special hardware or software: Only three basic requirements are needed to use the platform, i.e. to take an examination: a PC, internet access and an up-to-date browser. The result is a centralised examination solution that can be used decentrally and will eventually be used in all guilds of the federal state. The implementation of the project itself went smoothly from the start and met with a high level of acceptance among the trainees, but also among those responsible for the examinations. The survey of the trainees and those responsible for the examinations showed that the examination platform is particularly fail-safe and that malfunctions can be easily remedied. The platform is easy to use and enables those responsible to organise examinations themselves and thus also to guide the examinees. It enables centralised as well as decentralised inspection control, which makes efficient, data protection-compliant and fast control processes possible. Through various statistical key figures of the overall evaluation of the examinations, the quality of the examination questions as well as the performance level of the trainees, also in relation to other years, can be evaluated and ultimately increased.

“We feel encouraged by the jury's decision to have developed our own online examination platform. It corresponds exactly to the requirements of the examination system in the skilled crafts sector. At the same time, it is an important building block for the digitalisation of training in the motor vehicle trade and other trades with increasing technical complexity. The online examination is an important milestone on the way to the future!“

Marcus Büttner, Chief Executive at Verband des Kfz-Gewerbes Nordrhein-Westfalen e.V.

Conclusion

Digitisation is on everyone's lips and is often used as a buzzword. The conversion of the central final examinations to a digital format is not only a big leap, but a decisive one. With 46 guilds, the North Rhine-Westphalia Motor Vehicle Association is particularly challenged to create equal and cent¬red examination requirements. The use of a digital examination platform makes it possible to centralise and improve the quality of training. "With the online examination platform, which is groundbreaking for all trades, KFZ-NRW has successfully taken the path towards the "future". The digitalisation of training is the first step towards the holistic recruitment and securing of skilled workers, which will be consistently pursued in the interest of the entire sector". says Frank Mund, President of the Verband des Kfz-Gewerbes NRW. For this reason, it is gratifying to learn that the online examination platform is already being used successfully in other areas of training and further education, but also in other trades. The consideration of already existing PC pools as a basis as well as the use of free open-source software and thus the reduction of costs to a possible minimum is also remarkable.

For this reason, the jury of the eLearning AWARD 2022 is pleased to award the prize in the category "eTesting" to the project "Digital Examinations for Trainees in the Automotive Industry in North Rhine-Westphalia".

Info:

Requirements: Creation and provision of a central examination platform, which can be used decentrally at all guilds of the state association of the motor vehicle trade of North Rhine-Westphalia. The aim is to gradually expand this platform to all 42 guilds and to include other examination areas in addition to the final examinations.

Special features: The basis is the use of existing technology: PC pools, internet and up-to-date browsers. In addition to ONYX from BPS GmbH, open source software was used. This made it possible to save costs but still achieve a particularly good result.