Structured and unstructured viva voce assessment: A double-blind, randomized, comparative evaluation of medical students

Introduction

The assessment of students is very important first to train quality medical graduates to meet the society demand and second to provide the feedback to all the stakeholders of medical education policymakers.^[1] It provides differentiation, discrimination, and monitoring of the students as per their capacity and talent for progressive learning. In addition, it determines the students and teacher efforts in imparting the teaching-learning.^[2] The multiple sampling strategies or assessment forum is used to capture competencies in medical training evaluation depending on the nature of the content such as pre-clinical, paraclinical, or clinical. There are various kinds of assessment forums which can be broadly divided into the written exercises, assessment by supervising clinicians, clinical simulations (CSs), and multisource (360 degree) assessment.^[3]

The objective and content of assessment direct the selection of appropriate instruments from ever growing myriad of options such as patient management problems, modified essay questions, checklist-based evaluation (CBE), objective structured clinical examinations (OSCEs), student projects, constructed response questions, multiple choice questions, critical reading papers, global rating scales, extended matching items, tutor reports, students’ portfolios, short case assessment, long case assessment, students’ log book, trainer’s report, student’s audit, simulated patient surgeries, video assessment, CSs, self-assessments, peer assessment, standardized patients, viva voce/oral examinations (VV), short answer questions/short essay questions, key feature test, mini-clinical evaluation exercise, direct observation of procedural skills, clinical work sampling, 360-degree evaluation (360 degree)/multisource assessment, traditional viva voce (TVV)/oral examinations(OEs), and skill-based assessment.^[4]

Assessment and learning are interlinked, and just by changing the way the students are assessed, it can reorient the engagement of students with a particular subject. The students start focusing more thoughtfully, get clinical oriented; develop their skills and attitude accordingly. Therefore, assessments are done through written examination, practical examination, and OE to cover their training in entirety.^[1]

The teaching and learning of medical subjects are undergoing many changes including introduction of case-based learning, problem-based learning, evidence-based learning, and newer assessment forums such as OSCEs,^[4] objective structured theory examinations (OSTEs), and objective structured practical examination (OSPE) to make the whole process students centric so as to enhance performance and outcome rather than just upgrading their recall and knowledge. The future of country’s health care depends on the quality of medical graduates which, in turn, depend on the quality of medical education imparted.

The assessment pattern that is being followed in this college is as per the medical council of India (MCI) requirement. The written exercises are given in two parts each assigned with 40 marks (2 × 40 = 80 marks) for the subject of pharmacology at the end of 1½ years of teaching. The practical assessment is only for 25 marks and TVV examination consists of 10% marks (i.e., 15 marks). The 30 marks are given for internal assessments which are divided into 15 marks for practical and 15 marks for theory. Thus, a total of 150 marks are prescribed by MCI in university examination as an assessment tool to measure learning which finally expects to establish pharmacological/therapeutic practice.^[1]

The OE or viva voce is used to test attitude, skill, and communication of students which cannot be evaluated by written theory examination.^[5] It consists of grand viva voce examination primarily focused on theory course. However, there are hardly any problem-solving and higher-order learning domains in this assessment. TVV is purely subjective and variation is high due to the variability of the examiner to examiner regarding their limited choice or content of questions. Furthermore, it is marred with biases (gender and racial) and needs two or more trained examiners.^[3] However, the objective structured viva examination (OSVE) described in 2005 by Oakley and Hencken is reliable oral assessment method to decrease biases.^[6]

The continuous attempts are made by the teaching faculty for the search of more valid and reliable methods such as e-assessment for harnessing the data to improve the outcome ^[7] The OSPE and OSCE have been described in 1975 and extended in 1979.^[8] The reliability of structured written theory examination (STE) was rated in 1997,^[9] while structured viva voce (SVV) was introduced in 2005^[6] as the new assessment tools for health-care graduates. The structuring of these assessment tools (SVV and STE) is obtained by breaking the essay type or viva voce questions into different parts starting with an action verb such as list, illustrate, or define and has interdepending leading connections. Moreover, each part of the single question is assigned with separate mark (e.g., 2+3+5 = 10 marks) weightage. The SVV has been shown as a valid and reliable tool that has been implemented in small group of students.^[10]

The structured essay questions of written exercises are considered suitable for pre-clinical subjects to evaluate higher-order cognitive processes, while OEs are conducted to gain the credible feedback by the subject experts.^[3,4] The objective of the study was to comparatively evaluate the structured tool of theory viva voce (OE) and written theory examination (structured essays) with respect to unstructured OE in third semester MBBS students.

Methodology

The study was conducted during the third semester examinations (sessional-I) of 2^nd year MBBS students as a formative assessment in 2017. The marks obtained in STE paper-1 (40 marks) and paper-2 (40 marks) were added to get the percentage, and it was named as STE which was considered as a benchmark for comparison. The course content was divided into two separate theory examination papers. The paper-1 contained general pharmacology, autonomic nervous system, local anesthetics, and smooth muscle relaxants. The paper-2 has cardiovascular system, anti-dyslipidemia drugs, shock, fluid therapy, diuretics, antidiuretics, and respiratory system.

A total of 135 students divided into four batches were assessed by SVV and unstructured oral practical assessment (UPA) over 4 continuous days following the written STE paper-1 and paper-2 in their semester examinations.Their results were analysed with respect to the intergroup variability responses between STE and SVV; STE and UPA taking STE as a benchmark and between SVV and UPA taking SVV as a benchmark.

The same students were also assessed in theory viva voce during the practical examinations without any limit on time by a CBE to ensure its reliability. Each question was divided into two parts: Initial query followed up by the leading question. The initial five questions were from the basic concepts of general pharmacology and fundamentals of other relevant systems. The next two questions were kept from the autonomic nervous system, respiratory system, diuretics, and antidiuretics. The last question was kept from the cardiovascular system, anti-dyslipidemia drugs, shock and fluid therapy, local anesthetics, and smooth muscle relaxants. The mark distribution was kept accordingly as 0.5 marks for each first ten basic queries responded satisfactorily (0.5 × 10 = 5), 0.75 for next four questions answered correctly (0.75 × 4 = 3), and the two division of the last question were kept as 1 mark each (1 × 2=2). The marks obtained in this section were designated as SVV. A checklist sheet of all students was prepared and kept hidden from the students. Only those boxes were ticked against the questions which were answered rightly. The final totaling of marks was done at the end of the examination to remove the further possibility of bias.

The viva was conducted for continuous 4 days. The students were divided into four batches. Each batch contained 33–35 students. The 20 same colored cards were made each containing equitable 16 queries. The students were asked to sit in front of the examiner to pick one card out of the shuffled lot which was kept facing downward as a method of random permutation. The process was double-blind as neither the students nor did the examiner know which card or set of questions would be selected. The viva voce examinations for 4 days were conducted by the same examiner to remove the teacher-teacher bias.

The structured tools were used in two assessment forums, traditional theory as well as viva voce examinations, to remove the disparities and examiner bias. It was done to possibly minimize the student-teacher, teacher-situational, and student topic bias in OE. An average time of 10–12 min was assigned for each student although there was no limit for replying the relevant questions. Each card contained separate sets of eight questions each split into two parts; the second was the leading question of the first one. The questions were set with increasing difficulty levels as per the Bloom’s taxonomy of educational objectives of cognitive domain having similar pattern mutually agreed by all the faculty members [Table 1].

Table 1: Framing of the structured viva voce cards containing 8 questions (10 marks)

The questions were set as per clinically and pharmacological relevance. The initial five questions were easy, next two were of moderate difficulty, and the last question was difficult. The entire faculty (assistant professor and above) agreed on the content, marking, difficulty index, and equitability of the cards for uniform assessment [Table 2]. The answer keys were prepared and assessed before the examinations started.

Table 2: The content of two cards used in this study as a structured viva voce assessment tool

The training of medical students to make them complete physician is by mastery of multiple competencies (criterion reflected test). Each competency is made up of a series of different specific learning objectives spanning the horizontal and vertical arrangement of subjects. A matching teaching, learning, and assessment tool is required for evaluating each educational objective.^[3-4,7]

The structured essay evaluates cognitive domains while OEs are used to assess attitude, skills, and communication. Although there is no gold standard for assessing wholesome development of a physician, structured written exercises are considered to be consistent with the student’s learning pattern. Therefore, most of the colleges and universities at least follow this assessment forum. The other forum such as structured or unstructured OE should correlate with written exercises for all the students to become a complete physician.

It is simply to suggest that each domain of learning is important but indispensable by other domain. Thus, percentage variance was used as intergroup as well as intragroup data comparison for cognitive, affective, and psychomotor domains of learning.

The same batches of students were evaluated on different occasions by three methodologies using assessment tools of STE, SVV, and UPA. The percentages of marks obtained in STE, SVV, and UPA were treated like three groups and compared for intergroup and intragroup variability. After calculation of percentage STE results, percentage SVV marks, and percentage UPA, the percentage intergroup variability was calculated as per the formula (% variation in assessment [theory vs. viva voce] = % assessment in theory − % assessment in viva voce/% assessment in theory × 100). This formula is very commonly used for calculating the percentage variability and is also available in Microsoft Excel program where percentage difference between the new number and the benchmark number is divided by the benchmark number (V₂−V₁/V₁ × 100). The variability results were compared based on the aggregate marks of these three parameters such as (a) STE versus SVV, (b) STE versus UPA, and (c) SVV versus UPA. The students were divided based on the results of percentage variability into six groups as in the earlier published study^[11] and designated as Group 1 (+100–+51); Group 2 (+50–−50) that was further divided into Group 2a (+50–+6), Group 2b (+5–−5), and Group 2c (−6–−50); Group 3 (−51–−100); Group 4 (−101–−150); Group 5 (−151–−200), and Group 6 (<−200) [Table 3]. The Group 1, Group 2, Group 3, and beyond (Group 4–Group 6) were designated as inappropriate, appropriate, inappropriate, and erroneous results, respectively.

Table 3: Appropriateness and distribution of the percentage intergroup variability

Results

The students in appropriate category are 72 (53%) students in %vSTE-SVV, 18(13%) in %vSTE-UPA, and 20 (14%) in %vSVV-UPA as shown in Table 3. It can also be observed that erroneous results are more in %vSVV-UPA (110, 81%). It clearly shows that the magnitude of appropriate assessment is comparable to the earlier study although it was compared with STE instead of TVV. However, it was better than the UPA. The SVV when compared to STE outcomes shows more students in appropriate category and less erroneous results. When UPA was compared to SVV, it showed very less number in appropriate category and 81% of students in erroneous category.

The average mean, standard deviation, and coefficient of variance are 62.78, 11.46, and 18.25 for UPA%; 37.33, 17.64, and 47.26 for STE%; and 36.53, 23.34, and 63.91 for SVV%; respectively. There is least discrimination between students in UPA% and more discrimination in SVV% as per the coefficient of variance values which are progressively increasing in the following order UPA% < STE% < SVV% [Figure 1].

Close

Figure 1:

Percentage of marks in structured theory examination, structured viva voce, and unstructured practical viva voce assessment

Export to PPT

All the three assessment tools were analyzed for intergroup correlation of student’s results in percentages to each other, and all of them were found to have a highly statistical significant correlation (P = 0.001). However, maximum correlation was between SVV and STE with correlation coefficient (r) = 0.74 [Table 4].

Table 4: Outcome of three categories of assessment tools (structured theory examination, structured viva voce, and unstructured practical assessment)

In addition to that, we also analyzed the result for variability of students’ answer pattern. We found that there were two such students who did not answer any question from the basic level queries but one student exclusively responded to one question in moderate section and the other student responded only in difficult section. There were 33 students who only answered the basic queries related to recall and recognition, and there was progressive decline in percentages of students answered the higher learning domain level questions [Table 5].

Table 5: Frequency of students responded to increasing difficulty levels of questions in structured theory viva voce

Discussion

The medical students follow the assessments as their only source of guidance to curriculum.^[12] It becomes the main motivation for learning and professional achievement. The change in learning is consistent with the change in assessment. Therefore, more and more assessment of learning is taking place rather than the testing of knowledge.^[13] It should assess the higher-order learning and competencies to make a person skilled doctor who will interact in ethical, professional, and accountable manner with the society.^[14]

The TVV has been restricted to only borderline pass/fail students and for the distinction holders in some of the developed countries as there are lots of factors resulting in variability in marking pattern in traditional oral assessment such as teacher-teacher, subject topic, situational, student teacher biases.^[15] Not many studies have determined the quantum of the problem and alternatives of its mitigation. Some studies reflect the opinions of the students regarding the TVV where they believe that it has a potential to be biased on many counts such as types of questions, predetermined image of the students, mood of the examiner, haphazard conduct, and predictability of questions from students to students,^[16] while other studies recommend objective SVV to be implemented in each medical subject of a large number of medical colleges.^[17]

Our study agrees with the previous studies where TVV does not collate with the theory assessment,and almost 60% of students are awarded marks erroneously as in our case where UPA is similar to TVV.^[18] SVV and STE show equitable erroneous results when compared to each other. Our study also shows that neither student is awarded 100% marks nor 0% in UPA% [Figure 1 and Table 4] and STE%. However, students can get 100% marks as well as 0% in SVV%. Thus, SVV is a good tool to discriminate students, objective uniform assessment, and scientific and satisfactory process.^[19]

It has been established that the STE reflects a better assessment of students than the traditional written test.^[9] Therefore, we have compared the STE with the SVV and UPA. Our finding correlates with the earlier studies where SVV correlates with traditional written theory more appropriately. Our finding supports the use of SVV as well as STE tool as a preferable methodology. In addition, erroneously examined proportion in structured component is very less in comparison to the other studies supporting the idea of structuring the assessment tools. Our finding also suggests that even the practical oral assessment should also be framed as structured for uniformity, appropriateness, and assessment of higher order of cognitive domain learning.

SVV provides advantages of wide coverage of topics and reduced anxiety, improves teacher-student relationship, and offers multiple choices, objectivity, uniformity, and opportunity to answer without threat and fear of the examiners.^[20-22] It has been established that fear and anxiety have an inverse relationship with the performance.^[23] SVV provides a comfortable environment and prevents the student–teacher and student–topic biases, and chance factor get minimized. There is more transparency, fairness, and positive environment in the medical school affecting the student when it is incorporated into the system putting the faith of the students back to the assessment system.^[24]

Conclusion

This study shows that STE outcomes are comparable to the SVV. The SVV and STE were found to have maximum correlation, suggesting that structured format tools are equitable, reliable, valid, and uniform for assessing student’s capacity and skills. However, least correlation was found between UPA and SVV. Therefore, traditional theory paper settings and TVV pattern should be replaced with structured pattern (OSTE and OSVE) in both the cases. In addition, UPA outcomes were least discriminatory with respect to the potential and capability of the students. Thus, all the four parts (written theory, oral theory, written practical, and oral practical) should be structured for better assessment of MBBS students.

The SVV examination pattern provides equitable questions, balance time, comfortable environment, wide choices for answering, uniform assessment, measuring precise capability, and objectivity and increases the faith of the students in assessment system. It diminishes such biases as teacher favoritism, student’s verbal skills, subject topic asked, and examiners mood on the day of assessment. It balances theory and practice with Evidence Based Assessment such as OSTE with OSPE and OSCE.