Contact Form

Contact us if you have any questions

Inter-Rater reliability of a professionalism OSCE developed in family medicine training University of Medicine and Pharmacy

Original Research
  • Pham Duong Uyen Binh, Pham Le An, Tran Diep Tuan, Jimmie Leppink,
  • Pages 20-24

A POSCE was developed and administered in 2015 to assess six professional attributes for the Family Medicine (FM) residents, University of Medicine and Pharmacy (UMP), Vietnam. This study aims at exploring inter-rater reliability in FM POSCE developed in this context when analytic rubrics were applied.

Background: Past POSCEs showed raters’ variability on applying the global marking items and holistic rating. Using analytic rubrics, unlike holistic type, will provide more rationale for assigning a certain score might influence raters’ variability. Nonetheless, it is little known to what extent switching to this rubric type might influence the inter-rater reliability of POSCE.  

Methods:  Before the FM professionalism module (pretest) and after this module (posttest), 36 and 42 FM residents took the POSCE respectively. The raters in the pretest included 12 teachers of FM training center. Four faculty members from different faculties were belatedly added to the post-test together with the 12 former raters.  Raters’ training occurred in two different times, the former took place only for the 12 FM raters before the pretest and the latter was before the posttest for the 4 belatedly-recruited. During the POSCE, one pair of raters observed all performances per station. Inter-rater reliability was measured by the differences in total scores between raters per pair using paired t-test and Pearson correlation coefficient.  

Results: In POSCE pretest, no significant difference was found between raters’ scores in most pairs of raters, contrasting with that in the posttest. Most differences were noticed in the pairs of raters, in which one of the raters was the belatedly-recruited. In the pretest, moderate to strong positive correlation between raters’ mean scores were found (r=0.55-0.85), similar range was seen in the post-test (r=0.47-0.87), however, the correlation slightly weakened.   

Discussion and conclusion: The FM POSCE has high inter-rater reliability on the utilization of analytic grading rubrics. An analytic rubric might help minimize the discrepancies among raters. Moreover, training raters might have been an alternative influential factor on the raters’ consensus.

Graphical abstract

Content validity of a professionalism OSCE developed in family medicine training University of Medicine and Pharmacy at Ho Chi Minh city Vietnam

Original Research
  • Pham Duong Uyen Binh, Pham Le An, Tran Diep Tuan, Jimmie Leppink,
  • Pages 6-11

Background: Assessments of professional behaviors such as professionalism Mini clinical examination (PMEX) and OSCE (POSCE) are playing an important role in driving the practice professionalism in medical training. Simulation-based assessment or POSCE has been used to evaluate several professional attributes. However, few evidence of content validity proving that whether POSCE is really measuring specific professional attributes have been reported in medical education literature.

Methods:  The four-step process of developing FM POSCE was analyzed to highlight the validity evidence according framework of Downing (2003). Group of 5 independent FM experts from Vietnam, Boston University, US and Liege, Belgium evaluated the blueprints, scenarios and item lists on a scale (1-totally disagree to 5-totally agree) regarding to what extent test blueprint, cases and item lists were relevant to the content domains and cultural context. The results of their evaluation were considered as an evidence of content validity. The mean and standard deviation of the scores given by them were calculated using SPSS, 20.0.

Results: Important evidence of content validity were found in the process of developing POSCE. Content experts’ evaluation showed that all professional attributes represented medical professionalism. However, the adequacy of professional attributes to evaluate the broad construct of professionalism was controversial (M=3.75, SD=0.95). Cases are relevant to assess these professional attributes and culture. Only the cases of “Respecting the patient” and “Making altruistic decision” contained some inappropriate marking items (M=3.75, SD=0.95; M=3.00, SD=0.92).  

Discussion and conclusion: FM POSCE developed in Vietnamese context can assess six specific professional attributes. This study suggested a process of developing POSCE that has several features such as using both sources of expertise and medical literature to build up the content of POSCE to improve the content validity.

Graphical abstract