1. INTRODUCTION
Medical professionalism has been drastically challenged by profound changes in healthcare systems. Teaching and assessing professionalism in medical school plays an important role in preparing the medical graduates to cope with these challenges. In the assessment toolbox of professionalism, OSCE and Mini-CEX (POSCE and PMEX) have been used in assessing professional behaviors in several contexts. Particularly, a POSCE was developed for Family Medicine (FM) residents in the University of Medicine and Pharmacy, Ho Chi Minh city (UMP) in response to a professionalism training module. Simulation-based assessment is able to assess multiple attributes, “Honesty”, “Keeping confidentiality”, moral reasoning or involving patient in decision making, to name a few [7]. Nonetheless, it is little known about the content validity to support the usefulness of POSCE in measuring professional attributes.
For the simulation-based assessment that relies on standardized patient (SP), several evidence must be documented and presented according to the framework of Downing, 2003 and Jonsson and Svingby, 2007 to prove that the assessment has the content validity [1]. These evidence primarily showed whether all components such as cases, item lists and grading rubrics of this assessment were based on expert and real practice [1, 4]. Also, the representativeness of the assessment to the construct of interest is another evidence.
This study aims at investigating content validity of Vietnamese POSCE by gathering these above mentioned evidence.
2. METHOD
A cross-sectional study was conducted in following stages.
The POSCE was developed and administered in the Training Center of Family medicine (FM) which is a training unit of the University of Medicine and Pharmacy (UMP), Ho Chi Minh city, Vietnam from September, 2014 to August, 2015.
The four-stage process for developing POSCE blueprint, cases, marking items and rating rubrics was analyzed to collect evidence of content validity according to Downing et al, 2003 and Jonsson and Svingby, 2007.
Moreover, an independent expert panel who had no involvement in the process of making POSCE including 5 members including 3 local FM faculty members and 2 international experts were invited to evaluate the test blueprint, cases and item lists. Foreign experts in Global health and Family medicine from the University of Boston, United State and University of Liege, Belgium were invited due to their long period of collaborating with many medical universities in Vietnam including UMP for over ten years in developing Family medicine training in Vietnam. For their convenience, the test blueprint, cases and item lists were forward- and back-translated into English.
POSCE comprised six stations for testing six professional competencies,”Keepingconfidentiality”,”Providingaccessto healthcare for underserved community”, “Disclosing medical errors”, Breaking bad new”, “Making altruistic decision”, and “Admit one’s self-limitation. These competencies belong to six professional attributes, “Respecting the patients”, “Honesty”, “Empathy-Compassion”, “Responsibility to the community”, “Altruism” and “Self-awareness of one’s limitation”.
3. MATERIALS
A five-point rating form, (1) totally not agree to (5) totally agree, in which the expert panel rated POSCE components such as test blueprint, scenarios and marking items against predetermined criteria focused on (1) relevancy, (2) representativeness and (3) adequacy to assess the construct of professionalism and cultural context.
4. RESULT
Figure 1 presents the developing process of POSCE. Four core components of POSCE including test blueprint, cases, marking items and grading rubrics were developed one by one and through three main steps, composing-revising-reviewing. All components of POSCE were composed and revised based on content experts’ opinion and findings from educational studies. The American Board of Internal Medicine (ABIM) definition, ABIM taxonomy and a study of Vo, et al., 2014 on professional attributes in Vietnamese context, were referred to in composing the test blueprint including professional attributes and competencies to be tested as shown in Table 1. Before releasing the final version of each component, expert panel independent from two above steps, conducted the final review. Notably, the grading anchors in the rubrics were described with verbal and non-verbal expression that sampled the actual practice. In summary, most evidence adherent to validity framework of Downing et al, 2003 and Jonsson and Svingby, 2007 can be found in this process. Table 2 summarizes all evidence of content validity of POSCE
| Sources of content validity evidences | Documented in FM POSCE | |
|---|---|---|
| Downing et al., 2003 [1] | Proof that each of the SP cases fits into a detailed content blueprint of professionalism attributes | X |
| Content-expert agreement that these specific cases are representative of professionalism relating cases in outpatient settings | X | |
| Expert clinical faculty have created, reviewed and revised the SP cases together with the checklists and ratings scales | X | |
| The SP cases were reviewed and critically critiqued and competently edited | X | |
| During the time of administration, the SP portrayals are monitored | X | |
| Jonsson and Svingby, 2007 [4] | Rubric is used for qualitative rating of authentic work. Analytical scoring with anchors which use actual work samples improves validity | X |
According to ABIM, medical professionalism encompasses multiple attributes and each of these can be measured by different competencies. Table 3 describes to what extent the panel review rated these attributes and competencies against five criteria. The expert panel evaluated that these attributes tested in this POSCE belong to the construct of professionalism and they adequately sample this construct. Also, they agreed that all competencies strongly demonstrated the tested professional attributes. For instances, the competency of “Disclosing medical error” strongly indicates the doctor’s “Honesty”. Notably, the expert panel might have differed in their opinion whether there have been enough competencies and cases for assessing professional attributes.
The experts’ score on rating the cases and the checklists were shown in Table 4
Scenario four and six were rated to be highly relevant for testing the attributes of “Compassion” and “Self-awareness of limitation” (M=4.40, SD=0.55) whereas scenario five was considered to relatively represent “Altruism” (M=3.55, SD=1.29). Moreover, all scenarios were rated as culturally relevant. The case of “Keeping confidentiality” was rated to be highly relevant to Vietnamese culture. However, the case of “Disclosing medical errors” which is used to measure Honesty was given the lowest mean score of relevancy to Vietnamese culture (M=3.67, SD=0.58).
The expert panel agreed that items for each scenario were adequate. Also, they agreed that most items, developed for each scenario are indeed, the appropriate steps to resolve that scenario. However, steps for resolving the scenario of “Altruism” was rated to be the least relevant to the case (M=3.00, SD=0.82).
Regarding the relevancy to Vietnamese culture, the items in “Disclosing errors” and “Altruism” case were rated to reflect Vietnamese culture relatively (M=3.67, SD=0.58; M=3.33, SD=0.58). Otherwise, the items of the remaining cases were rated to strongly reflect the local context.
5. DISCUSSION
There were evidence that sources of expertise, literature and practice were considered comprehensively in developing POSCE cases, checklist and rubrics. For instances, attributes and competencies to be tested were selected based on framework of Nhan, Vo et al, 2014 and ABIM Taxonomy [7]. Moreover, clinical dilemmas in Vietnamese outpatient settings were used to develop the cases. Descriptors in the grading rubrics were taken from FM residents’ sampling responses in POSCE pilot. As for developing the marking items in POSCE, the method used in this study also resembled the method of developing checklists suggested in a previous study [2].
The results of expert panel review also indicated the representativeness to the construct of professionalism and cultural relevancy of POSCE. Particularly, scenario four and six were rated to be the most relevant for testing the attributes of “Compassion” and “Self-awareness of limitation”. This can be explained as follows. Scenario four was developed to assess the attribute of Empathy-Compassion. The chosen task was delivering the diagnosis of gastric cancer to a 50-year old man. A growing body of evidence showed that physicians more likely expressed their sympathy and concern in disclosing a diagnosis of terminal illness to patients. Therefore, delivering bad news is an opportunistic moment to evaluate doctors’ expression of compassion toward a patient. The sixth scenario represented family doctors’ duty of managing Hand-foot-mouth disease in children. Children were normally brought to the commune health center (CHC) presenting with severe symptoms such as high fever and seizures and require referral to higher level of fully-facilitated health institutes. Therefore, observing residents’ performances in this scenario can provide evidence of their own awareness of limitation.
Notably, the scenario of “Keeping confidentiality” was rated the most relevant to Vietnamese culture. It is natural in Vietnam that women become caring for all members in their families as much as possible. This nature urges the women to inquire about the other family members’ personal health information. Normally, doctors encounter that their patients’ wives, mothers or daughters request them to disclose their relatives’ healthcare information without their consent. This situation was replicated in the scenario of “Keeping confidentiality”.
However, the adequate numbers of attributes and cases in POSCE to assess professionalism seemed to remain debating among experts. This finding supports the wide-spread viewpoint that no single assessment can measure the broad construct of professionalism adequately [5].
6. CONCLUSION
FM POSCE can assess six specific professional attributes with adequate content validity based on the several evidence of content validity found in the POSCE developing process. This study suggested that developing the POSCE requires a process with input from various sources such as expertise or medical literature to improve the content validity.
