co-organiser of the Ottawa 2020 Conference “Assessment of health professionals
and evaluation of programmes: Best practice and future development”. Ottawa
2020 provides an international forum for all involved in assessment and
evaluation throughout the continuum and across all the healthcare professions
to come together to discuss current practice and future developments. State of
the art sessions by experts in plenaries, workshops and symposia are integrated
with presentations from participants from around the world who have come to
share their experiences and to learn from those of others. Teachers in the
undergraduate curriculum, postgraduate trainers, CPD providers, assessment
specialists, students, deans, curriculum planners, researchers and
administrators are all represented and sessions relevant for them are included
in the conference programme.
important work on Consensus Statements and Recommendations as to best practice
in assessment and evaluation will be continued at Ottawa 2020, with the
opportunity for all participants to take part in this initiative in the areas
of Performance Assessment, Technology in Assessment and Programmatic
than 1,000 participants from over 50 countries expected at the conference, both
those highly experienced in assessment and evaluation and those new to the
area, we believe Ottawa 2020 is a conference not to be missed.
Please see the website www.ottawa2020.org for the provisional programme, registration and abstract details. Abstracts for oral presentation, poster presentation and conference workshops should be submitted by 15 September. The earlybird registration ends on 15 December.
In this wide-ranging conversation, Vanessa and I discuss her 25 years in health professions education and research. We look at the changes that have taken place in the domain over the past 5-10 years and how this has impacted the opportunities available for South African health professions educators in the early stages of their careers.
“That for me is probably the most rewarding thing about my own career; not what I’ve achieved but what I’ve seen others achieve.”
In this episode, we take a slightly different perspective to the topic of health professions education. Instead of speaking to someone who has completed a PhD in HPE, I talk to Vanessa Burch, who has spent almost all of her career establishing HPE as a field of study in the South African context.
Vanessa has a National Excellence in Teaching and Learning Award from the Council of Higher Education and the Higher Education Learning and Teaching Association of South Africa (HELTASA), and holds a Teaching at University (TAU) fellowship from the Council for Higher Education of South Africa. She is a Deputy Editor at the journal Medical Education, and Associate Editor of Advances in Health Sciences Education. Vanessa was Professor and Chair of Clinical Medicine at the University of Cape Town from 2008-2018, and is currently Honorary Professor of Medicine at UCT. She works as an educational consultant to the Colleges of Medicine of South Africa.
In this wide-ranging conversation, Vanessa and I discuss her 25 years in health professions education and research. We look at the changes that have taken place in the domain over the past 5-10 years and how this has impacted the opportunities available for South African health professions educators in the early stages of their careers. We talk about developing the confidence to approach people you may want to work with, from the days when you had to be physically present at a conference workshop, to explore novel ways to connect with colleagues in a networked world. We discuss Vanessa’s role in establishing the Southern African FAIMER Regional Institute (SAFRI), as well as the African Journal of Health Professions Education (AJHPE) and what we might consider when presented with opportunities to drive change in the profession.
Finally, we talk about Vanessa’s method of writing. As a prolific researcher and author, I thought it’d be useful to learn how she approaches writing and publishing, and she shares some really practical advice on how to write intentionally. Even though we didn’t cover Vanessa’s own research interests, you can get a sense of her research activity by visiting her profiles at ResearchGate, LinkedIn and Google Scholar.
In this episode of the SAAHE podcast I speak to Simone Titus about her PhD research project on the use of game-based learning. Simone talks about how this approach can lead to improved student engagement and collaboration, as well as some of the challenges she faced. She also describes how she wrote her final thesis, including the final year in Dublin with a mobility funding grant.
Simone Titus is a teaching and learning specialist in the Faculty of Community and Health Science at the University of the Western Cape. She graduated with a PhD in Education from the University of Cape Town where she developed an interest in the use of emerging technologies as a tool to mediate learning.
Her special research interests are focused on game-based learning and using emerging technologies to foster cross-cultural interaction, learning and engagement in higher education. During the past seven years, Simone has taught undergraduate and postgraduate students and she has a growing number of Masters and PhD students who are under her supervision. Her current portfolio involves developing teaching and learning strategies in health science and interprofessional education. She has been the recipient of, amongst others; the South African and Netherlands Programme for Advanced Development (SANPAD), AESOP Mobility to the University College of Dublin and has been awarded grant funding for various other projects.
On the 12th of September, the Western Cape SAAHE regional committee hosted a half-day seminar on the topic of assessing “soft” skills. We had 40 participants who spent a few hours discussing different approaches to assessing things like professionalism, interpersonal skills and graduate attributes. This post presents the notes taken during the discussion from the seminar, with some additional information added after the fact, together with links to external sources. We share it here in the hope that others may find it useful.
The point was made that these should perhaps not be called soft skills given that soft has the unintended connotation of being “unimportant” or “easy”. Should these be referred to rather as: Complex skills? Intrinsic roles (e.g. as described in competency frameworks)? Graduate attributes? Critical cross-field outcomes?
Tools that can be used in the context of a course / module / block
Has the added advantage of boosting students’ confidence but we also need to bear in mind that patients may need to be prepared in advance. In the experience of some, patients tend to be overly complimentary about student performance (e.g. “everything was perfect”).
Scope of undertaking: No indication of how many students
UWC Occupational therapy department developed a matrix over four years considering the following: UG graduate attributes and core competencies; Weighted according to the four years; Used Bloom’s taxonomy as a guide for levels e.g., know about communication in the first year; application and integration in the fourth year.
Scope of the undertaking: About 50 per year level
At the end of every term or every semester, lecturers sit and hand over what was covered and what still needs to be covered in upcoming modules to bridge gaps in learning, module expectations and what students struggle with. Individual students who are struggling are raised in regular staff meetings
Scope of the undertaking: 50 students
Create opportunities for students to develop but then don’t actually assess
It is questionable with respect to the degree to which these things can be taughtThese opportunities might consist solely of 1) do something, 2) get feedback on what you did, 3) reflect.
Practicalities of deploying approaches
Designing and managing a distributed curriculum
Nursing at UWC has Interpersonal skills sessions for first-year students e.g. death and dying; washing a baby – observation of how student washes baby, makes eye contact with the baby, etc.
three times a week; four scenariosuse simulated patientsfacilitated by mentorsfizzles out in later yearsScope of undertaking: over 1000 students in programme; 300-350 in first year
UCT has two courses that run in the first year and incorporate soft skills: Becoming a Professional and Becoming a Health Professional (all disciplines are working together in these courses)
A challenge is whether the principles addressed here are followed through in later years, particularly in the clinical yearsIn addition, there is some evidence that isolated modules like this aren’t very effective at developing what is essentially an integrated skill that must be used across domains
Understanding fully what you are trying to help students to learn
For example, iare communication skills about mastering isiXhosa as a language or about understanding culturally appropriate communication e.g. the role of eye contact and body position in communication? We need to understand the role of cultural humility (see here and here for more information) in the context of understanding and assessing soft skills – ensuring that students recognise that difference of any sort has an impact and moderating yourself and your behaviour accordingly.Role modelling is very important in this regardAre we assessing things like “punctuality” and “dress code” under the broad name of “professionalism”?
Symbolism is important – soft skills should be communicated as being integral rather than peripheral
Shouldn’t be “putting it on the side as a different mark” on a rubricAll aspects of the curriculum should be integrated; a signal that clinical expertise is necessary but not sufficientAssessment system should be designed in such a way that a student cannot pass on clinical reasoning alone yet have poor communication skills or empathy (see below)Using soft skills tools that do not contribute to decision making can be a two-edged sword e.g., if Nursing students must log 4000 clinical hours, this equates to an impact curriculum; student focuses on the technique so much they forget there is a patient in the bed; rote behaviours are emphasised, rather than soft skills, in order to demonstrate the technique (think here about the quote adapted from Rees & Knight, 2007: “Do we want [health care workers] who are professional, or will we settle for [health care workers] who can act in a professional manner?”)Important to consider the instrument and purpose of the assessment; if the student doesn’t understand that communication is part of the manual technique, then they will not focus on / address / bother with communication skills.
We could consider using longitudinally collected narrative data obtained from clinicians, patients, peers, etc.
We currently compartmentalise information about students because we have modular systems within universities and institutions There is little or no longitudinal collation and consideration of data
Consider narrative, qualitative approaches to collecting assessment data: Maybe each clinician writes a short paragraph about the students’ behaviour e.g. “the student rolled their eyes a bit every time I spoke to them” – this says something about a student’s professionalism without having to assign a number to a definition of what “professional” means.
once in 4 years or if three people out of 25 have a problem with an aspect of student behaviour, maybe it’s not an issue; but if 18 out of 25 note issues and make qualitative, open-ended claims about student behaviour, that is a different story
An example of a Physiotherapy school in New Zealand collating narrative feedback alongside quantitative data was discussed in some depth in a podcast you can listen to here.
The individual preceptor or tutor isn’t burdened with making pass-fail decisions about a student – all they have to do is write a short narrative where they don’t even have to define the behaviour i.e. is this about “communication” or “professionalism”? All they need to do is describe an observed behaviour.
lowering the stakes for the preceptors
This helps us to evaluate the aggregate picture over time rather than make decisions based on single snapshots of performanceProgrammatic approach to assessment suggests applying criteria used to illustrate rigour for qualitative research – trustworthiness; credibility; dependability; confirmability
How do we use the information?
Only make decisions with high resolution data
Can’t make defensible decisions about soft skills using tiny amounts of data collected on a single clinical placementCees van der Vleuten uses the idea of the pixel resolution of the “picture” we have about a student. He suggests that we gather and collate small numbers of pixels over time to generate a high-resolution picture of what is going on.
This takes pressure off any one clinician as being “responsible for” a student’s failure
Helps avoid the problem of “failure to fail”
If I have to make a defensible decision, I rather make no decisionThen, the student reaches the final year and everyone agrees the student shouldn’t be there
Each clinician / tutor / preceptor just contributes their 3 or 7 or 4 pixels worth of data towards building the composite picture, without the pressure of pass-fail decision making attached to the data collection
These small pixels don’t form the basis for any single decision but are rather summed over the duration of the programme; so, rather than “professionalism” counting for 5% of a single assessment task, you have many tasks where those marks are added up over time.
Separate data collection from decision making
this kind of principle could have systemic implicationsfor all of the principles we mention, there are implications for programme design i.e. it’s hard to fit these ideas into our traditional curriculum models.
Use collective decision making
Anonymise the observations; give them to a panel of five clinicians to make a judgement about competent / not e.g. is this someone you would hire? No? Not competent.Is it defensible?
if you have 60-70 observations from 23 preceptors over 2 years, it will be (again, consider qualitative criteria for rigour as mentioned above)once such a system is in place, there are bound to be legal challenges but we cannot let that dictate what is sound practice and our current system faces legal challenges anyway – it is not as though it is perfect
Avoid spending too much time on students who are clear fails or clear passes (i.e., spending hours “agreeing vigorously”); rather focus energy on borderline students eg 5% or 1SEM or some such measure either side of the cut score
Incorporating this into assessment programmes and bureaucratic systems
This should be incorporated into rules of progression or for the award of a qualification rather than used at module level
We need to circumvent the tyranny of an assessment bureaucracy that demands a numerical mark to enter against a credit bearing course code on the university’s computer system
Options to consider
make a “competent” judgment (don’t use numerical scores) for each of a series of soft skills it a prerequisite / DP requirement for access to the next year of study or to the exit exams
make a “competent” judgement for each of a series of soft skills a subminimum requirement for the award of a qualification (irrespective of scores in other domains)
this would ensure that there are meaningful consequences for not being competent in soft skills that are valued in the programme and ensure that the student who scores 80% overall but is very unprofessional or very poor communicators does not qualify
Could convert letter grades (what the assessor gives e.g. A = Excellent, B = Good, C = Borderline, D = Weak) to number grades (what the system needs to be captured e.g. 80%, 70%, 60%, etc.)
This would require some planning about what kind of remediation to put in place
This award is made for a full-length article or a chapter, published in the five years to 31 Dec of the previous year, with a South African first author. (This will change in 2019 with Zimbabwe joining as a region). This year, for the first time, a joint award was made.
This publication summarises empirical research on the inclusion of sexual orientation and gender identity-related health disparities and health concerns in the health sciences curricula at the University of Cape Town. Understanding the exclusion of these topics has direct practical relevance to South African health professions education: sexual orientation and gender identity are important social determinants of health (Logie, 2012; Pega and Veale, 2015), and healthcare providers thus need the knowledge, attitudes and practical skills to provide competent care to lesbian, gay, bisexual and transgender patients. The American Association of Colleges of Medicine released a detailed guide on how to integrate topics related to sexual orientation and gender identity into medical curricula (Eckstrand and Sciolla, 2014). Understanding the current gaps in our curricula is the first step full-length curriculum reform that meaningfully incorporates these topics.
paper addresses the identity shifts that students in an academic
development programme undergo. Whilst such programmes endeavour to
support the notion of widening access, and thereby offering students
support to overcome learning barriers, cognisance should
be taken of the psychological toll this has on students’ well being.
The transition from school to university poses particular problems,
specifically with regards to alienation. Many students encounter an
unfamiliar dominant academic culture that does not
foster a sense of belonging. Educators should be aware the alienation
could impact negatively on students’ performance – especially if these
students are further put into support programmes that separate learning
activities from the majority of the student
The 2018 SAAHE Distinguished Educator Award was made to Prof Fatima Suleman. Fatima is Professor in the Discipline of Pharmaceutical Sciences, in the School of Health Sciences at the University of KwaZulu-Natal (UKZN). She has demonstrated a commitment to the development of teaching, mentoring and supervision skills in new academics and assisting other academics with developing their Teaching Portfolios for Performance Management and Promotion. She has developed international partnerships for undergraduate and postgraduate teaching, within and beyond Africa. At UKZN, she conceptualized and coordinated the development of two completely online Masters programmes in Pharmacy and Health Sciences. Fatima has been involved in various international grants to develop health professions education. With a NORHED grant, she facilitated the development of online or blended Masters programmes in Mozambique and Malawi; she was involved in the Medical Education Partnership Initiative grant; she was a panel member on WHO Technical Working Group on Health Workforce Education; and Principle Investigator Council for AFREhealth.
Other awards she has garnered include the Distinguished Teacher Award for UKZN in 2010; the Distinguished Teacher award by Academy of Pharmaceutical Sciences of South Africa in 2011; and a TAU Fellowship 2015 (Teaching Advancement at University (TAU) Fellowships).
In this episode, I talk to Dr Mpho Jama about how a humanistic pedagogy could be key to facilitating student success through enhanced support. She suggests that it is in the human relationships between teachers and students that we must look to provide higher, more subtle levels of support for students.
Dr Jama is the head of the Division of Student Learning and Development in the Faculty of Health Sciences at the University of the Free State. Mpho does research on student retention, Humanistic pedagogy and Qualitative Social Research. Her PhD thesis is entitled: Designing an academic support and development programme to combat attrition among non-traditional medical undergraduates.
Jama, M. & Beylefeld, A.A. (2007). “Thou shallt know thy student”. What pre-university attributes characterised the first-year medical students that were denied examination access in 2007, and what competencies did they lack? Poster presentation.
In order to graduate physiotherapy students who are able to thrive in increasingly complex health systems, professional educators must move away from instrumental, positivist ideologies that disempower both students and lecturers. While the potential for pedagogical transformation via the integration of digital technology is significant, we must be critical of the idea that technology is neutral and be aware that our choices concerning tools and platforms have important implications for practice.
Earlier this year the Critical Physiotherapy Network published Manipulating practices: A critical physiotherapy reader. The book is a collection of critical writing from a variety of authors dealing with a range of topics related to physiotherapy practice and education. One of the interesting features of this collection is that it is completely open access, which means that the authors, and not the publishers, have the intellectual property rights to make choices about what is permissable to do with the content of the book. While the entire book is available in different formats, including PDF, HTML, EPUB and XML, there is no audio version.
This SAAHE podcast is a recording of one chapter in the collection, entitled “A critical pedagogy for online learning in physiotherapy education“. We are using the SAAHE blog to experiment with sharing content in different formats, and would love to hear your feedback on whether or not this is something you would like to see more of.
In order to graduate physiotherapy students who are able to thrive in increasingly complex health systems, professional educators must move away from instrumental, positivist ideologies that disempower both students and lecturers. Certain forms of knowledge are presented as objective, value-free, and legitimate, while others – including the personal lives and experiences of students – are moved to the periphery and regarded as irrelevant for professional education. This has the effect of silencing students’ voices and sending the message that they are not in control of their own learning. While the integration of digital technology has been suggested as a means for developing transformative teaching and learning practices, it is more commonly used to control students through surveillance and measurement. This dominant use of technology does little more than increase the cost-effectiveness and efficiency of information delivery, while also reinforcing the rigid structures of the classroom. Physiotherapy educators who adopt a critical pedagogy may use it to create personal learning environments (PLEs) that enable students to inform their own learning based on meaningful clinical experiences, democratic approaches to learning, and interaction with others beyond the professional programme. These PLEs enable exploration, inquiry and creation as part of the curriculum, and play a role in preparing students to engage with the complex and networked systems of the early 21st century. While the potential for pedagogical transformation via the integration of digital technology is significant, we must be critical of the idea that technology is neutral and be aware that our choices concerning tools and platforms have important implications for practice.
In this episode I speak to Corné Postma from the University of Pretoria. We discuss his PhD research where he looked at the use of case-based learning to develop clinical reasoning in undergraduate Dentistry students. Corné used both quantitative and qualitative data to determine that students’ clinical reasoning ability improved after using a case-based approach to learning.
Corné is an Associate Professor in the Department of Dental Management Sciences, School of Dentistry, at the University of Pretoria. He is a specialist in Community Dentistry by training and his primary teaching responsibility lies in the domain of Comprehensive Patient Care, which includes patient communication, patient administration, clinical reasoning and patient management. He is also involved in developing other non-clinical skills such as self-awareness, ethics, professionalism, leadership, team work and health advocacy skills in dental students.
Corné has a very broad clinical research interest, which correlates with the generalist requirement of Comprehensive Patient Care. He has a particular affinity for health professions education research, which is closely linked to the development of different kinds of soft skills in students. His research outputs can be viewed on Google Scholar. Corné is a SAFRI (Sub-Saharan African Foundation for the Advancement of International Medical Education and Research Regional Institute) as well as a TAU (Teaching Advancement at University) fellow.
In this episode of the SAAHE podcast I speak to Prof. Scarpa Schoeman, Director of Undergraduate Medical Education at the Wits Medical School, Faculty of Health Sciences, University of the Witwatersrand, where he leads and directs the Graduate Entry Medical Programme. Scarpa and I talk about the (almost) universal pass mark (cut score) of 50% and the problems with this as a standard. We also discuss possible alternatives to standard setting that take into account the validity and reliability of the assessment scores, as well the difficulty of the test.
Scarpa has published a variety of peer-reviewed articles and presented at international conferences on the topic of medical education and assessment. His research interests include assessment and standard setting (the Cohen method in particular), as well as the educational environment for medical students. His clinical interests and practice focuses on Emergency Medicine. He also acts as Assessment consultant to the Colleges of Physicians, Obstetrics and Gynaecology, Paediatricians and Anaesthetists of South Africa. He is a Fellow of the Higher Education Academy in the United Kingdom and is a part-time tutor in Assessment and Standard setting for the CME at Dundee University.