Mosr professions initially developed on the basis of trust. Examinations were rare. Doctors (and other professions) learned through experience. For healthcare, as science advanced and increased in importance, examinations in basic sciences were introduced. Thereafter, once qualified, further advance depended upon the trust of patients and other doctors. Even when I trained, the only examination after qualification was the MRCP. All other junior posts were obtained because other doctors were prepared to say that one was competent and trustworthy. The next phase (1997 and 2020) involved more formal assessment of competence in a defined range of activities. However, as early as 2005 a move started towards establishing entrustability, trusting a doctor to undertake complex and prolonged activities. Having left trust for the apparently more sound idea of competence, we have returned to trust, but now trust based on evidence. This post focuses on three articles in the July 2021 special issue of Academic Medicine, summarised in this overview. (here)
Context – education and outcome
Education and rehabilitation are similar in many ways – (here) – and an additional similarity is that the important outcomes are difficult to define and measure.
In rehabilitation the important outcomes are, ultimately, determined by the patient. They usually relate to social roles and interactions, emotional state, and satisfaction. In healthcare education, the important outcomes are defined by two groups of people: those who employ healthcare professionals; and those who depend upon the person’s professional expertise, usually but not only patients. The quality of the outcome depends upon a mixture of inter-personal skills, specific knowledge coupled with the ability to use it well, and specific technical skills.
I trained under an apprenticeship model. The consultant or consultants I worked with taught and I learned, but there was no assessment. When applying for a post, they would, one hoped, say that one was competent and safe, and ‘good at the job‘. Other less relevant factors also influenced decisions. In educational terms, this is equivalent to just saying that a student ‘attended the course and seemed to learn’.
Ten years later, when I started training and educating trainees, the emphasis was on teaching and assessing competencies. The assessments were initially vague. The trainee’s competence was usually rated by reference to expectation:- being below, at, or above the level expected ‘for the stage of training‘, but the expected levels were never defined or explained.
There was a gradual move to measuring competence in terms of being competent to undertake the activity independently, where rating this was a realistic option. This in turn led to more attention being paid to activities that could be measured, at the expense of other activities that were important.
Two problems arise. The first is the increasing number of competencies identified and specified. The recent rehabilitation medical syllabus (here) identified 39 competencies, and many of those could include many lower level competencies, which could also be specified and measured.
The second problem is more serious. As professional expertise becomes more and more atomised, the learner and the educator forget the real purpose of being competent at a discrete activity, which is to undertake some complex and unpredictable high-level activity. In rehabilitation, for example, a person might be competent at selecting and using many individual assessment tools, but they might never be able to undertake an assessment of a patient that informs the formulation of the situation, because the person fails to appreciate the reason for assessment.
Another example comes from a time, 25 years ago, when I was an external assessor evaluating a speech and language training course. The external assessors met with recently qualified graduates of the course who all said that their first few months working in the NHS had been terrible because, for example, they had no idea how to run an out-patient clinic, nor could they diagnose and manage a patient who did not have a classical and isolated language impairment. The presence of cognitive losses, additional dysarthria, tearfulness, and incontinence completely overwhelmed them. Training has moved on since then!
High level outcomes
One response to this excessive focus on discrete activities was to reframe accreditation of being qualified as achieving a few ‘high-level outcomes’, activities that encompassed many individual activities within a complex whole. A high-level outcome in rehabilitation would be for a patient to undertake a shopping trip. This requires a person to have many skills – planning and organising, remembering, communication and inter-personal skills, mobility and dexterity etc.
In an educational setting, the expert rehabilitationist should be able to draw up a plan to guide rehabilitation over a few months. This depends upon knowing about disease and its prognosis, knowing or discovering the available rehabilitation options, establishing the patient’s priorities and wishes, using information from all other team members, and prioritising the actions of the team.
In medical rehabilitation training, the training outcomes have now been restricted to about 14 high level outcomes in each speciality, of which six outcomes are generic, the same in every medical speciality. The remaining eight outcome define the speciality. These speciality-specific outcomes are discussed elsewhere on this site. (here and here)
The essential characteristics of these high-level outcomes, known in medical specialities as Capabilities in Practice, are that they:
- are complex, integrative activities that typically take a long-time (hours);
- involve a large number of separate competencies;
- usually involve working with other people; and
- are not easily measured using any single measure.
This last characteristic – the difficulty in measurement – poses a challenge, because patients, employers and regulators all wish to know that the patients will receive a safe, effective, and efficient service from the doctor.
Concerns with using assessed competencies as the best way to evaluate doctors in training emerged about 15 years ago. Gradually the idea of entrustment emerged. An overview of entrustment published in 2016, gives information about its evolution and basic features. (here)
The essence of the arguments is that:
- patients necessarily are vulnerable, and cannot control or supervise the doctor’s actions; and so
- patients have to trust doctors to maintain their (the patient’s) interests and safety at all times.
At the same time, senior doctors undertaking training have to trust that their trainees (and, indeed, other medical and non-medical colleagues):
- will act to maintain patients interests and safety at all times; and
- will ask for help or support when they cannot manage without support.
The trainer will gradually entrust work to the trainee, allowing them to work independently. When they have sufficient evidence, through experience, that indeed the trainee can be fully trusted at the activity, then the trainer is said to have grounded trust, and the trainer can say that the doctor is entrusted to undertake that activity,
This system of assessing a doctor’s expertise does not imply that competence is unimportant. In order to undertake the complex, high-level activity safely and successfully, the doctor will need to draw on a whole range of knowledge and skills. A doctor training in respiratory medicine would never become entrustable if they were not competent to place a chest drain safely. However, just because they can place a chest drain, they would not be entrustable. They also need, for example, to judge when it is needed or not needed, what the risks and benefits are for the patient, what to assess over time, when and how to remove it etc.
Entrustability requires a whole range of knowledge and skills that cannot easily to assessed and measured using a framework of individual competencies.
Entrustment in practice – purpose
One of the papers has an interesting discussion on how the assessment of entrustability is perceived by trainees, and how these perceptions can be very different from that of the trainer. (here) The authors suggest that assessments should be considered on three axes, and for each to think: to what extent does the trainer, and to what extent does the trainee, consider the assessment
- formative – summative spectrum; to be educational, or to be evaluative?
- low stakes – high stakes; to matter in terms of progression in or completion of training
- ad hoc – structured; undertaken relatively informally, or undertaken within a clear set of rules.
They stress that problems are most likely to arise when there are different perceptions. Additionally they point out that any high stakes assessment must be, and must be seen and agreed to be, carried out within a clear, fair set of rules. They suggest the need to:
- Align the process with the purpose. The less structured an assessment, the more it should be formative, perhaps as part of ongoing, longitudinal coaching;
- Align stakes with purpose. Only summative (evaluative) assessments should be high stakes, and all formative assessments should be low stakes;
- Align process with stakes. The obvious corollary is that the process needs to be appropriate for the importance; any high stakes assessment should be planned and undertaken within a clear set of rules.
Because a trainee is vulnerable and often uncertain, it is important always for a trainer (assessor) to discuss explicitly with the trainee what the purpose of and stakes associated with an individual assessment are. The more that a trainer focuses on reducing hierarchy, and creating a learning environment, the less likely that problems will arise.
This does not mean that a low stakes assessment is actually completely discounted. Even low stakes assessments will be considered when reaching an overall judgement, but there will be (should be) a large number so the stakes associated with each assessment are low.
Finally, the authors also review the words, summative and formative, and suggest a subtle but important change in terms for assessments: an assessment of learning (summative), and and assessment for learning (formative). The importance is that the term will remind both the trainee and the trainer of the purpose of the assessment.
Entrustment in practice – words or numbers?
A second paper considers the same matter (summative-formative) from a different perspective; how to report the outcome of an entrustment assessment. They point out that there is a natural tendency to quantify outcome, using numbers, even if comments are ‘allowed’.
Their conclusion is set out so well in the summary, that I have reproduced it here:
“Finally, the authors address the tyranny of documentation created by programmatic assessment and urge caution in yielding to the temptation to reduce words to numbers to make them manageable. Instead, they encourage educators to preserve some educational encounters purely for feedback, and to consider that not all words need to become data.” (here)
The typical work-based assessment form two components: a scale, words or numbers (i.e. summative), and a space for comments (i.e. potentially formative). The authors report that 98% of work-based assessment in psychiatry had no comments, only numbers, despite the space for comments. I have not surveyed UK assessments, but many that I have seen either have no comments, or bland comments such as ‘doing well’.
Numbers (or verbal quantification such as ‘good’, ‘satisfactory’, ‘below expectation’) have two major weak points. They are often managed mathematically, which is usually invalid. More importantly, they give no information on context, the circumstances of the assessment. The number cannot fairly be interpreted.
Comments can offer much more information:
- the context of the assessment;
- additional qualitative information about performance;
- justification for or explanation of any quantitative score;
- feedback to the trainee on performance
- what went well
- what might have been done better or differently
- suggestions to the trainee on changes or further learning
The authors provide evidence that comments, far from being unhelpful and subject to bias, are often more informative about a person’s standard than the quantified outcome. They are also consistent, not unreliable.
Several changes and improvements are suggested.
Simply placing the comment box above or before the quantification box increases the use of comments. Second, each comment box should be labelled with its purpose such as: contextual information; information on performance; and educational, feedback information.
Third, he authors suggest that, in the US but I suspect also in the UK, we are focusing too much on documenting training assessments as evidence for judging entrustability, and that if used properly as part of the process of learning, then too much information is available for review when making an overall judgement on entrustability. They refer to “the tyranny of documentation”. The solution suggested is to identify a proportion of work-based assessment as purely educational (formative) in purpose.
In other words, they are suggesting:
- all assessments should use comments, but comments should be focused on the purpose:
- if summative, then on context, quality, and explanation
- if formative, which will be the more frequent purpose, then the comment focus on learning
- There should be enough higher stake, summative assessments to ensure a proper holistic assessment, but not too many to be reviewed.
- Numbers, or single words summarising the standard, should be avoided or should be interpreted by reference to comments which should be made first.
A third article discusses the implementation of entrustment as a means of judging progress and achieving sufficient quality to practice independently. (here)
It draws on experience in the Netherlands, where a plan to reduce expenditure on training was necessary. One approach was to reduce the length of training. The medical establishment suggested that a blanket reduction was unwise, as it did not allow for variation between trainees. Instead a goal of reducing average length of training by introducing (a) entrustment decisions so that trainees could progress once entrusted, and (b) flexibility in the training programme so that training duration could be shortened when appropriate, but allowed to continue if necessary. A third strand was to make decisions as dependable as possible.
One fact to note is that, in the Netherlands, rehabilitation has 12 core entrustable professional activities, but the number varies from six in pathology to 30 in medical genetics. Emergency medicine has seven with 17 further sub-specialist activities. In the UK, all specialities have the same six generic capabilities, and rehabilitation has eight speciality capabilities.
The article gives many details of the work undertaken by three project groups between 2015-2018; it was a substantial undertaking. The main conclusions appear to be that:
- an electronic portfolio was vital; without it the project would fail. They now have one.
- flexible, time-variable posts could be arranged but it required a major involvement of all parties including health providers;
- change will only succeed if the culture of all parties changes to support more trainee-centred training;
- one major cultural change is to allow trainees more autonomy in their practice, while trainees. Using entrustment may assist this change.
The system in the Netherlands for certifying entrustment of an activity is summarised in the article (here):
“During the rotation, the resident collects documentations for achievements in the e-portfolio. When the resident feels ready for a summative entrustment decision for an EPA, he or she submits annotated proofs of achievement to the Clinical Competency Committee (CCC). The committee meets and considers 5 elements of the request:
- meeting the required knowledge, skills, and behavior;
- clinical experience and relevant exposure related to the EPA;
- any tests relating to the EPA;
- educational and scientific activities undertaken related to the EPA;
- committee members’ personal experience and judgment concerning the trainee.
Most information about 1–4 is derived from the e-portfolio. In addition, the committees are advised to attend to the general criteria that have been acknowledged as important for entrustment:
- capability (specific knowledge, skills, experience, situational awareness),
- integrity (truthful, benevolent, patient centered),
- reliability (conscientious, predictable, accountable, responsible),
- humility (recognizes limits, asks for help, receptive to feedback), and
- agency (proactive toward work, team, safety, personal development), acknowledging that a well-grounded decision does not only weigh “objective” criteria but also aim to include intersubjective judgment.“
Entrustment can be used as a means to evaluate all assessments of clinical activities, from those currently termed ‘competencies’ to those currently termed ‘high level outcomes’ (i.e. Capabilities in Practice).
Entrustment is a judgement made by another senior, experienced doctor (in medical training) based on relatively formal structured work-based assessments (of discreet activities) or, for high-level outcomes, on a holistic assessment of the totality of the evidence. This holistic evaluation should itself be run according to known rules, and must take into account comments; indeed comments made may be more important than actual categorical classification.
All work-based assessments should be recorded on forms that place comments before any classification of performance. Further, both trainer and trainee should agree beforehand whether a work-based assessment is primarily developmental and educational, or evaluative. Any evaluative assessment should have a clear structure. To reduce information overload when judging entrustment of high-level outcomes, it would help to identify evaluative work-based assessments as a separate category.
In the UK system, where entrustment is only used in relation to high-level outcomes and not to work-based assessments, the forms evaluating or reporting on work-based assessments should be restructured to encourage comments. Comments should cover the context, justification for and/or explanation of the evaluative categorisation if given, and comments should give helpful feedback on a trainee’s performance.
This post has explored three articles on using entrustment to evaluate the performance of doctors (and other healthcare professionals) during their training. It has emphasised the importance of agreeing the purpose for any work-based assessment undertaken, and of structuring any evaluative, high-stakes assessment. It suggest that most assessments should be considered developmental, an integral part of a programme of learning; as part of this the importance of using written comments has been highlighted and the risks associated with scores have been emphasised. The question of a variable training programme, one that is adapted to the speed of the trainee’s learning, is still open but, if variation is to be considered, the healthcare system needs a cultural change to allow trainees to be more autonomous at the later stages, and to allow more flexibility in starting and ending jobs.