RENAISSANCE SCHOOL OF MEDICINE AT STONY BROOK UNIVERSITY
CURRICULAR REFORM 2012-2013: Annotated Bibliography
Workgroup #1: Resources for Learning
Research Studies on Resources for Learning
E-Learning in Medical Education:
Velan, G.M., et al. 2010. A web-based module on lymphoma for senior medical students: benefits for learning. Med. Sci. Ed. 20(1): 32-41.
Authors created an interactive web-based module (“roll-over feature” introduced definitions, answers to questions) on lymphoma using Adobe Captivate and studied its effectiveness by comparing a control group and study group (matched for gender and academic ability). Learners in the control group were given two weeks to learn about lymphoma using “traditional resources” (books, internet) while the learners in the study group had access to the module. Interactive web-based module had a significant positive impact on learning and feedback on the module was very positive.
2. Morgulis, Y., et al. 2012. Impact on learning of an E-learning module on leukaemia: a randomized controlled trial BMC Med. Ed. 12: 36.
Authors created an interactive web-based module on leukemia using Adobe Captivate and studied its effectiveness by comparing study and control groups. This study used more subjects than the above, related paper. Authors found a significant improvement in learning for learners who used the web-based learning module compared to learners who were directed to standard textbooks and online resources. Features of effective e-learning modules are authenticity of cases, interactivity, feedback, and integration.
3. Morrow, J.B., et al. 2010. Evaluation of a web-based family medicine case library for self-directed learning in a third-year clerkship. Family Med. 42(7): 496-500.
Paper presents the reaction of learners to a series of “Design A Case” modules. Seventy five percent of learners preferred the cases to didactic lectures. There was no difference in board scores between learners who used the cases and learners who did not.
4. Skye, E.P, et al. 2011. Developing online learning modules in a family medicine residency. Family Med. 43(3): 185-192.
Authors evaluated the effectiveness of 16 web-based modules in a family medicine residency using pre-post testing and examined the learners’ responses to the modules. Resident satisfaction with the learning modules was very good and the impact on learning was positive for most, but not all, of the modules (note that they included expert review of the modules’ content to be certain the content was appropriate for the learners). This paper contains a very nice literature review regarding the creation of effective web-based learning modules that will be very useful as we grapple with our resources (e.g., technical capabilities, institutional support) and possible barriers to implementation. Also, feedback was obtained from the module authors; this information will be very useful for implementation purposes.
5. Wong, G., et al 2010. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med. Ed. 10: 12.
A scholarly review of the literature to identify theoretical models of how the Internet may support learning in medical education. The authors make recommendations to medical educators seeking to develop e-Learning modules. In particular, the ability of learners to interact with instructors, fellow students and/or a virtual tutorial and obtain feedback on their understanding and performance is highly valued.
6. Shutter, J., and Stevenson, F.T. 2011. Conversion of an image-based lecture to a virtual microscopic tutorial; a practical approach including student feedback for consecutive classes Med. Sci. Ed. 21(3): 206-209.
Three pathology cases were converted to online tutorials each containing a description of a clinical case followed by virtual slides and corresponding questions. Learners viewed the slides and answered questions about the microscopic findings in groups; instructor then reviewed the virtual slides in a laboratory setting and answered the questions with active student participation. Majority of learners (92%) with prior hands-on experience with microscopes preferred the virtual slides for a variety of reasons. Importantly, group work promoted active learning.
7. Ruiz, J.G., et al. 2006. The impact of E-learning in medical education. Acad. Med. 81(3): 207-212. This is a comprehensive, well-written review article on E-Learning in medical education that supplements the above papers. The article describes terms and tools used in E-learning, cites studies that have evaluated the effectiveness of E-learning in medical education, and a list of medical institutions that provide E-learning resources (MedEdPortal, Multimedia Educational Resource for Learning and Online Teaching [MERLOT], International Virtual Medical School [IVIMEDS], Health Education Assets Library [HEAL], End of Life/Palliative Education Resource Center [EPERC]).
8. Cook, D.A., et al. 2010. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad. Med. 85(5): 909-922.
The authors identified characteristics of E-Learning tools associated with improved learning outcomes (interactivity, practice exercises, repetition, feedback) but the authors cautioned that heterogeneity between studies mitigate the associations. The impact of online discussion, tutorials, games and simulation, and blended learning was inconclusive.
9. Johnson, M., et al. 1997. Computer-based interactive tutorial versus traditional lecture for teaching introductory aspects of pain. Assoc. Learn. Tech. J. 5: 22-31.
The authors evaluated a computer-based tutorial for teaching clinical aspects of pain and found no difference in knowledge gain or learning experience between the study and control groups (control group received a traditional lecture on the topic).
Lecture Recordings in Medical Education:
1. Franklin, D.S., et al. 2011. Use of lecture recordings in medical education. Med. Sci. Ed. 21(1): 21-28.
Authors conducted a survey study to determine how first- and second-year medical students at one school used lecture recordings (audiovisual synched web-based lecture recordings), how the availability of these recordings influenced lecture attendance, and how their use impacted student performance. Recordings were widely used (80% of learners). Learners perceived the recordings as improving their ability to learn, however use of the recordings did not improve performance on exams. The study also identified reasons learners used the recordings (flexibility in study habits, filling in gaps in lecture notes, clarification of difficult concepts).
2. McNulty, J.A., et al. 2011. A three-year study of lecture multimedia utilization in the medical curriculum: associations with performances in the basic sciences. Med. Sci. Ed. 21(1): 29-36.
Authors used server logs to determine how frequently learners accessed video and audio files of lectures in first- and second-year courses over a three-year period. Most learners (64-78%) downloaded less than 10% of the available files and there was an inverse trend between the frequency with which students used the recordings and performance in the courses.
3. Lovell, K., and Plantegenest, G. 2009. Student utilization of digital versions of classroom lectures. Med. Sci. Ed. 19(1): 20-25.
Authors conducted a descriptive study of learner use of video recordings with audio and perception of their usefulness. Learners who used the digital versions reported a “positive subjective effect on study habits and exam scores”. A significant fraction of learners (16%) felt that the digital versions were less effective than live lectures.
4. Newman, L.R., et al. 2012. Developing expert-derived rating standards for the peer assessment of lectures. Acad. Med. 87(3): 356-363.
Ten criteria are identified that characterize effective lecturing and lecture quality. Implication for us: E-Lectures should be evaluated using these criteria when provided as resources for learning.
5. Newman, L.R., et al. 2009. Developing a peer assessment of lecturing instrument: lessons learned. Acad. Med. 84(8): 1104-1110.
Individuals must be trained to rate lectures, a process in which their ratings of sample lectures (six used in this study) are compared to “expert” ratings and a discussion with the trainee points out any differences. Implication for us: trained faculty members should determine the quality of E-Lectures provided as resources for learning.
Computer Assisted Learning (CAL)
1. Choules, A.P. 2007. The use of elearning in medical education: a review of the current situation. Postgrad. Med. J. 83: 212-216.
2. Dev, P., et al. 2001. Computers in medical education. In: Shortliffe, E.H., Perreault, L.E. (eds). Medical Informatics: Computer Applications in Health Care and Biomedicine. New York: Springer-Verlag.
3. Potomkova, J., et al. 2006. Web-based instruction and its impact on the learning activity of medical students: a review. Biomed. Pap. Med. Fac. Univ. Palacky Olomouc Czech Repub. 150: 357-361.
4. Greenhalgh, T. 2001. Computer-assisted learning in undergraduate medical education. BMJ 322: 40-44.
Summary of these four review articles: The proliferation and availability of computer technology has dramatically changed the ways in which information is packaged and delivered. As a consequence, medical educators have begun to leverage the variety of digital information formats and training systems when considering changes to the curriculum and didactic methodologies.
Some of the advantages of CAL are:
The 3 “anys”: any time, any place, any pace
Reusable learning objects
The ability to enhance some interactive aspects of the traditional lecture (diagrams, 3D modeling, virtual simulations)
Some degree of dynamic interactions and immersive virtual environments
Presentation of concepts in “digestible” portions
Some concerns with CAL are:
Disconnects with asynchronous interactions with instructors and classmates
Possible reduction in emphasis on human interactions in clinical care (doctor-patient sensibilities and bed-side manner)
Intellectual property and ownership of course materials
Challenges of creating relevancy and context between courses and syllabi
Myriad of links to other information sources possibly creating learning distractions
Consideration should be given to hybrid or mixed-mode instruction methods, finding the appropriate balance between in-person and CAL deliveries, perhaps depending upon subject focus and degrees of assignment interactivity.
Increased use of CAL in medical education requires investments in organizational change management and staffing skilled in the use and administration of computer technologies.
Syllabi
1. Duker, G. 2011. Syllabi in Pitt Med Education. Accessed online at www.ame.pitt.edu/documents/SyllabiinEducation_Duker.pdf on March 27, 2013.
This document provides a small amount of data concerning the learner’s response to syllabi at the University of Pittsburgh School of Medicine. There was a positive correlation between overall course quality and syllabus quality as rated by the learners in nine pre-clerkship courses with one exception in one year.
2. Chahla, M., et al. 2010. The effect of providing a USB syllabus on resident reading of landmark articles. Med. Ed. Online 15: 4639 – DOI: 10.3402/meo.v1510.4639.
The authors created a syllabus containing 187 primary scientific articles and organized into medical subspecialties. Syllabus contained links to the full-test articles and a section called “My Portfolio” that encouraged users to save information on patients seen, presentations given, and procedures performed. Syllabus was distributed to 53 internal medicine residents on USB drives. The study found that the USB syllabus was used by all of the learners and it increased both the number of original articles read by the learners each month and their perception of the value of original articles as an educational resource.
3. Eberly, M.B., et al. 2001. The syllabus as a tool for student-centered learning. J. Gen. Ed. 50(1): 56-74.
This paper describes general education syllabi at one institution and contains a good literature review in the introduction. The literature identifies three areas in which syllabi are important:
1. Administrative - the syllabus can be viewed as a contract between the learner and the institution; e.g., “final grades will be determined by…”
2. Course development – syllabi are used to evaluate the curriculum and inform the learner of the topics to be covered, assignments, etc.
3. Communication – syllabi communicate expectations, obligations, what to expect from a course
Multimedia Learning Tools
1. Grunwald T., and Corsbie-Massay, C. Guidelines for Cognitively Efficient Multimedia Learning Tools: Educational Strategies, Cognitive Load, and Interface Design. Acad. Med. 81:213-223.
Key Points:
“The most effective approach is to build a new curriculum for the MMLT and resist the urge to ‘go virtual’ by merely translating an existing, lecture-based syllabus to a multimedia interface.”
“Good interfaces do not simply tell the learner what to do; they provide the environment necessary to assist the learner in productive independent study.”
A well-designed MMLT eliminates the need to collate textbooks, notebooks, illustrations, and other learning tools and places these resources at the learner’s fingertips.
Students learn better when corresponding information is presented simultaneously in space and time.
Learner preferences must also be taken into account with respect to information presentation
A common misconception is that allowing the learner full control over the MMLT will increase her or his learning and motivation, but this may be disorientating and confusing to the learner.
2. Sestini, P. et al. 1995. Multimedia presentation of lung sounds as a learning aid for medical students. Eur. Respir. J. 8: 783-788.
This study asked whether simultaneous presentation of sounds together with their expanded waveforms and spectrum analysis enhances the understanding and learning of medical students approaching the study of lung sounds for the first time. Exposure of inexperienced medical students to a multimedia presentation significantly increased their learning of lung sounds compared to students receiving only conventional teaching. The multimedia used in this study (combined audio and visual cues) was simple by today’s standards. The authors carefully state that this is not a replacement for bedside teaching.
Teaching/Learning Methods
1. Metzler, R., et al. 2012. Teaching on three-dimensional presentation does not improve the understanding of according CT images: a randomized controlled study. Teach. Learn. Med. 24: 140-148.
The study looked at whether “training on 3D presentation enhances the understanding of 2D images”. The study showed no difference in ability to come up with a correct interpretation of a CT image between groups of students trained in 3D and 2D imaging.
2. Ko, P.Y., et al. 2011. Comparison of a modified longitudinal simulation-based advanced cardiovascular life support to a traditional advanced cardiovascular life support curriculum in third-year medical students. Teach. Learn. Med. 23: 324-330.
This study compared using traditional ACLS training for 2 days with a longitudinal ACLS course over 2 weeks using “high fidelity simulation”. The longitudinal training included having materials available online, smaller class sizes, and exposure to real ER cases in the ER department. Students in the longitudinal group did better in the mega code test. Possible reasons were cited as: more individual attention due to the smaller groups, dedicated time to practicing on simulators. This was an example of “practice, repetition, and active learning is emphasized over a classroom setting of passive learning” (Ericsson (2008) as cited by Ko, p. 329).
3. Tunguntla, R., et al. 2009. Computer-based animations as student aids in learning bladder anatomy and physiology Med. Sci. Ed. 19(4): 167-169. The authors asked whether computer-based animations of the anatomy and physiology of the bladder filling and emptying promoted better learner compared to static images. No differences were found between the study and control groups in ability to learn the material or to apply the knowledge.
4. Romanov, K., and Nevgi, A. 2007. Do medical students watch video clips in eLearning and do these facilitate learning? Med. Teach. 29: 490-494.
Authors created a course in medical informatics that contained six eLearning modules with integrated video clips and collaborative learning tools. Viewing the video clips and use of the collaborative discussion tools was associated with a higher course grade. Female learners were more likely to view the videos than male learners.
Novel Learning Resources
1. Jones, B.G., et al. 2009. Impact of podcasting on first-year medical student note-writing skills. Med. Sci. Ed. 19(1): 15-19.
Authors creating six audio podcast episodes (15-20 minutes in length) called “SOAP Note 101” and analyzed their ability to improve note-writing skills among first-year medical students. Learners who listened to the podcasts did not have better note-taking skills during subsequent OSCEs than learners who did not listen to the podcasts. Learners liked the podcasts.
2. Kerfoot, B.P., et al. 2012. An online spaced-education game to teach and assess medical students: a multi-institutional prospective trial. Acad. Med. 87(10): 1443-1449.
The authors created 100 MCQs covering four preclinical topics and core information deemed important for all medical graduates. An email system sent emails twice a day to students who volunteered to participate in the study. Each email contain a MCQ and a link to a web page; students submitted an answer to the question and were then given the correct answer, summary of learning points, discussion of each of the choices, and hyperlinks to educational materials. Questions that were answered incorrectly were sent to the student three weeks later; questions answered correctly were re-sent six weeks later. Questions were retired once they were answered correctly two times in a row. Goal of the game is to retire all questions as quickly as possible. Students were periodically given information about how other “players” were doing. The results show a significant impact of the spaced-education game on learning.
3. Bogoch, I.I., et al. 2012. Morning report blog: a web-based tool to enhance case-based learning. Teach. Learn. Med. 24: 238-241.
The authors describe the use of a blog created by the Chief Medical Resident after morning report several times a week. The blog contained information relating to recent morning reports (medical content, links to relevant literature, medical images). The study showed that “trainees rated the blogs a useful learning tool and cited it to be among the top 3 educational resources accessed during their rotations”. This sounds like a good web-based tool to help learners review the morning report as well as review cases later on. Patient identifying information was not included in the blog. Using the blog was also helpful in monitoring topics that were important or interesting to students.
4. Cook, D.A., et al. 2010. Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad. Med. 85(10): 1589-1602.
This meta-analysis found that learners appreciated virtual patients but not as a replacement for real-life patient interactions. The use of virtual patients has a positive effect on learning when compared to no intervention.
5. George, D.R., and Green, M.J. 2012. Beyond good and evil: exploring medical trainee use of social media. Teach. Learn. Med. 24: 155-157.
We usually consider how social media might be misused. We don’t consider how it might benefit medical education. This article encourages medical educators to look at how trainees view social media and how they use it. It encourages us to do research on how students see themselves using social media for life long learning, for communications with peers, for professional development, etc. It encourages us to ask the trainees themselves to consider the ethical issues especially when medicine is so intent on privacy regarding patient information, etc. This is a way that students can actively help us develop our policies and add to the curriculum regarding ethical standards of social media usage in medicine.
6. Bow, H.C., et al. 2013. A crowdsourcing model for creating preclinical medical education study tools. Acad. Med., in press.
The authors creating a site in Google Drive that was accessible by all medical students at their institution and allowed them to create, edit and access sets of questions and answers linked to specific lectures in the preclinical curriculum. The questions were converted into flashcards using a Java-based program. Over 16,000 questions were created. A class that had access to the flashcards had higher exam scores than a previous class without access to the Q&A database.
7. Cheston, C.C., et al. 2013. Social media use in medical education: a systematic review. Acad. Med., in press.
A review of the literature found that use of social media, e.g., blogs, wikis, Twitter and Facebook, improved grades, attitudes and skills. Social media were used to engage learners, provide feedback and collaboration/professional development.
Team Training
1. Morrison, G., et al. 2010. Team training of medical students in the 21st century: would Flexner approve? Acad. Med. 85: 254-259.
The authors argue that changes in physician culture and medical errors due to health system failures create a need for team training in medical education. Team training can lead to better communication, coordination and decision-making skills, and improve error-management skills. Currently, few medical students receive team training.
2. Lerner, S., et al. 2009. Teaching teamwork in medical education. Mount Sinai J. Med. 76: 318-329.
The authors of these two review articles argue that changes in physician culture and medical errors due to health system failures create a need for team training in medical education. Team training can lead to better communication, coordination and decision-making skills, and improve error-management skills. Currently, few medical students receive team training.
3. Robertson, B., et al. 2010. The use of simulation and a modified TeamSTEPPS curriculum for medical and nursing student team training. Simul. Healthc. 5: 332-337.
The authors describe a 4-hour didactic/small group team training exercises program that teaches teamwork to medical and nursing students. Pre-post testing demonstrated increases in knowledge and attitude.
Commercial Board-exam Preparation Resources
1. Werner, L.S., and Bull, B.S. 2003. The effect of three commercial coaching courses on Step One USMLE performance. Med. Ed. 37: 527-531.
The authors evaluated the impact of a 3-4-week commercial coaching course on Step 1 performance. There was no difference in performance when comparing students who took a course and those who studied on their own. Their analysis would have detected an improvement of as little as two questions on the exam.
2. Tompkins, J. 2011. Money for nothing? The problem of the board-exam coaching industry. NEJM 365: 104-105.
This article is an opinion piece that summarizes the available literature on the board-exam coaching industry and argues that medical schools should play an active role in helping students prepare for Step 1 and alleviate anxiety associated with the exam.
3. McGaghie, W.C., Downing, S.M., and Kubilius, R. 2004. What is the impact of commercial test preparation courses on medical examination performance? Teach. Learn. Med. 16: 202-211.
This article is an analysis of the literature on commercial test preparation courses up to 2002. They found that the studies were not rigorous and that “evidence in support of the courses is weak or nonexistent”.
Workgroup #2- Active Learning and Skill Development
Article I- A Model for Small-Group Problem-Based Learning in a Large Class Facilitated by One Instructor. Nicholl TA and Lou, K. Am J Pharm Educ 76:1-6, 2012
Summary: Active learning can be greatly facilitated by small-group problem- or case-base courses; however, for large classes such small group sessions require a relatively large number of faculty facilitators. Costs and space requirements became a barrier to providing effective PBL courses throughout the curriculum. One instructor was utilized for a 1h “lecture session” per week to facilitate 16 group discussions (5 to 6 students each) in same large lecture hall for two 3h periods. Students could text questions to the instructor. In response, the instructor could project the answer for the whole class to see, or initiate and facilitate a class discussion. A multiple-choice case-content quiz was administered to students during the weekly lecture session to test knowledge related to the learning objectives.
Article II Impact of increased authenticity in instructional format on preclerkship students' performance: a two-year, prospective, randomized study. -LaRochelle JS, Durning SJ, Pangaro LN, Artino AR, van der Vleuten C, Schuwirth L. Acad Med. 2012 Oct;87(10):1341-7.
Summary: Prospective randomized cross-over trial of 2nd year medical students taking a clinical reasoning course. The purpose was to study 3 instructional formats in a case based small group course. The formats differed in increasing authenticity (paper cases, video of a doctor-patient encounter, live standardized patient). They conclude: 1. paper cases alone may disadvantage lower-performing students, and adding an additional format along with the paper cases may help this group. 2. Consider using both paper cases to give the students more organized information and add a more authentic presentation such as video or standardized patient to better engage the students.
Article III - Peer Assisted Learning: A planning and implementation framework: AMEE Guide No. 30 Ross, M.T., and Cameron, H.S. Medical Teacher 2007, 29:527-545
Summary: The authors define PAL as ‘People from similar social groupings who are not professional teachers and are helping each other to learn and learning themselves by teaching.’ Implicit in the development of PAL is the concept that medical graduates must ultimately ‘be able to demonstrate appropriate teaching skills’. PAL approaches help develop skills in lifelong learning and team work. PAL tutors typically have less expansive knowledge of subject matter, less developed teaching skills and less authority than expert tutors.
Article IV- May W, Park JH, Lee JP. 2009. A ten- year review of the literature on the use of standardized patients in teaching and learning: 1996 – 2005. Med Teach 31(6):487-92.
Cleland JA, Abe K, Rethans JJ. 2009. The use of simulated patients in medical education: AMEE Guide No. 42. Med Teach 31(6):477-86
Summary: Medical training has traditionally depended on patient contact. However, changes in healthcare delivery coupled with concerns about lack of objectivity or standardization of clinical examinations lead to the introduction of the “standardized” or “simulated” patient (SP). Standardized patients have been used in the education of health professions students, residents, and faculty. Standardized patients are most commonly used for teaching communication skills (interviewing skills), followed by clinical skills (PE skills, ethics, teaching skills, cultural competency, multidisciplinary cooperation). The majority of studies reported outcomes with changes in knowledge, skills or attitudes of the learners.
Article V- Team-based learning: A practical guide: AMEE Guide No. 65 Parmalee, D, Michaelseon, LK, Hudes, PD. 2012, Medical Teacher 34:e275-e287.
Summary: TBL is a learner-centered, instructor-directed strategy for small group active learning in large group educational settings. Learners are accountable; expected to prepare outside of class and collaborate with their team members to solve authentic problems and make decisions in class. Only one content-expert instructor is needed for the whole class in one room. Students learn how to work in teams through the process of TBL – they do not need additional instruction nor does the instructor need be a group process expert. A backward design, outcomes-based approach is used to stay focused on what the learners should be able to do. One must use TBL’s key components and follow the process for TBL to be successful.
Workgroup #3-
Topic: Professional Identity Formation + Values
Title / Citation: Goldstein EA, Maestas RR, Fryer-Edwards K, et al. Professionalism in medical education: an institutional challenge. Acad Med. 2006;81(10):871–6.
Summary: The University of Washington School of Medicine decided to revamp their curriculum and focused on connecting role-model faculty with medical students. These chosen faculty would be examples for students to build their professional identity, especially in the clinical environment. There is also an emphasis on professionalism as an entity that may be distinct from institutional level or hierarchal levels, but not independent from influencing each other. By addressing professionalism and improvement across all levels, a true institutional culture of professionalism and safe criticism can be born.
Take-aways / Implications for Subgroup 3 Professionalism can only be taught in a clinical context-- at the patient's bedside. If the school wishes to start building a student's professional identity during the preclinical years, then there must be actual interaction with real patients in challenging, unstaged encounters. There should be effort made to choose proven, role-model faculty to impart values, perspectives and techniques to medical students. If students do not see the clinical relevance of abstract topics, they will not engage themselves. In addition to medical faculty, patients can also be used as teachers. As a target of professionalism and with a unique lens, patients can provide valuable and poignant feedback to medical students as well. In the end, all of this is only possible if a true culture of institutional improvement and constructive feedback is implemented across all levels. By maintaining learning environments at standards of the highest professionalism, it will self-sustain itself and the community will conform to its ideals. Some schools/hospitals actually have departments that monitor and advance professionalism; Stony Brook should follow suit if it has not already done so.
Topic: Professional Identity Formation + Values
Title / Citation:> Cohen JJ. Professionalism in medical education, an American perspective: from evidence to accountability. Medical Education 2006;40:607-617.
Summary: Academic medicine has a role in shielding its members from the corruptive force of commercialism and poor ethics of the marketplace. A more impervious and better-prepared professional identity can be made by improving the selection of students during the medical school admission process, enhancing the methods of teaching professionalism, and purging the educational environment of unprofessional practices which threaten regression and recidivism into bad habits.
Take-aways / Implications for Subgroup 3: In order to teach professionalism, we must adopt a definition of professionalism and make it as ubiquitous as possible within the entire field. Otherwise, abstract or murky definitions of professionalism will be ignored or unattainable due to lack of clarity. If we believe in the concept that recruiting a certain type of student will determine the type of medical student or doctor they will most likely become, then we should look to change our admissions criteria. Once academic readiness is determined, students should be evaluated more finely on their involvement in team-settings, and in community service. The school should change its nonspecific mission statement, and adopt something they can strive towards collectively and individually. Students who learn professionalism, regardless of setting, will become disconcerted by what they see is actually, or realistically practiced on the hospital floors. Everyone must embody professionalism (or act it on the premises), and unprofessional behaviors should be purged to prevent setbacks in progress. Students will take professionalism more seriously if it's inspected. One particular instrument called the P-MEX can serve as a starting point to measure professionalism.
Topic: Professional Identity Formation + Values
Title / Citation: Stephenson A, Higgs R, Sugarman J (2001). Teaching professional development in medical schools. The Lancet 357: 867-870.
Summary: Small group, self-directed, and reflective teaching methods can effectively teach professional values. Problem-based learning and standardized patients present the patient as a whole person rather than just an illness. Longitudinal formative assessments work better than end-of-year summative assessments.
Take-aways / Implications for Subgroup 3: MCS currently incorporates many of the suggested methods, including lectures, small-group discussions/reflections, and case studies. Design team activities with assessment by faculty, patients, and group members. Admit more compassionate applicants.
Topic: Professional Identity Formation + Values
Title / Citation: Wear D & Zarconi J (2008). Can compassion be taught? Let's ask our students. Journal of General Internal Medicine 23(7): 948-953.
Summary: Role modeling, both positive and negative, can be very powerful forces. Single-minded focus on efficiency and volume of work can hamper compassion. Focus on improving overall learning/practicing environment rather than relying on individual role models. Design activities that prompt self-reflection, preferably led by good role models.
Take-aways / Implications for Subgroup 3: Expand ICM clinical opportunities in years 1 and 2. Integrate patient contact with content teaching. Choose good role models for ICM preceptors. Encourage self-reflection on real patient encounters.
Topic: New Step 1
Title / Citation: Susan Kies, EdD. “Proposed changes to the United States Medical Licensing Examination: impact on curricula and libraries.” J Med Libr Assoc. 2010 January; 98(1): 12–16.
Summary: The proposed changes to the Medical Board Exam schedule imply the elimination of Step 1. The NBME further recommend that schools place a greater emphasis on the integration of basic sciences with clinical practice during all four years and that the USMLE assessments (at that point only at the end of medical school and intern years) be more in line with national competencies.
Take-aways / Implications for Subgroup 3: Not sure how the USMLE plans to assess core competencies like professionalism and interpersonal skills, but we could certainly stand to learn from that in the development of our new curriculum. Peer Assessments, especially in the context of inter-class learning communities, has some real potential to cultivate these competencies in students, because students are is a very good position to evaluate the less tangible competencies in each other.
Topic: Grading use
Title / Citation: Reed, D. A., T. D. Shanafelt, et al. (2011). "Relationship of pass/fail grading and curriculum structure with well-being among preclinical medical students: a multi-institutional study." Acad Med 86(11): 1367-1373.
Bloodgood, R. A., J. G. Short, et al. (2009). "A change to pass/fail grading in the first two years at one medical school results in improved psychological well-being." Acad Med 84(5): 655-662.
Summary: Assessment drives learning. If medical schools with tiered grading systems switch to P/F grading in the first two years, would students still learn? Yes!
No difference btw graded and P/F classes in the following measures:
· performance in first- and second-year courses
· grades in 3rd year clerkships
· scores on USMLE Step 1 and Step 2CK
· attendance at academic activities
· success in residency placement
o 73% of residency directors do not favor graded or P/F schools
o though residency directors in some specialties, such as surgery, may give preference to students from graded schools
Increase in psychological well-being, less burnout and stress
· Increased time taking exams --> lower well-being
· Medical schools that emphasize exams and grades may be cultivating learning environments that potentiate anxiety and stress
Among all curricular elements examined (total contact days/hours, percent didactic learning, percent clinical experiences, percent testing experiences, number of tests, type of grading scale), grading scale had strongest association with students’ well-being
Take-aways / Implications for Subgroup 3: Advocate for P/F grading in first two years, as a means of improving students' well-being and decreasing burnout (see next article) without compromising academic achievement.
Topic: Grading use
Title / Citation: Dyrbye, L. N., F. S. Massie, Jr., et al. (2010). "Relationship between burnout and professional conduct and attitudes among US medical students." JAMA 304(11): 1173-1180.
Summary: Burnout is associated with higher levels of self-reported unprofessional conduct, such as saying you had ordered a test when you actually had not or reporting a result as normal when you actually accidentally omitted it from the PE, or copying from another student during a closed-book exam. Burnout is associated with less altruistic attitudes, such as feeling less likely to want to provide medical care for the underserved.
Take-aways / Implications for Subgroup 3: By decreasing burnout rate, P/F grading has the potential to ultimately create more professional and altruistic Stony Brook students/physicians and thereby improve Stony Brook's community and reputation.
Topic: Pre-matriculation Requirements
Title / Citation: Muller D, Kase N. (2010). "Challenging Traditional Premedical Requirements as Predictors of Success in Medical School: The Mount Sinai School of Medicine Humanities and Medicine Program." Academic Medicine 85(8): 1378-1383.
Summary: Researchers at Mt. Sinai compared/contrasted students from their HuMed program with regular med students. The HuMed program is a Mt. Sinai specific-program which admits students into Mt. Sinai without the traditional course requirements (bio, chem, orgo, physics) and without an MCAT requirement. The authors sought to determine if this lack of traditional premed courses affected academic outcomes. They looked at class quartile ranking, MSPE letter evaluation, AOA, Gold Humanism, Step 1, 3rd year clerkship honors, etc. and found that the only statistically significant differences in academic outcomes were that the HuMed students had a lower Step 1 average (221 vs 227), they were more likely to take a leave of absence, and were more likely to go into specialties such as primary care or psychiatry. Otherwise, no differences were found in (AOA, class rank, 3rd year honors, MSPE evaluation).
Take-aways / Implications for Subgroup 3: While not conclusive, Mt. Sinai's seeming success with the HuMed program clearly challenges the notion that students entering medical school need to have completed a certain set of requirements. The HuMed program's GPA requirements are also quite modest - requiring undergraduates admitted into the program to maintain a modest 3.5 GPA. As GPA and MCAT score are used heavily, particularly for admissions screening purposes, the findings of this study beg the question as to how much weight should be placed on these traditional metrics of evaluation for medical school applicants.
Topic: Peer Assessment
Title / Citation: Shue CK, Arnold L, Stern DT. Maximizing participation in peer assessment of professionalism: the students speak. Academic Medicine 2005; 80(10 supp): S1 – S5.
Summary: Students are generally reluctant to participate in peer assessment. However, willingness to participate were improved if the students’ preferences for how peer assessments were done were considered. At this specific institution this included: reporting of unprofessional behavior to an impartial counselor, 100% anonymous process, individuals who receives unprofessional behavior report should also receive corrective instruction.
Take-aways / Implications for Subgroup 3: Our students should be surveyed to assess what characteristics they believe should be included in peer assessments to increase student “buy-in” and their likelihood to take the exercise seriously.
Topic: Peer Assessment
Title / Citation: Nofziger AC, Naumburg EH et al. Impact of Peer Assessment on the Professional Development of Medical Students: A Qualitative Study. Academic Medicine 2010; 85(1):140-147.
Summary: Peer assessment can be particularly useful in professional development among medical students especially in the interpersonal dimensions when used for formative assessment. Positive transformations following peer feedback was more likely following negative feedback. Transformation was more likely among those individuals who received specific feedback (feedback which seemed “generic” were often ignored). Participants should be trained on how to provide specific, constructive feedback for maximal efficacy. Institutional culture must be centered on safety surrounding providing negative feedback while also commending those who show excellence in professionalism. Student “buy-in” increased and assessments were felt to be more worthwhile when the institution also showed support for the process. Opportunities to discuss the feedback with an advisor or friend made feedback received more helpful. Concerning behaviors reported should be addressed so that students can modulate their behavior
Take-aways / Implications for Subgroup 3: If peer assessments are to be used, culture at SBU must be changed from top down to support its implementation. Additionally, formal training is helpful to teach students how to provide feedback to peers. Finally, SBU will have to consider carefully how they introduce students to this process if the process is to be taken seriously.
Topic: Peer Assessment
Title / Citation: Arnold L, Shue CK, et al. Can there be a single system for peer assessment of professionalism among medical students? A multi-institutional study. Academic Medicine (2007); 82(6): pp. 578-586.
Summary: Having surveyed 1,661 students across several institutions, there were several characteristics which were agreed to be essential in designing peer feedback: 100% anonymous system, immediate feedback provided, focuses on both professional and unprofessional behaviors, and peer assessment is used formatively only.
Take-aways / Implications for Subgroup 3: Students generally expressed preferences for how feedback was to be given at other schools. However, our students should be polled as well to identify what characteristics they believe would be most valuable to them.
Workgroup #4- Integrated Curriculum
Article I- Evolution of the New Pathway Curriculum at Harvard Medical School: the new integrated curriculum. Dienstag, J.L. Perspectives in Biology and Medicine 2011; 54: 36-54
Summary: This paper summarizes the second iteration of Harvard Medical School’s New Pathway Curriculum that is designed to integrate the teaching of basic and population science and clinical medicine throughout the entire student experience. This process re-establishes meaningful and intensive faculty-student interactions and reengages the faculty. They have developed a new model of clinical education that offers longitudinal continuity of patient experience with cross-disciplinary curriculum, faculty mentoring and student evaluation. In addition, they provide opportunities for all students to pursue in-depth, faculty-mentored scholarly projects. They note that medical school curricula are dynamic and continually evolving requiring adaptations to current and future needs in health care by individuals and populations.
Article II - Medical Students' Evaluations of Curriculum Innovations at Ten North American Medical Schools. Ross R, Fineberg H (1998) Academic Medicine 73(3): 258-265.
Summary: The authors surveyed groups of students at 10 different medical schools in North America to gather student opinion on curriculum reforms in the early 90's. At smaller schools, where curriculum innovation was more comprehensive and broad, students actually objected to the emphasis on new teaching methods over standard instruction. Whereas at larger schools, where curriculum reform would occur more piecemeal, opinion was essentially the opposite - students wanted more reform and less standard instruction. Medical students at ALL schools invariably appreciated educational opportunities for individuation, connection, and diversity. Students want effective instruction - in other words, curriculum reform should be very focused on the quality of what it delivers vs. just switching to a certain pedagogy.
Article III- Medical School Curricula - Do Approaches Affect Competence. Hecker K, Violato C (2009) Fam Med 41(6):420-6.
Summary: AAMC and USMLE data from 1994-2004 was statistically analyzed for 116 medical schools with varying curriculum approaches (Discipline based, Organ-system based, Discipline based first year and organ system based second year, Other/multi-track, and PBL). The authors found that the variation on Step 1-3 scores was small across schools. The greatest variation in scores was noted within schools.
Article IV – Deconstructing integration: A framework for the rational application of integration as a guiding curricular strategy. 2012 Acad. Med. 87: 729-734.
Summary: Integration is a strategy of curriculum development. Each school needs to select methods appropriate to its goals, structure and constraints. It should be viewed at the Program level: what is the mission of the school; the course level: course objectives, content and sequencing determined by the role of the course in the program (what type of integration is needed to achieve course objectives), and at the session level (objectives, content, sequencing and strategies associated with the course).
Article V: Bolender et al. Curriculum integration = course disintegration: What does this mean for anatomy. 2012. Anat Sci Ed.
Summary: For anatomy, each derivative of curricular integration can be shown to involve progressive disruptions of the temporal and topographical relationship between organs systems in a body region, of the temporal relationship with other courses in a harmonized curriculum, and of the relationships between components of organ systems when integration is implemented in thematic curricula.
Article VI – Cuddy et al. Changes in Anatomy instruction and USMLE performance: Empirical evidence on the absence of a relationship. 2013, Anat Sci Educ 6: 3-10.
Summary: Examined effect of variation in anatomy course hours, curriculum type, lab experience on Step 1 and Step 2 CK scores. Found no meaningful relationships suggesting that performance on Step exams is not related to variable examined. For Step 1, Step 1 anatomy subscores and Step 2 CK, examinees from schools with stand-alone courses performed slightly better than those from schools with integrated curricula.
Article VII – Bergman et al. How much anatomy is enough? 2008 Anat Sci Educ 1: 184-188.
Summary: Asked the following: Are levels of students’ anatomical knowledge related to use of innovative or traditional educational approaches? and, Which educational approaches offer best chance of raising level of anatomical knowledge? Assessments of actual knowledge among students in the study indicated no relationship between level of knowledge and whether in innovative or traditional educational approach. Found some correlation between time spent learning anatomy and knowledge level. Highest level of anatomical knowledge attained with subject taught in a clinically meaningful way; didn’t matter if PBL or traditional approach.
Article VIII- Hecker and Violato, Medical School Curricula: Do curricular approaches affect competence in medicine? 2009 Fam Med 41: 420-426
Summary: Investigated the effects of different curricular approaches using path analysis on performance on Step 1-3 exams (discipline based, organ-system based, discipline based-first year and organ based second year, other/multi-track, Problem-based learning). Results indicated that type of school curriculum did not influence competence in medicine as reflected on Step 1-3 scores; greater variation in performance among students within a school than between students. The type of curriculum employed is less important than the quality of the curriculum implementation.
Workgroup #5: Clinical Experience and Translation Pillars
Spencer AL et al. Back to Basic Sciences: An Innovative Approach to Teaching Senior Medical Students How Best to Integrate Basic Science and Clinical Medicine.
Academic Med. 2008. 83(7) 662-9.
Assigning basic sciences to the preclinical curriculum only, creates a situation where students have difficulty retaining basic science information and integrating it with clinical medicine. The article discusses efforts to integrate basic science into the clinical years of medical education throughout US and Canadian Medical Schools. It also describes the approach at the University of Pittsburgh in greater detail.
Bandiera G, Boucher A, et al. Integration and Timing of Basic and Clinical Sciences Education. Med Teach. 2013, 1-7, early online.
Literature review, interviews, consultation. Framing medical education as ‘basic’ plus ‘clinical’ sciences is anachronistic and does not acknowledge the breadth of physician competencies. Rather, all sciences have ‘foundational’ and ‘applied’ components. Need to integrate the two over time. Even current innovative systems (vertical, horizontal, spiral) – maintain the distinction. All agree that sound basic science knowledge is critical to future medical practice. Physician as researcher may become more essential to the professional identity in the future. The traditional unique role of physician is being challenged. Role shifting from knowledge repository to knowledge integrator or broker. What becomes more important is physician as knowledge generator and academic problem solver as means to achieve leadership. Recent publications about brining in basic science with technology- simulation, web-based. Consider streaming students based on research interest or envisioned future specialty. Match educational objectives of each group to their basic science needs including newer disciplines (social sciences).
Sakles JC, Maldonado RJ, Kumari VG. Integration of Basics Sciences and Clinical Sciences in a Clerkship: A Pilot Study. J International Association of Medical Science Educators.
Emergency Medicine clerkship at UCD. Case based-approach. 4-week long required clerkship in 4th year. Did a pilot project first (1/7 groups did it). Split session format – 3-7 days in between 2 sessions. 1st session case is presented cold with images of physical findings and lab data. Students identify most relevant clinical and basic science issues for exploration. 2nd session – share information they researched, integrate it with clinical manifestations, pathophysiology, management. In real study, basic scientists facilitated the discussions. 1.5-2 hours once weekly for 2 weeks. Did modify the format a bit – sent out information about case beforehand. 97% felt integrated sessions enabled them to achieve deeper understanding of basic science principles relevant to the case. 86% stated the sessions improved their ability to evaluate and manage another like patient case. 69% said more experiences like this would motivate them to explore basic science principles relevant to clinical problems. 95% said understanding basic science principles relevant to clinical problems would contribute to better patient care. 79% recommended integrating teaching in other 4th year clerkships.
Haspel, R.L., Bhargava, P., Gilmore, H., Kane, S., Powers, A, Sepehr, A., Weinstein, A.R., Schwartzstein, R.M., and Roberts, D.H. Successful implementation of a longitudinal, integrated pathology curriculum during the third year of medical school. 2012. Arch Pathol Lab Med 136: 1430-1436.
A pathology curriculum is described that integrates into a longitudinal clerkship year. The curriculum includes an introductory session during a transition week course that describes the role of anatomic and clinical pathology including the tests and resources available; a series of longitudinal student case conferences, and a pathology elective. They conclude that the third and fourth year of medical school are ideal for linking the pathology learned in the preclinical courses to patient care and a better understanding of pathology as a profession. This type of curriculum offers the opportunity to incorporate areas that are not typically included in third year medical student training including nutrition, geriatrics, ethics, and pathology.
Corbet EC et al. Recommendations for Preclerkship Clinical Skills Education for Undergraduate Medical Education – Task Force on the Clinical Skills Education of Medical Students
The emphasis in the preclerkship curriculum should be to prepare the student to acquire sufficient clinical expertise to participate in clerkship level clinical care activities. According to Pangaro’s RIME schema, the student should be at the level of consistent reporter and sometimes interpreter. This results from sequentially more challenging clinical learning experiences, repeated practice opportunities, and observation and feedback based on definable performance outcomes. Additionally, at this stage the six ACGME competencies should be introduced.
Specifically, the preclerkship clinical experience should emphasize. . .
Professional and ethical behavior
Communication skills
Clinical problem solving skills including the ability to generate a differential diagnosis
The ability to take a clinical history, both focused and comprehensive
5) The ability to perform a mental and physical examination
6) The ability to select, justify and interpret selected clinical tests and imaging
7) The ability to understand and perform a variety of basic clinical procedures
8) The ability to record, present, and manage clinical information
This learning should take place in a variety of formats. Ideally the majority of the time should be spent doing direct patient care and this should be applied as early and often as possible. Direct patient care can occur in a variety of settings and there are benefits with each setting. For example, the Emergency Department and acute care clinics can be excellent settings for practicing problem focused histories and physical exams, diagnostic reasoning, and test interpretation. Critical Care Units can be excellent settings for examining patients with abnormal findings. Of all the settings, those that allow for longitudinal patient care experiences are the most useful for providing students opportunities to learn the majority of the competencies.
Additional formats can include, in descending order of importance, simulation, small group sessions and large group learning.
Students should be frequently assessed, both formative and summative, and OSCEs are the best format for this.
The program should be frequently assessed by a clinical skills curriculum committee. This committee should include the course director, clerkship directors, residency directors, education deans and interested students.
Finally, the program should have adequate resources. This includes financial support, administrative support, IT support, time in the curriculum and faculty time.
Workgroup #6: Themes, Competencies, and Intercession Topics
Article I- Kitzes JA, Savich RD, Kalishman S, Sander JC, Prasad A, Morris CR, Timm C (2007). Fitting it all in: integration of 12 cross-cutting themes into a School of Medicine curriculum. Medical Teacher 29: 437-442.
At the University of New Mexico School of Medicine, “cross cutting themes” designed by locals with reference to core competencies are taught through horizontal and vertical integration throughout all four years of medical school. During the first year of implementation, themes groups selected three priority themes based on current gap analysis, and encouraged course instructors to integrate these themes into the curriculum.
Article II - Bradley P, Mattick K. (2008). Integration of basic and clinical sciences. AMEE
Integration prevents learning which is divided into discrete blocks and parcels of information, focuses on learning concepts and principles, and how these can be applied in the clinical setting. 1) Horizontal integration brings together different disciplines to engage the learner in a holistic manner with the formation of connections between different subject areas 2) Vertical integration is the fusion of the basic and clinical sciences to provide a setting and context for both the basic sciences and the clinical cases.
Article III - Robinson DM, Fong CT. (2008) Genetics in medical school curriculum. A look at the University of Rochester School of Medicine and Dentistry. J Zheijiang Univ Sci B. 9(1): 10-15.
Double Helix Curriculum (DHC) named because it consists of “intertwining strands” of basic and clinical science education throughout the four-year curriculum. Six cross-cutting themes: Aging, Diversity, Ethics and Law, Health Systems, Nutrition, and Prevention were selected and are among the LCME Hot Topics. Medical genetics is not taught as a single course, but as a motif that is part of many different courses and themes. No outcomes reported or cited for the DHC curriculum.
Article IV - Wiener CM, Thomas PA, Goodspeed E, Valle D, and Nichols DG. (2010) “Genes to Society”—The Logic and Process of the New Curriculum for the Johns Hopkins University School of Medicine. Academic Medicine 85:498–506.
Johns Hopkins implemented a new curriculum in 2009 with a main focus to disengage from conventional thinking of humans as either “healthy” or “ill”, and instead be considered along a continuum ranging from “asymptomatic or latent” to “critically ill”, or some point in between. The Curriculum Reform Committee initially concentrated on the “vertical” components of each course and the Horizontal Strands Subcommittee’s task was to integrate broad themes, many of the LCME hot topics, across the entire curriculum.
Article V – NYU Website - “Curriculum for the 21st Century”
http://school.med.nyu.edu/studentsfaculty/office-medical-education/curriculum
The website refers to “Pillars” which are thematic disease-based topics of medical importance which are woven throughout the curriculum. Some examples given are: Metabolism and Obesity, Cancer Biology, Cardiovascular, and Microbial Pathogenesis. According to a NYU news online news report: http://newsandviews.med.nyu.edu/curriculum-21st-century pillars also include: genetics, nutrition, epidemiology, imaging, informatics, preventive health, biomedical ethics, and health disparities among different populations.
Article VI - UVA "Next Generation Cells to Society Curriculum”
http://www.medicine.virginia.edu/education/medical-students/UMEd/nxgen/AdvancingCurriculum-page
According to the website - "Next Generation", employs a system-based learning experience with deeper integration of basic sciences into clinical medicine, is oriented to clinical performance, and uses the best evidenced-based models for medical education to foster student learning. It seems that the curriculum content is integrated around organ systems rather than themes. While the curricular documents refer to “themes”, these are really topics to be discussed during the listed session and refer to organ systems rather than actual themes.
Article VII – Crespo L, Nelson L, Peppler R. University of Central Florida College of Medicine (2010). Academic Medicine: 85 (9) S147-151
http://med.ucf.edu/academics/md-program/integrated-curriculum/#
The UCF integrates basic and clinical sciences across all four years. The two year practice of medicine module integrates EBM and basic science with clinical care, a focused individualized research experience is offered. The clinical curriculum is complemented by basic science lectures and Longitudinal Curricular Themes (LCT’s) are woven throughout the curriculum. Each LCT has a theme director responsible for ensuring content.
Article VIII – Alston SR, Littles AB, Fogarty JP, Watson RT. Florida State University College of Medicine. (2010) Academic Medicine:85 (9) S144-146 http://med.fsu.edu/?page=medicalEducation.integration
Themes are integrated into existing course structure and reinforced in small group sessions. They are in the process of curricular redesign with goal of complete integration. 75% of the clerkships are outpatient and most are community based. Some themes are better integrated than others. Best example is geriatrics, which is strongly supported through Reynolds funding.
Article IX - University of Pittsburg Medical Center
http://www.omed.pitt.edu/curriculum/themes.php
A key strength of the UPSOM integrated curriculum is the approach taken to integrating themes throughout the four-year curriculum which provides an opportunity to build upon a foundation of prior learning while providing a level-appropriate and well-synchronized introduction of new content. The result of this approach is to better demonstrate how the thematic content, though often new to the curriculum, truly represents a core element of medical practice for the 21st century physician.
Article X – Wolpaw TM, Hull AL, Wilson-Delfosse AL, Dannefeer EF, Wolpaw DR, Isaacson JH, Papp KK, Bierer SB, Ornt DB. Case Western University School of Medicine and Cleveland Clinic. Acad Med:85 (9) S439-445.
http://casemed.case.edu/admissions/education/up_program.cfm
The central themes of public health, civic professionalism and leadership, along with the core concepts of health and disease prevention, are longitudinally woven throughout the entire curriculum. The systems issues of patient safety, quality medical care, and health care delivery are emphasized and integrated throughout the curriculum.
Article XI – UNSOM Statement: Three Organizing Principles of Medical School Curricula
Three organizing principles: 1) Systems-Based Curriculum: organized by organ systems, clinical content and skills integrated with basic science subjects. Woven throughout are “threads” e.g. humanism, health systems and equity, public health and preventive medicine, etc., which constitute a distinct parallel course. 2) Clinical Topic Integration: organized by topics that occur together e.g., ‘Molecules to Cells,’ ‘Human Development and Structure,’ with curriculum organized around the broader areas of medical science, medical arts and clinical medicine, basic science exposure all four years and a longitudinal course for clinical skills, professionalism, etc. 3) Broad-Based Philosophy-Integrated Throughout the Curriculum: includes an overarching thematic dimension that filters down into organ system and basic science units.
Workgroup #7: Grading and Assessment of Students
Assessment Tool/Method: In-House Exams
Citation:< Jozefowicz RF et al.The Quality of In-house Medical School Examinations. Acad Med 2002;77:156–161.
Summary: Most medical schools test their students throughout the curriculum using in-house examinations written by the faculty who teach the courses. The authors assessed the quality of in-house examinations used in three U.S. medical schools. A total of 555 questions were analyzed. The in-house examinations were of relatively low quality. The quality of examination questions can be significantly improved by providing question writers with formal training.
Assessment Tool/Method: Multiple Choice Tests and Constructed Response Questions
Citation: Stanger-Hall KF. Multiple-choice exams: an obstacle for higher-level thinking in introductory science classes. CBE – Life Sciences Education 2012, 11:294-306.
Summary: This study tested the effect of exam format on critical-thinking skills. Multiple-choice (MC) testing is common in introductory science courses, and students in these classes tend to associate memorization with MC questions and may not see the need to modify their study strategies for critical thinking, because the MC exam format has not changed. To test the effect of exam format, two sections of an introductory biology class were used. One section was assessed with exams in the traditional MC format, the other section was assessed with both MC and constructed-response (CR) questions. The mixed exam format was correlated with significantly more cognitively active study behaviors and a significantly better performance on the cumulative final exam (after accounting for grade point average and gender). This suggests that the MC-only exam format indeed hinders critical thinking in introductory science classes. Introducing CR questions encouraged students to learn more and to be better critical thinkers and reduced gender bias. However, student resistance increased as students adjusted their perceptions of their own critical-thinking abilities.
Assessment Tool/Method: Clinical Vignettes
Citations: Peabody JW, Luck J, Glassman P, et al. Measuring the quality of physician practice by using clinical vignettes. Ann Intern Med. 2004;141(10):771-780.
Summary:> Interesting technique. Would require considerable IT input and maintenance. Probably better at assessing clerks and above. Vignettes could be partly or wholly fictitious: “homework.” Resource input involved but possibly only at certain steps.
>
Assessment Tool/Method: Portfolios
Citation: Driessen EW, van Tartwijk J, van der Vleuten C, Wass V. Portfolios in medical education: why do they meet with mixed success? A systematic review. Med Educ 2007;41:1224–33.
Summary: Looks great as an assessment tool until we reach the main stumbling block, resources. A portfolio needs intensive faculty mentoring over the length of the curriculum if students are to receive proper coaching for managing the portfolio experience. This program introduces a committee assessment on top of numerous one-on-one mentoring tasks. Validity and reliability of the grading in this kind of assessment is suspect in many instances. In sustaining reliability, the portfolio became rigidified: students must answer certain questions or lose points.
Assessment Tool/Method: OSCE and Clinical Evaluation Forms
Citation:
Kreiter C, Bergus G. A study of two
clinical performance scores: assessing the psychometric characteristics of a
combined score derived from clinical evaluation forms and OSCEs. Med Educ Online 2007:12:1-5
Summary: The results of this study demonstrate that assessment information based on simulated and actual patient encounters can be combined into a composite. Since a composite score may provide a more valid measure of clinical performance, this study supports using a combined CEF and OSCE measure.
Assessment Tool/Method: Standardized Patients
Citation: Cleland JA, Abe K, Rethans JJ. 2009. The use of simulated patients in medical education: AMEE Guide No. 42. Med Teach 31(6):477-86
Summary: SPs are now used widely for teaching and assessment purposes. SPs are usually, but not necessarily, lay people who are trained to portray a patient with a specific condition in a realistic way, sometimes in a standardized way (where they give a consistent presentation which does not vary from student to student). SPs can be used for teaching and assessment of consultation and clinical/physical examination skills, in simulated teaching environments or in situ. All SPs play roles but SPs have also been used successfully to give feedback and evaluate student performance. Clearly, given this potential level of involvement in medical training, it is critical to recruit, train and use SPs appropriately. The authors provided a detailed overview on how to do so, for both teaching and assessment purposes including: how to monitor and assess SP performance, both in terms of validity and reliability, and in terms of the impact on the SP; and an overview of the methods, staff costs and routine expenses required for recruiting, administrating and training an SP bank.
Assessment Tool/Method: Clinical Performance Observation
Citation: Pulito A, Donnelly M, Plymale M, Mentzer R. What do faculty observe of medical students’ clinical performance? Teaching and Learning in Medicine: 18(2) 99-104.
Summary: The purpose of this study was to identify which aspects of students' clinical performance faculty actually observe. Analysis of the questionnaires and evaluations indicated that faculty members gauge medical knowledge, professionalism, and clinical reasoning skills from direct interaction with students. History-taking and physical examination skills are inferred from the quality of verbal presentations. Faculty have little basis for evaluating other important aspects of clinical performance.
Assessment Tool/Method: Faculty Direct Observation
Citation: Howley LD, Wilson WG. Direct observation of students during clerkship rotations: a multiyear descriptive study. Acad Med. 2004 Mar;79(3):276-80.
Summary: Although alternative assessments of clinical skills are becoming more common in medical education, faculty ratings based on direct observation are still prominent. The data in this study reflect that these observations may actually be occurring quite infrequently, if at all. Decreasing the evaluative weight of faculty and resident ratings during the clerkship rotation may be necessary. Otherwise, efforts should be made to increase the validity of these ratings.
Assessment Tool/Method: Self-Assessment
Citation: Davis D, et al. Accuracy of physician self-assessment compared with observed measures of competence: a systemic review. JAMA:296: 1094-102; 2006.
Summary: While suboptimal in quality, the preponderance of evidence suggests that physicians have a limited ability to accurately self-assess. The processes currently used to undertake professional development and evaluate competence may need to focus more on external assessment.
Assessment Tool/Method: Informed Self-Assessment
Citation: Sargeant J, Eva KW, Armson H, Chesluk B, Dornan T, Holmboe E, Lockyer JM, Loney E, Mann KV, van der Vleuten CP. Features of assessment learners use to make informed self-assessments of clinical performance. Med Educ. 2011 Jun;45(6):636-47.
Summary: Participants generally perceived the effectiveness of both formal and informal assessment and feedback activities as being both person- and context-specific. There appeared to be no generally effective or ineffective strategies. The success of each approach, as seen through the learners’ eyes, was moderated predominantly by external factors, mainly by supervisors’ engagement in their learning and improvement, awareness of appropriate performance standards, and skill in facilitating specific approaches and providing feedback. In other words, the value and validity of the approach were not inherent in the approach itself, but in how the approach was used.
Assessment Tool/Method: Patient Questionnaires, Faculty Observations and Student Self-Assessment
Citation: Medical students’ clinical performance in general practice: Triangulating assessments from patients, teachers and students. Med Teach:32(4):333-9; 2010.
Summary: The purpose of the study was to triangulate feedback using patient questionnaires, written self-assessment and teachers' observation-based assessment, and to describe the content of this feedback. Patients scored students' performance high compared with students' self-assessments. Teachers' scores were in accordance with patients' scores. Teachers' written evaluations of students were often general. There is a potential for improving teachers' feedback in terms of more specific and concrete comments.
