Hello everyone. This is a brief orientation video for how to use the clinical performance assessments in Acuity (One45). I'm Dr. Theresa Pesavento, the Director of Medical Education for Assessment and Continuous Quality Improvement for the MD program at UWSMPH. I along with Dr. Kirstin Nackers, the scope block leader and co-chair of the Clinical Assessment Task Force, and Dr. Kathy Stewart, a phase 2 and 3 director, are leading the multi-year plan for changes and improvements to the clinical performance assessments.

These improvements and changes resulted from the Clinical Assessment Task Force that Dr. Kirstin Nackers and I led in spring 2023. Among these task force recommendations were priorities for the form itself. And some of these were in the category of simplifying, condensing, and clarifying. These were also our top priorities when we were moving the forms from OASIS into Acuity for January 2024.

So you will see us talk through some of the key changes to address things that we made. I wanted to offer a reminder of the knowledge base that our Acuity team has put together for many of these resources. You will find this video along with other communication links found here soon linked at that knowledge base.

First, I will do a very brief screenshot of what you will see when you log into your Acuity account. We are using the demo account here, Wisconsin Badger, for a faculty account in your to do list. It also pops open the screen that you see on the right hand side showing the forms to complete. We'll be using the demo Bucky Badger Student Association to demo the remainder of this orientation.

The other way is that Acuity will send a link it will come from this no reply at One45.com email address. In the body of that email, there will be a link you can click directly to go to the form. Note that the subject line will be new evaluation and this is not spam. This is how Acuity generates those emails to connect you directly to your forms to fill out.

This is a view of the old OASIS form to show you a simplification we made in Acuity forms for the demographic information collected at the very beginning of the form. You can see here the version of the Acuity form which I will show enlarged on the next slide in just a minute.

A few of the simplifications we have made are items such as evaluator role, the specialty in which the student is rotating, the type of rotation, et cetera, are collected by Acuity anyway, so we have eliminated them from the new Acuity form because those are collected in the background. Therefore, we remove some of these at the very beginning, like I said, of assessor demographics or of rotation details.

You see that the form starts with the number of half days worked with a student. We've kept that item and so you will need to fill it out with the dropdown menu rounded to the nearest whole number. And then the form questions begin starting with the professionalism question.

There are three different types of assessment items on the Acuity forms just like there were on the OASIS forms. The first and the majority of the items are competency-based items, like you see here. These are connected to our milestones which are connected to our graduation competencies. These competency-based assessment items use the milestone language for the response anchors. We did not make changes to the milestones, we did not make changes to the competencies. We made a few changes I want to highlight to, again, condense and simplify the forms for ease of use.

One is this milestone you see here. In OASIS had three questions associated with them. We are able to combine and reduce to two questions capturing the same milestones and using the same language for the Acuity form. Wherever possible throughout, we changed the question stems from a question that might be required in a yes or no, does a student do this or does a student not do this? Two questions that indicate observable performance on a scale such as how well does or which option on the below scale accurately describes, for example.

We did clarify and simplify some of the response anchors, which I will walk you through in a minute wherever possible. Again, we did not change the milestone. So while the response anchor may look slightly different or slightly shorter, it is still indicating the same milestone as the OASIS form. There are a few differences in item order and scale displayed between the old OASIS forms and the new Acuity ones.

First, is not observed not applicable, is the first option rather than the last. Second, the competency-based scale is always a seven point scale, seven items here, and the first item will always be unacceptable. This is the unacceptable milestone using the milestone language. We wanted to make it consistent that it's always the first item on the seven point scale.

We have also included some parenthetical intermediary level response anchors that you will see here highlighted between above and below defined levels. Acuity requires that we show the same scale for all items. So we were required to do this for technical reasons, whereas in OASIS, we had hidden some of these intermediary levels that did not defined language to them. For pedagogical reasons, we also wanted to include these to stay true to the SMPH competencies and milestones.

And retain the ability to indicate that development was progressing between one milestone and the other in order to provide learners with more specific formative feedback. So these are to be used, you can use them as they are written, that the learner is between the above and below defined levels.

Next, we added some bolded text to clarify where language might be different, key differences in language between two milestones. You see with guidance here is bolded and then growing toward independently is bolded to visually highlight those differences. We also, for simplification, added some ellipses and I will explain these quickly.

Some of the SMPH competencies and milestone language repeat a portion at multiple milestone levels. We wanted to stay true to the competencies. Again, we didn't change the milestone language or the competencies. We wanted to stay true to that as written, while balancing readability of the forms and keeping the response anchors as short as possible. So there are places where ellipses are used to indicate when a skill builds on the preceding skill without repeating the full text of the preceding response anchor.

There's an example here at the bottom where reading... even for complex patient cases is meant to be understood that that is added to the end of the preceding milestone builds on that preceding skill to be read as such history taking as consistently organized even for complex patient cases. Again, we did this to avoid very long response anchors wherever possible.

We also added some parenthetical explanation. This might be in the question itself, it may be in the response anchor to help provide consistency for interpretation of terms. You see here an example where we define hypothesis driven in the question. An example where we define consistently meaning over 90% of the time. An example where we give some examples.

The second type of assessment item is a global domain of competence assessment items. These will look exactly like they did in a OASIS. They remain unchanged, assessed on and normative, meets expectation scale. There are five items on this normative scale there that you see in the blue box ranging from unacceptable to outstanding. Once again, a formatting change happening in Acuity is not applicable, not observed. Will always be the top option. You see that right above the blue box.

We wanted just to take this opportunity to offer just a brief reminder of how best to interpret these five normative items. One is where a learner should be in the performance based on their phase and their learning trajectory. So assessing based on that. Working from the top down, unacceptable will be an outlier, has a lot of overlap with unprofessional behavior. A student that needs improvement is maybe somebody well meaning and trying, but not on track, not quite meeting expectations.

The majority of students will fall into the meets expectations category with some exceeding expectations. Once again, performing above the level for their stage in their learning. Outstanding. You see in parentheses here is the top 5% of students that you would have assessed or top 5% of students on that learning scale. So this would be truly the best of the best performing above and beyond where they're expected to be.

The third type, again, unchanged is overall rating MSP, informative feedback comments. So you see here that these will look exactly like they did in the OASIS forms. That's the end of our very brief tour of the new clinical performance assessments for Acuity. We encourage you to reach out to us with any questions.

Once again, I'm Teresa. My email is listed right there. You may contact me or Kirstin or Kathy, particularly with any process or pedagogy questions that we can help answer about the forms. The email address for our Acuity SMPH implementation team is listed at the bottom and will be linked in many of the materials and in the knowledge base. They can help answer technical, logistical, acuity login questions. If you're unsure of whom to contact, you are welcome to email the Acuity team and they will funnel any questions to us that we can help reply to and we'll get back to as quickly as possible.

Thank you so much for your time and watching this orientation.

Log in to Acuity directly: https://uwsmph.one45.com/
Acuity Knowledge Base: https://kb.wisc.edu/smph/academicaffairs/130298

Theresa Pesavento: pesavento@wisc.edu
Kirstin Nackers: nackers@pediatrics.wisc.edu
Kathy Stewart: ksstewart@wisc.edu

Acuity SMPH Implementation & IT Team: AcuityONE45Support@med.wisc.edu