The Learner Activity View for Advisors (LAVA) Overview
During the Spring 2024 term a learning analytics pilot is being conducted with the intent of providing academic advisors with access to and awareness of trends across the course array of a student in two broad areas: performance and participation.
The “Learner Activity View for Advisors” (LAVA) resource will be made available to the pilot academic advisors via the Advising Gateway. Advisors will have no direct access to Canvas courses or any associated material. LAVA exposes only high-level trend indicators of patterns across a student’s enrolled course array. Communications about the pilot to pilot students and to the instructors whose courses they are enrolled in for Spring 2024 will be sent out in January.
Some of the major goals of the LAVA pilot include:
- Evaluating this approach to learning analytics for advisors (providing awareness of broad trends ACROSS a student’s enrolled course array)
- Understanding the utility and efficacy of this approach and of the LAVA resource
- Working with academic advisors to determine what interventions/actions could look like based on insights provided by access to this new information
- Understanding the impact of access to learner activity data on advisors (bandwidth, impact of seeing flags, ability to engage/intervene, etc.)
- Assessing whether this insight into learner activity is helpful to advisors in their work
- Assessing whether this insight into learner activity proves helpful to students (getting ahead of potential bad outcomes, improving outcomes, etc.)
Notes:
- This project is part of UW-Madison's broader efforts around learning analytics. More information about learning analytics at UW-Madison is available from the Vice Provost for Teaching and Learning and the Learning Analytics Center of Excellence.
- This document describes a learning analytics approach to help support student success. Students are unique individuals and are more complex than their data - be sure to consider them holistically.
- Please review the UW-Madison Learning Analytics Guiding Principles and the Office of Undergraduate Advising Guiding Principles before using LAVA.
- For more information about the use of data, review the Teaching and Learning Data Transparency Statement.
What is the Learner Activity View for Advisors/Canvas Flags (LAVA)?
The Learner Activity View for Advisors/Canvas Flags is an awareness function that displays performance and participation data to authorized users through a new capability in the Advising Gateway.
Search for a student - display shows Canvas Flags (see detailed image, below)
If a student has an active Canvas flag, a checkmark will appear in the Flags column in the search results on the Advising Gateway. The Flags column and checkmarks will appear on all of the Gateway’s three search tabs: All Students, My Advisees, and Last Viewed.
Clicking on the Flags column heading will sort the search results into two groups; students with active flags will be listed first (in alphabetical order), followed by all students without active flags (in alphabetical order).
Canvas Flags details on search page
By incorporating these data directly into the Advising Gateway, it offers authorized users a view of student academic activity and performance during the active term, within the normal workflow and familiar Advising Gateway interface.
Student profile page - showing Canvas Flags tab with the active flag indicator
Features:
- This is a learning analytics approach, using course activity and gradebook data from Canvas (and other integrated tools) and accessed within the Advising Gateway. More details about the data, and when performance and participation flags will appear are provided below.
-
Data appear both in a Flags column in Advising Gateway search results, and in a new Canvas Flags tab on the Student Profile page within the Advising Gateway. If low performance or participation trends are present across multiple courses (relative to their peers within those specific courses), a flag will appear in that tab on the student profile page.
- The flags may help an advisor start a conversation with a student, if there is a potentially concerning trend or pattern of behavior across multiple courses.
About the Flags
Two types of flags may appear:
- Participation - course activity
- Performance - grades
To view flag data and details, click on the tab to show the accordion menus that are labeled with the flag names.
Canvas Flags tab - with no raised flags
If flags are present, there will be a label telling you which flag has been raised. In the example below, the Performance flag has been raised.
In the example below, there are two flags raised - both Participation and Performance flags have been raised.
Flagged Courses
When the flag menus are expanded, you can see the list of courses the student is in, and which specific courses contribute to the active flag.
- Course lists will appear under both the Participation and Performance sections, since there may be a flag raised in one area but not in the other, for the same course. (For example, a student’s performance is not yet flagged, however they have not participated in Canvas for the last five days, compared to their peers, for several courses.)
- Courses that meet the criteria in the flag details will appear in the Flagged Course column. All other courses will appear in the No flag or no data column.
Canvas Flag details example - showing courses
Two or More Courses - Looking at Trends
LAVA/Canvas Flags is designed to show trends or patterns of behavior across a student's array of enrolled courses. A student generally must have two or more courses that meet the criteria in the flag details for the flag to be “raised”.
Caveats:- If a student is only enrolled in one course, a flag may be raised if the threshold is met in that one course.
- If a student is enrolled in two courses, a flag will be raised if the threshold is met in only one course.
Flag Details
The flag details box describes the course data that is used to determine if courses have flags.
Important reminders to keep in mind:
- Instructors teach using many different strategies and pedagogical approaches. How a course is set up, and how much an instructor uses Canvas and other linked tools for teaching activities will impact the data that can trigger these flags.
- The more an instructor uses Canvas and other linked tools, the more useful Canvas Flags may be.
- Don’t assume anything based on just one data point; remember to validate and ask questions. This is a starting point to help with a conversation.
Participation Flag Details (Course Activity)
Student activity in the course is compared with other students enrolled in that course. Participation data uses activity in Canvas, along with several other tools if they are integrated within the Canvas course (more details below).
- Participation is measured as any activity in an online tool in the course.
- Wherever possible, students are compared to the other students enrolled in all sections of a course.
What is Participation in a Course?
Canvas participation is based on page views in a course, along with student activities. Student activities may include:
- Accessing a course page
- Announcement - posting a new comment
- Assignments - submitting an assignment
- Commenting - posting a new comment on an announcement or discussion
- Collaborations - viewing and/or editing a collaborative document
- Conferences - joining a web conference
- Discussions - posting a new comment
- External tools that are linked within the course (see more info, below)
- Pages - creating a page
- Quiz - starting a quiz, submitting a quiz
More About External Tools
Other tools may be linked within a Canvas course; for example Kaltura MediaSpace, Piazza, TopHat, Pressbooks, Engage eText or Zoom are other Learn@UW tools that instructors use.
- When a student opens an external tool from within the Canvas course, that participation will be counted.
- Activity occurring within external tools is NOT generally captured.
Activity in other online tools that are not connected to Canvas - or are not launched from Canvas - will NOT be included as participation for flag calculations. For example, a course that uses a standalone WordPress website for course activities won’t show participation data.
A participation flag will be raised if:
- The student is 5 days behind the median of their peers in aggregate participation. In other words, they are well behind their peers in accessing the course activities.
- There must be at least two courses that the student hasn’t logged into for five days. (If a student is only enrolled in one course, a flag may still be raised if they’ve not interacted with that one course.)
How the Participation Flag is Calculated
- LAVA calculates the median most recent activity time for all students in the course.
- If the students’ Last Activity DateTime is 5 or more days before the median, the flag may be raised (if this occurs across two or more courses, when the student is enrolled in more than one course).
- This flag will be most useful for courses where many students are frequently logging in (for example, they all take a quiz synchronously in a live class).
- If the student does not have a last activity date (has never interacted with the course), and the course start date is set, the students’ last activity date defaults to the course start date.
- If the course start date is not set, the default start date is the start of the term.
Additional Details
- Peers for the activity quartiles are based on the course the student is enrolled in Canvas. This means if sections have been split across multiple courses (for example, the discussion sections have been moved to their own course, separate from the lectures) they may not be compared with all their peers in the course.
Performance Flag Details (Grades)
Performance data uses the Canvas Gradebook. It uses the current calculated grade for a course.
- The more assessment and grading activity that is handled in Canvas, the more useful this flag will be.
- Grades are included based on the grade posting policy set in Canvas.
- Grades can be included from external tools, if they are integrated within the Canvas course (e.g. Top Hat, Atomic Assessments, ScanTron, Kaltura Quizzes.)
- Grades can be manually entered or imported into the Gradebook.
What is Performance in a Course?
- Performance is measured as any and all grades entered in the Canvas Gradebook.
- Wherever possible, students are compared to all sections of a course
A performance flag will be raised if:
- A student is a full letter grade (8 percentage points) behind the bottom quartile of their peers.
- Students are scoring in the bottom quartile - their grade performance is in the lowest 25th percentile for the course.
- There must be at least two courses that meet the performance threshold (If a student is only enrolled in one course, a flag may still be raised.)
How the Performance Flag is Calculated
- LAVA calculates the bottom quartile grade (the lowest 25th percentile) for a course.
- It then subtracts a single grade (e.g. from A to B) which is 8 percentage points.
- If the students’ grade is less than the above calculation, a flag may be raised (if this occurs across two or more courses, when the student is enrolled in more than one course)
Additional Details
- If the Gradebook is not turned on, nobody gets flags (for example, a face-to-face course that doesn’t really use Canvas).
- Peers for grades are based on the course and enrollments as originally defined in the Student Information System (SIS). If different sections are later split apart (e.g. the discussion sections have been moved to their own course, separate from the lectures) and grades are given in different sections, we will get duplicate grades for the same SIS course. In this case, the maximum score from the two sections will be used, as there is no way programmatically to determine how an instructor will later combine these grades before final grade submission. We default to the maximum score to reduce the risk of raising a flag unnecessarily.
- Instructional choice about grading (and activity for that matter) is a personal choice of each instructor; consistent grading practices should not be assumed.
Where can I find more information about learning analytics?
For additional information on using learning analytics at UW-Madison, check out:
- For help with the Advising Gateway, please start with the overview document: MyUW Madison - Using the Advising Gateway
- For additional information on learning analytics at UW-Madison, please visit the learning analytics guide, What are the pedagogical uses of learning analytics?
- Visit the Learning Analytics at UW-Madison page on the Vice Provost for Teaching and Learning website.
- View the The Learning Analytics Center of Excellence information about projects, resources and events.
- Check out the Learner Engagement Analytics Dashboard (LEAD) for instructors.
- Review the Learning Analytics Guiding Principles.