Facilitators: Cid Freitag, Center for Teaching, Learning & Mentoring (CTLM) | Kari Jordahl, Learning Analytics Center of Excellence (LACE)

What questions do you have about the student experience? What data or information would help answer those questions, and are there strategies you can incorporate into your course to help you find answers? 

Slides  |  Recording

Presenter: Jonathan Gallimore, Psychology 

Jonathan shared his analysis and facilitated a conversation from considering questions in his course; for example, does attendance predict exam grades? Can students' access of course materials predict exam scores? How do late or missing assignments impact overall grades? 

Slides  |  Recording 

Participants from both years of the program discussed their action research projects, and shared some of the lessons learned and impacts. They spoke about the questions they explored, the data they accessed, and the actions they took or are planning to take.

• Liza Chang and Diana Frantz Anderson with CALS
• Sarah Ann Zurawski with School of Education/OT
• Christa Wille with Engineering (in collaboration with Kate Fu)
• Amanda Margolis with Pharmacy (in collaboration with Andrea Porter)
• Beth Janetski with Pharmacy (in collaboration with Casey Gallimore)

Slides | Recording

Presenters: Kari Jordahl, Dan Voeks & Kim Arnold; Learning Analytics Center of Excellence (LACE)

We have done numerous rounds of student engagement to find out what kinds of data might be useful for students. Find out more about what students have asked for, what types of approaches, tools and visualizations that students are interested in. We also discussed My Learning Analytics (MyLA), a student-facing tool that was created by the University of Michigan and that we're planning to pilot during fall semester.  

Slides | Recording

Presenters:  Cid Freitag, CTLM & Kari Jordahl, LACE

What resource, event or article has been impactful for you recently? We looked at a Big Ten Academic Alliance report about learning analytics efforts across numerous institutions, and briefly discussed several Unizin Summit sessions. Then we shifted towards thinking about AI and analytics, and looked at an example of a Tableau tool that uses AI.

Here's some interesting questions and resources to explore: 

• What are our peer institutions doing with learning analytics? UW-Madison is part of the Unizin Consortium and is also a member of the Big Ten Academic Alliance. 
  • Check out the Big Ten Academic Alliance Learning Analytics Interest Group Landscape Report to learn more about how LA is supported and organized, tools and approaches, programming, and opportunities and barriers. 
  • Did you attend the Unizin Summit? This year we heard about tools and approaches to support students, instructors and advisors, along with how AI is being incorporated in  teaching. Recordings are available on YouTube.
• Envisioning the future - AI is everywhere already- how do you see that intersecting more with learning analytics? Are you running into AI additions in the tools you use? 
  • The Instructure Community (Canvas) has an Artificial Intelligence in Education group with some interesting posts.
  • One tool example - Tableau is promoting their Einstein tool - to help ask queries. 
  • Check out this Kaltura blog from 2022 “AI in Education - All you Need to Know” which makes some good points, and is also not going to be all you need to know….
  • 2024 EDUCAUSE Horizon Action Plan: Unified Data Models - What is a unified data model for learning analytics, and why does it matter for higher education? What opportunities does it offer?  “Unified data models for learning analytics have been identified as one of the key technologies and practices anticipated to have a significant impact on the future of higher education. The 2024 Action Plan envisions a future.” 

Shared notes | Recording

Presenter:  Kari Jordahl, LACE

We started out the fall semester talking about strategies for getting started working with learning analytics.

• We reviewed the campus definition and cultural change model for LA, and looked at some examples of questions, data to help answer those questions, and potential actions that could be taken.
• We considered different stakeholder groups - instructors, students and advisors - and their specific questions and needs.
• We reviewed a framework for using learning analytics throughout the semester in a typical Canvas course, and some examples of where to find data for various questions.
• We used Canvas New Analytics as an example, to see where to find answers to some common questions that instructors often have.
• We also discussed ethical guidance around using learning analytics and talked about caveats and potential limitations of using analytics.

Slides | Recording

Resources & Opportunities:

• Getting started with learning analytics: 
  • What are the pedagogical uses of learning analytics? 
  • Learning Analytics Guiding Principles - review the four principles, along with stakeholder responsibilities, data that's in-scope and out-of-scope for learning analytics.
  • Share the Teaching & Learning Data Transparency Statement with students to make sure they are informed about how data is used and protected.
  • Peruse the Knowledge Base for learning analytics topics and specific tool analytics, including Canvas New Analytics and Kaltura Analytics.
• Find out about various approaches and review examples of learning analytics in practice
  • Explore the Learning Analytics Functional Taxonomy, for 6 different approaches for LA; each category has UW-Madison examples, and also links to other institutional examples.
  • Participate in other Learning Analytics Community of Practice sessions (offered monthly during the academic year, generally the 2nd Friday of each month)
  • Access resources from past sessions, including a session summary and links to the slides and recordings. Learn more about topics that you've missed.
• Join the UW-Madison Learning Analytics Team - share information about events, projects, LA in the news.
• Request a consultation - contact the Help Desk
• Find out more about the Learning Analytics Center of Excellence (LACE) 

Presenters: Dan Voeks, LACE & Mary K. Thompson, SLA

Data stewards Dan and Mary provided an overview of what data governance is, and how it's structured and managed across UW-Madison. We heard about our institutional data policy and considered different types of data - institutional data, teaching and learning data, and student record data. Then we discussed some real-life situations about accessing and using data - what do you need to know when you're using data to support teaching and learning? We considered these scenarios and discussed things to consider for each one:

1. You often work with student data in your area and have access to student data including grades as part of your role on campus. A close, trusted colleague wants access to the data.
   
2. You are a campus instructor, and you want to engage students in the classroom by trying out a new software tool you found online.
3. You are working closely with an instructor over the course of a semester on belonging and what that looks like for students in a large course. The instructor has become interested in publishing the data collected with you as a research paper.
   
4. You have been asked by leadership to collect teaching and learning data about student performance by demographic groups. What are some questions you might want to pose and what are some of the risks you should consider?

Please check out the slides/recording to see and hear thoughtful responses and the discussion.

Slides  | Recording

Resources

• Access the Data Governance, Academic Planning & Institutional Research (DAPIR) page to learn more about Data Governance. The Data Governance dropdown provides access to a program overview, descriptions of the various data domains across campus and their data trustees and data stewards. Links to policy and training opportunities are also provided. 
• Review guidance around using learning analytics and teaching and learning data. 
  • Learning Analytics Guiding Principles (May 2019)
  • Teaching and Learning Data Guiding Principles (May 2024)
  • Teaching and Learning Data Transparency Statement

Training 

• Numerous learning opportunities are posted on the Data Literacy & Training page (all online).
  
  • Data@UW: Data Governance and Responsible Use - online course in Canvas - self-register to access this asynchronous, brief course. 
  • Data Literacy track
  • Data Management track
  • Links to other training - eg. FERPA, HIPAA, Cybersecurity and more. 

Presenter: Cid Freitag, CTLM

Cid Freitag from the Center for Teaching, Learning and Mentoring facilitated a discussion about exploratory and explanatory data analysis methods, emphasizing the role of visualizations for both methods. Exploratory analysis is about making meaning - finding insights, using analysis to view trends and relationships. Explanatory analysis is about a selected, specific view of the data that provides a message for an anticipated audience. We reviewed many examples of visualizations, and considered how they complement statistical analytics, can reveal patterns and lead to new insights. 

We explored a hypothetical research process that began as an initial exploration of a dataset, walking through numerous steps and considering the role of visualizations in exploring the data through multiple types of visualizations. Moving from exploration to insights, checking the hypothesis and communicating to specific audiences involved multiple iterations and different visualizations and tools. A data storytelling framework provided a structure for creating visual communication that met the project goal and the audiences’ need - focusing attention on key points. 

Slides | Recording 

Resources: 

• Yi, Ji Soo, et al. "Understanding and characterizing insights: how do people gain insights using information visualization?." Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization. 2008.
• North, Chris. "Toward measuring visualization insight." IEEE computer graphics and applications 26.3 (2006): 6-9.
• Shao, Hongbo, et al. "Data Storytelling in Data Visualisation: Does it Enhance the Efficiency and Effectiveness of Information Retrieval and Insights Comprehension?." Proceedings of the CHI Conference on Human Factors in Computing Systems. 2024.
• Zdanovic, Dominyk, Tanja Julie Lembcke, and Toine Bogers. "The influence of data storytelling on the ability to recall information." Proceedings of the 2022 Conference on Human Information Interaction and Retrieval. 2022.
• Danyel Fisher and Miriah Meyer, Making Data Visual. O’Reilly, 2018
• Blogs:
  • The Data School: Communicating via Explanatory vs. Exploratory Analysis
  • Storytelling with Data: exploratory vs explanatory analysis
  • DataJournalism.com: Using Data Visualization to Find Insights in Data
  • Data Storytelling: Separating Fiction From Facts

We had a lively session at the November LA CoP, where we met in Zoom and had several topical breakout sessions with lots of experts on hand to facilitate discussions. The goal for this session was to allow participants to connect and discuss topics of interest in smaller groups. While we didn't record the session, we did collaboratively take notes in a shared doc, which includes notes and links to useful resources:  https://go.wisc.edu/v2z11f 

• Assessment and Evaluation - When are surveys helpful? What can surveys do well/not do well? (Mary K Thompson & Chad Shorter)
• Using Learn@UW Tools and T&L Data - What kinds of data can you get from X tool? What are common problems or questions instructors come to you for that X tool can help solve? (James McKay, Andrew Jason Turner, Andrew Reinke)
• Active Learning and Classroom Observations - How can I structure classroom activities that facilitate learning and generate useful data? How does classroom observation work? Does it benefit students or instructors? (Lisa Jong & John Martin)
• Teaching and Learning Data Environment - What are common questions that our T&L data could help instructors answer? What are some challenges to answering certain questions due to our T&L data environment? (Chris Lalande & Dariane Drake)
• Applied examples of LA in the classroom -
  • Measuring Student Team Dynamics - What is a challenge or a lesson learned while working with student teams? Some ideas for thinking about group work, how to iterate/improve on it, and measure its effectiveness. (Christa Wille)
    
  • T&L Project Iteration and Transparency - What is a challenge faced or a lesson learned while improving the T&L in class? How to iterate on a T&L project? How to continue to engage with students and be transparent about this work? (Sarah Zurawski)
    

Join this discussion where we'll review some course structure/organization practices that also lead to better T&L data. Bring your examples of what has worked or not worked well for you - we'll look at 4 general recommendations for course design that also can help capture better data: 

• Gradebook practices
• Consistent naming conventions
• Course structure and organization 
• Incorporating third party tools

Presenters: Kim Arnold, Learning Analytics Center of Excellence (LACE) | Jeff Shokler, Office of Academic & Career Success (OACS) | Mary K. Thompson, Student Learning Assessment (SLA)

We discussed the high-level organization and structure of the DEEP framework, and heard updates about a couple of the working groups and their projects.

Slides | Recording

Presenter: Dariane Drake, Learning Analytics Center of Excellence (LACE)

Dariane talked about types of biases and helped us consider how we learn and reinforce them. We looked at case studies and examples, and reviewed tips for increasing awareness.

Slides | Recording 

Presenters: Diana M. McFarland & Janae Winston, Educational Psychology MS program in Learning Analytics

We heard from two students in the first cohort of the Educational Psychology MS program on learning analytics: Diana Marie McFarland and Janae Winston. They shared their capstone projects and reflected on their unique perspectives with learning analytics.

Slides | Recording

Group discussion 

We created a google doc and shared notes, insights and links and heard about the microgrant program, the Unizin Spring meeting, several NYU LEARN recent events about subversive LA and equity oriented LA.

Shared notes | Recording 

Presenters: Dan Voeks, Learning Analytics Center of Excellence (LACE) |  Mary K. Thompson, Student Learning Assessment (SLA)

Dan and Mary provided a brief review/introduction to our institutional data policy and data governance, and then we spent the rest of the session discussing and considering some real-life situations. For example: 

• You often work with student data and have access to student data, including grades as part of your role on campus. A close, trusted colleague wants access to the data. What are some things to consider? 
• You are a campus instructor and you want to engage students in the classroom by trying out a new software tool you found online. What are some things to consider?

Slides | Recording

Presenters: James McKay, Learn@UW  | Andrew Reinke, Learn@UW |  Keegan Shaw, Learn@UW 

James McKay showed us Piazza (for discussions); Canvas Analytics & New Analytics (which are no longer new but still branded that way); and TopHat (student response system).  

•  Piazza starts at 8:45 mins
•  Canvas Analytics/New Analytics - starts around 13.25 mins
•  TopHat starts at 24 mins

Andrew Reinke provided an overview about Kaltura Analytics, and showed us a couple of ways to access analytics which varies based on how you’ve embedded media into your course. He shared slides and a Kaltura terminology doc as additional resources.

• Kaltura Analytics starts at 27:56 min

Keegan Shaw discussed Atomic Assessments analytics, which integrates directly into Canvas and offers advanced quizzing functionality.

• Atomic Assessments - starts at 41 mins

Slides | Recording 

Presenter: Clare Huhn, Data, Academic Planning & Institutional Research (DAPIR)

Clare reviewed the overall structure of Tableau and how to access UW-Madison workbooks. These institutional resources can answer a variety of questions, for example: how are curricular subjects related to departments? What are the learning outcomes for a course? Are there differences in average grades between terms? Are there gaps in D/F/drop rates by demographic groups? 

You have access to the workbooks we explored with your UW-Madison credentials from either RADAR or the Tableau server web pages (you must use VPN if you are not on campus). Here's the workbooks we explored:

• • UW-Madison Academic Structure
  • UW-Madison Course Catalog 
  • Course Grade Distribution
  • Undergraduate Course D/F/Drop Rates
  • Courses Completed by Bachelor’s Degree Recipients
  • Course Demographics Profile

Slides | Recording

Presenters: Kari Jordahl, Learning Analytics Center of Excellence | Cid Freitag, Center for Teaching, Learning & Mentoring

How you design your course will affect what data you can see in various learning analytics tools. In this session we discussed tips for designing a course to help create useful data, including; 

• Review and reflect; what is your question? What tools/data are available?
• Identify a pivotal assignment or time in the course
• Use logical naming conventions and clear navigation
• Reminders about how to link to content (eg. Kaltura videos)

We also discussed using course blueprints or instructional design frameworks (Horton’s Absorb - Do - Connect) when designing a course, to ensure learning analytics is part of the overall course design and strategy.

Resources:

• Data Informed Design for Online Course Improvement (Chris Millet & Jessica Resig) book chapter in A Practitioner's Guide to Instructional Design in Higher Education.
• Transparency for Learning and Teaching Project: TILT Higher Ed

Slides | Recording 

Presenter: Cid Freitag, Center for Teaching, Learning & Mentoring

The Learner Engagement Analytics Dashboard (LEAD) can help identify students who may be struggling. Which visualization is most useful for you? Do you use other tools or methods to identify struggling students? 

Cid facilitated a discussion about how to translate data into action - using a framework that discussed Canvas tools and achievable actions - before the course starts, at the beginning of the term, during the term, and after the end of the course.  (Muljana, Pauline Salim, and Greg Placencia. "Learning analytics: Translating data into “just-in-time” interventions." Scholarship of Teaching and Learning, Innovative Pedagogy 1.1 (2018): 6.)

Slides| Recording

Presenters: Mary K. Thompson, Student Learning Assessment  | Saundy Solum, DoIT Academic Technology | Regina Lowery, Student Learning Assessment

Direct evidence of student learning (DESL) is an assessment approach that allows instructors to evaluate learning based on student performance on tasks directly connected to course learning outcomes. We learned how Canvas assignments can be linked to AEFIS Course Learning Outcomes, or CLOs and saw a demo of AEFIS. We also discussed the difference between grading and assessment, formative and summative assessment, the benefits of DESL, and how to get started using it. 

Slides | Recording 

Presenter: Cid Freitag, Center for Teaching, Learning & Mentoring (CTLM)

This session focused on one aspect of data literacy — being a consumer of data visualization products, such as those in the news, popular media, and data dashboards. By analyzing a visualization we focused on making meaning and thinking critically about the meaning and message it represents — preparing us to become a better consumers of visualizations. 

Slides | Recording 

Group discussion 

Spring conference season - what themes and new ideas were hot topics at events that you've attended (or that you wished you could attend)? Whether you had a chance to attend LAK'22 in March, the Unizin Summit in late April or the Indiana University LA Summit in mid-May (or other sessions we've missed) this conversation was open for all to share.

Shared notes | Recording 

Facilitators/presenters - team presentation facilitated by Kari Jordahl; Dan Voeks, Learning Analytics Center of Excellence | Shannon Harris, Learning Analytics Center of Excellence | Mary K. Thompson, Student Learning Assessment n

At this session we spent a little time providing information and updates on the LA landscape at UW-Madison, including the DEEP framework and our institutional data policy, and transparency around using data. We learned about enhancements for the Learner Engagement Analytics Dashboard (LEAD) and heard about Direct Evidence of Student Learning (DESL). We shared information about the new microgrant program - for exploring data-empowered educational practices/learning analytics in support of diversity, equity, inclusion and belonging. We also discussed the schedule and planned topics for fall LA CoP, while soliciting ideas for later sessions. 

Slides | Recording

Presenters: Chad Shorter, Learning Analytics Center of Excellence (LACE)  |  Maria Dahmen, Center for User Experience (CUE)

What types of learning analytics do students want? How do we empower students to use data about them, to support their own learning? 

We discussed what UW-Madison students think about learning analytics - what types of data and visualizations do they think are useful? How do they feel about data about them being used, when it’s used to support their learning? DoIT Academic Technology conducted several phases of student engagement over the past couple of years and this combined feedback from across those efforts.

Slides  |  Recording 

Presenter: Andrew Reinke, Learn@UW 

There are numerous ways to incorporate media into your course, and the analytics you can access will vary, based on your approach.  

• Individual media analytics
• Mediaspace My Analytics dashboard
• Canvas embedding options for best analytics
• Kaltura Gallery in Canvas Analytics
• Interactive video quizzes (for self-check/not for gradebook integration) 

Slides  |  Recording

Presenters: Cid Freitag, Center for Teaching, Learning & Mentoring (CTLM) | Kari Jordahl, Learning Analytics Center of Excellence (LACE)

We looked at a framework that can be used for learning analytics approaches. The Review-Amend-Apply framework (created at Indiana University) provides a useful and very approachable 3-step process. We discussed how it aligns (or doesn't align exactly) with the UW-Madison approach for learning analytics which focuses on a question that leads to data that leads to action.

• Review the information available to you about your students' behavior, your course, your teaching
• Amend the information you've gathered - are there other resources or sources of data? 
• Apply what you've learned - what modification or change might help? Consider evidence-based teaching practices. 

Slides | Recording

Presenters: James McKay, Shannon Harris, Cid Freitag, Kari Jordahl, all with DoIT Academic Technology

Sessions were offered frequently throughout the year.  

We explored the three visualizations in the Learner Engagement Analytics Dashboard (LEAD), and how to filter data to answer questions about courses, teaching, and students' activity in a  course. LEAD aggregates data from Canvas, Kaltura and Unizin Engage eText and offers unique insights about for-credit courses.

Slides (no recording)

Presenters: James McKay, Shannon Harris, Cid Freitag, Kari Jordahl, all with DoIT Academic Technology

Sessions were offered frequently throughout the year.  

We explored the three visualizations in the Learner Engagement Analytics Dashboard (LEAD), and how to filter data to answer questions about courses, teaching, and students' activity in a  course. LEAD aggregates data from Canvas, Kaltura and Unizin Engage eText and offers unique insights about for-credit courses.

Slides (no recording)

Presenter: Cid Freitag, CTLM

We focused on student engagement in a course, and identifying struggling students using LEAD visualizations. How can learning analytics help you identify students who may need more support? We also discussed actions you can take in Canvas throughout various times in the semester, to support students. 

(Muljana, Pauline Salim, and Greg Placencia. "Learning analytics: Translating data into “just-in-time” interventions." Scholarship of Teaching and Learning, Innovative Pedagogy 1.1 (2018): 6.)

Slides (no recording)

Presenters: Cid Freitag, Kari Jordahl, James McKay, Shannon Harris, all with DoIT Academic Technology

Do you use LEAD, Canvas New Analytics or Kaltura Analytics? Are you using Engage or other e-texts? We looked at some commonly asked questions about learning analytics and the Learner Engagement Analytics Dashboard (LEAD). 

Slides (no recording)

Presenters: Cid Freitag, Kari Jordahl, James McKay, Shannon Harris, all with DoIT Academic Technology

What worked well this semester? Are there changes you’d like to consider making in your course? Do you have instructional design strategies that worked well to leverage learning analytics? Interrogate your course data to discover which resources were most/least used, what times students accessed course content and activities. What strategies might help support teaching and learning?

Google doc (no recording)

Contributors: group discussion

Many spring conferences have learning analytics tracks or areas of focus, and one pandemic upside is that we can attend more conferences. This group discussion includesd updates from the Learning Analytics & Knowledge (LAK) conference, and the Unizin Summit 2021. 

Shared notes doc, links out to more resources (no recording)

Presenters: Kari Jordahl, LACE & Cid Freitag, CTLM

We facilitated a session at the Learning Design Community of Practice focusing on LA and course design. The learning analytics landscape and recent efforts at UW-Madison were discussed, and then we looked at analytics in tools that instructional designers typically have access to - eg. Canvas Analytics & New Analytics, Kaltura, etc. 

Slides  | (no recording)

Presenters: James McKay, Learn@UW; Kari Jordahl, LACE & Cid Freitag, CTLM

Have you explored learning analytics options available in Canvas? There are several locations in Canvas where you can access analytics, to answer questions you might have about your course and your students.

Slides | (no recording)

Presenters:  Kari Jordahl, Chad Shorter, Shannon Harris, all with Learning Analytics Center of Excellence (LACE)

This was a more general discussion around the use of learning analytics broadly. We discussed various ways to use learning analytics and talked about how course design impacts analytics, how analytics can provide insights and help you answer some questions, while potentially leading to more questions.

Slides  | Recording

Presenters: Andrew Reinke, Learn@UW & David Macasaet, L&S Learning Support Solutions 

We looked at several ways to incorporate media into a course with Andrew Reinke, and then looked at the visualizations and analytics that result from various methods.

David Macasaet discussed how he used Kaltura Analytics to review student engagement with video content in his course, and how student engagement improved after he made several modifications to the content.

Slides  | Recording