Topics Map > Course Refinement > Informed Design
Are my students understanding the content within my videos
Instructors often assign students the task of watching a video before class with the intent that this content will prepare them for class activities. This content can be a lecture, narrated PowerPoint, demonstration video, or interviews/case studies. When students do not complete this assignment or after viewing the content, they have not acquired the knowledge or understanding at the desired levels; this can cause problems for both the instructor and the student. Being unprepared often leads to failed classroom learning experiences, an unplanned readjustment in-class activities, and falling back to lecturing on the content covered by the video content.
This document will cover several approaches you can use to monitor students' compliance with watching video content. It will also provide ways of determining whether compliance equals preparation for planned classroom activities.
|
1) Identify the question
Monitoring compliance vs. measuring understanding
Once you have delivered content in ways that generate user data, the next step is to determine what question you want to use to answer. If you want to know whether students watched the video before class, that falls under a monitoring compliance action. If you want to know whether watching the video prepared students for future learning, that is a measuring understanding action. These two approaches have slightly different solutions.
2) Identify data collection process
Identifying data sources for understanding
Viewing a video doesn't mean that a student paid attention, understood the content, or was even present while the video was playing. There are additional steps you can take to guide students through the content and provide opportunities to reflect on the content. The solution presented here involves using the Kaltura Quiz Video tool, which can be used as self-checks for students to allow them to pause and answer a question to determine whether they listened and understood what they just watched. There are four kinds of interactions you can embed into your media:
- Multiple choice — a question that can be answered with only one correct answer out of several choices.
- True/false — a question that can be answered with either a "true" or "false" response.
- Reflection point — a video pause to reflect on an idea or concept covered. No answer is recorded.
- Open-ended question — answer a question in your own words.
3) Review the data
Review data for measuring understanding
Viewing a video doesn't mean that a student paid attention, understood the content, or was even present while the video was playing. There are additional steps you can take to guide students through the content and provide opportunities to reflect on the content. The solution presented here involves using the Kaltura Quiz Video tool, which can be used as self-checks for students to allow them to pause and answer a question to determine whether they listened and understood what they just watched. There are four kinds of interactions you can embed into your media:
- Multiple choice — a question that can be answered with only one correct answer out of several choices.
- True/false — a question that can be answered with either a "true" or "false" response.
- Reflection point — a video pause to reflect on an idea or concept covered. No answer is recorded.
- Open-ended question — answer a question in your own words.
Understanding the data
There are a few important caveats to point out here.
- Instructors should know that the information Kaltura MediaSpace presents is not real-time data. It may take some time for the data to be reported. Consider this before assigning points or calling people out for non-compliance.
- While you have access to individual viewing data, be aware that Kaltura MediaSpace analytics may not be able to collect a student's identity if they are running privacy or ad-blocking software. They need to configure their software to allow stats.kaltura.com and analytics.kaltura.com domains access. The viewing results would be collected but aggregated into an Unknown user category.
- Answers to quiz questions can be viewed in Kaltura Analytics. While results can be sent to the Canvas Gradebook, Kaltura QuizVideos doesn't reliably report 100% of students' quiz results. Therefore, we recommend using QuizVideos to measure knowledge and assign and record points formally. If this is the desired outcome, instructors should direct students to complete a Canvas Quiz after viewing the media.
Using Kaltura Quiz Video
Kaltura video quizzes can be used as self-checks for students to allow them to pause and answer a question to determine whether they listened and understood what they just watched. While Kaltura video quizzes can be used as Canvas assignments we do not currently recommend that due to issues outlined in Canvas - Adding a Kaltura Video Quiz As a Canvas Course Assignment [UW-Madison] . We recommend that you keep your video quizzes to a maximum of 4-6 minutes in length.
Note: Since Kaltura video quizzes don't work reliably for 100% of students we recommend you use video quizzes for student self-testing and preparation with no associated points or grade, and Canvas quizzes for anything with points or a grade.
Creating Kaltura video quizzes
- Launch Chrome or Firefox, go to Kaltura MediaSpace, and login: mediaspace.wisc.edu. If you need more information on how to log in to MediaSpace please see Kaltura - How to Log in to Kaltura MediaSpace [UW-Madison] .
- Click Add New and select Video Quiz:
- The Editor / Media Selectionscreen will display. You can :
- Search for a video to use as the basis for a video quiz
- Select an existing video in your collection to use as the basis for a video quiz.
- + Upload Media to to add a new video to use with a video quiz.
- Once you've selected a video you want to use as a quiz, clicking Select will open the Kaltura editor. Note that it can take a few minutes to load on longer videos:
Editing quiz settings
Toggle the settings menu to visible or hidden by clicking the icon next to Quiz:
Details settings
Details allows you to edit the name and whether a welcome message displays along with the welcome message for the quiz.
- If it isn't expanded click the black arrow next to Details:
Be sure to also notice the scroll bar on the right. If you scroll down you can also check or uncheck boxes to Allow Download of Questions List and provide some basic Instructions:
Scores settings
Scores allows you to configure:
- Do Not Show Scores - A "Thank You" message will display after submitting the quiz, or
- Show Scores - Your viewers will see the quiz score page after the quiz is submitted.
- The Include Answers checkbox determines whether viewers will be able to see correct/incorrect answers:
Experience settings
Under Experience you can:
- Allow Answers Change - Allow viewers to change their answers before submitting a quiz.
- Allow Skip - Allow viewers to skip questions.
- Scroll down to see the option to set No seeking forward:
Generally we do not recommend instructors disable the ability for students to seek forward as it reduces student control and can make troubleshooting video quizzes more difficult.
Adding questions to your video quiz
- To add a question to your video you can play the video and pause where you would like to place a question, or grab the time tag and drag it to the appropriate place in the video:
- Once you are at the appropriate point in the video, click the blue Add a Question button. You can choose from:
- Multiple Choice: Questions with only one correct answer. Questions have a 180 character limit for questions and 140 character limit for answers.
- True / False: Question with the choice of True or False. Questions have 450 character limit.
- Reflection Point: This is a question with no answer. The video will pause and allow you to point out specific items in the video to guide your viewer's attention. This is a non-graded question and will not be part of the quiz score. Questions have 500 character limit.
- Open Question: The student can type in a free text answer. This is a non-graded question and will not be part of the quiz score. Questions have a 200 character limit. Answers have a 270 character limit.
- The question editing screen will display for your question and will vary based on question type. For more detail please see Kaltura's documentation Kaltura Video Tools - Create a video quiz. For multiple choice questions you have options to:
- Click the shuffle button to shuffle the answers
- Click the light bulb button allows you to add text hints to the question as well as explanations.
- Add question text.
Note: This is listed above but it's worth restating that you are limited to 180 characters for questions and 140 characters for answers for multiple-choice questions. True-False questions have a 450 character limit. Reflection Point questions have a 500 character limit. Open questions have a 200 character limit for questions and 270 character limit for answers.
It's also important to note that you can't assign different point values to each question. If you decide to use the video quiz as a Canvas assignment (outlined in Canvas - Adding a Kaltura Video Quiz As a Canvas Course Assignment [UW-Madison] ), the point value will be divided up to each video question based on the total points you give the assignment in Canvas. - Add correct and incorrect answers.
- Add additional answers:
- Click the shuffle button to shuffle the answers
- Click Save when you are done editing your question:
- Once saved a cube icon will appear on the timeline indicating that a question will appear at that point in the timeline.
- You can click the cube to edit the question
- You can also use the timeline zoom to increase or decrease the magnification on the timeline:
- When you are done creating questions and have set the settings you can click Done in the upper right of the editor:
Editing an existing Kaltura video quiz
- Launch Chrome or Firefox, go to Kaltura MediaSpace, and login: mediaspace.wisc.edu. If you need more information on how to log in to MediaSpace please see Kaltura - How to Log in to Kaltura MediaSpace [UW-Madison] .
- Click your name in the upper right and select My Media:
- Search for or scroll until you locate the video quiz you want to edit. Select the pencil icon to the right of the video quiz you want to edit:
- Click the Launch Editor button from this screen:
Alternately if you clicked on the name of your video quiz you can click the Actions button and select the Launch Editor option from the drop-down menu. - Choose the Quiz button on the left:
- Follow the prompts in the editor to edit or add questions to the video as outlined in steps 5-10 above.
- Click Done when finished.
Integrating video quizzes with Canvas
Instructors have a few options to integrate Kaltura video quizzes with Canvas:
- Kaltura video quizzes can be embedded in a page or other Canvas area wherever the Canvas rich content editor exists. Embedding video quizzes this way will not send a grade for the student to the Canvas gradebook but can still be useful in helping students self-assess and determine whether they understand what they are watching. We recommend that you only use multiple choice, true-false, and reflection point questions for self assessment quizzes. Students can then be more formally assessed on the content via other assignments, Canvas quizzes, exams, etc. More information on embedding an video quiz is available in Canvas - Upload and Embed a New Kaltura Mediaspace Video in a Canvas Page [UW-Madison] . this will still collect individual analytics on student answers and viewing.
- Kaltura video quizzes can also be embedded by using embed code. How to find a media entry's embed code is described in Kaltura - How to Get a Link or Embed Code to a Video in MediaSpace [UW-Madison] . This will not collect individual student analytics.
- Kaltura video quizzes can be setup as a Canvas assignment. When setup this way video quiz grades are sent to the Canvas gradebook. See Canvas - Adding a Kaltura Video Quiz As a Canvas Course Assignment [UW-Madison] for information on setting up video quizzes as a Canvas assignment. Unfortunately due to Kaltura video quizzes not working reliably for 100% of students we do not currently recommend they be used as assignments. We are providing the information for those who want to see how it works.
Kaltura video quiz example
Play the following video to see an example of how a Kaltura video quiz works:
Viewing Quiz Results
Additional information on Kaltura video quiz analytics is available in Kaltura's document, Quiz Reporting and Analytics.
To review Kaltura video quiz analytics:
- Launch Chrome or Firefox and log in to Kaltura MediaSpace. You can also access your Kaltura MediaSpace media from inside Canvas by using Kaltura My Media as described in Canvas - Accessing Your Kaltura Media in Canvas with Kaltura My Media [UW-Madison] .
- Search for or navigate to the video quiz and click it to select it.
- Select Actions > Analytics underneath your quiz video. The video quiz analytics page will load.
- Click the Quiz Questions tab to see a summary of your quiz questions and student answers.
- Click the Export to CSV button to get a CSV spreadsheet summary of the questions, answers, and total percentage of students that got the questions right.
- Click Printable Version to get an HTML page of the questions, answers, and total percentage of students that got the questions right.
- Click a question to display a summary of responses.
- Click the Quiz Users tab to see more detailed information on how individual students answered a video quiz.
- Click the Export to CSV button to get a CSV spreadsheet summary of the User ID, first name, last name, answered right, answered wrong, and score data.
- Click Printable Version to get an HTML page summary of the User ID, first name, last name, answered right, answered wrong, and score data.
- Click the student name to display a summary of how they answered the video quiz questions.
- Click the trash can icon to delete the student's video quiz attempt. If you have limited the number of video quiz attempts and the student has used them up, this will allow the student to re-take the video quiz.