Topics Map > Course Refinement > Informed Design

Are my students understanding the content within my videos

Course Improvement

Determining whether students are understanding the content covered within an assigned video

Instructors often assign students the task of watching a video before class with the intent that this content will prepare them for class activities. This content can be a lecture, narrated PowerPoint, demonstration video, or interviews/case studies. When students do not complete this assignment or after viewing the content, they have not acquired the knowledge or understanding at the desired levels; this can cause problems for both the instructor and the student. Being unprepared often leads to failed classroom learning experiences, an unplanned readjustment in-class activities, and falling back to lecturing on the content covered by the video content.

This document will cover several approaches you can use to monitor students' compliance with watching video content. It will also provide ways of determining whether compliance equals preparation for planned classroom activities.

|

1) Identify the question

Monitoring compliance vs. measuring understanding

Once you have delivered content in ways that generate user data, the next step is to determine what question you want to use to answer. If you want to know whether students watched the video before class, that falls under a monitoring compliance action. If you want to know whether watching the video prepared students for future learning, that is a measuring understanding action.  These two approaches have slightly different solutions.

2) Identify data collection process

Identifying data sources for understanding

Viewing a video doesn't mean that a student paid attention, understood the content, or was even present while the video was playing. There are additional steps you can take to guide students through the content and provide opportunities to reflect on the content. The solution presented here involves using the Kaltura Quiz Video tool, which can be used as self-checks for students to allow them to pause and answer a question to determine whether they listened and understood what they just watched. There are four kinds of interactions you can embed into your media:

  1. Multiple choice — a question that can be answered with only one correct answer out of several choices.
  2. True/false — a question that can be answered with either a "true" or "false" response.
  3. Reflection point — a video pause to reflect on an idea or concept covered. No answer is recorded.
  4. Open-ended question — answer a question in your own words.

3) Review the data

Review data for measuring understanding

Viewing a video doesn't mean that a student paid attention, understood the content, or was even present while the video was playing. There are additional steps you can take to guide students through the content and provide opportunities to reflect on the content. The solution presented here involves using the Kaltura Quiz Video tool, which can be used as self-checks for students to allow them to pause and answer a question to determine whether they listened and understood what they just watched. There are four kinds of interactions you can embed into your media:

  1. Multiple choice — a question that can be answered with only one correct answer out of several choices.
  2. True/false — a question that can be answered with either a "true" or "false" response.
  3. Reflection point — a video pause to reflect on an idea or concept covered. No answer is recorded.
  4. Open-ended question — answer a question in your own words.

Understanding the data

There are a few important caveats to point out here.

  1. Instructors should know that the information Kaltura MediaSpace presents is not real-time data. It may take some time for the data to be reported. Consider this before assigning points or calling people out for non-compliance.
  2. While you have access to individual viewing data, be aware that Kaltura MediaSpace analytics may not be able to collect a student's identity if they are running privacy or ad-blocking software. They need to configure their software to allow stats.kaltura.com and analytics.kaltura.com domains access. The viewing results would be collected but aggregated into an Unknown user category.
  3. Answers to quiz questions can be viewed in Kaltura Analytics. While results can be sent to the Canvas Gradebook, Kaltura QuizVideos doesn't reliably report 100% of students' quiz results. Therefore, we recommend using QuizVideos to measure knowledge and assign and record points formally. If this is the desired outcome, instructors should direct students to complete a Canvas Quiz after viewing the media.

Using Kaltura Quiz Video

Kaltura video quizzes can be used as self-checks for students to allow them to pause and answer a question to determine whether they listened and understood what they just watched. While Kaltura video quizzes can be used as Canvas assignments we do not currently recommend that due to issues outlined in Canvas - Adding a Kaltura Video Quiz As a Canvas Course Assignment [UW-Madison] . We recommend that you keep your video quizzes to a maximum of 4-6 minutes in length.

Note: Since Kaltura video quizzes don't work reliably for 100% of students we recommend you use video quizzes for student self-testing and preparation with no associated points or grade, and Canvas quizzes for anything with points or a grade.

   

Creating Kaltura video quizzes

Users can use Kaltura to add questions to a video with Kaltura's video quiz feature. For additional information on Kaltura video quizzes please see Kaltura's documentation Kaltura Video Tools - Create a video quiz
  1. Launch Chrome or Firefox, go to Kaltura MediaSpace, and login: mediaspace.wisc.edu. If you need more information on how to log in to MediaSpace please see Kaltura - How to Log in to Kaltura MediaSpace [UW-Madison] .
  2. Click Add New and select Video Quiz:
    A screenshot showing the user having clicked on the "Add New" drop-down menu. The cursor hovers over "Video Quiz which is outlined in orange to help point it out.
  3. The Editor / Media Selectionscreen will display. You can :
    1. Search for a video to use as the basis for a video quiz
    2. Select an existing video in your collection to use as the basis for a video quiz.
    3. + Upload Media to to add a new video to use with a video quiz.
    4. A screenshot showing the Kaltura MediaSpace Editor / Media Selection screen. Callouts indicate (1) the search dialog, (2) the Select button, and (3) the Upload Media button.
  4. Once you've selected a video you want to use as a quiz, clicking Select will open the Kaltura editor. Note that it can take a few minutes to load on longer videos:
    A screenshot showing the Kaltura Editor - specifically the quiz functions.

Editing quiz settings

Toggle the settings menu to visible or hidden by clicking the icon next to Quiz:

A screenshot showing the button to toggle the Kaltura Interactive Video Quiz settings to visible or hidden.

Details settings

Details allows you to edit the name and whether a welcome message displays along with the welcome message for the quiz.

  1. If it isn't expanded click the black arrow next to Details:
    A screenshot showing a Kaltura IVQ with the Details pane expanded to show the "Quiz Name" field and "Show Welcome Page" checked along with the welcome message.
    Be sure to also notice the scroll bar on the right. If you scroll down you can also check or uncheck boxes to Allow Download of Questions List and provide some basic Instructions:
    A screenshot showing the user having used the scroll bar under "Details" to scroll down to display the options for "Allow Download of Questions List" and "Instructions".

Scores settings

Scores allows you to configure:

  1. Do Not Show Scores - A "Thank You" message will display after submitting the quiz, or
  2. Show Scores - Your viewers will see the quiz score page after the quiz is submitted.
  3. The Include Answers checkbox determines whether viewers will be able to see correct/incorrect answers:
    A screenshot showing the Kaltura video quiz "Scores" options including whether or not to show the scores and whethe to let viewers see the correct answers.

Experience settings

Under Experience you can:

  1. Allow Answers Change - Allow viewers to change their answers before submitting a quiz.
  2. Allow Skip - Allow viewers to skip questions.
    A screenshot showing the Kaltura IVQ option to "Allow Answers Change" and "Allow Skip".
  3. Scroll down to see the option to set No seeking forward:
    A screenshot showing the Kaltura IVQ option "No seeking forward."
    Generally we do not recommend instructors disable the ability for students to seek forward as it reduces student control and can make troubleshooting video quizzes more difficult.

Adding questions to your video quiz

  1. To add a question to your video you can play the video and pause where you would like to place a question, or grab the time tag and drag it to the appropriate place in the video:
    A screenshot showing the Kaltura IVQ timeline. The time tag is outlined in red.
  2. Once you are at the appropriate point in the video, click the blue Add a Question button. You can choose from:
    A screenshot of the Kaltura MediaSpace editor. The user has clicked the blue "Add a Question" button and displayed the four types of questions that can be added: Multiple choice, Reflection Point, True/False, and Open-Ended Question.
    1. Multiple Choice: Questions with only one correct answer. Questions have a 180 character limit for questions and 140 character limit for answers. 
    2. True / False: Question with the choice of True or False. Questions have 450 character limit.
    3. Reflection Point: This is a question with no answer. The video will pause and allow you to point out specific items in the video to guide your viewer's attention. This is a non-graded question and will not be part of the quiz score. Questions have 500 character limit.
    4. Open Question: The student can type in a free text answer. This is a non-graded question and will not be part of the quiz score. Questions have a 200 character limit. Answers have a 270 character limit.
  3. The question editing screen will display for your question and will vary based on question type. For more detail please see Kaltura's documentation Kaltura Video Tools - Create a video quiz. For multiple choice questions you have options to:
    1. Click the shuffle button to shuffle the answers
    2. Click the light bulb button allows you to add text hints to the question as well as explanations.
    3. Add question text. 
      Note: This is listed above but it's worth restating that you are limited to 180 characters for questions and 140 characters for answers for multiple-choice questions. True-False questions have a 450 character limit. Reflection Point questions have a 500 character limit. Open questions have a 200 character limit for questions and 270 character limit for answers.

      It's also important to note that you can't assign different point values to each question. If you decide to use the video quiz as a Canvas assignment (outlined in Canvas - Adding a Kaltura Video Quiz As a Canvas Course Assignment [UW-Madison] ), the point value will be divided up to each video question based on the total points you give the assignment in Canvas.
    4. Add correct and incorrect answers.
    5. Add additional answers:
      A screenshot showing the Kaltura IVQ question editor with a multiple choice question.
  4. Click Save when you are done editing your question:
    A screenshot showing the Kaltura IVQ question editor. The "Save" button is outlined in red.
  5. Once saved a cube icon will appear on the timeline indicating that a question will appear at that point in the timeline.
    1. You can click the cube to edit the question
    2. You can also use the timeline zoom to increase or decrease the magnification on the timeline:
      A screenshot showing the Kaltura IVQ timeline with a question on it (in the form of a cube icon), along with the timeline zoom control outlined.

  6. When you are done creating questions and have set the settings you can click Done in the upper right of the editor:
    A screenshot showing a detail of the Kaltura IVQ editor. The "Done" button is outlined in red.

Editing an existing Kaltura video quiz

  1. Launch Chrome or Firefox, go to Kaltura MediaSpace, and login: mediaspace.wisc.edu. If you need more information on how to log in to MediaSpace please see Kaltura - How to Log in to Kaltura MediaSpace [UW-Madison]  .
  2. Click your name in the upper right and select My Media:
    A screenshot showing the user clicking on their name in the upper right to select "My Media" from the drop-down menu.
  3. Search for or scroll until you locate the video quiz you want to edit. Select the pencil icon to the right of the video quiz you want to edit:
    A screenshot showing a detail from the Kaltura MediaSpace "My Media" page. A video is displayed with the cursor over the pencil icon which is also outlined in orange. The user clicks the pencil icon to enter editing mode for that video or quiz.
  4. Click the Launch Editor button from this screen:
    A screenshot showing the Kaltura MediaSpace "Edit" window. The "Launch Editor" button is outlined in orange.
    Alternately if you clicked on the name of your video quiz you can click the Actions button and select the Launch Editor option from the drop-down menu.
  5. Choose the Quiz button on the left:
    A screenshot showing a zoomed in detail of the Kaltura MediaSpace editor. The "Quiz" button is outilned in red - it looks like a small three-dimensional box.
  6. Follow the prompts in the editor to edit or add questions to the video as outlined in steps 5-10 above.
  7. Click Done when finished.

Integrating video quizzes with Canvas

Instructors have a few options to integrate Kaltura video quizzes with Canvas:

  1. Kaltura video quizzes can be embedded in a page or other Canvas area wherever the Canvas rich content editor exists. Embedding video quizzes this way will not send a grade for the student to the Canvas gradebook but can still be useful in helping students self-assess and determine whether they understand what they are watching. We recommend that you only use multiple choice, true-false, and reflection point questions for self assessment quizzes. Students can then be more formally assessed on the content via other assignments, Canvas quizzes, exams, etc. More information on embedding an video quiz is available in Canvas - Upload and Embed a New Kaltura Mediaspace Video in a Canvas Page [UW-Madison] . this will still collect individual analytics on student answers and viewing.
  2. Kaltura video quizzes can also be embedded by using embed code. How to find a media entry's embed code is described in Kaltura - How to Get a Link or Embed Code to a Video in MediaSpace [UW-Madison] . This will not collect individual student analytics.
  3. Kaltura video quizzes can be setup as a Canvas assignment. When setup this way video quiz grades are sent to the Canvas gradebook. See Canvas - Adding a Kaltura Video Quiz As a Canvas Course Assignment [UW-Madison]  for information on setting up video quizzes as a Canvas assignment. Unfortunately due to Kaltura video quizzes not working reliably for 100% of students we do not currently recommend they be used  as assignments. We are providing the information for those who want to see how it works.

Kaltura video quiz example

Play the following video to see an example of how a Kaltura video quiz works:

Viewing Quiz Results

Additional information on Kaltura video quiz analytics is available in Kaltura's document, Quiz Reporting and Analytics. 

To review Kaltura video quiz analytics:

  1. Launch Chrome or Firefox and log in to Kaltura MediaSpace. You can also access your Kaltura MediaSpace media from inside Canvas by using Kaltura My Media as described in Canvas - Accessing Your Kaltura Media in Canvas with Kaltura My Media [UW-Madison] .
  2. Search for or navigate to the video quiz and click it to select it.
  3. Select Actions > Analytics underneath your quiz video. The video quiz analytics page will load. 
  4. Click the Quiz Questions tab to see a summary of your quiz questions and student answers.
    Kaltura MediaSpace video quiz "Quiz Questions" analytics tab. Callouts indicate (1) Export to CSV button, (2) Printable Version button, and (3) An example quiz question that has been clicked on to expand it to view the summary of answers.
    1. Click the Export to CSV button to get a CSV spreadsheet summary of the questions, answers, and total percentage of students that got the questions right. 
    2. Click Printable Version to get an HTML page of the questions, answers, and total percentage of students that got the questions right.
    3. Click a question to display a summary of responses.
  1. Click the Quiz Users tab to see more detailed information on how individual students answered a video quiz.
    Kaltura video quiz "Quiz Users" analytics tab. Callouts indicate (1) Export to CSV button, (2) Printable Version button, (3) an example student name that has been clicked on to expand, and (4) the trash can icon used to delete a student's video quiz attempt.
    1. Click the Export to CSV button to get a CSV spreadsheet summary of the User ID, first name, last name, answered right, answered wrong, and score data.
    2. Click Printable Version to get an HTML page summary of the User ID, first name, last name, answered right, answered wrong, and score data.
    3. Click the student name to display a summary of how they answered the video quiz questions.
    4. Click the trash can icon to delete the student's video quiz attempt. If you have limited the number of video quiz attempts and the student has used them up, this will allow the student to re-take the video quiz.

CTLM Instructional Resources KnowledgeBase Menu

Keywords:
comprehend, understand, video, content, student
Doc ID:
128794
Owned by:
Timmo D. in Instructional Resources
Created:
2023-06-05
Updated:
2024-08-23
Sites:
Center for Teaching, Learning & Mentoring
Resources on ways to measure course effectiveness and reflect on teaching.