Topics Map > Course Refinement > Informed Design
Are students watching my videos?
Yummy Instructors often assign students to watch a video before class to introduce content that will prepare them for class activities. This content can be a lecture, narrated PowerPoint, demonstration video, or interviews/case studies. When students do not complete this assignment or have not acquired the knowledge or understanding at the desired levels, this can cause problems for both the instructor and the student. Being unprepared often leads to failed classroom learning experiences, unplanned readjustment of class activities, and falling back to lecturing on the content covered by the video content.
This document uses the UW-Madison Informed Design Process to guide the instructor through determining whether their students completed the assigned video viewing. It will cover two approaches: monitoring students' compliance with watching video content and measuring students' understanding of the content within the video.
|
1) Clarify your question
Monitoring compliance
Monitoring for compliance involves reviewing data to see whether students have clicked on and viewed the assigned video. Tools like Kaltura MediaSpace provide analytic data to gain insight into student behavior. You can determine the following:
- Whether individual students viewed content,
- Whether they completed viewing content,
- How frequently they reviewed content, and
- Which parts of the content they reviewed?
Be aware, however, that there are no perfect solutions. These approaches provide a more informed view of student behavior but not a complete one. The information provided needs to be viewed and understood within a specific context.
Monitoring understanding
2) Identify data sources
Identifying data sources for compliance
The first step in monitoring student compliance with watching video content is to deliver it in a way that provides you information on their viewing. While there are several ways to deliver media to students to collect data in viewing/comprehension, this document focuses on the most common and simplest approach — an embedded link in a Canvas page to media housed in Kaltura MediaSpace. Using the Kaltura Embed Media tool within Canvas, you can upload existing media files into Kaltura MediaSpace, embed content into a Canvas page, and collect a variety of learning analytic data on the viewing behaviors of that content from students.'
Identifying data sources for understanding
Viewing a video doesn't mean that a student paid attention, understood the content, or was even present while the video was playing. There are additional steps you can take to guide students through the content and provide opportunities to reflect on the content. The solution presented here involves using the Kaltura Quiz Video tool, which can be used as a self-check for students to allow them to pause and answer a question to determine whether they listened and understood what they just watched. While this data can be delivered to the Canvas Gradebook, it is not recommended due to technical reliability issues. It is recommended solely as a self-guided measurement. Instructors, however, can still see the results through the Kaltura MediaSpace tool. There are four kinds of interactions you can embed into your media:
- Multiple choice — a question that can be answered with only one correct answer out of several choices.
- True/false — a question that can be answered with either a "true" or "false" response.
- Reflection point — a video pause to reflect on an idea or concept covered. No answer is recorded.
- Open-ended question — answer a question in your own words.
You may also create and assign a Canvas Quiz to measure students' understanding of video content. Canvas - Best Practices for Delivering Online Quizzes and Exams in Canvas [UW-Madison]
3) Collect the data
3) Data collection for compliance
Uploading and embedding video into a Canvas page
Kaltura MediaSpace is UW-Madison's media streaming service and is integrated with Canvas. Kaltura media can be embedded wherever Canvas allows you to use the rich content editor.
Note: to embed an existing Kaltura Mediaspace item in a Canvas course, please see Canvas - Embedding an Existing Kaltura MediaSpace Video in a Canvas Page [UW-Madison]
We recommend instructors new to using media with their Canvas course start here. Uploading and embedding Kaltura media in a Canvas Page or other Canvas tool is an easy way to make your Kaltura media available to your students. Embedded media provides more context and keeps the student in Canvas rather than sending them to an external link outside of Canvas. It also allows Kaltura to collect more detailed analytics.
- In your Canvas course, go to the Canvas page, discussion post, or quiz discussion you want to embed Kaltura media in. Click Edit.
- Move your cursor to the point in your Canvas item where you want to insert Kaltura media.
- Click the Embed Kaltura Media button which looks like the colorful Kaltura logo. If your browser is not of sufficient width you may need to click the three dot button on the right to expand the menu to see the button:
- A window titled Kaltura Media will open showing the media content associated with your Kaltura account.
- To upload media at this point click Add New and select Media Upload to upload a new video to your account:
- Read and click the "I agree to the above terms and conditions" disclaimer.
- Scroll down to click + Choose a file to upload or drag your MP4 file onto the window:
- A window will open allowing you to select the video file to upload. While Kaltura can process a wide range of video formats, we find MP4 videos generally work well. We recommend 1280 x 720 video as a good balance between file size and quality. We also recommend you try to keep your video files under 2 GB - the larger the file the longer it will take to upload and process.
- Your file will upload. While it uploads you can edit the name, description, and tags.
Upload Speed
Even when Kaltura MediaSpace is running smoothly, other internet service providers may experience slower service, which could limit your upload speed.
- Upload speeds and Kaltura's upload performance varies, but an upload speed of 10 Mbps should be able to upload a 250 MB file to Kaltura in about 4.5 minutes.
Processing
The amount of time it takes Kaltura MediaSpace to process a file depends on file size, duration, quality, and usage.
Generally it takes about 2-3 times the video's duration for Kaltura MediaSpace to display your video in low definition (high definition versions take longer to process). For example, a 15 minute video will usually take about 30-45 minutes to process (in low-definition). When Kaltura MediaSpace is being heavily used, processing times will lengthen and could take up to 24 hours.
If your video takes longer than 24 hours to process, please call the DoIT Help Desk. More detailed information is available in Kaltura's document involvingtranscoding times and quality of service.
- If you want to customize how your video is displayed, see the Editing Media Settings section in this document. Otherwise, select the </>Embed button to add the media.
- After embedding, you will see an icon indicating it is still processing. Once done processing the video will show up on your Canvas page. You can continue to edit the page and add text if you'd like. Once you're done click the Save button:
- Canvas will display your page with your embedded Kaltura video:
Editing Media Settings
You can edit your file name, description, and tags as it uploads. After your file is done uploading you can:
- Edit the file's name, description, tags, and add collaborators. More information on the collaboration feature is available in Kaltura - Media Collaboration: Changing Media Ownership, Adding Co-editors, Co-publishers, and Co-viewers [UW-Madison] .
- Save button on the bottom will simply save your changes to the metadata while your video uploads.
- </> Save and Embed button in the upper right will embed your video using the default settings in your Canvas Page or other area the Canvas rich content editor is available.
- Gear icon to edit your embed settings. An Embed Settings window will display:
- Choose Player: Choose your player based on the type of media you are embedding. As of 8/15/24 all players automatically include a transcript widget. You do not have to select a specific Transcript player. Video users can choose between:
Default player - This player is setup with the most commonly used options. Most users use this player. The download icon may appear to allow for download of closed caption or attached files, but the source video file can not be downloaded.
Download player - Use the download player if you want users to be able to download the original source media file locally to their computer. This can be useful if you have students without reliable high speed internet connections. They could go to a library or coffee shop to download your media to view later. If you choose the download player, viewers can download the original version of the video you uploaded to Kaltura by clicking the Download button in the upper right of the video: - Max Embed Size: Users can choose how large the media is displayed within the Canvas rich content editor.
- Thumbnail Embed: Turning thumbnail embed on renders an image that will display instead of the video player. The student needs to click on the image to load the player. Generally we recommend keeping this option turned off.
- Auto Play: Turning auto play on plays the media immediately when the student loads the page with the embedded media. We recommend you keep auto play off and let the student choose to play the media by clicking the "Play" button.
- Set as my default settings for media type: Video (or Audio, or Image): If you like the settings you have selected you can check this box to make future embeds use them without having to set them every time.
- Start at: Set the start time for the media item.
- End at: Set the end time for the media item. Note that users can still play the entire video!
- </> Embed: When you have set the options you want, click the </> Embed button to embed the media.
If after embedding your Kaltura media you decide you'd like to change the options you selected it is easiest to delete your embedded media from the page and re-embed.
- Choose Player: Choose your player based on the type of media you are embedding. As of 8/15/24 all players automatically include a transcript widget. You do not have to select a specific Transcript player. Video users can choose between:
Embedding an existing Kaltura video into a Canvas page
Kaltura MediaSpace is UW-Madison's media streaming service and is integrated with Canvas. Kaltura media can be embedded wherever Canvas allows you to use the rich content editor. We recommend instructors start here if they already have media in Kaltura MediaSpace
Note: to upload and embed an new Kaltura Mediaspace item in a Canvas course, please see Canvas - Upload and Embed a New Kaltura Mediaspace Video in a Canvas Page [UW-Madison]
Note: You can also embed other media sources, such as Youtube videos. Please see Instructure's Canvas Community site for instructions on adding other video sources, including Youtube media.
Embedding Kaltura media in a Canvas Page or other Canvas tool is an easy way to make your Kaltura media available to your students. Embedded media provides more context and keeps the student in Canvas rather than sending them to an external link outside of Canvas. It also allows Kaltura to collect more detailed analytics.
- In your Canvas course, go to the Canvas page, discussion post, or quiz discussion you want to embed Kaltura media in. Click Edit.
- Move your cursor to the point in your Canvas item where you want to insert Kaltura media.
- Click the Embed Kaltura Media button which looks like the colorful Kaltura logo. If your browser is not of sufficient width you may need to click on the three dot button on the right to expand the menu to see the button:
- A window titled Kaltura Media will open showing the media content associated with your Kaltura account. By default it is on the My Media tab. These directions assume you are embedding a video you already uploaded. You can also:
- Upload a new video (see above)
- Embed a video from a course media gallery: If you are using a course media gallery, first click on the Media Galleries drop-down menu at the top of the screen and click on your media gallery. More information on using media galleries is available in Kaltura - Using "Kaltura Gallery" Course Media Galleries and Analytics in Canvas [UW-Madison]
- If you are embedding a piece of media you already uploaded, scroll to or search for the media you want to embed. Click the </> Embed button next the media you want to embed:
Optional: If you would like to customize how your media is embedded in Canvas, instead of clicking the </> Embed button, click the gear button:
- An Embed Settings window will display:
- Choose Player: Choose your player based on the type of media you are embedding. Video users can choose between the "Default Player" and "Download Player". Both players automatically include a transcript widget if a closed caption file is available.
- Max Embed Size: Users can choose how large the media is displayed within the Canvas rich content editor.
- Thumbnail Embed: Turning thumbnail embed on renders an image that will display instead of the video player. The student needs to click on the image to load the player. Generally we recommend keeping this option turned off.
- Auto Play: Turning auto play on plays the media immediately when the student loads the page with the embedded media. We recommend you keep auto play off and let the student choose to play the media by clicking the "Play" button.
- Set as my default settings for media type: Video (or Audio, or Image): If you like the settings you have selected you can check this box to make future embeds use them without having to set them every time.
- Start at: Set the start time for the media item.
- End at: Set the end time for the media item.
- </> Embed: When you have set the options you want, click the </> Embed button to embed the media.
If after embedding your Kaltura media you decide you'd like to change the options you selected it is easiest to delete your embedded media from the page and re-embed.
- Choose Player: Choose your player based on the type of media you are embedding. Video users can choose between the "Default Player" and "Download Player". Both players automatically include a transcript widget if a closed caption file is available.
- The video will show up on your Canvas page. You can continue to edit the page and add text if you'd like. Once you're done click the Save button:
- Canvas will display your page with your embedded Kaltura video:
Embedding a YouTube video into a Canvas page
You can "save" YouTube videos into your Kaltura MediaSpace account and embed them into your Canvas page. Only public videos from YouTube can be embedded. While this feature does not download the video and requires that the original YouTube video exists, it provides a few major advantages:
- It removes advertisements.
- You can modify the title and description.
- You can embed student reflection prompts within the media.
- You can collect the same student engagement analytics as your media.
- It allows for adding professional captions to meet ADA requirements for accommodations.
Directions
- In your Canvas course, go to the Canvas page where you want to embed the YouTube media. Click Edit.
- Move your cursor to the point in your Canvas item where you want to insert Kaltura media.
- Click the Embed Kaltura Media button, which looks like the colorful Kaltura logo. If your browser is not of sufficient width, you may need to click on the three-dot button on the right to expand the menu to see the button:
- Select +Add New, then select YouTube.
- Enter the Video Page Link or the Content ID and click Preview. To obtain the URL, right-click on the video on YouTube and copy the URL.
- Click Preview to edit the details, including name, description, and tags.
- Click Save.
3) Data collection for understanding
Kaltura video quizzes can be used as self-checks for students to allow them to pause and answer a question to determine whether they listened and understood what they just watched. While Kaltura video quizzes can be used as Canvas assignments we do not currently recommend that due to issues outlined in Canvas - Adding a Kaltura Video Quiz As a Canvas Course Assignment [UW-Madison] . We recommend that you keep your video quizzes to a maximum of 4-6 minutes in length.
Note: Since Kaltura video quizzes don't work reliably for 100% of students we recommend you use video quizzes for student self-testing and preparation with no associated points or grade, and Canvas quizzes for anything with points or a grade.
Creating Kaltura video quizzes
- Launch Chrome or Firefox, go to Kaltura MediaSpace, and login: mediaspace.wisc.edu. If you need more information on how to log in to MediaSpace please see Kaltura - How to Log in to Kaltura MediaSpace [UW-Madison] .
- Click Add New and select Video Quiz:
- The Editor / Media Selectionscreen will display. You can :
- Search for a video to use as the basis for a video quiz
- Select an existing video in your collection to use as the basis for a video quiz.
- + Upload Media to to add a new video to use with a video quiz.
- Once you've selected a video you want to use as a quiz, clicking Select will open the Kaltura editor. Note that it can take a few minutes to load on longer videos:
Editing quiz settings
Toggle the settings menu to visible or hidden by clicking the icon next to Quiz:
Details settings
Details allows you to edit the name and whether a welcome message displays along with the welcome message for the quiz.
- If it isn't expanded click the black arrow next to Details:
Be sure to also notice the scroll bar on the right. If you scroll down you can also check or uncheck boxes to Allow Download of Questions List and provide some basic Instructions:
Scores settings
Scores allows you to configure:
- Do Not Show Scores - A "Thank You" message will display after submitting the quiz, or
- Show Scores - Your viewers will see the quiz score page after the quiz is submitted.
- The Include Answers checkbox determines whether viewers will be able to see correct/incorrect answers:
Experience settings
Under Experience you can:
- Allow Answers Change - Allow viewers to change their answers before submitting a quiz.
- Allow Skip - Allow viewers to skip questions.
- Scroll down to see the option to set No seeking forward:
Generally we do not recommend instructors disable the ability for students to seek forward as it reduces student control and can make troubleshooting video quizzes more difficult.
Adding questions to your video quiz
- To add a question to your video you can play the video and pause where you would like to place a question, or grab the time tag and drag it to the appropriate place in the video:
- Once you are at the appropriate point in the video, click the blue Add a Question button. You can choose from:
- Multiple Choice: Questions with only one correct answer. Questions have a 180 character limit for questions and 140 character limit for answers.
- True / False: Question with the choice of True or False. Questions have 450 character limit.
- Reflection Point: This is a question with no answer. The video will pause and allow you to point out specific items in the video to guide your viewer's attention. This is a non-graded question and will not be part of the quiz score. Questions have 500 character limit.
- Open Question: The student can type in a free text answer. This is a non-graded question and will not be part of the quiz score. Questions have a 200 character limit. Answers have a 270 character limit.
- The question editing screen will display for your question and will vary based on question type. For more detail please see Kaltura's documentation Kaltura Video Tools - Create a video quiz. For multiple choice questions you have options to:
- Click the shuffle button to shuffle the answers
- Click the light bulb button allows you to add text hints to the question as well as explanations.
- Add question text.
Note: This is listed above but it's worth restating that you are limited to 180 characters for questions and 140 characters for answers for multiple-choice questions. True-False questions have a 450 character limit. Reflection Point questions have a 500 character limit. Open questions have a 200 character limit for questions and 270 character limit for answers.
It's also important to note that you can't assign different point values to each question. If you decide to use the video quiz as a Canvas assignment (outlined in Canvas - Adding a Kaltura Video Quiz As a Canvas Course Assignment [UW-Madison] ), the point value will be divided up to each video question based on the total points you give the assignment in Canvas. - Add correct and incorrect answers.
- Add additional answers:
- Click the shuffle button to shuffle the answers
- Click Save when you are done editing your question:
- Once saved a cube icon will appear on the timeline indicating that a question will appear at that point in the timeline.
- You can click the cube to edit the question
- You can also use the timeline zoom to increase or decrease the magnification on the timeline:
- When you are done creating questions and have set the settings you can click Done in the upper right of the editor:
Editing an existing Kaltura video quiz
- Launch Chrome or Firefox, go to Kaltura MediaSpace, and login: mediaspace.wisc.edu. If you need more information on how to log in to MediaSpace please see Kaltura - How to Log in to Kaltura MediaSpace [UW-Madison] .
- Click your name in the upper right and select My Media:
- Search for or scroll until you locate the video quiz you want to edit. Select the pencil icon to the right of the video quiz you want to edit:
- Click the Launch Editor button from this screen:
Alternately if you clicked on the name of your video quiz you can click the Actions button and select the Launch Editor option from the drop-down menu. - Choose the Quiz button on the left:
- Follow the prompts in the editor to edit or add questions to the video as outlined in steps 5-10 above.
- Click Done when finished.
Integrating video quizzes with Canvas
Instructors have a few options to integrate Kaltura video quizzes with Canvas:
- Kaltura video quizzes can be embedded in a page or other Canvas area wherever the Canvas rich content editor exists. Embedding video quizzes this way will not send a grade for the student to the Canvas gradebook but can still be useful in helping students self-assess and determine whether they understand what they are watching. We recommend that you only use multiple choice, true-false, and reflection point questions for self assessment quizzes. Students can then be more formally assessed on the content via other assignments, Canvas quizzes, exams, etc. More information on embedding an video quiz is available in Canvas - Upload and Embed a New Kaltura Mediaspace Video in a Canvas Page [UW-Madison] . this will still collect individual analytics on student answers and viewing.
- Kaltura video quizzes can also be embedded by using embed code. How to find a media entry's embed code is described in Kaltura - How to Get a Link or Embed Code to a Video in MediaSpace [UW-Madison] . This will not collect individual student analytics.
- Kaltura video quizzes can be setup as a Canvas assignment. When setup this way video quiz grades are sent to the Canvas gradebook. See Canvas - Adding a Kaltura Video Quiz As a Canvas Course Assignment [UW-Madison] for information on setting up video quizzes as a Canvas assignment. Unfortunately due to Kaltura video quizzes not working reliably for 100% of students we do not currently recommend they be used as assignments. We are providing the information for those who want to see how it works.
Kaltura video quiz example
Play the following video to see an example of how a Kaltura video quiz works:
4) Review the data
Review data for measuring compliance
There are a few important caveats about the data.
- Instructors should know that the information Kaltura MediaSpace presents is not real-time data. It may take some time for the data to be reported. Consider this before assigning points or calling people out for non-compliance.
- While you have access to individual viewing data, be aware that Kaltura MediaSpace analytics may not be able to collect a student's identity if they are running privacy or ad-blocking software. They need to configure their software to allow stats.kaltura.com and analytics.kaltura.com domains access. The viewing results would be collected but aggregated into an Unknown user category. Consider communicating your use of this data to track compliance so they can adjust their browser settings.
- Individual user data only captures the user who clicked on the link. For example, if students are studying together, only the student logged into Canvas would be recorded.
Viewing Media Analytics
- Add the MyMedia tool to your Canvas sidebar by clicking Settings from the Canvas navigation menu. On the Settings page, click the Navigation tab. Location MyMedia from the list of hidden tools. Click on the three-dot icon to the right of the tool. Click Enable.
- Select the MyMedia from the Canvas navigation bar.
- Click on the desired media entry.
- Choose Analytics from the ACTIONS drop-down menu.
Option 2:
- Go to Kaltura MediaSpace (mediaspace.wisc.edu)
- Log into the service by selecting MyMedia. You will be prompted to log in with your NetID.
- Click on the desired media entry.
- Choose Analytics from the ACTIONS drop-down menu.
There is a lot of information provided by Kaltura MediaSpace if you embed your content within a Canvas page. This document will walk you through several reporting details. View a full list of reporting data. View a list of terminology used in reporting.
Media Views
Directly below the video, you will see a graph of users' (both general and unique) views of the video over time (based on the timespan identified at the top of the report.
User Engagement
Given the timespan you specify in Kaltura MediaSpace, you get the following data:
- Player impressions — how many times has a page with a link to your video been viewed?
- Plays — how many times has the video player started to play the video?
- Unique viewers — how many individuals have started to play the video?
- Minutes viewed — how many total minutes have users played the video?
- Average Completion Rate — what percent of people watched the entire video?
Engagement by User
You should see the option to View Engagement Per User below the Media Views graph. With content embedded in a Canvas page, each user's viewing behavior can be tracked. The data provided for each student includes information not only on the number of views but also an analysis of which parts of the video they watched and how often they watched it. This report shows:
- Not viewed — this section wasn't viewed by the user.
- Viewed once — this section was viewed once by the user.
- Viewed twice — this section was viewed twice by the user.
- Viewed more than two times — this section was viewed more than two times by the user.
Media performance over time
This report provides information based on the timespan defined above for the following details:
- Number of player impressions
- Number of plays
- Number of unique viewers — number of authenticated viewers; all anonymous viewers will be counted as a single viewer
- Number of minutes viewed
- Average completion rate — how much of the video was watched
- Average drop-off rate — measured by viewer reaching playback quartiles
Player Engagement Levels
The engagement funnel shows the student's level of engagement with the entire video during the identified timespan of the report. This can be a way to quickly check to see the depth of students' engagement with the content (keeping in mind the limitations of the data). The following metrics are shown in the funnel:
- Player impressions
- Plays
- Plays that reach <x%> percent play through — this can be selected by the user.
Review data for measuring understanding
There are a few important caveats to point out here.
- Instructors should know that the information Kaltura MediaSpace presents is not real-time data. It may take some time for the data to be reported. Consider this before assigning points or calling people out for non-compliance.
- While you have access to individual viewing data, be aware that Kaltura MediaSpace analytics may not be able to collect a student's identity if they are running privacy or ad-blocking software. They need to configure their software to allow stats.kaltura.com and analytics.kaltura.com domains access. The viewing results would be collected but aggregated into an Unknown user category.
- Answers to quiz questions can be viewed in Kaltura Analytics. While results can be sent to the Canvas Gradebook, Kaltura Quiz Videos don't reliably report 100% of students' quiz results. Therefore, we recommend you use Quiz Videos for measuring knowledge to formally assign and record points. If this is the desired outcome, instructors should direct students to complete a Canvas Quiz after viewing the media.
Viewing Quiz Results
Additional information on Kaltura video quiz analytics is available in Kaltura's document, Quiz Reporting and Analytics.
To review Kaltura video quiz analytics:
- Launch Chrome or Firefox and log in to Kaltura MediaSpace. You can also access your Kaltura MediaSpace media from inside Canvas by using Kaltura My Media as described in Canvas - Accessing Your Kaltura Media in Canvas with Kaltura My Media [UW-Madison] .
- Search for or navigate to the video quiz and click it to select it.
- Select Actions > Analytics underneath your quiz video. The video quiz analytics page will load.
- Click the Quiz Questions tab to see a summary of your quiz questions and student answers.
- Click the Export to CSV button to get a CSV spreadsheet summary of the questions, answers, and total percentage of students that got the questions right.
- Click Printable Version to get an HTML page of the questions, answers, and total percentage of students that got the questions right.
- Click a question to display a summary of responses.
- Click the Quiz Users tab to see more detailed information on how individual students answered a video quiz.
- Click the Export to CSV button to get a CSV spreadsheet summary of the User ID, first name, last name, answered right, answered wrong, and score data.
- Click Printable Version to get an HTML page summary of the User ID, first name, last name, answered right, answered wrong, and score data.
- Click the student name to display a summary of how they answered the video quiz questions.
- Click the trash can icon to delete the student's video quiz attempt. If you have limited the number of video quiz attempts and the student has used them up, this will allow the student to re-take the video quiz.
5) Respond to the data
Once you have reviewed the data and reflected on student performance, the next step is to act somehow. The data you collected may have highlighted some issues you want to address. Before making changes, consider the following:
- What other data might I need to make an informed response?
- What other factors might play a role in the response I'm seeing?
- How accurate or reliable is the data I'm using? What is it NOT telling me?
- How much time will it take for me to make a change?
- Can I respond now or wait until the semester's end?
Knowing how and when to respond to data appropriately can be complex. The Center for Teaching, Learning & Mentoring provides individual consultations to UW-Madison faculty and instructors to help you with these issues. If you would like to schedule a consultation, please contact us!