We sometimes get questions from faculty related to student engagements concerning Canvas course content access, such as

• “How do students engage in my Canvas course?”;
• “Do students tend to download or preview course files?”;
• “How often do my students check their submissions for comments?”

From a course design perspective, visualization of Canvas content access may help us identify effective Canvas course design strategies that utilize various Canvas features to facilitate student engagements.

The following chord diagram (graph 1) illustrates how much students are accessing different features in a Canvas course. The quantities represent the number of clicks on a given course element by each student.

graph 1: There are the “student sectors”, labeled S1, S2, …, S6, that represent six students. Canvas course content and features are called “content sectors”, which include categories such as announcements, assignments, discussions, and pages sectors. The width of content sector tracks represents the total clicks on a content made by all students. The width of the student sector track denotes the quantity of clicks by a student. The thickness of directional links represents the quantity of clicks.

A Chord Diagram is commonly used to represent relations between elements(https://cran.r-project.org/web/packages/circlize/vignettes/visualize_relations_by_chord_diagram.pdf). The data format used in the above example is an adjacency matrix. The value represents the number of clicks on a given course element by each student.

The R circlize package allows us to draw a chord diagram without much scripting in R. After an adjacency matrix is created (you may download the example data to practice), and the “circlize” package is installed in R, follow the steps to generate a basic chord diagram:

1. import the csv file to R — mat <- read.csv(“file directory”,header=T, row.names=1)
2. convert the csv to matrix — mat <- as.matrix(mat)
3. set the gaps between sectors — circos.par(gap.degree=c(rep(2,nrow(mat)-1),20,rep(2,ncol(mat)-1),20))
4. customize order of the sectors (optional) — order=c(“S1″,”S2″,”S3″,”S4″,”S5″,”S6″,”announcement”,”assignments”,”discussions”,”file_previews”,”file_download”,”file_tab”,”gradebook”,”submissions”,”pages”)
5. define R color for the sectors (optional) — grid.col=c(“aquamarine4”, “cadetblue4”, “darkolivegreen4”, “deepskyblue4”, “firebrick4”, “deepskyblue4”, “dimgrey”, “dimgrey”, “dimgrey”, “dimgrey”, “dimgrey”, “dimgrey”, “dimgrey”, “dimgrey”, “dimgrey”)
6. draw the chord diagram — chordDiagram(mat, order=order, grid.col=grid.col)
7. reset the default circos graphical settings — circos.clear()

To customize sector labels:

1. follow 1 to 5 steps described above
2. switch step 6 from ‘drawing a basic diagram’ to ‘drawing an empty track’: chordDiagram(mat,directional=1, order=order, grid.col=grid.col, direction.type=c(“diffHeight”,”arrows”),link.arr.type=”big.arrow”,annotationTrack=”grid”,preAllocateTracks=list(track.height=0.3))
3. go back to the first track and customize sector labels: circos.trackPlotRegion(track.index = 1, panel.fun =function(x, y) {
   xlim =get.cell.meta.data(“xlim”)
   ylim =get.cell.meta.data(“ylim”)
   sector.name =get.cell.meta.data(“sector.index”)
   circos.text(mean(xlim), ylim[1], sector.name, facing = “clockwise”, niceFacing = TRUE, adj =c(0, 0.5))
   }, bg.border = NA)

Resources:

• http://jokergoo.github.io/circlize/
• https://cran.r-project.org/web/packages/circlize/index.html – circlize package

By examining the distinct usage of Canvas course tools, the organization of navigation items, and the structure of course content, we categorized Canvas course design strategies into seven models:

Syllabus_based design: The Syllabus tool is used to list a topical outline of the course content and to communicate to students exactly what will be required of them throughout the course in chronological order. This design facilitates posting course descriptions, class guidelines, weekly reminders and assignments information.

Homepage_based design: This design presents a front page usually including a course outline and links to course activities. This page could be a wiki page or the Syllabus tool. Other navigation links are typically hidden from student view. This design is useful for courses that have a specific workflow by providing a central page that helps students understand how they can navigate through the course.

Module_based design: The design utilizes the Module tool to outline the sequence of course content and course activities. The Pages link is usually disabled from student view. This design is suitable for courses containing sequential activities with possible prerequisites.

Page_based design: This design uses the Page tool to list the sequence and structure of course activities. Course files are usually embedded and linked in the content pages. Students use the content pages to guide their coursework. The course instructor can also use Pages as a wiki collaboration tool, setting specific student access for each page. This design facilitates providing descriptions for course content.

Page/Module_mixed design: This design utilizes both Page and Module tool to construct course outline and guide students through their coursework. Both Pages and Modules navigation tabs are enabled to allow student access. This design is suitable for courses contain sequential contents, meanwhile allowing flexibility for students to navigate through the course content.

Discussion_based design: The Discussion tool is utilized to facilitate communication before or after face-to-face classes. Page and Module tools are usually not used in this design. This design may be appropriate for blended learning, helping students to begin thinking about an upcoming assignment or class discussion, or following up on questions that began in a face-to-face classroom.

File repository: This design utilizes the Files tool to share course documents and syllabi with students, The Pages and Modules tools are not used extensively.

This design is suitable for face-to-face classes that use Canvas mostly for provided documents to students.

Student online discussion interaction data can be quite rich, it is beneficial to visualize the data in a meaningful way that helps faculty make an informed decision to engage students in online discussions. We can try a dynamic network dashboard to explore the ideas.

In previous blog, I posted a couple of online discussion interaction layouts that were generated in R with igraph package. In this blog, I would like to share another approach that uses R in Tableau to create a dynamic network graph. Please click on the image to view a video clip that shows a dynamic interaction I created in Tableau using R.To build a dynamic network graph in Tableau, in addition to prepare the edge list, we need to get the x/y coordinate for each node. There are multiple way to obtain node x/y coordinate. Inspired by a blog posted by Boran Beran, in which he describes how to generate x/y coordinates in Tableau using R igraph, I decided to try the coding in Tableau to build a dynamic discussion interaction diagram.

Below includes a step-by-step instruction and sample script:

1. Install Rserve package in R and run Rserve – Rserve(). Make sure to install igraph and plyr package in R as well.
2. Prepare an edge list that includes the following fields: from (the interaction initiator/sender), to (the receiver), users (the field combines both from and to list), pathorder (1 for users=from, and 2 for users=to), weight (varies and depends on the elements you want to examine or the focus of a question)
3. Import the file to Tableau
4. Create a Tableau calculated field (GraphNodes) to generate x/y coordinates and betweenness calling R igraph:
   SCRIPT_STR(“library(igraph); library(plyr);set.seed(123);
   mydf <- data.frame(from=.arg1, to=.arg2, weight=.arg3, Order=.arg4);
   mydf <- aggregate(mydf[,3],mydf[,-3],sum);
   mydf <-mydf[(mydf$Order==’1′) & (!is.na(mydf$to)),];
   mygraph <- graph.data.frame(mydf);
   mygraph <- simplify (mygraph, remove.multiple=F, remove.loops=T);
   coords <- “+[Layout]+”(mygraph);
   c<-cbind(coords, data.frame(users=V(mygraph)$name));
   c<-cbind(c, betweenness(mygraph));
   allusers <- data.frame(users=.arg5);
   c<-join(allusers, c, by = ‘users’);
   paste(c[,2],c[,3],c[,4], sep=’~’)”,ATTR([From]), ATTR([To]),SUM([Weight]),ATTR([Pathorder]), ATTR([User]))
5. Create calculated fields to extract x and y coordinate from the calculated field ‘GraphNodes’
   X coordinate: FLOAT(LEFT([GraphNodes],FIND([GraphNodes],’~’)-1))
   Y coordinate: FLOAT(LEFT(RIGHT([GraphNodes],LEN([GraphNodes])-FIND([GraphNodes], ‘~’)),FIND(RIGHT([GraphNodes],LEN([GraphNodes])-FIND([GraphNodes], ‘~’)),’~’)-1))
6. Build a network diagram in Tableau: step-by-step instruction on how to build a network diagram in Tableau.

Resources about Using R in Tableau:

• Combining Tableau and R
• Tableau and R for data analytics
• Covariance and Correlation Coefficient Calculation in Tableau
• logistic regression in tableau using r

Student course access data can potentially help us identify effective course designs. We can leverage student click-stream data generated in LMS to examine the effectiveness of certain course designs.

For instance, the following graph demonstrates student course content access pattern. The course we selected in this example employed a page-based design approach: Each weekly study guide is presented in an individual page layout, reading materials are embedded/linked in the weekly study guide descriptions, and students can also access the reading materials by directly going to the file repository.

When a student accessed a file directly via the file repository, ‘folders’ event was emitted; When a student click on a file embedded in the study guide content page, ‘files’ event was emitted if the student chose to download the file, and ‘file_previews’ was emitted if the student clicked to preview the document.

Graph 1: The nodes with number indicate students – student nodes, the nodes with texts indicate course contents – content nodes. The size of content nodes implies total access to certain content made by students, and the size of student nodes implies the quantity of clicks to all course contents initiated by the student.

Please note that the number associated with each student node was fabricated.

This graph allows faculty to easily identify the student who made fewer content access than others. Also the graph shows that students prefer downloading files instead of previewing them. Students tend to access files embedded in content area (study guide pages) rather than going to the file repository (Files tab) to navigate through course files.

In previous blog, we talked about applying network visualization to course discussion interaction analysis. This blog demonstrates an example of using the visualization to analyze the impact of instructor involvement on student discussion interactions.

The following two graphs show student to student and instructor to student discussion interactions in two courses respectively. The two courses were offered in the same term under the same program and contain roughly the same number of enrollments. The discussion requirements specified in the two courses are identical. The results suggest that:

• Less instructor involvement coincides with more student-to-student interaction
• More instructor involvement coincides with longer student replies
• More instructor involvement coincides with greater student self-reflection

Graph 1: Each node represents a student who either received at least one feedback or provided at least one reply to another student. The size of each node suggests the quantity of interactions associated with the student. The thickness of each arrow line implies the length of a reply.

Graph 1 is presented in ‘kamada.kawai’ layout:

The two graphs above show that although course one students were fairly active in discussion activities, comparing to their counterparts in course two, course one students contributed more equally in terms of the length of replies (word counts of the threads) and the number of replies. In contrast, a few students in course two appear to have a greater quantity of the interaction, and yield a few longer replies.

Now, let’s take a look of instructor’s involvement in both courses. The course two instructors’ presence appear to be more evident than course one instructors, and instructors in course two provided more lengthy replies to their students than course one instructors.

Graph 2: the orange node in the middle represents the instructor who provided at least one reply to students. The size of each node suggests the number of replies made to students. The thickness of the arrow line implies the length of a reply.

Our Canvas discussion data shows that about 20% of courses that are published in Canvas use the Canvas discussion tool. However, little is known as to how students interacted with their peers in Canvas discussions, whether students were actively engaged in discussions, and how instructor involvements shape/facilitate a community of inquiry. To see if network analysis is useful to address some of these questions, we applied network graph approach to visualize discussion interaction data.

For the proof of a concept, we fabricated a small set of discussion interaction data. We converted the discussion data to an edge list. An edge list contains “from” and “to” columns that represent the two nodes connected in a network. Table 1 includes the sample set of discussion interaction data in an edge list form. The values in the first column are discussion feedback providers and the second column includes the feedback receivers.

The Graph 1 was derived from the sample data set and generated in R with the igraph package. Each node represents a discussion participant. The direction/edge arrow indicates a directed interaction from a feedback provider to the feedback receiver – the author of the target thread to which the provider replies. The feedback can be a reply to a new post, or a response to a reply. The size of each node implies the total counts of the directed interactions for the node. The graph reveals interesting elements related to students’ discussion engagement. For instance, we can quickly see that studentA tended to respond to most of his peers, but did not get much feedback from his peers at all. In contrast, studentE received responses from many of his peers, but only initiated one thread to the instructor. Maybe the initial thread that studentE posted was so interesting or debatable that grabbed the attention of other students. studentF appears to be less interactive than his peers, and provided no feedback to peer postings.

Graph 1:The size of each node implies the count/degree of the directed interactions for the node

To further explore the relationship between online discussion behavior and classwork performance, we experimented to add student grades as node attributes. We also added the word count of a reply as weight to each directed interaction.

To experiment with the nodes’ attributes in our analysis, we fabricated students’ grades, assigning them either an above median or below median value. We also added a weight for each unique interaction by counting the number of academic words by excluding English Stopwords in the thread. Graph 2 was generated by adding the weight values and the nodes’ attributes.

Graph 2: The color green means a performance above the median, and red denotes a performance below median. The size of each node represents the amount of the two-way interactions for the node. The thickness of the arrow line implies the number of academic words in each interaction.

With the same edge list, we can apply different igraph layouts to an interaction visualization. For instance, the following three graphs were derived from the same set of data, the circle layout gives us an overview of the students who either provided at least one feedback to their peers, or received at least one reply from their peers. The kamada-kawai layout allows us to quickly identify the students who are less interactive. The sphere layout helps us see the reply threads that contain the most words.

Table 1 includes a sample set of discussion interaction data in an edge list form. The values in the first column are discussion feedback providers and the second column includes the feedback receivers. Table 2 is the edgelist with associated edge values, the weight for each unique interaction, which was used to created a weighted network. Table 3 is the nodes’ attributes representing student performance, above median-Above or below median-Below respectively.

Table1:

Table 2 – nodes’ attributes:

Table 3 – a weighted edgelist:

install.packages(“igraph”)
library(igraph)
#load the edge list and nodes to R
nodes links

Resources: http://kateto.net/network-visualization

Would you like to know when Dartmouth students are likely to submit their assignments via Canvas, and whether the activity is related to the time and date when the assignment is due? If so, how does the assignment due times affect students’ submission activities? If you are interested in learning about the assignment submission facts, please click on the image below to view the 2015 assignment submission analytics.

These results were derived from 2015 course, course assignment info and submission data. Graded discussions, online quizzes and any assignments that have an ‘online’ submission type were included in the analysis. The assignment that does not have an ‘online’ submission type or a due date/time associated with was excluded from the analysis.

After a few outliers were identified and removed, the median submission time before due date is 30 minutes and the median submission time past due is 1.2 hours. The ‘withoutoutlier’ charts also show that the number of before due submissions is much greater than the total number of past due submissions and the variation in past due submission hour is wider than before due submission hour. All of which imply that majority Dartmouth students tend to submit assignments more often before than past the due time and the likelihood assignment submission time is 30 minutes prior to assignment due time.

Taking all four terms in the year of 2015 into consideration, the evening period from 8 pm to 10 pm is a popular time for assignment submissions, and 10 pm is the peak assignment due time (when assignments are due). Some months show some variability. For instance, in November, there is a peak submission time at 11 pm coupled with a 11 pm peak assignment due time. in April, 11 am arises to be another peak time for assignment submissions in addition to the popular evening submission hours. Faculty might consider these behaviors when choosing due times.

The chart below reveals that a number of assignments contain due date/time that were set between midnight and 8am Eastern time, which prompted some students staying up overnight in order to submit the assignments right around due time. Even though the hourly submission chart reveals that there are variability in median submission time for all submissions at a given submission hour, we can conclude that students tend to submit assignments 30 minutes prior to assignment due time more often than past assignment due time. Therefore, we need to suggest faculty to be mindful when choosing assignment due time.

The Course Users Flow report is a graphical representation of the paths a course user took through a course site, from the landing page to the various pages, and where along their paths they exited the course site. The Course Users Flow report allows us to examine course user navigation flow patterns within a course site, and further to analyze the efficacy of a course site design. (https://support.google.com/analytics/answer/1709395)

We attempt to use the user flow report to examine how Canvas facilitates students final exam preparation. The following two course user flow charts illustrates navigation traffic patterns in two Canvas courses that adopted distinct design approaches:

• Course one primarily uses Canvas native Files and Assignments feature to deliver course materials, publish weekly homework, and collect student assignment submissions;

• Course two utilizes homepage to distribute weekly lecture notes and exam study guide materials, links classroom lecture video recording via echo360 integration and organizes class discussion via an external tool – Piazza.

The course one user flow chart shows that checking course grade was the most popular path; while the user flow report for course two suggests that reviewing the class lecture video recording was the primary purpose of logging into the course site. From the level of engagement point of view, course two students were most engaged in taking final exam and reviewing the lecture captures, and in contrast,  course one students were more engaged in checking their grades and taking the exam than accessing course files.

This blog is a continuation of the previous blog:

In previous blog, we introduced the method to calculate the point-biserial value from students’ quiz submissions that were derived from three instances of the same course. The same course was offered in three consecutive terms: Spring 2013, Spring 2014 and Spring 2015. In the following texts, the three courses will be noted as SP13, SP14, SP15 respectively.

In this blog, we intend to demonstrate an application of the quiz statistics.

• The point-biserial correlation coefficient derived from students’ first attempts on all quizzes reveal two quizzes that contain question items yield a high point-biserial value (greater than .5); we’ll label these as “Quiz 1” and “Quiz 2”. Quiz 1 contains four questions worth 1 point each, whereas Quiz 2 contains five questions, worth 1 point each. The point-biserial correlation coefficient measures how well a single question can tell the difference (or discriminate) between students who do well on an exam and those who do not.
• In general, students participate in repeated practice to get a full score on the quizzes that allow multiple attempts. And quite a small number of students continue practicing after they get a full score. For instance, for Quiz 1, among 46 students who got a perfect score on their first attempt, only one student made second attempt; Among 61 students who got a full score on their second attempt, only 3 students took Quiz 1 the third time. The two graphs below show the average quiz score in relation to the attempt taken. We observe that after the 4th attempt, student scoring becomes somewhat random, which is quite likely due to fatigue/saturation.

For the few students who continued practicing after achieved a full score, they probably re-take the quiz for the sake of practice and care less about the score, or they might access the quiz primarily to review the embedded videos.

• Our further analysis on these two highly discriminating quizzes, Quiz 1 and Quiz 2, in the SP15 course indicates that students, who on their first attempt, receive a score of 1 (out of 4) on Quiz 1 and a score of 3 (out of 5) on Quiz 2, tend to perform poorly on the final exam.

• There is an inverse relationship between the number of attempts that a student takes to get a full score on the quiz and the student’s final exam grade.
  

  Implications:

  • Quizzes 1 and 2 can be used to predict problematic student performance: If a student receives a score of 1 (out of 4) on the identified highly discriminating Quiz 1 and a score of 3 (out of 5) on Quiz 2, the instructor can be notified in order to take additional remedial measures. Early interventions and extra help need to be employed in order for the identified students to make sufficient progress on the subsequent course content.
  • Furthermore, if  a student struggles to receive a full score on the quizzes by the 4th attempt, similar interventions are suggested.

  Recommendations:

  • In order to leverage quiz statistics to predict student course outcomes , we suggest that faculty who employ low-stakes quizzes embed one or two multiple choice question items into these quizzes so as to make them highly discriminating.

Canvas quiz statistics can help faculty understand how students perform on quiz items and help them choose quiz items that are highly correlated with overall student performance (high discriminating quiz items). The important statistics to analyze for each quiz item are its difficulty level and the correlation between the right/wrong attempts on a given quiz item and the total quiz scores (the point-biserial correlation coefficient).

Let’s look at the item statistics in detail.  Suppose that a faculty administered a quiz that contains 10 multiple choice quiz type items, and nine students took the quiz.

The sample data used in this blog is from http://www.eddata.com/resources/publications/EDS_Point_Biserial.pdf

The quiz items are presented in the sequence of its difficulty level from 0 to 1 (difficulty index=#correct attempts/#total attempts). The higher the difficulty index is, the easier the quiz item should be. The higher the point-biserial value is, the higher the item is correlated to the total scores.

• Application: Identifying quiz items with high point-biserial value

A large positive point-biserial value indicates that students with high scores on the overall test are highly likely to get the item right and that students who receive low scores on the overall test will tend to answer the item incorrectly. A low point-biserial value implies that there is not much correlation between a student answering the item correctly and the overall quiz score. For example, students who answer the item wrong might answer correctly on other more difficult quiz items and end up scoring higher on the test overall than might be predicted by the item in question. Therefore, items with low point-biserial values need further examination. Our sample data suggest that Item 3 deserves closer look, as its point-biserial value is the lowest (0.12) among other quiz items. Even though student-7 did not correctly answer an easy quiz item 3, the student correctly responded to more difficult quiz items, Items 4 and 6. If we can assume that the student did not make a guess on item 4 and 6, and that student-7 indeed understood item 4 and 6, item 3 should be treated as problematic item.

If a faculty is interested in creating an efficient shorter quiz, we can help the faculty gather previous quiz submissions and use the point-biserial value as reference to select the high discriminating quiz items, which are those with the high point-biserial correlation coefficient. Meanwhile, we need to take the difficulty index of each quiz item into consideration to ensure a balanced quiz which is neither too difficult nor too easy (http://jalt.org/test/bro_12.htm).

• Implication: Providing individualized support

The sample data suggest that item 4 is a high discriminating quiz item that may be used to predict students performance on the overall quiz. Its high point-biserial value (0.73) implies that if a student gets this quiz item right, the student is likely to answer other quiz items correctly and get a high overall score; If a student responds to this quiz item wrong, the student may find it difficult to answer correctly to subsequent quiz items. Therefore, faculty can use the information (responded incorrectly to the quiz item) to identify ‘weaker’ students, and guide them to proper materials/practices that facilitate their understanding on the content that quiz item 4 is designed to measure.

From Theory to Practice

In this study, we leveraged the point-biserial value to identify quizzes that contain highly discriminating question items. Furthermore, we examined the relationship between students’ activities and outcomes on these quizzes and their final exam performance.  We explored the possibility of predicting unsatisfactory student performance using quiz data early on during the term in order to pursue timely interventions. We describe a procedure applicable to other courses.

The particular course we analyzed contains three weekly quizzes, each consisting mostly of multiple choice question items. Students are asked to watch videos in preparation for class, interspersed with these quiz questions. We analyzed student performance on these quizzes through three instances of the course: Spring 2013, Spring 2014 and Spring 2015. In Spring 2015 course, optional practice questions were added to some quizzes; these optional questions are graded but bear no point value.

First, we examined the pattern of time-spent on all quizzes in SP13, SP14 and SP15 course respectively. Secondly, we calculated the rpbi for each question item, derived from the results of student first attempt, with the following formula (http://jalt.org/test/bro_12.htm). We identified two quiz questions that contain question items yield a high point-biserial value (rpbi > 0.5). A simple linear regression was performed to examine the correlation between students’ performance on the discriminating question items and their final exam grades. A number of plots on the mean of the response for two-way combinations of factors was drawn to illustrate possible interactions.

Where:

The rpbi point-biserial correlation coefficient derived from students’ first attempts on all quizzes reveal two quizzes which contain question items that yield a high point-biserial value (greater than .5); we’ll label these as “Quiz 1” and “Quiz 2”. Quiz 1 contains four questions worth 1 point each, whereas Quiz 2 contains five questions, worth 1 point each. Note: Original Quiz 2 contains six question items, but one question item grants either half or one point (0.5 or 1). In order to comply with the rpbi point-biserial formula, we took the question item out of the calculation, hence, rpbi point-biserial correlation coefficient for Quiz 2 was derived from only five question items.

The rpbi point-biserial correlation coefficients for the four question items of Quiz 1:

If we can harness these statistics to identify the ‘weaker’ students prior to the final exam, the faculty may be able to guide the students early on to facilitate a mastery of the course content.