Blackboard Predict (BB Predict) is a subscription-based predictive model designed to identify students at risk of receiving a grade of D or F in a course.  UMBC has a long partnership with Blackboard and had an active advisory role in the development of this tool.  While the product produced was high quality, it has been lightly used (although more recently DoIT has partnered with individual faculty and staff to explore ways to use it as part of our student success efforts and as a complement to our First Year Intervention Alert Program).  

Given current usage and financial constraints, we have made the decision to sunset BB Predict and will not renew our subscription.

One factor that limited the usefulness of BB Predict to improve student success through better early alerts is simple.  The model’s predictions only become precise enough to use with confidence at the semester's midpoint and are not “early enough.”  Recognizing that the earlier at-risk students can be identified with precision, the more time we have to help them and the greater the likelihood of their success, the Data Science Team within DoIT and supported by the Provost’s office constructed our own model, now called UMBC Predict.  The team’s goal was to increase the precision of the model’s estimates earlier in the semester, in this case, four weeks from the first day of classes, to allow for greater opportunities for students to seek assistance or change their approach to the class, and for these changes or assistance to have an impact.

UMBC Predict does not attempt to replicate the model and platform of our vendor partner.  The strategy employed by the Data Science Team was to focus on fewer courses in the initial stages of the work.  In particular, the team focused on courses with elevated DFW rates, large numbers of students, and importantly, courses where instructors make relatively heavy use of our learning management system (Blackboard) to engage students and to record assignments and grades throughout the semester.  It has always been true that instructors’ use of the learning management system provides valuable information to students about their progress in the course, but it is also the case that these interactions provide important information that the predictive model can use to sharpen predictions at an earlier point in the semester.

A second important factor underlying our decision to sunset BB Predict is pragmatic and necessary.  With the current pandemic and its associated negative effects upon the state budget, reducing our expenditure on a subscription-based model in favor of an internal model with equivalent or better performance for certain courses allows us to maintain our support for students while freeing up resources to support people and programs.

UMBC Predict is already being integrated into department-level enhanced early alert programs on a pilot basis.  This term, faculty in the Mathematics and Statistics Department and the Dean’s Office in the College of Mathematical and Natural Sciences plan to partner with DoIT and the Analytics and Business Intelligence Group to use targeted communications to students who may be at risk after four weeks.  The communication, delivered over the course instructor’s signature, encourages students to take advantage of their office hours along with other tips developed by the Academic Success Center to improve students’ chances of success both in the course and at UMBC.

We in DoIT know that numbers alone can’t capture the time and effort UMBC students, faculty and staff have put into coping with the disruption caused by the Coronavirus global pandemic. However, after the start of campus-wide online instruction last week, we wanted to provide a “bird’s eye view” of the various systems and tools that now make up everyone’s virtual UMBC.

For example, consider the following changes we’re seeing in just a few related tools (Collaborate, Webex and Panopto) compared to the first week of March, when “academic continuity” and “business continuity” were still becoming common terms like “social distancing.”

In terms of campus Blackboard usage, UMBC already started from a strong position. Apart from averaging 25k total logins per day last week, prior research has shown that UMBC’s Blackboard usage is higher compared to most institutions. Obviously, this reflects the hard work and dedication of faculty (especially preparing for last week), but we also want to extend special thanks to faculty serving as Academic Continuity “Ambassadors.” Through a brief survey last week, we learned Ambassadors have already provided support to more than 100 colleagues and peers, primarily through one-to-one direct assistance. The most popular topics include Collaborate, Assessments and Panopto. 

DoIT’s Instructional Technology & New Media group has also seen similar high volume interest in these topics. To date, we’ve hosted 216 and 194 participants in virtual “drop ins” and “pop up” focused training workshops, respectively. For a glimpse of what these sessions are like, check out this brief (3 min) clip of DoIT's Collin Sullivan demonstrating how to use OneNote in Blackboard for shared annotations, taken from a recorded virtual drop-in workshop for the Math Department last Friday, March 27. In addition to several DoIT ITNM staff responding to a robust "back channel" chat, the workshop was attended by 16 math faculty. Special thanks again to Math faculty member (and Academic Continuity Ambassador) Liz Stanwyck and Department Chair Animikh Biswas for the invitation and support. More info, including new evening drop-ins twice a week, are available on the DoIT Academic Continuity and Training sites.

UMBC’s Frequently Asked Questions (FAQ) knowledge base has also increased 45% from March 2019 (66,154 pageviews) vs. March 2020 (95,974 pageviews). By contrast, phone calls and Request Tracker (RT) ticket volume for DoIT’s Technology Support Center (TSC) have been relatively flat compared to FAQ usage. Tip: See the FAQs about FAQs, including why campus support offices and staff should use the FAQs to help users help themselves.

Beyond everyone's continued health and safety, we know there is obvious concern with how well students are able to make the transition to online learning. We know there are challenges, but if you haven’t seen it already, please check out (and share) DoIT’s tips for students about getting online, getting a computer, and getting software. 

In particular, in terms of software, we’ve just added a faster, more user friendly Virtual Lab Environment for all students and faculty to use for classes and instruction. The new system allows students to use all University software without having to download or install those applications on their own computers. It leverages UMBC’s computing resources, and can be accessed from any type of device, including Chromebooks, tablets, mobile devices or regular computers, making it easier than ever for students to get access to the tools they need for their classes. Instructions for accessing the system can be found at umbc.edu/go/vde-faq. 

To help faculty quickly determine if, and how frequently, students are keeping up during the move to campus-wide online instruction, DoIT’s Analytics and Business Intelligence group has also recently provided new Report Exchange (REX) reports focused on days since last login, dependence on mobile devices, and “Student at a Glance” reports showing patterns and trends across all courses a student is enrolled in that use Blackboard (Reminder: accessing REX reports from off campus requires logging in with the Virtual Private Network (VPN) to emulate an on-campus connection). Similar, more detailed reports the new reports are derived from are also available inside every Blackboard course. 

Finally, as more of us move to work and life online, DoIT’s IT Security group has provided important updates about such topics as home network security, telework security resources, and the rise in COVID-19-related scams, including insurance offers, unexpected calendar invites, and fake Docusign phishing messages. Similarly, DoIT’s Business Systems Group (BSG) made sure Human Resources has what they need to track leave, process payroll and make sure employees still get paid. BSG also quickly developed and added Docusign exception request forms so researchers can come to campus to maintain facilities and experiments, and students can seek approval of applied learning experiences.

When DoIT announced the phased rollout of Blackboard (Bb) Ultra on April 4 of this year, we did so based on the number of courses typically using Bb in each college as of Spring 2019, starting with least to greatest, to afford more time to colleges using Bb the most. To date, 270 courses have moved to Bb Ultra in Fall 2019, nearly doubling the 147 who did so in Spring 2019. 

However, of the 270 Fall 2019 Ultra courses, 61% are from the College of Arts, Humanities and Social Sciences (CAHSS), which was not expected, followed by 11% from the College of Engineering & Information Technology (CoEIT), and 3% from CNMS. The Division of Professional Studies (DPS), Division of Undergraduate Academic Affairs (DUAA), Erickson School and Social Work collectively represent 65 courses (24.5%), which was DoIT’s priority focus for Fall 2019. 

In recent years, the Office of Institutional Research and Decision Support  (IRADS) has installed or built a nifty set of “bird’s eye view” reports about teaching & learning in the campus’ Report Exchange (REX) data warehouse that any instructor might find useful. These include but are not limited to various campus-wide Blackboard usage reports such as the following:

Blackboard

• Bb Learn Course Use By College (shows Bb use by students across campus)

• Bb Instructor At A Glance (shows overall use for each instructor’s courses)

• Student At A Glance (shows individual student activity across all Bb courses)

Grade Comparison and Distribution

Similarly, the following "summary level" REX reports could be interesting to instructors who want to benchmark their own courses with peers from any discipline or department across campus:

• Course to Course - Grade Comparison (inspired by Dr. Hrabowski's definition of student success: "not only passing a course but also the next one that requires it.").*

• Outcomes After Course Completion (includes retention and graduation)

• Grade Distribution by Organization (by college and campus-wide).

• Courses by DFW Rate and Average Grade (by course view only)

• Most Popular REX Reports (see what reports others are using)

The aggregate data these summary REX reports provide could provide useful context for understanding any course, but by default, none of them will have “drill to detail” student ID information.

Background

UMBC faculty may recall that DoIT used to publish a list of most active Blackboard courses for every term. However, we discontinued this after Spring 2013, when we completed our transition to Blackboard’s Analytics for Learn (A4L) product that UMBC’s own experience helped influence. A4L reports for students and faculty are now embedded in every Blackboard course (see Control Panel → Evaluation → Analytics), but there is still nothing that provides a “bird’s eye view” of Bb usage across campus -- except the Bb Learn Use by College Report mentioned above.

Why might this be important? Increasingly, UMBC and other institutions are looking at institutional Learning Management Systems (LMS) like Blackboard as a real-time indicator or even predictor of student engagement, not just a static content repository of syllabi and powerpoint presentations. If so, then understanding effective design of LMS courses -- as measured, in part, by students’ use of them -- could be helpful as well.

Accessing REX

Until recently, access to REX itself was granted “upon request,” so one might first want to know what lies beyond the login screen before doing so. However, the summary reports above offer a sneak peek at what REX can do, and are now available to any instructor with access to the PeopleSoft advising system by default.**