Almost every question about following up digital learning lands sooner or later in the same place: which standard do you use? The standard decides which language the training and the learning platform speak with each other, and therefore also what can be measured at all. Here we go through the most common standards and what it is customary to follow up with each one.
What a standard actually does
A standard for digital learning solves three things. It describes how a training programme is packaged so that it can be uploaded in different systems, how the system starts the training, and how the training reports back what has happened. Without a shared standard, every course needs to be specially adapted for every platform.
Worth noting is that the standard says nothing about the quality of the training. Video, simulations, interactivity and responsive design work equally well regardless of standard. What differs is what can be measured and where the learning may take place.
AICC — the origin, and where “cmi” comes from
AICC (Aviation Industry Computer-Based Training Committee) produced the first shared rules for computer-based training in the late 1980s. The specification was called CMI, Computer Managed Instruction, and described which data elements a course should report: status, score, time and where the user was in the material.
That is where the abbreviation cmi comes from, and it lives on in two places. Partly in SCORM, where every data element is still named things like cmi.completion_status and cmi.score.raw. Partly in the name cmi5, which we return to further down.
The AICC organisation was wound up in 2014 and new training programmes are rarely built to AICC today. You may, however, still encounter the standard in older systems, especially in industry, aviation and defence. The technical distinction is that AICC communicates via calls against the server rather than via JavaScript in the browser, which historically made it possible to run the course on a different server from the learning platform.
SCORM — the established standard
SCORM (Sharable Content Object Reference Model) was launched in 2000 and is still the most used standard in e-learning. It gathered AICC’s data elements in a package format: a zip file with a manifest file that describes the content, which in principle every authoring tool can export and in principle every learning platform can read.
SCORM is built on the course running inside the learning platform and communicating with it via the browser. That is also the standard’s clearest limitation. If the learning happens somewhere else – in a business system, in an app or in day-to-day work – there is no channel to report on.
SCORM 1.2 or SCORM 2004?
SCORM 1.2 is the oldest, simplest and has the widest support. It has a shared status field that mixes up two questions: the course can be either completed or passed, but not both at once. For a training programme with both a completion requirement and a pass threshold, that becomes a compromise.
SCORM 2004 separates the two. One field answers whether the course is completed, another whether the participant has passed. The version also has support for progress measurement and for sequencing, that is rules for which order the modules may be taken in. The space for saving where the participant was at an interruption is also considerably larger, which matters in longer courses.
In practice: choose SCORM 1.2 if maximum compatibility is most important and SCORM 2004 if you need to distinguish completed from passed or control the order between modules. Always check what your learning platform actually supports. Support for 2004 is common, but full support for sequencing is not.
xAPI — learning outside the course
xAPI (Experience API, formerly called Tin Can) was published in 2013 and turns the perspective around. Instead of a course reporting its status to the learning platform, any source can send short statements about what has happened.
Every statement has the form actor, verb, object. “Anna completed module 3.” “Johan chose the wrong option on question 2.” “The team opened the checklist on the phone.” The statement can be complemented with result, context and time, and is sent to a data store called an LRS (Learning Record Store). Because the statements can come from an app, a simulator, a business system or a classroom training, xAPI also captures the learning that happens outside the learning platform.
The price of the flexibility is that xAPI does not decide for you what should be measured. Two vendors can describe the same thing in two different ways, and then the data becomes hard to compare. That is why an xAPI initiative always needs to start with an agreed data model: which verbs you use, what they mean and which activities should be reported.
cmi5 — xAPI with rules for course delivery
cmi5 was published in 2016 and solves exactly that problem, for the course context. It is a profile on top of xAPI that adds what SCORM had and xAPI lacked: a package format, rules for how the learning platform starts the course, and a defined set of verbs with given meanings, including launched, initialised, completed, passed, failed and terminated.
The result is a standard that behaves like SCORM for the administrator but stores data like xAPI. The course also does not need to sit in the learning platform, which makes it easier to run the same content in several systems.
Two things to reckon with: cmi5 is not backwards compatible, so existing SCORM packages need to be rebuilt. And support in learning platforms is still considerably narrower than for SCORM. Check the support before you decide.
Two related standards you may encounter
LTI (Learning Tools Interoperability) is not about packaging courses but about plugging external tools into a learning platform, with login and permissions in place. In the latest version the tool can also send results back to the platform’s gradebook. Most common in higher education and among training providers.
QTI (Question and Test Interoperability) describes how questions and tests are stored so that a question bank can be moved between systems without being rebuilt. Relevant mainly when you have large volumes of questions that need to survive a platform change.
What is customary to measure?
This is the question that is most often forgotten. The standard sets the ceiling, but you decide yourselves what you actually use.
The basic level, which all standards cover
- Status — is the course started, completed, passed or failed?
- Score — result on a knowledge check, often both raw score and pass threshold.
- Time — time in the course, per session and in total.
- Bookmark — where the participant was at an interruption, so that the course can be resumed.
This goes further than many think. For a shorter training programme where the result does not underpin important decisions, completion rate and quiz results are often fully sufficient.
The question level, which SCORM also covers
- Interactions — which question, which answer the participant chose, whether it was correct and how long it took.
It is this level that makes it possible to see that a particular question is systematically misunderstood, which more often reveals an unclear training segment than a lack of knowledge among the participants. Many never use interaction data even though SCORM supports it, often because the learning platform does not show it in its standard reports.
The behaviour level, which requires xAPI or cmi5
- Video behaviour — which segments are replayed and where participants drop off.
- Choices in scenarios — which path the participant took, not only whether it ended up correct.
- Activity outside the course — that the support material is used at work, not only during the training.
- Activity in other systems — exercises and segments in business systems or simulators.
- Effect over time — the link between completed training and fewer support cases, lower error rates or fewer incidents.
The last point gives the greatest value and requires the most preparation. Therefore start with the questions the organisation wants answered and measure what helps you answer them. It is easy to collect large volumes of data and considerably harder to do something meaningful with it.
How to choose
Do you need to follow up completion and results in a course that runs in the learning platform, and is broad system support most important? Choose SCORM. Do you need to distinguish completed from passed or control the order between modules? SCORM 2004.
Do you need to capture learning that happens outside the learning platform, or measure behaviour and effect rather than only completion? Then xAPI is the right path. If you want that data collection but still deliver and administer the courses in a predictable way, cmi5 is the combination – provided your systems support it.
One common misunderstanding is worth killing: you do not need to choose a single standard for the whole organisation. It is entirely reasonable to run SCORM for compliance training where completion rate is what is asked for, and xAPI where you want to understand how the training is actually used.
We build training in all of these standards and are happy to help you choose the right level of follow-up based on what you actually need to measure.