What is IRT and how will it affect the trucking industry?
Over the past few years, there has been a lot of talk about CSA scores and possible changes. The FMCSA is working on an IRT model to possibly supplement and/or replace current CSA scores. What do fleet operators and safety managers need to know to stay in compliance with regulations?
Current implementation date for the new model:
UNKNOWN.
The FMCSA is still in the development phase and has not officially adopted a new model. While it is possible to guess what the new model will look like, any early representations would likely produce results very different from the final model.
Here is a little background. . .
In 2015, Congress ordered a review of CSA by the National Academy of Sciences (NAS) as part of a larger transportation bill, The FAST Act. NAS made a number of findings and recommendations.
In the 132-page report entitled “Improving Motor Carrier Safety Measurement” published in June of 2017, the NAS panel noted that the current system has been criticized for, among other things:
- Using highly variable assessments
- Not accounting for crashes where the motor carrier is not at fault
- Including carriers that have very different tasks in the same peer groups
- Using measures that are sensitive to effects from one or more individual states
- Using measures that are not predictive of a carrier’s future crash frequency
- Using measures that are not reflective of a carrier’s efforts to improve its safety performance over time.
NAS Recommendation:
“The FMCSA should develop the suggested Item Response Theory (IRT) model over the next 2 years. If it is then demonstrated to perform well in identifying motor carriers for alerts, FMCSA should use it to replace SMS in a manner akin to the way SMS replaced SafeStat (CSA’s predecessor).”
It has been 2 years since NAS recommended IRT, what is the current timeline?
So far, the IRT model development process has been a slow one. The current state of the implementation process is nowhere near the 2-year timeline initially suggested in the NAS report. Here is the very tentative plan for the future of IRT:
-
Spring/Summer 2019 meeting (date still TBD)
- Plan to present a small scale model
-
September 2019
- Full scale model completion
-
September 2019 – 2020/2021. SPECIFIC DATE UNKNOWN.
- Implementation period
- Carriers should anticipate lengthy implementation and comment periods.
- Implementation period
There is NO TIMELINE on when IRT model will begin to impact carriers.
What is IRT and where else is it used?
Item Response Theory (IRT) refers to a family of complicated mathematical models that, while not widely used in the transportation industry, have been successful in the education and medical fields.
In an attempt to simplify the IRT concept, FMCSA officials and NAS academics often have compared it to SAT tests used in high schools. A student’s SAT score is not based on the sheer number of correct answers. Rather, those who have a mastery of a subject such as math or science are determined by how many of the difficult questions they can answer correctly.
How will IRT replace or improve on CSA?
The IRT shift will likely end CSA’s “crash risk” approach by changing the way we look at scoring. Instead of having only seven separate BASIC scores, IRT will focus the BASIC category scores AND one overall “safety culture score” to prioritize carrier intervention. The same critical data points that are currently used to determine the BASIC scores will continue to be the foundation of the IRT score.
The IRT-backed scoring system will look for patterns and assign weights and priorities — weighting them according to patterns across the industry. Using IRT should also help to reform the severity of the scoring.
For example, violations in on one or two areas, with otherwise excellent marks in all other areas, will mean that the fleet’s overall score will not be too severe. However, a fleet with mediocre scores across the board will point to safety issues, and likely trigger a compliance review.
Should you worry about what your IRT scores will look like NOW?
The FMCSA is still in the development phase and has not worked out all the details of the IRT methodology. As it works towards the small-scale pilot stage, the FMCSA will be defining, calibrating and adjusting the scoring process. Because the FMCSA has not released details on its current model or any changes it has made along the way, releasing a score management product prematurely will only lead to false outcomes. While there are vendors in the market that are offering early access to IRT based scores, it is unlikely their IRT model will match what the FMCSA develops (should the FMCSA eventually decide to implement IRT).
Creating an early IRT score management product utilizing inspection, violation and crash data without a mapped out methodology is similar to sending two different drivers out with no defined route or map. Each driver would likely take a different route and end up in different destinations.
SuperVision is committed to helping your fleet manage and improve CSA scores. When an IRT model is defined, SuperVision will have the tools your safety team needs to navigate the changing compliance landscape.
CSA PERFORMER
LEARN MORE ABOUT CSA PERFORMER