The EPSS Model

Motivation

Vulnerability remediation has some fundamental truths. First, there are too many vulnerabilities to fix them all immediately. Past research has shown that firms are able to fix between 5% and 20% of known vulnerabilities per month.1 Secondly, only a small subset (2%-5% of published vulnerabilities2) are ever seen to be exploited in the wild. These truths set up both the need and the justification for good prioritization techniques since firms both cannot and do not need to fix everything immediately.

Vulnerability remediation

Therefore, what is the optimal prioritization strategy when remediating vulnerabilities? Unfortunately, there is no one single answer to that question, but instead lies with a collection of metrics to help inform and guide the prioritization decisions, which is where this research comes in.

The Exploit Prediction Scoring System (EPSS) is a community-driven effort to combine descriptive information about vulnerabilities (CVEs) with evidence of actual exploitation in-the-wild. By collecting and analyzing these data, EPSS seeks to improve vulnerability prioritization by estimating the likelihood that a vulnerability will be exploited.

Data Sources

EPSS is a data-driven solution. Through community partnerships and the work of EPSS SIG members, EPSS gathers data the following four kinds of data:

  1. Information that describes a vulnerability (descriptions, products, vendors, etc)
  2. Information about the vulnerability in the wild (prevalence, complexity, severity)
  3. Information about community reactions (social chatter, depth of discussions, exploits and tool publication)
  4. Ground-truth of exploitation in the wild (exploits in malware, IDS/IPS alerts, honeypot activity)

For a detailed list of data sources, please see past EPSS publications.

Model Performance

EPSS version 1.0 was developed in the summer of 2019 by five researchers and presented at the BlackHat conference in Las Vegas on August 8th, 2019, as well as other conferences and workshops. Since then, focus has shifted to community organization, establishing a computing infrastructure, and preparing for a more robust and repeatable 2.0 release. This section covers the model developed in the 1.0 version.

But first some background. Consider first the sample of CVEs used in EPSS 1.0 and what the overlap looks like between all CVEs rated as a CVSS 7 and above3). Next consider a remediation policy that seeks to patch all CVSS 7+ CVEs. In order to measure the quality of this approach, we need to need to consider the ground truth, which we do by tracking vulnerabilities we know have been exploited in the wild. This enables us to put each vulnerability remediation decision into one of four categories, as shown in the figure below.

Categories

As the figure above shows, the strategy to remediate based on CVSS 7+ produces many false positives and still leaves about half of the exploited vulnerabilities open and waiting to be remediated.

Using these four categories we can derive two more meaningful metrics, what we’ve termed the efficiency and coverage (or what information theory calls precision and recall respectively). Efficiency considers how efficiently resources were spent by measuring the percent of remediated vulnerabilities that were exploited. Remediating mostly exploited vulnerabilities would be a high efficiency rating (resources were allocated efficiently), while remediating random, mostly non-exploited vulnerabilities would result in a low efficiency rating. Efficiency is calculated as the number of exploited vulnerabilities prioritized (TP) divided by the total number of prioritized vulnerabilities (TP+FP). Coverage is the percent of exploited vulnerabilities that were remediated and is calculated as the number of exploited vulnerabilities prioritized (TP) divided by the total number of exploited vulnerabilities (TP + FN). Having low coverage indicates that not many of the exploited vulnerabilities were remediated with the given strategy.

Measuring and understanding both efficiency and coverage allows different firms to approach vulnerability remediation differently based on their risk tolerance and resource constraints. Firms that do not have many resources may wish to emphasize efficiency over coverage, attempting to get the best impact from the limited resources available. But for firms where resources are more plentiful and security is critical for success, the emphasis can be on getting high coverage of the highest risk vulnerabilities. With many companies probably striking a balance between the two.

For example, before EPSS, resource constrained firms may opt to only remediation vulnerabilities rated at CVSS 9 and above, while firms with more resources may focus on remediating CVSS 4 (or whatever) and above.

Precision and recall

We can measure the performance of EPSS Version 1.0, and use CVSS version 3.0 (base score) as a reference point. Both systems produce values in a range (EPSS is a value between 0 and 1 while CVSS produces a score between 0 and 10), so depending on where a cut-off is established, the coverage and efficiency will vary.

The plot shown here identifies red points for various cut-off points for CVSS, while the blue line represents the sliding cut-off points available to EPSS. The highest cut-off points for both start on the left and the points/line move to the right as that cutoff point is lowered.

One way to read the plot is that if we want to match the coverage for CVSS 9+ (meaning the same amount of exploited vulnerabilities remediated) with EPSS, we start by finding CVSS 9+ on the chart. It’s rated around 25% coverage and 9% efficiency (25% of exploited vulnerabilities are remediated and 9% of the vulnerabilities we remediated were actually exploited, and 91% of the remediated vulnerabilities may have safely been delayed). If we go straight up from CVSS 9+ to the line representing EPSS we see at 25% coverage, the efficiency of EPSS is about 37%. Indicating that a firm will be able to reduce the same amount of exposure as CVSS 10+ but do so much more efficiently. The following plot shows the difference visually. Notice how what is remediated with EPSS is a much smaller amount, meaning savings in resources.

Figure 4

But let’s take it the other way with CVSS 9+, let’s use the same amount of resources but look at the improvement in coverage. If we allow the efficiency to drop to 9%, with EPSS we would be getting coverage somewhere about 85%, so roughly for the same amount of tolerance in efficiency, we are able to close off a lot more vulnerabilities that are known to be exploited.

EPSS References

  1. Jay Jacobs, Sasha Romanosky, Benjamin Edwards, Michael Roytman, Idris Adjerid, Improving Vulnerability Remediation Through Better Exploit Prediction, Journal of Cybersecurity (in press).
  2. Jay Jacobs, Sasha Romanosky, Benjamin Edwards, Michael Roytman, Idris Adjerid (2019), Exploit Prediction Scoring System (EPSS), available at https://arxiv.org/abs/1908.04856.
  3. Michael Roytman, Jay Jacobs (2019), Predictive Vulnerability Scoring System, Black Hat USA, August 3-8, Las Vegas, 2019, available at https://i.blackhat.com/USA-19/Thursday/us-19-Roytman-Predictive-Vulnerability-Scoring-System-wp.pdf.
  4. Jay Jacobs, Sasha Romanosky, Benjamin Edwards, Michael Roytman, Idris Adjerid (2019), Improving Vulnerability Remediation Through Better Exploit Prediction, PrivacyCon Federal Trade Commission, June 27, 2019, available at https://www.ftc.gov/news-events/events-calendar/privacycon-2019.
  5. Jay Jacobs, Sasha Romanosky, Benjamin Edwards, Michael Roytman, Idris Adjerid (2019), Exploit Prediction Scoring System (EPSS), 2019 Workshop on the Economics of Information Security, Boston, June 3-4, 2019.

  1. Kenna security and P2P reference ^
  2. Other references here ^
  3. As rated by the National Vulnerability Database ^