Guidelines and Practices for Multi-Party Vulnerability Coordination and Disclosure

Version 1.1
Released Spring, 2020

Introduction

Foundational work on best practices, policy, and process for coordinated vulnerability disclosure has focused primarily on bi-lateral coordination (i.e., between one researcher and one vendor) and less on the increasing complexities of multi-party coordination. Factors such as a vibrant open source development community, the proliferation of bug bounty programs, increasing supply chain complexity, and the support challenges facing CSIRTs and PSIRTs are just a few of the complications. Examples such as Heartbleed)1 highlight these coordination and disclosure challenges.

This document is the outcome of an effort between the National Telecommunications and Information Administration (NTIA)2 and FIRST to address such challenges. The purpose of this document is to assist in improving multi-party vulnerability coordination across different stakeholder communities.

This document differs from the ISO vulnerability disclosure and handling standards (ISO/IEC 29147 and ISO/IEC 30111). The ISO standards are written mainly for one stakeholder group: vendors. The ISO standards also focus on bi-lateral disclosure for vulnerabilities in products. This document is a collection of best current practices that consider more complex and typical real-life scenarios that extend past a single researcher reporting a vulnerability to a single company.

This document contains a common set of guiding concepts and best practices derived from use cases and examples scenarios. This guidance is targeted at vulnerabilities that have the potential to affect a wide range of vendors and technologies simultaneously.

Notice: This document describes what the Forum of Incident Response and Security Teams, Inc. (FIRST.Org) believes are best practices. These descriptions are for informational purposes only. FIRST.Org is not liable for any damages of any nature incurred as a result of or in connection with the use of this information.

Definitions

Within the context of this document, the following definitions apply. Definitions that are available in ISO/IEC 29147:20143 are used with minimal modification.

Stakeholder roles and communication paths Figure 1: Stakeholder roles and communication paths

Guiding Concepts and Best Current Practices

The following guidance is derived from the cases, variants, responses, and preventions discussed later in the section on Multi-Party Disclosure Use Cases. The most important practices, particularly those that occurred repeatedly, are captured here. Stakeholders should carefully consider their actions, particularly notification and public disclosure, due to the widespread impact on other stakeholders in multi-party cases.

Establish a strong foundation of processes and relationships

Maintain clear and consistent communications

Prior to disclosure

After disclosure

Build and maintain trust

Minimize exposure for stakeholders

Respond quickly to early disclosure

Use coordinators when appropriate

Multi-Party Disclosure Use Cases

Vulnerability disclosure can be a complicated process, especially when multiple parties (usually multiple vendors) are involved. This section of the document is organized as a set of vulnerability disclosure use cases, in rough order, from simple to complex. Significant attention is given to coordinated, multi-party disclosure (see Use Case 2: Vulnerability with coordinated disclosure). Disclosure often deviates from the expected or ideal process; therefore, within each use case are variants that are common exceptions to the ideal use case. Within each variant are causes, preventions, and responses. The collected set of preventions and responses are presented as practices that can be used to reduce the occurrence and cost of expected variants.

Practices are denoted as strong recommendations (“should”) or suggestions (“can,” “could,” or “may”).

After describing multi-party coordination and disclosure use cases and variants, recommended practices are collected into the concluding section: Guiding Concepts and Best Current Practices.

Use Case 0: No vulnerability

Description

This case is included for completeness. If there are no vulnerabilities, there is no need for coordination.

Use Case 1: Vulnerability with no affected users

Use Case 1 Vulnerable product, but no affected users Figure 2: Use Case 1 Vulnerable product, but no affected users

Description

A vulnerability software or hardware with no users is a security hole that does not affect anyone else in any way. Examples of this case include products that are (a) non-production, experimental (e.g., OWASP WebGoat11 ), (b) internal or for personal use, (c) never published or sold, or (d) under development.

Vulnerability is discovered and fixed before the product is deployed. Vendor takes steps to prevent recurrence of the vulnerability. No advisory required for users.

Coordination is not required, except:

Variant 1: Product is deployed before vulnerability is discovered or fixed

Description

The product is shipped and available with one or more existing vulnerabilities. The vendor discovers the vulnerabilities and corrects them. The vendor releases an updated version of the product and takes steps to prevent reoccurrence. The vendor, then, publishes an advisory.

Causes
Prevention

Use Case 2: Vulnerability with coordinated disclosure

Vulnerability with coordinated disclosure Figure 3: Vulnerability with coordinated disclosure

Description

Many security vulnerabilities are discovered after the product is released. Multiple stakeholders—including finders, upstream vendors, vendors, defenders, and users—are involved in the coordinated disclosure effort. Stakeholders are encouraged to follow guidelines set out by international bodies like FIRST and ISO to formulate the basis of their disclosure practice.

In a generalized coordinated disclosure process, the following stakeholders perform certain roles.

Finder

Vendors

Defenders

Users

Variant 1: Finder makes the vulnerability details public prior to remediation

Description

There may be instances in which a finder publicly releases details of a vulnerability prior to remediation, which can increase the risk to affected users. Although a known active exploitation may prompt the finder to publicly disclose prior to remediation, other causes for disclosure include inability to establish contact with vendor and financial or other motivations. Preventing public release prior to remediation is ideal, but in cases where early public release occurs, quick response and communication of potential mitigations are paramount.

Causes
Prevention
Response

Variant 2: Users do not deploy remediation immediately

Description

Providing remediation alone is not sufficient to reduce risk; deployment is also necessary. There may be instances in which users either do not deploy the remediation or the vendor suggested mitigations immediately after being made available by the upstream vendor. In general, users are strongly encouraged to apply, whenever possible, a risk-based approach in deciding how quickly to deploy vendor-supplied remediation or mitigations when made available to help reduce potential risk of exploitation. Vendors responsible for issuing remediation or mitigations for critical and high severity vulnerabilities should communicate the availability of such, as broadly as possible, along with clear deployment and recommendations.

Causes
Prevention
Response

Variant 3: Missing communication between upstream and downstream vendors

Use Case 2, Variant 3 Missing communication between upstream and downstream vendors Figure 4: Use Case 2, Variant 3 Missing communication between upstream and downstream vendors

Description

Direct communication or a security disclosure could be missing between upstream vendors and downstream vendors or between vendors and users. A coordinator could facilitate receiving and distributing information back and forth to relevant parties at the various stages of remediation.

Causes
Prevention
Response

Variant 4: Vendor makes the vulnerability details public prior to remediation

Description

Multi-party vulnerability disclosure often involves complex interaction among stakeholders. It is possible for a vendor to disclose the vulnerability details publicly prior to remediation. In many cases, such disclosure is accidental and a plan for damage control should be in place. A review of the incident afterwards should take place to prevent occurrences in the future.

Causes
Prevention
Response

Variant 5: Vendor does not remediate a reported vulnerability

Description

There may be situations in which the vendor does not provide remediation to a vulnerability. There are many causes for such a scenario, including the vendor no longer existing, the affected product no longer being supported, the vendor being unable to verify the finder’s report, or the vendor not considering the report to be a vulnerability. Establishing clear communication and dialogue between the reporter and vendor is foundational to establishing a plan of action, whether that be remediation or mitigation.

Causes
Response

Variant 6: Missing communication between peer vendors impedes coordination

Description

Missing or poor communication between peer vendors can negatively impact coordination efforts. In some cases, this is due to lack of awareness of the uses and impacts of a common component or technology that makes it difficult to identify and coordinate with affected peers. Use of third-party coordinators and the investment in developing and maintaining an awareness of peer vendors are just two ways of managing these complexities in multi-party coordinated response.

Example 1. A vulnerability named “httpoxy” affected many CGI or CGI-like environments.

According to httpoxy.org, it was first discovered in 2001. Over the years, the issue was rediscovered many times. Its impact on other peer CGI implementations was never investigated. In 2016, when an exploit was discovered in the wild, the issue was widely investigated across various CGI implementations and 14 CVE IDs were assigned.

Example 2. CVE-2008-1447

CVE-2008-1447 is a vulnerability in DNS protocol that was first mitigated by UDP source port randomization idea implemented in djbdns in 1999. While importance of this mitigation was emphasized on public mailing lists, many other DNS implementations lacked this mitigation until 2008. When a practical exploit for this vulnerability was demonstrated in 2008, the source port randomization mitigation was widely implemented.

Causes
Prevention
Responses

Variant 7: Coordinator makes vulnerability details public prior to remediation

Description

In this variant, a coordinator discloses vulnerability information publicly before remediation is ready. As in previous variants, disclosure may be accidental, or a coordinator may intentionally disclose due to the perceived defensive benefit.

Causes
Prevention
Responses

Variant 8: Finder reports a vulnerability to one vendor that may affect others using the same component

Description

In this variant, the finder discovers a vulnerability that may affect more than one vendor’s product. The finder reports the vulnerability to one vendor, but not to others also affected. Other vendors may not be aware of the issue, and customers may not understand when a vulnerability is addressed.

Causes
Prevention

Use Case 3: Public disclosure of limited vulnerability information prior to remediation

Use Case 3 Public disclosure of vulnerability and impact prior to remediation Figure 5: Use Case 3 Public disclosure of vulnerability and impact prior to remediation

Description

Some information about the vulnerability is published without giving any hints about the exploit. This use case is different than what is typically called “full disclosure.”

As a middle way between full public disclosure and a privately coordinated disclosure, a finder or a vendor may publish some preliminary notice about the existence of a vulnerability and its disclosure timeline. Information disclosed may contain names of vulnerable product or component, worst-case impact, and location of future advisories, but not provide any hints about exploiting the vulnerability, such as source code changes or vulnerability type. This disclosure scenario is common when a large number of vendors are affected and maintaining confidentiality can be difficult.

Such advance notice helps all the responding parties (i.e., upstream vendors, downstream vendors, users and defenders) to plan and prepare to respond to the disclosure. Preparation may involve identifying potentially affected products and assets, identifying personnel responsible for analyzing the security fixes, making code changes or patching, testing, and solution delivery. The variations, including causes, preventions, and responses from Use Case 2, also apply to Use Case 3.

Example 1. Vendor advance warning

On April 28, 2016, the OpenSSL project team announced a new software release with fixes for several ‘high’ severity security defects that would be made available on May 3rd, 2016.15 The users and downstream vendors had five days to plan and prepare for taking response measures, thus minimizing the preparation time required for the responders.

Example 2. Vendor expected cadence

Oracle publishes Critical Patch Update Advisories on a pre-determined quarterly schedule.16 A pre-release announcement is also published five days prior to each Critical Patch Update release with a summary of affected products and risks. This notification serves as a trigger to initiate a customer’s patching procedure.

Example 3. Researcher advance warning

OnJuly 17, 2019, Orange Tsai and Meh Changi released a blog post called Attacking SSL VPN - Part 1: PreAuth RCE on Palo Alto GlobalProtect, with Uber as Case Study!17 The post provided details about a vulnerability in one vendor and product and announcing plans to disclose more at Black Hat and DEF CON 2019.

Use Case 4: Public disclosure or exploitation of vulnerability prior to vendor awareness

Use Case 4 Public disclosure or exploitation of vulnerability prior to vendor awareness Figure 6: Use Case 4 Public disclosure or exploitation of vulnerability prior to vendor awareness

Description

When a vulnerability is discovered in a deployed product, the finder makes the information about the vulnerability accessible to anyone by methods such as publishing on the Internet, mailing lists, academic papers, or conferences. Disclosed information may include affected products and versions, proof-of-concept test cases that can trigger or demonstrate the vulnerability, and detailed explanation of the defect or attack methodology. This disclosure is made without waiting for development or deployment of a remediation or mitigation. This type of disclosure is often referred to as “full disclosure”18 or a “zero-day.”

One of the main intentions is to make users aware of the vulnerability as early as possible as a way to minimize exposure, with an assumption that there could be unknown attackers who may already know about the vulnerability and could be exploiting it.

An Internet survey of about 400 researchers, indicates that only 4% of the researchers follow full public disclosure, versus 92% of researchers that follow some form of coordinated disclosure. While such disclosures are rare, vulnerability responders (vendors, defenders, users) should be prepared to handle disclosures anytime.

Example 1. A paper19 presented at AppSec California in January 2015, described remote code execution under certain context related to Apache Commons Collection. Apache Commons project was not informed20. On November 2015, a blog post21 was published containing exploits based on this paper for multiple products. None of the vendors or open source projects were directly notified prior to disclosure.

Variant 1: Finder publishes vulnerability details and vulnerability is exploited

Description

In this variant, a finder publicly discloses detailed vulnerability information without first having notified the vendor. Attackers can use this information to develop exploits and attack systems before vendors have prepared a remediation. Typically, attackers can develop attacks faster than vendors can develop a remediation and users can deploy them. This variant is commonly called a “zero-day” disclosure.

Causes
Prevention
Response

Variant 2: Previously undisclosed vulnerability used in attacks

Description

In this variant, a vulnerability becomes publicly known because of its use in attacks. This variant is also referred to as a “zero-day” vulnerability or exploit since vendors and defenders have not had a warning in advance. This is usually a very harmful scenario since vendors, defenders, and users rush to respond while under attack. Exploitation of a vulnerability in an attack can be considered a disclosure of the vulnerability or a confirmation of its existence. The attacker typically wants the vulnerability and its exploitation to remain undetected and undisclosed.

Causes
Prevention
Response

Acknowledgements

The Vulnerability Coordination SIG thanks all of its members and specifically the following contributors:

Pete Allor

Christa Anderson, Microsoft
Jerry Bryant, Intel
Vic Chung, SAP
Mark Cox, Red Hat
Jeroen van der Ham, NCSC-NL
Jean-Robert Hountomey, Brocade
Kent Landfield, McAfee
Magid Latif, Intel
Art Manion, CERT/CC
Klee Michaelis, Cisco
Beverly Miller Alvarez, Lenovo
Bruce Monroe, Intel
Chandan Nandakumaraiah, Palo Alto Networks
Kymberlee Price, Bugcrowd
Shawn Richardson, NVIDIA
Vivian Smith, Dell Technologies
Katie Trimble Noble, Intel
Krassimir Tzvetanov
Tania Ward, Dell Technologies
Brian Willis, Intel

The SIG also thanks Allan Friedman of the National Telecommunications and Information Administration (NTIA) at the U.S. Department of Commerce.

Supporting Resources


  1. http://heartbleed.com/ ^
  2. https://www.ntia.doc.gov/other-publication/2016/multistakeholder-process-cybersecurity-vulnerabilities ^
  3. http://www.iso.org/iso/catalogue_detail.htm?csnumber=45170 ^
  4. https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=csaf ^
  5. https://nvd.nist.gov/ ^
  6. https://cve.mitre.org/ ^
  7. https://cve.mitre.org/cve/cna/rules.html ^
  8. https://www.first.org/cvss/ ^
  9. https://blog.joda.org/2018/09/java-release-chains-features-and-security.html ^
  10. https://uefi.org/security ^
  11. https://www2.owasp.org/www-project-webgoat/ ^
  12. https://blog.joda.org/2018/09/java-release-chains-features-and-security.html ^
  13. OpenSSL CVE-2016-2108: A vulnerability was fixed in OpenSSL June 2015 releases, but it was not recognized as a vulnerability until May 2016. Downstream vendors who upgraded their OpenSSL code base to the latest stable release in June 2015 had effectively resolved this vulnerability eleven months ahead of vendors who selectively patched only the CVE assigned vulnerabilities. ^
  14. https://www.first.org/global/sigs/vrdx/vdb-catalog ^
  15. https://mta.openssl.org/pipermail/openssl-announce/2016-April/000069.html ^
  16. http://www.oracle.com/us/support/assurance/leveraging-cpu-wp-164638.pdf ^
  17. http://blog.orange.tw/2019/07/attacking-ssl-vpn-part-1-preauth-rce-on-palo-alto.html ^
  18. Strictly speaking, “full disclosure” means publication of vulnerability details before remediation is available, either before or after notifying vendors. ^
  19. http://frohoff.github.io/appseccali-marshalling-pickles/ ^
  20. https://commons.apache.org/proper/commons-collections/security-reports.html#Apache_Commons_Collections_Security_Vulnerabilities ^
  21. https://foxglovesecurity.com/2015/11/06/ ^
  22. Coordinated vulnerability disclosure is often considered part of the deployment, maintenance, or support phases of a Secure Software Development Lifecycle. ^