Guidelines and Practices for Multi-Party Vulnerability Coordination and Disclosure

Version 1.0
Released Summer, 2017

Introduction

Events in the recent past have highlighted the need for real improvements in the area of vulnerability coordination. Historically, foundational work on best practices, policy, and process for vulnerability disclosure have focused on bi-lateral coordination and did not adequately address the current complexities of multi-party vulnerability coordination. Factors such as a vibrant open source development community, the proliferation of bug bounty programs, third party software, and the support challenges facing CSIRTs and PSIRTs or bug bounty programs are just a few of the complications. Examples such as Heartbleed1 highlight coordination challenges.

This document is the outcome of an effort between the National Telecommunications and Information Administration (NTIA)2 and FIRST to address such challenges. The purpose of this document is to assist in improving multi-party vulnerability coordination across different stakeholder communities.

The Industry Consortium for Advancement of Security on the Internet (ICASI) proposed to the FIRST Board of Directors that a Special Interest Group (SIG) be considered on Vulnerability Disclosure. After holding meetings at the FIRST Conferences in Boston and Berlin, ICASI formally requested FIRST to charter a SIG to review and update vulnerability coordination guidelines. The first part of this work is collaboration with the National Telecommunications and Information Administration (NTIA) to address multi-party coordination. Subsequent work will address bi-lateral coordination and approaches to notification. This document differs from the ISO Vulnerability disclosure and handling standards (ISO/IEC 29147 and ISO/IEC 30111) in that the ISO standards provide basic guidance on the handling of potential vulnerabilities in products. This document is a collection of best current practices that consider more complex and typical real-life scenarios that extend past a single researcher notifying a single company about a discovered vulnerability.

This document is a compendium of coordination resource documents and recommended methods for reporting/updating coordination directories. The guidelines contain a common set of ‘guiding concepts', and vulnerability coordination best practices that include use cases or examples that describe scenarios and disclosure paths. This document is targeted at vulnerabilities that have the potential to affect a wide range of vendors and technologies at the same time.

Notice: This document describes what the Forum of Incident Response and Security Teams, Inc. (FIRST.Org) believes are best practices. These descriptions are for informational purposes only. FIRST.Org is not liable for any damages of any nature incurred as a result of or in connection with the use of this information.

Definitions

Within the context of this document, the following definitions apply. Definitions that are available in ISO/IEC 29147:20143 are used with minimal modification.

Stakeholder roles and communication paths Figure 1: Stakeholder roles and communication paths

Figure 1 shows the relationships and communication paths between stakeholder roles.

Multi-Party Disclosure Use Cases

Vulnerability disclosure can be a complicated process, especially when multiple parties (usually multiple vendors) are involved. This section of the document is organized as a set of vulnerability disclosure use cases, in rough order, from simple to complex. Significant attention is given to coordinated, Multi-Party Disclosure (see Error! Reference source not found.). Disclosure often deviates from the expected or ideal process, so within each use case are variants that are common exceptions to the ideal use case. Within each variant are causes, preventions, and responses. The collected set of preventions and responses are presented as practices that can be used to reduce the occurrence and cost of expected variants.

Practices are denoted as strong recommendations (“should”) or suggestions (“can,” “could,” or “may”).

At the conclusion of the use cases and variants, practices are rolled-up into the concluding section: Guiding Concepts and Best Practices.

Multi-Party Disclosure Use Cases

Vulnerability disclosure can be a complicated process, especially when multiple parties (usually multiple vendors) are involved. This section of the document is organized as a set of vulnerability disclosure use cases, in rough order, from simple to complex. Significant attention is given to coordinated, multi-party disclosure (see Use Case 2: Vulnerability with coordinated disclosure). Disclosure often deviates from the expected or ideal process, so within each use case are variants that are common exceptions to the ideal use case. Within each variant are causes, preventions, and responses. The collected set of preventions and responses are presented as practices that can be used to reduce the occurrence and cost of expected variants.

Use Case 0: No vulnerability

Description

This case is included for completeness, if there are no vulnerabilities, there is no need for coordination.

Use Case 1: Vulnerability with no affected users

Use Case 1 Vulnerable product, but no affected users Figure 2: Use Case 1 Vulnerable product, but no affected users

Description

A vulnerability software or hardware with no users is a security hole that does not affect anyone else in any way. Examples of this case include: products that are (a) non-production, experimental (e.g., webgoat), (b) internal or for personal use, (c) never published or sold, or (d) under development.

Vulnerability is discovered and fixed before the product is deployed. Vendor takes steps to prevent recurrence of the vulnerability. No advisory required for users.

Coordination is not required, except:

Variant 1: Product is deployed before vulnerability is discovered or fixed

Description

The product is shipped and available with one or more existing vulnerabilities. The vendor discovers the vulnerabilities and corrects them. The vendor releases an updated version of the product and takes steps to prevent reoccurrence. The vendor, then, publishes an advisory.

Causes
Prevention

Use Case 2: Vulnerability with coordinated disclosure

Vulnerability with coordinated disclosure Figure 3: Vulnerability with coordinated disclosure

Description

Many security vulnerabilities are discovered after the product is released. Multiple stakeholders such as finders, upstream vendors, vendors, defenders, and users are involved in the coordinated disclosure effort. Stakeholders are encouraged to follow some guidelines set out by international bodies like ISO, to formulate the basis of their disclosure practice.

In a generalized coordinated disclosure process, the following stakeholders perform certain roles.

Finder

Vendors

Defenders

Users

Variant 1: Finder makes the vulnerability details public prior to remediation

Description

There may be instances in which a finder publicly releases details of a vulnerability prior to remediation, which can increase risk to affected users. Although a known active exploitation may prompt the finder to publicly disclose prior to remediation, other causes for disclosure include inability to establish contact with vendor and financial or other motivations for finder disclosure. Preventing public release prior to remediation is ideal, but in cases where early public release happens, quick response and communication of potential mitigations is paramount.

Causes
Prevention
Response

Variant 2: Users do not deploy remediation immediately

Description

Providing remediation alone is not sufficient to reduce risk, deployment is also necessary. There may be instances in which users do not deploy either the remediation or the vendor suggested mitigations immediately after being made available by the upstream vendor. In general, users are strongly encouraged to apply, where possible, a risk-based approach in deciding how quickly they should deploy vendor-supplied remediation or mitigations when made available to help reduce potential risk of exploitation. Vendors responsible for issuing remediation or mitigations for critical and high severity vulnerabilities should communicate the availability of such, as broadly as possible, along with clear deployment and recommendations.

Causes
Prevention
Response

Variant 3: Missing communication between upstream and downstream vendors

Use Case 2, Variant 3 Missing communication between upstream and downstream vendors Figure 4: Use Case 2, Variant 3 Missing communication between upstream and downstream vendors

Description

Direct communication or a security disclosure could be missing between upstream vendors and downstream vendors or between vendors and users. A coordinator could facilitate receiving and distributing information back and forth to relevant parties at the various stages of remediation.

Causes
Prevention
Response

Variant 4: Vendor makes the vulnerability details public prior to remediation

Description

Multi-party vulnerability disclosure often involves complex interaction among stakeholders. It is possible for a vendor to disclose the vulnerability details publicly prior to remediation. In many cases, such disclosure is accidental and a plan for damage control should be in place. A review of the incident afterwards should take place to prevent occurrences in the future.

Causes
Prevention
Response

Variant 5: Vendor does not remediate a reported vulnerability

Description

There may be situations in which the vendor does not provide remediation to a vulnerability. There are many causes for such a scenario including the vendor no longer existing, the affected product no longer being supported, the vendor being unable to verify the finder’s report, or the vendor not considering the report to be a vulnerability. Establishing clear communication and dialogue between the reporter and vendor is foundational to establishing a plan of action, whether that be remediation or mitigation.

Causes
Prevention
Response

Variant 6: Missing communication between peer vendors impedes coordination

Description

Missing or poor communication between peer vendors can negatively impact coordination efforts. In some cases, this is due to lack of awareness of the uses and impacts of a common component or technology that make it difficult to identify and coordinate with affected peers. Use of third party coordinators and the investment in developing and maintaining an awareness of peer vendors are just two ways of managing these complexities in multi-party coordinated response.

Example 1. A vulnerability named ‘httpoxy’ affected many CGI or CGI like environments. According to httpoxy.org, it was first discovered in 2001. Over the years the issue was rediscovered many times. Its impact on other peer CGI implementations was never investigated. In 2016 when an exploit was discovered in the wild, the issue was widely investigated across various CGI implementations and 14 CVE identifiers were assigned.

Example 2. CVE-2008-1447 CVE-2008-1447 is a vulnerability in DNS protocol that was first mitigated by UDP source port randomization idea implemented in djbdns in 1999. While importance of this mitigation was emphasized on public mailing lists, many other DNS implementations lacked this mitigation until 2008. When a practical exploit for this vulnerability was demonstrated in 2008, the source port randomization mitigation was widely implemented.

Causes
Prevention
Responses

Variant 7: Coordinator makes vulnerability details public prior to remediation

Description

In this variant, a coordinator discloses vulnerability information publicly before remediation is ready. As in previous variants, disclosure may be accidental, or a coordinator may intentionally disclose due to the perceived defensive benefit. Also, similar to other variants setting and expectation, good communication can reduce accidental disclosures.

Causes
Prevention
Responses

Use Case 3: Public disclosure of limited vulnerability information prior to remediation

Use Case 3 Public disclosure of vulnerability and impact prior to remediation Figure 5: Use Case 3 Public disclosure of vulnerability and impact prior to remediation

Description

Some information about the vulnerability is published without giving any hints about the exploit. This use case is different than what is typically called “full-disclosure.” As a middle way between full public disclosure and a privately coordinated disclosure, a finder or a vendor may publish some preliminary notice about the existence of a vulnerability and its disclosure timeline. Information disclosed may contain names of vulnerable product or component, worst case impact, and location of future advisories, but not provide any hints about exploiting the vulnerability such as source code changes or vulnerability type. This disclosure scenario is common when a large number of vendors are affected and maintaining confidentiality can be difficult.

Such advance notice helps all the responding parties (i.e., upstream vendors, downstream vendors, users and defenders) to plan and prepare to respond to the disclosure. Preparation may involve identifying potentially affected products and assets, identifying personnel responsible for analyzing the security fixes, making code changes or patching, testing, and solution delivery.

The variations, including causes, preventions, and responses from Use Case 2 also apply to Use Case 3.

Example 1. Vendor advance warning: On April 28, 2016, OpenSSL project team announced a new software release with fixes for several ‘high’ severity security defects that was made available on May 3rd, 2016. The users and downstream vendors had five days to plan and prepare for taking response measures, thus minimizing the preparation time required for the responders.

Example 2. Vendor expected cadence: Oracle published Critical Patch Update Advisories on a pre-determined quarterly schedule. According to Oracle8, a pre-release announcement is also published five days prior to each Critical Patch Update release with a summary of affected products and risks. This notification serves as a trigger to initiate a customer’s patching procedure.

Example 3. Researcher advance warning: On 22nd March 2015, Stefan Metzmacher published an advance warning on website badlock.org, that a crucial security bug in Windows and Samba would be disclosed on April 12th, 2016. System administrators responsible for Windows or Samba server infrastructure were advised to be ready to patch their systems.

Use Case 4: Public disclosure or exploitation of vulnerability prior to vendor awareness

Use Case 4 Public disclosure or exploitation of vulnerability prior to vendor awareness Figure 6: Use Case 4 Public disclosure or exploitation of vulnerability prior to vendor awareness

Description

When a vulnerability is discovered in a deployed product, the finder makes the information about the vulnerability accessible to anyone by methods such as publishing on the Internet, mailing lists, academic papers, or conferences. Disclosed information may include affected products and versions, proof of concept test cases that can trigger or demonstrate the vulnerability and detailed explanation of the defect or attack methodology. This disclosure is made without waiting for development or deployment of a remediation or mitigation. This type of disclosure is often referred to as “full disclosure”9 or a “zero-day.”

One of the main intentions here is to make users aware of the vulnerability as early as possible as a way to minimize exposure, with an assumption that there could be unknown attackers who may already know about the vulnerability and could be exploiting it.

An Internet survey of about 400 researchers, indicates that only 4% of the researchers follow full public disclosure versus 92% of researchers that follow some form of coordinated disclosure. While such disclosures are rare, vulnerability responders (vendors, defenders, users) should be prepared to handle disclosures anytime.

Example 1. A paper10 presented at AppSec California in January 2015, described remote code execution under certain context related to Apache Commons Collection. Apache Commons project was not informed11. On November 2015, a blog post12 was published containing exploits based on this paper for multiple products. None of the vendors or open source projects were directly notified prior to disclosure.

Variant 1: Finder publishes vulnerability details and vulnerability is exploited

Description

In this variant, a finder publicly discloses detailed vulnerability information without first having notified the vendor. Attackers can use this information to develop exploits and attack systems before vendors have prepared a remediation. Typically, attackers can develop attacks faster than vendors can develop a remediation and users can deploy them. This variant is commonly called a “zero-day” disclosure.

Causes
Prevention
Response

Variant 2: Previously undisclosed vulnerability used in attacks

Description

In this variant, a vulnerability becomes publicly known because of its use in attacks. This variant is also referred to as a “zero-day” vulnerability or exploit, since vendors and defenders have not had a warning in advance. This is usually a very harmful scenario since vendors, defenders, and users rush to respond while under attack. Exploitation of a vulnerability in an attack can be considered as a disclosure of the vulnerability or a confirmation of its existence. The attacker typically wants the vulnerability and its exploitation to remain undetected and undisclosed.

Causes
Prevention
Response

Guiding Concepts and Best Current Practices

The following guidance is derived from the cases, variants, responses, and preventions discussed previously. The most important practices, particularly those that occurred repeatedly, are captured here. Stakeholders should carefully consider their actions, particularly notification and public disclosure, due to the widespread impact on other stakeholders in multi-party cases.

Establish a strong foundation of processes and relationships

Maintain clear and consistent communications

Prior to disclosure

After disclosure

Build and maintain trust

Minimize exposure for stakeholders

Respond quickly to early disclosure

Use coordinators when appropriate

Acknowledgements

The Vulnerability Coordination SIG thanks all of its members and specifically the following contributors:

Pete Allor, Honeywell
Christa Anderson, Microsoft
Jerry Bryant, Microsoft
Vic Chung, SAP
Mark Cox, Red Hat
Beverly Finch, Lenovo
Jeroen van der Ham, NCSC-NL
Kent Landfield, McAfee
Magid Latif, Intel
Art Manion, CERT/CC
Klee Michaelis, Cisco
Bruce Monroe, Intel
Chandan Nandakumaraiah, Juniper
Kymberlee Price, Bugcrowd
Krassimir Tzvetanov, Fastly
Tania Ward, Dell EMC
Brian Willis, Intel

The SIG also thanks Allan Friedman of the National Telecommunications and Information Administration (NTIA) at the U.S. Department of Commerce.

Supporting Resources

 


  1. http://heartbleed.com/ 

  2. https://www.ntia.doc.gov/other-publication/2016/multistakeholder-process-cybersecurity-vulnerabilities 

  3. http://www.iso.org/iso/catalogue_detail.htm?csnumber=45170 

  4. OpenSSL CVE-2016-2108: A vulnerability was fixed in OpenSSL June 2015 releases, but was not recognized as a vulnerability until May 2016. Downstream Vendors who upgraded their OpenSSL code base to the latest stable release in June 2015 had effectively resolved this vulnerability eleven months ahead of vendors who selectively patched only the CVE assigned vulnerabilities. 

  5. https://nvd.nist.gov 

  6. http://cve.mitre.org 

  7. https://www.first.org/global/sigs/vrdx/vdb-catalog 

  8. http://www.oracle.com/us/support/assurance/leveraging-cpu-wp-164638.pdf 

  9. Strictly speaking, “full disclosure” means publication of vulnerability details before remediation is available, either before or after notifying vendors. 

  10. http://frohoff.github.io/appseccali-marshalling-pickles/ 

  11. https://commons.apache.org/proper/commons-collections/security-reports.html#Apache_Commons_Collections_Security_Vulnerabilities 

  12. https://foxglovesecurity.com/2015/11/06/ 

  13. Coordinated vulnerability disclosure is often considered part of the deployment, maintenance, or support phases of a Secure Software Development Lifecycle.