Also available in PDF.
The Services Frameworks are high-level documents detailing possible services that computer incident response teams (CSIRTs) and product incident response teams (PSIRTs) may provide. They are developed by recognized experts from the FIRST community. FIRST strives to include feedback from all sectors, including CSIRTs with a national responsibility, private sector CSIRTs, and PSIRTs as well as other stakeholders. These documents were intended to provide a foundation for the development of new training material. However today they are used in a much wider scope, for example when defining an initial service catalogue for new teams.
In the creation of the CSIRT Services Framework it became clear that PSIRTs do provide quite different services and typically operate in quite different environments. It was thus decided to create a separate document covering PSIRTs. The two documents will be aligned, highlighting the many similarities shared. The development of the frameworks is driven by the Education Advisory Board.
The Frameworks exist to assist organizations in building, maintaining, and growing the capabilities of their CSIRTs or PSIRTs. The Frameworks are guides and identify various models, capabilities, services, and outcomes. In this way, teams are free to implement their own model and to build capabilities that meet their stakeholders’ unique needs. The Frameworks seek to assist security incident response teams (SIRTs) by identifying core responsibilities, providing guidance on how to build capabilities to meet those responsibilities and offering insights on how teams can add and communicate value to their larger organizations.
A Product Security Incident Response Team (PSIRT) is an entity within an organization which, at its core, focuses on the identification, assessment, and disposition of the risks associated with security vulnerabilities within the products, including offerings, solutions, components, and/or services which an organization produces and/or sells.
A properly deployed PSIRT is not an independently operating group, disconnected from the development of the organization’s products. Instead it is part of the organization’s broader secure engineering initiative. This structure ensures that security assurance activities are integrated into the Secure Development Lifecycle (SDL).
Product security incident response is often associated with the maintenance phase of the SDL because most product security vulnerabilities are reported as quality escapes after the product has been released to the market. However, PSIRT can be impactful in the earlier requirements gathering of architecture, design, planning, and risk-modeling phases. PSIRT functions may also add value by providing guidance and oversight for the handling of internally found security issues.
SERVICE AREAS – SERVICES – FUNCTIONS – SUB-FUNCTIONS
Service areas regroup services related to a common aspect. They help to organize the services along a top-level categorization to facilitate understanding. The specification for each service area would include a “Description” field consisting of a general, high-level narrative text describing the service area and the list of services within the service area.
A service is a set of recognizable, coherent functions towards a specific result on behalf of or for the stakeholder of an incident response team.
A service is specified by the following template:
A function is an activity or set of activities aimed at fulfilling the purpose of a particular service. Any function might be shared and used in the context of several services.
A function is described by the following template:
A sub-function is an activity or set of activities aimed at fulfilling the purpose of a particular function. Any sub-function might be shared and used in the context of several functions and/or services.
The difference between PSIRT and CSIRT
The focus on products is the key differentiator between the PSIRT of an organization and other incident response teams represented in the same organization, such as a CSIRT. Generally, an Enterprise CSIRT is focused on the security of computer systems and/or networks that make up the infrastructure of an organization.
While there are important differences between an Enterprise CSIRT and PSIRT, it is important to recognize that there is also synergy between the two entities. The important point to take away is that both PSIRTs and CSIRTs do not operate independently of other parts of an organization, and throughout this framework we will highlight areas of collaboration and synergy that should be nurtured.
Figure 1: Organizational Structure
PSIRTs can be as unique and varied as the products they help protect. Between organizations within the same sector or industry there will be variations in business characteristics, operating models, product portfolios, organizational structures, and product development strategies. As a result, there is no single one-size-fits-all product security incident response strategy or team template for all organizations to follow. However, three PSIRT models are used by most companies: Distributed, Centralized, and Hybrid.
The Distributed model utilizes a small core PSIRT that works with representatives from the product teams to address security vulnerabilities in products. In this model, the smaller PSIRT Operations has several core responsibilities:
Figure 2: Distributed Model
An organization with a large and diverse product portfolio can benefit from the Distributed model because the cost of the PSIRT mission is defrayed across the organization. This model also allows the PSIRT mission to scale by leveraging the skilled people in the product engineering teams.
The challenge with the Distributed PSIRT model is that the people responsible for performing the triage and delivering the fixes for security vulnerabilities are not directly controlled by and do not report to the PSIRT Operations.
The Centralized model has a larger PSIRT staff drawn from multiple departments that report to one or more senior executives responsible for the organization’s product security. This model might have a structure similar to the following:
Figure 3: Centralized Model
This model works well with a smaller organization and/or an organization with a homogenous product portfolio. This model concentrates and cultivates a high level of security skill and expertise into one area of the organization. The challenge with this model is in the cost of maintaining a centrally specialized team that does not scale as well if the product portfolio grows and/or becomes more diverse.
Figure 4: Hybrid Model
It is important for a PSIRT to have the autonomy to maintain an independent and objective position on the organization’s product security vulnerabilities. As such, in developing the organization’s PSIRT strategy and structure, the organization should consider how the team should best be integrated into the organization and its reporting structure. It is important that a PSIRT report to an executive of the company who confirms the authority of the PSIRT.
As a PSIRT continues to mature and scale, and the mission evolves, the composition or reporting structure of the team could change. The driving force of change and maturity of a PSIRT is its key stakeholders and, unfortunately, the impact of a severe vulnerability on a broad spectrum of the organization’s stakeholder base. Stakeholders are often defined by the model adopted by the organization as well as the size of the organization.
Considering the stakeholders’ needs and requirements is a critical part of defining the strategy and structure of a PSIRT. The model that an organization adopts to form the PSIRT can dictate the identity of the stakeholders and the amount of influence they have. It is critical to continue to maintain positive relationships. Service Area 1: Stakeholder Ecosystem Management, contains more detail on the ecosystem of stakeholders and how to manage them.
One final consideration in the formation of the product incident response team and strategy is influencers. This is different than stakeholders, in that stakeholders are discretely named people or groups of people. In contrast, influencers are industry and government standards, legislation, regulations, and trends. These influencers may impose greater requirements on the formation, strategies, policies, and operational characteristics of a PSIRT than the stakeholders.
The model used will define the scope and operational activities of a PSIRT, but not necessarily change the actions an organization needs to take with respect to addressing security vulnerabilities in their products. The model refines the scope of the capabilities, actions and responsibilities directly attributed to the PSIRT rather than those distributed throughout the organization.
Figure 5: General PSIRT Activities
Ongoing Process and Policy Development
PSIRTs establish the organization’s policies with respect to product security. The needs of the business drive and dictate the requirements of PSIRT and not the other way around. Before the PSIRT policies can be implemented, they must be reviewed and imbued with authority by the organization’s leadership. Approved policies must be followed with clear procedures that, when followed, ensure the organization’s compliance with these policies.
Along with PSIRT policies and procedures, the PSIRT needs to build workflow and management systems that streamline the execution and completion of the actions required to address product security vulnerabilities. These implementations will make it easier for the organization to adopt product security as part of their normal day-to-day business activities.
The greatest mistake that can be made when deploying the PSIRT mission, policies and procedures is to have it viewed as a separate responsibility or requirement. Therefore, it is critical to educate all members of the organization on product security basics and the role they play. The entire organization must be included, enabled, and empowered to meet the PSIRT policy requirements.
The Importance of Metrics
It is critical to measure the success of the product security incident response mission. Metrics reporting does not define the requirements, but supports the program, helps determine the required resources, and may help identify places that need process/tool improvements. Creation and tracking of metrics may also help in the maturing of a PSIRT by uncovering issues or bottlenecks with respect to the deployment and adoption of a PSIRT. Service 1.7 Stakeholder Metrics and Service 5.3 Vulnerability Metrics go into more detail on the types of metrics that would be valuable to track.
In this document, we are defining the use of certain terms. Note that Service Areas, Services, and Functions identify what is being done at different levels of detail, while Tasks and Actions identify how it is being done at different levels of detail. Tasks and Actions are being published later in an accompanying document and can/will be updated more frequently:
Advisory- [^1]announcement or bulletin that serves to inform, advise, and warn about a vulnerability of a product.
Bug Bars- criteria that define the types of bugs that qualify as a security vulnerability. Bugs that meet these criteria will be processed as a vulnerability through the PSIRT standard operating procedures.
Coordinator- [^2]optional participant that can assist vendors and finders in handling and disclosing vulnerability information.
Embargo- a hold on the publication of vulnerability details until affected vendors are able to release security updates or mitigations and workarounds to protect customers.
Finder- [^3]an individual or organization that identifies a potential vulnerability in a product or online service. Please note that finders can be researchers, reporters, security companies, hackers, users, governments, or coordinators.
Open Source- refers to works that are licensed in such a way that they may be freely redistributed and modified, where the source code is made available publicly, and is freely distributed and does not discriminate against any persons, groups or fields of endeavor, and is technology-neutral. Open source software is often maintained by a community of individuals and entities who collaboratively create and maintain it.
Partners- Original Equipment Manufacturers (OEM)s, suppliers, Original Design Manufacturers (ODM)s.
Product- [^4]a system implemented or developed for sale or to be offered for free.
Quality Gate- a set of criteria that must be met before the product moves to next phase of development or release.
Remediation (or Remedy)- [^5]a change made to a product or online service to remove or mitigate a vulnerability. A remediation typically takes the form of a binary file replacement, configuration change, or source code patch and recompile. Different terms used for “remediation” include patch, fix, update, hotfix, and upgrade. Mitigations are also called workarounds or countermeasures.
Risk- [^6]the “effect of uncertainty on objectives”. In this definition, uncertainties include events (which may or may not happen) and uncertainties caused by ambiguity or a lack of information.
Risk Acceptance- [^7]a risk response strategy whereby the project team decides to acknowledge the risk and not take any action unless the risk occurs.
Risk Register- [^8]a document in which the results of risk analysis and risk response planning are recorded.
Secure Development Lifecycle (SDL)- a development process that helps developers build more secure products and address security compliance requirements while reducing development costs.
Service Level Agreement (SLA)- a contract between a service provider (either internal or external) and the end user that defines the level of service expected from the service provider.
Stakeholders- [^9]PSIRT stakeholders are the groups that build and modify the product or product components and ensure an appropriate product communication strategy, and groups who can benefit from product security. In short, PSIRT Stakeholders either contribute to or benefit from product security and incident response.
Third-Party- any upstream supplier or producer that provides components incorporated into a product or solution/service.
Vendor- [^10]a person or organization that developed the product, or service, or is responsible for maintaining it.
Vulnerability- [^11]weakness of software, hardware, or online service that can be exploited.
This section identifies and describes the foundation of core components that an organization needs to plan, establish, and effectively operate a PSIRT.
A. Executive Sponsorship
Obtain sponsorship from the organization’s executives and key decision makers.
To obtain executive sponsorship, the organization should inform or educate the executives by providing them with a plan and other supporting information to help them understand the purpose, importance, potential risks of security vulnerabilities and benefits of operating a PSIRT. (See “PSIRT Charter” and “Budget” below.)
See Service 1.1 Internal Stakeholder Management for related information.
Identify stakeholders and the relationship your PSIRT will have with these groups.
This should include external stakeholders, such as the organization’s customers, external security researchers, CSIRTs, and other PSIRTs, as well as internal stakeholders, such as software developers, engineers, customer support, legal, and public/corporate/media relations.
See Service Area 1 Stakeholder Ecosystem Management (Service 1.1 Internal Stakeholder Management, Service 1.2 Finder Community Engagement, Service 1.3 Community and Organizational Engagement, and Service 1.4 Downstream Stakeholder Management) for related information.
C. PSIRT Charter
Develop a charter or other document (e.g. strategic plan, implementation plan, or concept of operations document).
The PSIRT charter/plan should define the following:
D. Organizational Model
Determine and document the organizational structure and model that the PSIRT will use.
The documented organizational model should describe the PSIRT’s internal reporting structure and identify the authority under which the PSIRT operates. See “PSIRT Organizational Structure” for descriptions of some common organizational models (e.g. distributed model, centralized model, hybrid model). See Service 1.5 Incident Communications Coordination within the Organization for more related information.
E. Management and Stakeholder Support
Obtain support “buy-in” from organizational management and internal stakeholders.
See Service 1.1 Internal Stakeholder Management for related information.
Identify the costs of resources required to operate the PSIRT and obtain the appropriate funding to finance these resources.
The budget should include expenses for staffing the PSIRT (salaries and benefits, plus other encumbered costs), equipment, and other capital expenses (e.g. information technology systems/devices, software licenses), and training budget (including travel expenses).
Identify the staffing resources needed to provide your PSIRT services and obtain a skilled staff.
This includes identifying the various staff positions or roles and responsibilities for individual members of the PSIRT, as well as the competencies (knowledge, skills, and abilities (KSAs)) and any other requirements (e.g. education, experience, certifications) expected of those roles. Full-time employee, vendors, contractors, or a combination of these may fill these positions or roles.
As part of the staffing plan (or as identified in a separate document), training requirements should be identified and planned, including general training for all PSIRT staff and role-based training for individuals (e.g. initial onboarding/mentoring; ongoing training, education, and awareness; specific training for professional development).
See Service 6.1 Training the PSIRT for related information.
C. Resources and Tools
Identify and acquire other necessary resources and tools.
These resources and tools include the following:
A. Policies and Procedures
Document the policies, processes, and procedures relevant to conducting PSIRT operations.
Documenting the policies and procedures will ensure common understanding among all PSIRT staff, enable consistency and repeatability of the products and services provided by the PSIRT, and serve as a training resource for new PSIRT staff.
B. Evaluation and Improvements
Identify metrics for evaluating performance and/or effectiveness to identify improvements.
The PSIRT should continuously and/or periodically assess or evaluate how it is performing (providing its products and services) and identify any potential areas for improvement.
Evaluation metrics and methods can be informal (e.g. collecting feedback from stakeholders) or formal, and can occur as needed (e.g. documenting lessons learned (see Function 1.1.3 Incident Postmortem Process) or on a designated schedule.
The information provided in this PSIRT Framework document can be one source of the criteria or capabilities used to evaluate a PSIRT’s operations.
This service area describes the services and functions a PSIRT can fulfill to appropriately engage with both internal and external stakeholders. Execution of services under this umbrella are in effect throughout the lifecycle of an incident or the maturity lifecycle of the PSIRT. This service area is dedicated to ensuring all stakeholders of the PSIRT are appropriately informed and engaged in the incident response process.
Prior to formally providing these services, the PSIRT must first identify the unique stakeholders that are relevant for their businesses. Stakeholders include such parties as executive or business leadership, internal development teams, external component providers or developers, or even the organization’s customer base. It can be extremely useful to compile a matrix of stakeholders-to-products/versions to streamline the communication process. Prior to communicating with these stakeholders, it would be beneficial to understand the viewpoints or artifacts or methods by which they desire to be engaged with (web portal, personalized email, internet chat, ticketing system, etc.). For the purposes of this document, stakeholders are divided into several groups (your specific business circumstance may identify others): finders, peers/partners, internal teams, and consumers of your products.
Figure 6: Internal Stakeholder Management
Define processes related to engaging with internal stakeholders to ensure both awareness and assistance during incidents. Successful internal stakeholder engagement will improve communication and response efforts by clearly communicating the PSIRTs role within the organization and making internal connections between product teams and security analysts.
Maintain active dialog with internal teams involved around the development, testing, packaging, and maintenance of the organization’s offerings. Internal stakeholders are not only engineering resources, but could also be testing/quality assurance, release engineering, stakeholder-facing support teams, sales and marketing, or other technical subject matter experts in the field.
Sub-Function 126.96.36.199 Engage Corporate/Business Leaders and Executives
For a PSIRT to be effective, it must understand and be able to react to the current organizational environment. Working with business leaders and executives helps the PSIRT on several levels. It helps legitimize the PSIRTs work within the organization by virtue of executive sponsorship. It allows the PSIRT to share information with leaders to help inform business decisions. It also allows leadership to express changes in policy and organizational direction that might alter the PSIRTs mission.
Sub-function 188.8.131.52 Engage Public Relations, Legal, and Corporate Communications
Engaging with the array of internal Communications and Legal teams will ensure that the PSIRT is compliant with current branding and messaging standards as well as the regulatory/legal environment the organization must be compliant with (e.g. privacy, federal space). Each of these stakeholders offers unique paths to critical stakeholders of the PSIRT, and lines of communication should be established prior to critical events or incidents to ensure all parties can effectively work together.
Sub-Function 184.108.40.206 Engage Lines of Business
Engaging with development stakeholders ensures issues are appropriately documented, prioritized and addressed. For example, engineers from the PSIRT or authorized delegates need to coordinate vulnerability remediation with the software engineering groups responsible for the faulty code. In times of incidents, these partnerships also assist in the speedy transmission of information and effective, quick remediation of the issue. Stakeholders here include Program or Product Managers, SDL oversight groups, Project Managers, Product Owners, and others with similar business-related responsibilities.
Sub-Function 220.127.116.11 Engage Development/Engineering
Engineers from the PSIRT need to coordinate vulnerability remediation with the software engineering groups responsible for the flawed code. Engaging with development stakeholders ensures issues are appropriately documented, prioritized, and addressed. In times of incidents, these partnerships also assist in the speedy transmission of information and effective, quick remediation of the issue.
Sub-Function 18.104.22.168 Engage Customer-Facing Teams – Sales, Support
Engineers from the PSIRT need to provide explanation and artifacts to stakeholder support teams so that as issues develop and become public the support organization can respond to stakeholder inquiries and support requests. “Support” could include front-line (a.k.a. “Help Desk”) personnel, premium support resources (e.g. Technical Account Management, Stakeholder Success Managers, etc.), internal/external Sales teams, or in-field resources (consulting, sales engineering, etc.).
Sub-Function 22.214.171.124 Internal Working Group Participation
In more mature organizations, engineers from the PSIRT can build and strengthen relationships with internal stakeholders by participating in various internal initiatives or working groups. This helps to reaffirm/establish the technical expertise of the PSIRT and build networking/communication channels for future efforts.
Maintaining and enforcing a Secure Development Lifecycle is a cornerstone of establishing stakeholder confidence and trust in an organization’s products. Without the ability to demonstrate repeatable application of security standards through a product’s lifecycle, stakeholders may lose faith in the organization’s products, they may impose harsher requirements on the organization (burden of proof, right to audit, etc.), and could ultimately lead to loss of revenue and stakeholder confidence.
Sub-Function 126.96.36.199 Participate in SDL Activities
SDL is a critical governance process that helps an organization produce stable, repeatable offerings that adhere to common standards. The PSIRT’s participation in the creation and maintenance of the organizational SDL helps ensure that appropriate security practices and checks are followed.
Sub-Function 188.8.131.52 Participate in SDL Governance
SDL is a critical governance process that helps an organization produce stable, repeatable offerings that adhere to common standards. The PSIRT’s participation in the governance and enforcement of the organizational SDL helps ensure that appropriate security practices and checks are followed, and that exceptions are documented and appropriately reviewed.
As vulnerabilities are discovered within the organization’s offerings, the PSIRT requires a process to review these issues be they code, process, or personnel-related, to provide feedback to participating stakeholders and organizational leaders. Some severe or very public security vulnerabilities may require more in-depth analysis about how the company reacted to and corrected the issues. A postmortem is a meeting involving all internal stakeholders that participated in remediation and communication efforts, and seek to document what went well, what could have been done better, and what changes will be made for future events.
Sub-Function 184.108.40.206 Establish Product Security Defect Review Process
Establishing a consistent process to review postmortem issues helps ensure that products are continuously improved with lessons learned.
Sub-Function 220.127.116.11 Review Timing of Processes and Release Updates Track areas of strength and weakness.
Sub-Function 18.104.22.168 Review High-Profile Incidents
Coordinate organizational lessons-learned, response and review for high-profile incidents, and provide reporting data to the business and other stakeholders as required.
Figure 7: Example of External Stakeholders for the PSIRT
Services related to engaging the research community as a stakeholder. Finders have many varied roles and unique perspectives – they may be academics, development professionals, professional security finders, or hobbyists. Finders may be researching theoretical attacks or flaws in the hopes of publication and academic achievement, while others may be professional security finders that are motivated by financial or corporate means. Still others may be hobbyists or enthusiasts participating in their spare time, perhaps to gain respect and accolades from their communities. Finder community engagement is a proactive approach to Product Security Incident Response.
Implement activities designed to maintain active dialogue with finders that have expertise in the security of a company’s products and access to different channels. PSIRTs can conduct numerous activities to more deeply engage with their finder communities. These activities could include inviting well-qualified finders into private contracts, engaging with them at conferences and other events, or even sponsoring academic research.
Nurturing relationships between peer PSIRTs can help in information-sharing and potential mutual assistance and/or coordination for incidents. Working with these peer organizations can help fill in vital data to remediate vulnerabilities and exposes the organization to the peer’s expertise as the two groups consult on issues. The PSIRT should establish communication channels (both normal and secured) with key peer PSIRTs. Establishing and nurturing relationships with industry peers is critical for information sharing and coordinating on issues that affect both organizations.
Sub-Function 22.214.171.124 Document and Define Peer PSIRTs
Collect contact information and engagement processes for future use. The PSIRT should engage and interact with the larger PSIRT community to share best practices and insights around lessons learned. As vulnerabilities arise, they are often solved in a collaborative, multi-group manner, allowing the PSIRT to extend its internal capabilities by leveraging these external peers for information and/or assistance.
Sub-Function 126.96.36.199 Define Coordinated Disclosure Process
PSIRTs should carefully document vulnerability information-sharing parameters and agreements. The PSIRT should honor embargo parameters set out by the vulnerability finder and/or reporting organization (and expect to have theirs honored).
Sub-Function 188.8.131.52 Establish Security Information-Sharing Process
The PSIRT should establish methods to securely share vulnerability and other confidential information with parties involved in the coordinated disclosure arrangement. This could be such options as out-of-band, non-electronic communication, encrypted email/portals, or private mailing lists.
Sub-Function 184.108.40.206 Participate in Industry SIGs and Working Groups
Working with peers on topics of industry interest supports and nurtures contacts and furthers the professionalization of the industry by collaboratively solving problems.
Working with government CSIRTs helps build trust to share information and helps the PSIRT earn the trust and respect of valued peers. Other organizations with relevant interests or communities include FIRST, MITRE, Advancing Open Standards for the Information Society (OASIS), the Industry Consortium for Advancement of Security on the Internet (ICASI), International Organization for Standardization (ISO), amongst others. Groups that participate could be viewed based on national, enterprise, regional, or industrial sectors.
Sub-Function 220.127.116.11 Engage Communities and Partners
The PSIRT should research where the desired external groups engage in dialog and make efforts to participate in those forums.
Security researchers come in many varieties – academics, hobbyists, and professional security practitioners, to name a few. These persons are the primary finders of vulnerabilities across the industry. Researchers will attempt to contact the owner of a product, but for assorted reasons they will not always reach the appropriate party. PSIRTs will passively receive reports from these individuals or groups and be forced to work on externally controlled timeframes. It is in the PSIRT’s best interest to take a proactive approach with security researchers who are involved in studying areas that affect the PSIRT’s products, and to positively engage with those groups to have greater visibility into discovered issues.
Sub-Function 18.104.22.168 Engage with Security Vendors
Large commercial security vendors work with stakeholders during breaches and often will have forensic data to which the PSIRT may not normally have access. Developing relationships with these vendors helps build trust and mutual respect and can ideally help the PSIRT gain access to critical threat data that may otherwise not be available to them.
Sub-Function 22.214.171.124 Document Relevant Security Vendors
Knowing and properly engaging with security vendors can speed communications and efforts around vulnerability reporting/remediation as they report issues to the PSIRT. It is important to understand what these vendors will have access to and keep. The organization’s relationship to the bug bounty vendor should be thoroughly documented and vetted prior to entering into a relationship so that all involved parties understand how they should behave, what resources they can access, how data is shared, and with whom it is shared.
Sub-Function 126.96.36.199 Document Methods to Engage with Security Vendors
The PSIRT should research where the desired external groups engage in dialog and make efforts to participate in those forums.
Building a relationship with bug bounty vendors to enhance communication and data-sharing efforts around vulnerability management.
Sub-Function 188.8.131.52 Document and Define Relevant Bug Bounty Programs
Document and define bug-bounty vendors applicable to the offerings the organization provides.
Sub-Function 184.108.40.206 Engage Bug Bounty Vendors
Identify channels to engage these bug-bounty vendors in active dialogue.
CSIRTs are a special category of “downstream” stakeholders that are purely focused upon security concerns. While these groups can typically be interacted with via standard stakeholder engagement practices and customer management, the PSIRT should understand the unique requirements and perspective of these security-focused groups that will contact and consume information from the PSIRT. This includes disclosure formats and timelines (see Service 5.3 Disclosure), as well as communication channels for specific requests.
Two stakeholder groups that PSIRTs will interact with deserve additional attention. Sometimes referred to as “upstream” and “downstream”, community participation is essential to nurture joint remediation efforts or assist in mutual aid with others within the organization’s peer groups. “Upstream” is a term used for groups or individuals from which your organization sources components or projects for its products. “Downstream” refers to individuals, groups, or organizations that source your organization’s output as part of their offerings. Downstream engagement is covered in Service 1.4 Downstream Stakeholder Management.
A vibrant upstream community can help feed innovation into product streams as well as assist with the burden of complex vulnerability remediations, often compensating for the lack of crucial subject matter expertise within the organization. Likewise, cultivating professional relationships with individuals and teams from other organizations can help expand the capabilities of the PSIRT by allowing access to external perspectives, expertise, and historical knowledge. This can be achieved through proactively engaging the security community as a stakeholder and establishing relationships with partners and peer PSIRTs.
Often products will include code or components that were not created by the organization. The originators of these materials are sometimes called third parties, suppliers, or upstream vendors, original equipment manufacturers (OEMs) or simply partners. It is helpful to identify these partners within your ecosystem and determine how the organization would contact and engage them when vulnerabilities are discovered in the third-party’s code.
Sub-Function 220.127.116.11 Document and Define Upstream Communities and Partners
Upstream communities and partners provide code and/or knowledge and expertise that is incorporated into the organization’s offerings. It is critical to know and engage with these suppliers to ensure speedy and effective interactions as security vulnerabilities are reported to and worked on with the PSIRT. Ideally these relationships are documented in contracts, covered by non-disclosure agreements and other protections for the organization.
Sub-Function 18.104.22.168 Engage Communities and Partners
Each upstream community or partner may have different methods or tools they use to develop and communicate about their software/offerings. The PSIRT should understand how to engage with these external groups and ensure it has appropriate contacts/methods to collaborate on security issues involving those external parties.
Sub-Function 22.214.171.124 Participate in Upstream Communities
Participation with upstream communities and partners helps build valuable inter-group trust, as well as helping augment the capabilities of that external team with expertise the organization may have.
Sub-Function 126.96.36.199 Participate in Community and Industry Events
Conferences and professional organizational meetings are excellent places for PSIRTs to interact with stakeholders and partners, receiving direct feedback for the organization as well as building goodwill and a positive reputation amongst the external community that may be leveraged for future coordination/collaboration.
Sub-Function 188.8.131.52 Engage Community Security Teams
It is critical that the PSIRT understands how to contact upstream software/hardware/service providers’ security teams (PSIRT, CSIRT, security engineers), and whom to contact. Establishing lines of communication and rapport between the PSIRT and these groups helps ensure smooth interactions during times of crisis or vulnerability remediation.
“Downstream” has many connotations, but that does not mean that the PSIRT should ignore these vital stakeholder groups. “Downstream” refers to any product, organization, or individual that takes the PSIRT’s company’s products and offerings and uses them for their own purposes. This most frequently takes the form of customers or consumers of the goods and services offered, but this is not always the case. Often another company could use or license the PSIRT’s company’s products and resell them as an offering through this third party, or in the case of open-source software, where this commonly occurs, one group will provide and maintain software and a large group of ancillary parties will leverage those resources, also known as being downstream from the source.
Sub-Function 184.108.40.206 Document and Define Downstream Communities, Consumers, and Partners
Downstream communities and partners consume code, and/or, knowledge and expertise that is incorporated into the organization’s offerings. Ideally these relationships are documented in contracts, covered by non-disclosure agreements and other protections for the organization.
Sub-Function 220.127.116.11 Engage Downstream Communities
Each downstream community or partner may have different methods or tools they use to develop and communicate about their software/offerings. The PSIRT should understand how to engage with these external groups and ensure that the have appropriate contacts/methods to collaborate on security issues involving those external parties.
To engage your stakeholder base as a stakeholder, PSIRTs must establish processes and methods to interact with the stakeholder community around product security response. Stakeholders of the organization’s products are some of the most important to keep happy, as they represent current and future revenue opportunities for the organization.
Stakeholders of your products and services should have avenues to share information and opinions and obtain support how the organization handles security vulnerabilities. Proactively working with the organization’s stakeholders helps provide a positive brand experience and sustain/improve stakeholder loyalty.
Sub-Function 18.104.22.168 Provide Clear Lifecycle and Support Policies
The organization should clearly and publicly describe what the stakeholder’s expectations should be regarding the fixing of security vulnerabilities and for how long products are supported. Refer to Service Area 4 for more information.
Sub-Function 22.214.171.124 Stakeholder Engagement
Stakeholders of the organization’s products and services will have questions, require assistance, or need remediation of reported security flaws. The PSIRT should actively engage with stakeholder requests, provide clear and accurate guidance around security vulnerabilities, and provide risk mitigations until such time the remedy can be provided to the stakeholder.
A security incident touches on many internal groups and, possibly including products, within the organization. PSIRTs are a central point to coordinate vulnerability remediation efforts as well as serving as a hub for sharing information about an event to authorized internal stakeholders.
To engage effectively with stakeholders, the PSIRT must provide an assortment of communication channels. Different stakeholders may prefer certain outlets over others. The PSIRT should account for the widest possible audience as communications are crafted and released. The PSIRT also should be equipped to intake security reports, comments, and questions from a variety of sources.
Sub-Function 126.96.36.199 Provide Clear Communication Channels
Stakeholders should have avenues to submit questions, check the status of flaws, and report issues to the PSIRT. If a stakeholder is impacted by or discovers a security vulnerability, they should easily be able to make and send a report to the PSIRT.
Sub-Function 188.8.131.52.2 Provide Internal Communication Channels
To engage internal stakeholders, the PSIRT should provide communications channels to advertise the remediation status of vulnerabilities. Internal stakeholders should be able to easily contact the PSIRT and understand what to expect from inquiries.
Sub-Function 184.108.40.206.3 Provide External Communication Channels
To engage external stakeholders, the PSIRT should provide communications channels to advertise the remediation status of vulnerabilities. This would include vetting/qualifying activities around external communication to ensure their validity and that they are appropriately routed to internal associates.
Oftentimes, the PSIRT must handle information that is considered confidential (i.e., issues that are under embargo). The PSIRT needs to be able to securely and privately communicate with finders, other organizations, or with assorted internal resources. Abiding by the disclosure agreements and only communicating via private methods helps build confidence from finders. Protecting the confidential vulnerability information from unauthorized parties also helps ensure the issue can be appropriately and effectively managed, per the terms of the embargo. Secure channels can also help protect the identity of finders that do not wish to be revealed. A retention policy should be established to ensure data is properly disposed of after its use has ended.
Sub-Function 220.127.116.11 Provide Secure communications Channels
The PSIRT should ensure that vulnerability finders and partners working on vulnerabilities impacting the organization’s offerings have private and secure methods to share information.
The PSIRT should have access to system(s) of record for all product defects and be able to create and use a system for tracking and information-sharing around security vulnerabilities.
Sub-Function 18.104.22.168 Provide Security Flaw Defect Tracking for Products
Security defects should be tracked, and these systems should be accessible (within the least privilege model) with internal and external parties (if applicable) to update and track progress. External finders should receive adequate communication around the status of the reports they have filed with the PSIRT.
Sub-Function 22.214.171.124 Create and Publish Security Defect Tracking Process
The PSIRT should ensure that vulnerability finders and partners working on vulnerabilities impacting the organization’s offerings have private and secure methods to share information.
After an issue has been addressed, the PSIRT should make information available about what the security vulnerability is, what its severity and impacts are, what possible risks could be exploited, and how to resolve the issue or mitigate it until such a time that fixes can be made available.
Sub-Function 126.96.36.199 Provide Multiple Communication Outlets
Different stakeholders will prefer different methods of interaction/communication as vulnerabilities are disclosed to the public. The PSIRT should ensure that in addition to traditional advisory-style updates, other methods are used to ensure maximum engagement and awareness from stakeholders around the vulnerability. After vulnerabilities have been remediated, the PSIRT should use multiple methods to advertise the fix.
Sub-Function 188.8.131.52 Provide Feedback to Stakeholders
Feedback helps improve processes and response in the future. It can highlight areas the PSIRT is strong at and should continue performing in areas where the PSIRT needs to develop and improve further.
Acknowledging the finder helps establish their and their organization’s (as applicable) credibility within the community in addition to expressing appreciation for partnering with the PSIRT on the flaw.
Acknowledgement of the person(s) responsible for discovering a security vulnerability is a vital element within the security vulnerability workflow. The small expression of gratitude builds trust and respect within the community and shows that the organization is responsive to security concerns.
Sub-Function 184.108.40.206 Provide Acknowledgements
Written acknowledgement of a finder’s efforts and involvement in the discovery of a security vulnerability is the single most effective and inexpensive tool the PSIRT has to reward these individuals. It is traditional to include acknowledgement of the finder(s) in security advisories, software release notes, and CVE text. The PSIRT will need to understand how internal attribution of found vulnerabilities will be communicated.
To generate positive outcomes for stakeholders and encourage further sharing of research, the PSIRT can elect to develop a program to reward or incent this collaboration in the hopes that it will continue and expand in the future.
Sub-Function 220.127.116.11 Create Finder Rewards Program
The PSIRT can sponsor a rewards program designed to encourage positive behavior in security finders. Rewards could be monetary, promotional merchandise, or any number of things a security finder might value above their acknowledgement in discovering the issue.
Sub-Function 18.104.22.168 Start a Monetary Bug Bounty
One form of a reward can be monetary compensation. Some organizations will pay finders that disclose vulnerability information to them.
Sub-function 22.214.171.124 Start a “Points Board”
Another form of compensation is a “Points Board”. This gamifies finding and reporting security vulnerabilities and encourages friendly competition by promoting “leaders” and providing rankings for finders to brag over.
Providing details around PSIRT volume, performance, or other measurements is critical in keeping stakeholders aware of the effectiveness of the PSIRT. Different stakeholders will have unique viewpoints that must be addressed with potentially differently formatted artifacts (or views). The PSIRT must understand how each stakeholder group desires to consume this information. These metrics could be Key Performance Indicators (KPIs) for the PSIRT. Function 2.5.1 speaks to Operational Reports and how the PSIRT should consider providing such reports ensure smooth operations. Function 2.5.2 reviews Business Reports that the PSIRT can consider providing to stakeholders.
The first step to effectively articulating how a PSIRT is delivering services is to understand the unique viewpoints of each stakeholder group. Some stakeholders may be concerned about timeliness of security patches, while others may be focused on financial dimensions of the PSIRT’s operation. Each viewpoint is valid and requires different artifacts to effectively communicate the desired information. Each stakeholder group should be polled to understand what aspects of the PSIRT they require data on, and the best method to share that information.
Sub-Function 126.96.36.199 Gather Stakeholder Metrics Requirements
Stakeholders will be concerned with a specific set of data that other stakeholders may not be. For example, such metrics could be around performance of the extended patch remediation team, costs, and quality.
The processes and actions required to document the requested metrics for all stakeholder groups. Wherever possible, the tooling the PSIRT uses should be able to collect and provide information about the PSIRT’s processes and performance. Ideally metrics should be stored in a centralized location (a database, spreadsheet, or other tool) so that historic performance can be periodically reviewed, and so that the differing stakeholder views can be easily addressed with minimal additional labor.
Sub-Function 188.8.131.52 Gather Stakeholder Metrics
The PSIRT should create processes and methods to collect the required metrics at the prescribed intervals (SLAs/OLAs).
Sub-Function 184.108.40.206 Store Stakeholder Metrics
The PSIRT will need to conduct historical analysis on performance and other trends, so it is useful to develop a repository for this data so that it can continue to be leveraged in the future.
Data without context is meaningless. Incorrect assumptions can be inferred, and services may not be adjusted to meet changing business or stakeholder demands. Once the PSIRT has collected the required data, effort must be taken in reviewing that data and providing necessary context around what that data means to the stakeholder.
Sub-Function 220.127.116.11 Analyze and Review Metric Data
The PSIRT should spend time and effort to review collected data and provide context along with the metric reporting.
Sub-Function 18.104.22.168 Analyze Data trends and Historic Performance
As historic data is gathered, unique trends or chronic issues may be identified that the PSIRT or its partners can address.
Sub-Function 22.214.171.124 Provide Data Context
Provide context to data so that stakeholders can appropriately understand what will be provided to them and offer a possibility to address questions or concerns.
After metrics data has been collected and analyzed, it must be delivered to stakeholders in an agreed-upon format. This format can be referred to as an artifact, or as a view to address a stakeholder viewpoint. These artifacts could take the form of a web page, an email, a more formalized report, or another method.
Sub-Function 126.96.36.199 Provide Stakeholder with Metric Artifacts
Each stakeholder has a unique viewpoint they represent. Each viewpoint needs to be addressed with a view of the data in the form of some reporting artifact. These artifacts may need to be adjusted to match differing viewpoints. Artifacts could include reports emailed or posted to a web page, dynamic web portal, executive briefs, charts, graphs, or any number of other data delivery mechanisms.
Sub-Function 188.8.131.52 Review Metrics and Lesson Learned
One of the PSIRT’s most important goals should be to constantly improve the process of vulnerability management. Reviewing performance metrics and stakeholder feedback helps the PSIRT identify areas to focus on or improve.
This service area describes the services and functions a PSIRT may perform to discover potential vulnerabilities. Operation of this service area will trigger the vulnerability handling process described in other sections of this document. Maturity of a PSIRT may be measured via the availability and efficiency of the difference services prescribed in this service area.
For a PSIRT the main scenario is the intake of reports affecting a stakeholder’s product. A key element for the intake of vulnerability reports is to set up and maintain the required infrastructure, define and advertise contact points, and define and maintain readiness.
PSIRTs must create awareness of their existence and be available to external parties or internal escalation paths. A clear and defined communication channel may help finders, partners, or stakeholders report a vulnerability to PSIRTs.
Sub-Function 184.108.40.206 Define Preferred Form Report Submission
Expect to receive vulnerabilities through various channels and of variable quality. It is still helpful to define the best way to process a report. This can be a web form, a public ticketing system, an email address, a support hotline, or any other means of submission.
Sub-Function 220.127.116.11 Publish Contact Details
The preferred contact information for the PSIRT should appear in product documentation, advertised on the company’s web page, indexed in search engines, registered in major CSIRT/PSIRT lists, and communicated to Common Vulnerability Enumeration (CVE) issuing entities such as CVE Numbering Authorities (CNA) and announced in security communities.
Sub-Function 18.104.22.168 Register Common Points of Contact
It is helpful to reserve common terms related to PSIRT such as “psirt@”, “incidents@” or “security@” within your company’s domain name. Such reservation will help to direct relevant PSIRT communication to you.
Sub-Function 22.214.171.124 Connect the PSIRT within the Company
Ascertain that stakeholder service (for stakeholder request or vulnerability reports), the communications department (for media requests), and your product development teams (for escalating critical internal findings) are aware of the PSIRT and know how to contact it.
Sub-Function 126.96.36.199 Define and Maintain Readiness
Depending on the industry and the requirements set forth by the stakeholders, establish on-call or follow-the-sun duty to maintain the necessary readiness to respond to critical reports.
Sub-Function 188.8.131.52 Prepare for Encrypted Submissions
Vulnerability reports often contain sensitive information about the operational environment and products in which the vulnerability was observed. To avoid accidental information leakage or disclosure, promote means to submit reports in an encrypted manner, such as S/MIME or PGP protected emails or an HTTPS-enabled web form.
Vulnerability reports are received from diverse sources and in various forms. Regular monitoring of incoming communication channels and timely response to incoming reports is crucial. Response times to external finders should be defined in an SLA, internal to the company.
Sub-Function 184.108.40.206 Monitor Communication Channels
Check the advertised means of contacting the PSIRT regularly, as well as other available channels such as general-purpose email inboxes or company social media accounts.
Sub-Function 220.127.116.11 Process Reports in Isolation
Vulnerability reports will be investigated by the PSIRT, which is therefore easy to target through a malicious submission. Prepare policies and technical procedures to protect the working environment from such attempts by providing means to securely process vulnerability reports.
Sub-Function 18.104.22.168 Timely Acknowledgement of Reports
The detailed analysis of the report is often complex and time-consuming, but mere acknowledgement of the report can be quickly achieved. Prompt reaction shows the report is taken seriously and greatly helps create a relationship of trust. Subsequent communication throughout the handling process can be built upon this first engagement and it shows the PSIRT is committed to comprehensible resolution.
Vulnerabilities disclosed to the vendor directly or from reporting parties are straightforward to take in. However, it is important to realize there are additional vulnerabilities that may be disclosed via informal channels like news outlets, technical blogs, expert databases, social media or technical publications and conferences.
Publicly available exploit databases or commercial feeds should be monitored actively to discover potential zero-day vulnerabilities that require investigation. A fully functional exploit may lead to a company’s proactive communication with its stakeholders.
Relevant security conferences should be monitored to identify submissions of interest. Besides directly referencing products or brands, submissions might be discussing broader topics such as protocol flaws that may require work of a PSIRT. If the abstract raises questions, it is a good idea to engage with the finder at an early stage to clarify if action needs to be taken. In addition, presence on conference and pro-active engagement with authors can promote direct contact to the PSIRT for future research.
Pay attention to publication by finders who have a track record of relevant publications or extensive expertise with either the industry or specifically a company’s products and services. Their scientific works, blogs posts, or mailing list participation may hint on possible vulnerabilities or weaknesses that require attention.
Especially in cases of catastrophic incidents to stakeholder installations or personnel, the mass media is often first to pick up. Monitoring of mass media can help to detect situations where the stakeholders of the PSIRT are potentially an important or predominant supplier.
Vulnerabilities roughly fall into three categories: (1) vulnerabilities in a product’s very own source code, (2) vulnerabilities in product components maintained by vendor-internal sources, and (3) vulnerabilities in components provided by vendor-external sources (third-parties). From a product’s perspective, (2) and (3) are external components, but vulnerabilities in these components can ultimately impact the superseding product. Although a product owner has only indirect control over the remediation of the underlying issue, the stakeholder sees some degree of ownership over the supply chain and the remediation of the vulnerability with regard to the affected product. This is especially the case when the vulnerable component cannot be remediated independently from the including product. Included open-source components are also considered third-party components.
Keep a list of vendors, products, and versions provided by external and internal parties that are included in products. This is essential to quickly identify affected products for inherited vulnerabilities.
Obtain timely information about vulnerabilities in third-party components by subscribing to vendor advisories or establishing specific communication channels to suppliers. Subscribe to security mailing lists for open source projects. This can be supported by the use of vulnerability information providers.
It might not always be possible to subscribe to vendor advisories for third-party components. This is when the vendor does not publish advisories, the vendor went out of business or the open source community around the component is not proactive. Resources such as the National Vulnerability Database (NVD) or commercial intelligence sources can help identify vulnerabilities that have not been advised.
Product components from vendor-internal sources will in most cases not issue public advisories on resolved security issues. In order to obtain information about vulnerabilities in the vendor-internal supply chain, set-up specific communication channels with such suppliers.
Establish automated channels to distribute identified third-party vulnerability notifications directly to the development teams of affected products. Often it is sufficient to follow the instructions of the upstream vendor to fix the issue in the downstream product. In accordance with the prioritization policy, define when vulnerabilities should be triaged differently and escalated to PSIRT handling. The latter is especially important if a stakeholder needs to take action to obtain a fixed version of the product in order to secure operation.
A PSIRT may actively engage in internal discovery of new vulnerabilities as an opportunity to address security issues with products to lessen management of external relations and potentially reduce overall coordination effort. Such activities should complement security verification activities that are part of the SDL. PSIRT activities may include product-security assessments prior to product release or in the maintenance phase, as well as providing security testing tool expertise to Research & Development. Internally found vulnerabilities impacting end users should be treated in the same manner as externally found vulnerabilities, including scoring and reporting, coordinated with the fix publication.
Product security assessment is the practice of actively seeking to discover currently unknown vulnerabilities. This can include a wide range of techniques and tools such as penetration testing or vulnerability scanners. These grey-box/black-box security assessment techniques simulate company-external hacking as they refer to a methodology where the attacker has little, or no knowledge of the system being attacked.
Sub-Function 22.214.171.124 Security Assessment of your Products
The analysis results of a security assessment challenging the security controls of your product can be of great help to developers looking to improve the posture of their product before it is released to the market or when preparing a remedy.
Sub-Function 126.96.36.199 Security Assessment of Third-Party Components
For components that are obtained from third parties it is recommended that there is a need for an increased dedicated security assessment, in addition to general procurement management procedures. This is especially needed for critical components to ensure high quality due diligence.
Both commercial entities and communities are constantly developing new security analysis and offensive tools. The PSIRT should maintain up-to-date knowledge of available tools. This is useful for conducting assessments of products, validating findings from external finders, or directing development teams choosing the right tools for their internal tests.
Sub-Function 188.8.131.52 Training of PSIRT Staff on Security Testing Tools
Training of staff is a key element in maintaining up-to-date knowledge of available security testing tools. Service 6.3 Secure Validation elaborates on PSIRT staff training in more detail.
Figure 8: Vulnerability Discovery Metrics Process
Providing details around PSIRT volume, performance, or other measurements is critical to keep stakeholders aware of the effectiveness of the PSIRT (also see Operational Foundation Section III: Evaluation and Improvements). Different stakeholders will have unique viewpoints that must be addressed with potentially differently formatted artifacts (or views). The PSIRT must understand how each stakeholder group desires to consume this information. These metrics could be KPIs for the PSIRT.
Operational reports provide information on the volume as well as the types of vulnerabilities being discovered. These reports may be published on a regular basis internally within the PSIRT as well as with internal stakeholders.
Sub-function 184.108.40.206 Total of Discovered Vulnerabilities vs. Confirmed
This data helps capture the volume that a PSIRT handles from a resource perspective. This data may be broken down to business unit level, product type, or specific products.
Sub-function 220.127.116.11 Total of Confirmed Vulnerabilities Broken Down by Third-Party Component
This data helps capture the risk associated with embedded specific third-party components.
Sub-function 18.104.22.168 Total Confirmed Vulnerabilities Broken Down by CWE
This data can be fed upstream to the Security Development Lifecycle and impact Training & Education. This data may be broken down to business unit level, product type, or specific products.
Sub-function 22.214.171.124 Total Discovered Vulnerabilities Broken Down by Vulnerability Discovery Approach
This data helps identify easy-to-spot vulnerabilities and can be fed upstream to the Security Development Lifecycle. This data may be broken down to business unit level, product type, or specific products.
Sub-function 126.96.36.199 Total Discovered Vulnerabilities broken Down by Source
This data helps describe how well-known the PSIRT is.
Business reports provide information on the vulnerability response health of an organization as it relates to handling and responding to security vulnerabilities.
Sub-function 188.8.131.52 On Time Response Rate This data captures how well the PSIRT is doing in time for initial response to vulnerability reports within the respective SLA timeframes.
Sub-function 184.108.40.206 Total Down Time of PSIRT Communication Channels
This data captures whether the PSIRT communication channels were available as defined in the SLA.
Sub-function 220.127.116.11 Time to Triage Rate
This measures the time from initial report intake to completion of triage activities. This data captures the performance and/or workload of PSIRT staff.
Sub-function 18.104.22.168 Number of Full Disclosures, Vulnerabilities Exploited in the Wild, And Vulnerabilities Identified Through Media
This data captures the risk to a stakeholder’s products.
Vulnerability intake and triage comprise the case management function of a PSIRT. While the order of operations is very similar among PSIRTs, there are variations, such as the exact point when a “case” is created or the personnel performing different functions within a case. Where organizations receive a high volume of vulnerability reports, they may consider performing initial triage to validate reports before cases are created. In contrast, in organizations where the volume of vulnerability reports is low, a case may be made created before triage. The ultimate goal among PSIRT is to create an efficient and defined process.
Organizations define appropriate qualification criteria to the type and scope of issues they are willing to address. Such qualification criteria will help set the security baseline and help with triaging incoming vulnerability reports effectively.
Figure 9: Vulnerability Qualification Process
Quality gate and bug bars are used to establish minimum acceptable levels of security quality, and prioritization criteria for security vulnerabilities. Defining these criteria before products are released provides transparency to the vulnerability handling process by pre-determining what the PSIRT will qualify as a product vulnerability that should be remediated.
Sub-Function 22.214.171.124 Document Product Security Vulnerability Definitions
The quality gate or bug bar should be documented, stored in a central location, and be part of the standard training for developers/engineers.
Sub-Function 126.96.36.199 Engage with Product Development Teams In the event that there are multiple products and product development teams within an organization, engaging across all of them to standardize the definition of a product security vulnerability is critically important.
A mature PSIRT should adopt the mindset of continuous improvement to revise its qualification criteria where appropriate to reflect past experience, industry best practices, product changes, and stakeholder feedback. It is important to communicate changes to internal and external stakeholders to manage their expectations.
Sub-Function 188.8.131.52 Collect Data
Collect data on the triage process including number of incoming reports, how many qualify as a vulnerability, how many do not qualify, and any discrepancies encountered.
As an organization’s PSIRT matures, the team may notice a group of habitual finders responsible for reporting an above-normal volume of vulnerabilities. It is recommended to consider the finder’s reputation and historical high quality of submissions, that some functions be bypassed such as qualification and triage to move directly to root cause analysis and remediation development. This may help to improve process efficiency and foster finder relationships.
Develop and maintain a database of individuals and organizations who have reported vulnerabilities to you in order to track history, outcomes, and any other case handling considerations for that finder.
Some finders may be prolific or consistent (vetted/credibility) in finding and reporting software bugs in your products or services. For example, they may use custom fuzzing tools and report crashes without a specific write-up or proof of concept. When the finder is well known to you and you have determined that the majority of the issues they report will be fixed, consider skipping the qualification/vetting process altogether and moving straight to remediation.
Consider building profiles on finders to inform handlers how to best work with them. Profiles might contain things such as geographic location, languages spoken, conferences they have presented at, methodologies used to find vulnerabilities, products/technologies they typically focus on, if they practice coordinated vulnerability disclosure, if they like to present their findings at conferences, if you pay them bounties or have you offered other incentives, etc. Consult with legal and/or compliance teams to determine what information can be collected and how long it can be kept.
Organizations may want to consider defining and publishing guidelines for what constitutes a minimum quality vulnerability report in order to provide finders guidance on the type of information you need to quickly assess their report. A baseline might include, but not be limited to, a write up, reproduction steps, platform(s) tested on, and proof of concept.
Beyond qualification, unless otherwise specified, PSIRT needs to ensure the finder’s report is reproducible in order to validate and understand the conditions which lead to the vulnerable state.
Figure 10: Vulnerability Verification/Reproduction
A PSIRT may not have sufficient technical expertise to reproduce all incoming vulnerabilities. PSIRTs may need to consult, work with, or rely on expertise within product development or other teams so it is important to have a clearly aligned and defined agreement to ensure the needed expertise is readily available. Ideally a dedicated full or part-time resource is recommended. However, if due to budget constraints this is not possible, at a minimum, subject matter experts should be pre-identified as part of the PSIRT process who can serve on short notice for limited periods of time in case of an incident.
A dedicated test environment should be set up for PSIRT or dedicated team to reproduce the vulnerability. The test environment should be isolated, to avoid malicious activities and in validating a finder’s report. Where appropriate, a dedicated network environment, simulations, or virtualization can be used to create a safe environment.
Teams engaged in reproducing reported vulnerabilities need to have tools and updated product licenses at their disposal to perform these operations (e.g. a debugger).
It is recommended that sensitive information, such as vulnerability reports, proof of concepts files, etc. should be stored securely and access limited to only those who need it and ensure security of the information at rest and in transit. For example, see ISO 27001.
During reproduction, the team doing the analysis should work to determine which products are impacted and if any variants of the vulnerability exist. See also Product Lifecycle Management section 4.1.1.
This service area captures the different services required to deliver and announce a remedy to both stakeholders and downstream vendors. The delivery mechanism for a remediation should be determined based on the impact of the vulnerability to stakeholders when exploited. Processes should be established to ensure that a remedy is delivered on a predictable schedule so both stakeholders and downstream vendors can plan accordingly for the test and deployment of these remedies.
Figure 11: Example of a Core Remedy Release Process
This service focuses on providing guidance around how the vendor plans to establish a cadence for releasing a remedy for supported product versions in the market. Stakeholders, especially in the enterprise space, need to plan for the deployment of a remedy. Some deployments, like cloud, may have automatic updates or a different patch management policy.
Figure 12: Setting the Foundation for Consistency
Companies may have different support policies and agreements with stakeholders. Based on these factors, a PSIRT may partner with business units/lines of business and stakeholder support to determine how and if, they will support products which have fallen out of the support scope or support obligations. This could depend on the severity of the vulnerability and may involve inputs from business units/lines of business and stakeholder support.
Sub-Function 184.108.40.206 Product Inventory
Build a product inventory of all the products released to market to ensure all supported applicable products are assessed and remediated.
Sub-Function 220.127.116.11 Support Models
Understand the different types of product support models including paid services, extended warranties, maintenance agreements or contracts with specific stakeholders.
Sub-Function 18.104.22.168 Product Lifecycle
Identify at what point a product is no longer supported within the product lifecycle.
PSIRTs may partner with product teams and stakeholder support to identify the different options for delivering a remedy to stakeholders. The criteria for determining when to deploy a remedy through the means identified should also be developed.
Sub-function 22.214.171.124 Product Packaging Formats
Understand the different packaging formats relevant to delivering a remedy (e.g. binary executable, source code diffs, etc.).
Sub-function 126.96.36.199 Delivering a Remedy
Understand the different mechanisms for delivering a remedy such as hot fix, patch, maintenance releases, firmware updates, and how to distribute a remedy.
Sub-function 188.8.131.52 Deploying a Remedy
Identify across the different products how a remedy can be deployed, i.e. remotely, customer installable, automatic updates or requirements onsite.
Stakeholders and downstream vendors need to plan for a remedy so that they can maintain the security posture of their environment. By setting a cadence for when a remedy will be delivered, this will enable stakeholders to schedule and plan resources for the necessary updates to their environments.
Sub-function 184.108.40.206 Remedy Delivery Cadence
Partner with product management teams and release management to determine the cadence for when a remedy should be delivered. Some remedies are integrated as part of a feature release and will be aligned to those release schedules, while others may require an emergency fix which is considered an out-of-band release.
Sub-function 220.127.116.11 Document Exceptions
Identify and document the exceptions for when a remedy would not be delivered through the normal cadence.
This service relates to the management of reported vulnerabilities by finders and includes the response analysis as well as mitigation. It also defines which versions will be remediated and may take into consideration how the remedy will be delivered. It may also consider any workarounds that can be immediately applied by the stakeholder prior to the remedy being delivered.
Figure 13: Remediation Process for the Reported Vulnerability
The impacted product may include a single software application, firmware or multiple hardware programs with different versions of software or firmware. A number of parameters need to be considered when crafting a remediation plan to ensure that your stakeholder needs are met.
Sub-Function 18.104.22.168 Validate Vulnerability
Validate the vulnerability report or incident against the quality gate or bug bar. See Function 3.1.1 Quality Gate and Bug Bars.
Sub-function 22.214.171.124 Remediate Product Versions
Identify affected products and versions as well as any variants that may need to be remediated at the same time.
Sub-function 126.96.36.199 Review Support Agreements
Review the support agreements and models associated with affected product versions. Refer to Sub-Function 188.8.131.52 Support Models.
Sub-function 184.108.40.206 Root Cause Analysis
Understand the design or implementation flaw that caused the vulnerability.
Sub-function 220.127.116.11 Determine the Mechanism for Rejecting a Vulnerability
For example, a vulnerability may be a false positive or security design flaw.
Sub-function 18.104.22.168 Remediation Analysis
Determine the means to mitigate or remediate the risks created as a result of the vulnerability.
Sub-function 22.214.171.124 Remedy Workarounds
Identify if there are any workarounds that can be implemented to mitigate the vulnerability while a remedy is under development.
Sub-function 126.96.36.199 Exceptions
Identify any exceptions where a vulnerability cannot be remediated. Refer to Function 4.2.4 Risk Management Process.
Prior to releasing a remedy for a reported vulnerability, it should be validated by quality assurance (QA), security testing and if applicable, the finder who reported the vulnerability. This describes the process and mechanisms for internally validating the remedy as well as partnering with the finder to validate and sign off on the remedy.
Sub-function 188.8.131.52 Validate Reported Vulnerabilities that have been Remediated
Validate to ensure all instances of the reported vulnerability have been remediated across all affected product versions.
Sub-function 184.108.40.206 Remedy Signoff
Obtain signoff of the remedy by the responsible QA engineer or team. Remedy validation should be integrated into the standard testing/QA practice.
Sub-function 220.127.116.11 Validate Remedy with Finders
Partner with third-party finder or stakeholder to validate the remedy.
As part of releasing a remedy for a reported vulnerability, the disclosure timeframes may vary depending on your organization’s business requirements. For example, some disclosures may align to when the remedies are available, others may align the disclosure to come after the remedies have been released, particularly if the remedies have been staged, or in some cases the disclosures may be prioritized based on stakeholder relationships (for example partners or critical entities). Regardless, key stakeholders across the industry including the finder need to be kept informed of the timeframes.
Sub-function 18.104.22.168 Disclosure Type
Determine the preferred mechanism for disclosing the vulnerability. This may be based on severity or type of vulnerability.
Sub-function 22.214.171.124 Coordinate the disclosure, if applicable.
Sub-function 126.96.36.199 Post Remedy to Internal Database
Partner with stakeholder support or other stakeholders to post the remedy to the web portal, stakeholder support site or release to manufacturing (RTM) as examples.
Sub-function 188.8.131.52 Release Remedy Disclosure
Partner with stakeholder support or stakeholders to release the disclosure of the reported vulnerability.
It is a PSIRT responsibility to provide stakeholders with sufficient information so they are able to evaluate the risks to their systems resulting from vulnerabilities in their system and in the products the PSIRT organization supports. Risk management assessments should be conducted across the organization when a vulnerability is not remediated within a specific timeframe (per Service Level Agreements or Objectives). This includes having a transparent mechanism to quantify the risk as well as escalating up to the appropriate stakeholders included in the organization’s Risk Register.
Sub-function 184.108.40.206 Authoritative Roles
Identify which roles have the authority to accept the risk, for example Chief Information Security Office (CISO)/Chief Security Office (CSO) or Risk Manager, and which roles should be informed of the risk.
Sub-function 220.127.116.11 Define Risk Management Process
Define risk management practices for handling and responding to risks within the organization, including the set of conditions which would trigger the process.
Sub-function 18.104.22.168 Assess and Quantify Risk
Assess and quantify risks by conducting an assessment of the risks to understand the threat and impacts to the business.
Sub-function 22.214.171.124 Document the Risk in the Risk Register
Assist the CSO, Risk Manager, or other stakeholders in tracking both status of risk evaluation and subsequently implementing recommendations.
Sub-function 126.96.36.199 Recommendations
Update risk register with the findings and recommendations.
The PSIRT needs to have a mechanism to expedite remediation time to address “critical vulnerabilities”, which can be defined as active exploits in the wild, zero day, and non-coordinated public disclosures. This service provides guidance for the incident, as well as alerting stakeholders and coordinating activities associated with the response, mitigation, and recovery of an incident to reduce the time from report to delivery of the remedy.
Figure 14: Incident Handling
When incident management is required, establish a situation room including PSIRT, Legal, Communications, Development, Stakeholder Support, Supplier, and other roles as needed. This can be a physical or virtual location, as long as all parties are available to respond as needed in a secure manner. Typically, both physical and remote options are necessary to ensure stakeholder attendance. Resources should be identified ahead of time in order to adequately support the incident management process.
Sub-function 188.8.131.52 Incident Management Plan
Develop a plan to manage critical vulnerabilities and develop the ability to mobilize all the resources required to address them. It is important that incident response readiness is conducted to validate the preparedness of this plan to handle unexpected events and emergencies.
Sub-function 184.108.40.206 Identify Resources Required to Handle and Manage the Incident
Resources may include meeting rooms, private lines and additional manpower. For long-term incident handling, food and accommodations should be considered.
Sub-function 220.127.116.11 Involve Stakeholders in Incident Response Plan
Identify all key stakeholders required to participate in handling the incident as part of your incident response plan. See Service 1.1 Internal Stakeholder Management and Service 1.5 Incident Communications.
Sub-function 18.104.22.168 Assign Clear Roles and Responsibilities to Manage the Incident
Personnel must know their roles and order of operations when a response is needed. Training and tabletop exercises should be conducted to prepare key response participants.
When an incident is declared, the main focus of the PSIRT in partnership with their stakeholders is to reduce the impact of the incident and work to restore the business function of a product as well as their stakeholders.
Sub-function 22.214.171.124 Information Collection
Intake, cataloging, and storage of information related to the incident.
Sub-function 126.96.36.199 Analysis
Incident handling is dependent upon analysis activities, which are defined in the “Analysis” section.
Sub-function 188.8.131.52 Response
Services related to reducing the impact of an incident and working to restore business functions within the constituency.
Sub-function 184.108.40.206 Incident Tracking
Documenting information about actions taken to resolve an incident, including critical information collected, analysis performed, remediation and mitigation steps taken, closure, and resolution.
Sub-function 220.127.116.11 Incident Postmortem Process
Action to review to identify improvements to processes, policies, procedures, resources, and tools to help mitigate and prevent future compromise.
All stakeholders and action owners must know the latest plans and progress to keep on track. Engage management as needed to break down any barriers that may impede open collaborative communication during an incident.
Sub-function 18.104.22.168 Publication of Information to Internal Stakeholders
Management of lists used to distribute announcements, alerts, data feeds or other publications for situational awareness.
Sub-function 22.214.171.124 Public Relations are Well Managed and Coordinated
Ensure information is disseminated to the media and stakeholders, but only through authorized organizational channels. This includes social media postings.
Sub-function 126.96.36.199 Communicate Recovery Activities
Recovery activities are communicated to internal stakeholders, executives, and management teams.
Sub-function 188.8.131.52 Collect Incident Postmortem Feedback
Incident postmortem briefings are conducted by PSIRT, and feedback is collected to improve incident response as well as Security Development Library (SDL) activities (for example, what SDL activity could/should have prevented the issue in the first place?).
Data to be collected should include, but may not be limited to, issue volume, classification, fix time, affected products or services.
Figure 15: Operational and Business Metrics
Operational reports provide information on the volume as well as the types of vulnerabilities being reported and confirmed across the different products and versions. These reports should be published on a regular basis internally within the PSIRT, as well as with internal stakeholders.
Sub-function 184.108.40.206 Total Number of Vulnerabilities Reported vs. Confirmed (by Product/Business Units)
This data helps capture the volume that a PSIRT handles from a resource perspective.
Sub-function 220.127.116.11 Total Confirmed Vulnerabilities Broken Down by Third-Party Component
This data helps capture the risk associated with embedded specific third-party components.
Sub-function 18.104.22.168 Total Confirmed Vulnerabilities Broken Down by CWE (by product/BU)
This data can be fed upstream to the Security Development Lifecycle and impact Training and Education.
Business reports provide information on the health of the vulnerability response capability of an organization.
Sub-function 22.214.171.124 On-Time Impact Assessment
This metric captures how well product teams are doing in completing impact assessments within the respective impact assessment SLA timeframes.
Sub-function 126.96.36.199 On-Time Fix Plan
This metric captures how well product teams are doing in providing a fix plan within the specified SLA.
Sub-function 188.8.131.52 Remediation Tracking
This metric captures how well product teams are doing in providing a fix within the specified SLA timeframes.
Sub-function 184.108.40.206 On-Time Remediation Rate
This metric captures how well product teams are doing in meeting the overall objectives or agreements for delivering a fix from time of report to delivery of a fix. This can be broken down by severity or by vulnerability type (product line, type of vulnerability).
Sub-function 220.127.116.11 Number of Incidents
This data captures the risk to the organization.
It is important to create a transparent and collaborative environment where vendors, coordinators, and finders can share information with their stakeholders and each other and negotiate mutually agreeable disclosure plans. By partnering in this way, the primary needs of resolving vulnerabilities, protecting stakeholders, and acknowledging finders can be achieved. The vendor should publish their vulnerability disclosure policy so it can be referenced by coordinators, and by other vendors, as well as finders.
Figure 16: Vulnerability Notification Process
Figure 17: High-level example of Vulnerability Coordination
This service involves determining the appropriate notification process to provide timely information about mitigation strategy, remedies, and workarounds to stakeholders so they are kept informed and can plan accordingly. In some cases, contractual agreements may exist between vendors – for example, an upstream vendor would be required to notify a downstream vendor of disclosed vulnerabilities or known incidents. The intent of the notification process is to ensure that all stakeholders and vendors can understand and manage the risk imposed by the vulnerability.
An intermediate vendor such as an OEM or partner may develop and/or produce a part, subsystem or software that is used in another vendor’s end product. In such instances, their PSIRT should make arrangements to share vulnerability information with their vendors. They should be aware of the vulnerability-handling policy of the different vendors. Sometimes these expectations are captured in a contractual agreement. The timeline for remediation and disclosure should be negotiated as soon as possible.
Sub-function 18.104.22.168 PSIRT Reporting to Intermediate Vendors
PSIRTs may learn of vulnerabilities reported by their stakeholders and should notify the intermediate vendor PSIRT of those vulnerabilities.
Sub-function 22.214.171.124 Intermediate Vendor Reporting
An intermediate vendor who supplies components or tools to a vendor may learn of vulnerabilities reported directly to them, and should notify their vendor PSIRTs.
Sub-function 126.96.36.199 Contractual Agreements
PSIRTs should identify all of their intermediate vendors and consider partnering with legal to ensure that clauses are added to contractual agreements to ensure a timely response on vulnerabilities.
Sub-function 188.8.131.52 PSIRT Notification to Stakeholders
Vendor PSIRTs may inform their stakeholders, especially if the intermediate vendor is not able to or takes a considerable time in remediating the vulnerability. In some cases, a vendor PSIRT may apply a tiered-notification process and notify those stakeholders that would be impacted the most by the given vulnerability.
A coordinator may be asked by a PSIRT to participate in notifying other vendors, as well as coordinating the timing of remedy for their advisories, especially if multiple vendors are involved. Coordinators such as the CERT Coordination Center (CERT/CC)[^12] or third-party coordinators provide value by arranging for a multitude of different organizations to partner and collaborate on addressing a vulnerability.
Sub-function 184.108.40.206 Coordinator Identification
Document and understand the different coordinators based on vulnerability disclosure policy.
Sub-function 220.127.116.11 Coordinator Engagement
Partner with a coordinator to ensure all affected vendor PSIRTs have been notified.
A finder such as a customer or third-party researcher may notify a PSIRT of a vulnerability through the channels documented in Service Area 2 Vulnerability Discovery.
Where appropriate, a vendor PSIRT should make arrangements to share vulnerability information with coordinators or other vendors. They should be aware of the vulnerability handling policy of the vendor. Timelines for remediation and disclosure should be negotiated as soon as possible.
Figure 18: Bilateral Coordination
A vendor PSIRT is responsible for maintaining communication with finders who report potential vulnerabilities. It is important for vendors to understand the finder’s intent, agenda, and stance on vulnerabilities in general, in order to promote and facilitate a coordinated disclosure on an agreed timeline. PSIRTs should consider acknowledging the finders who adhere to the public disclosure.
Sub-function 18.104.22.168 Report Receipt
Acknowledge receipt of vulnerability report from third-party finder.
Sub-function 22.214.171.124 Regular Updates
Provide the finder with regular updates on the status of the reported vulnerability.
Sub-function 126.96.36.199 Validation by Finder
Provide the remedy to the finder so they can validate it as well.
Sub-function 188.8.131.52 Finder Acknowledgement
Provide credit by acknowledging the contributions of the finder who reported the vulnerability. Vendor should verify with the finder that the credit is acceptable.
Figure 19: Multi-Vendor Coordination
Where appropriate, a vendor PSIRT should make arrangements for sharing vulnerability information with coordinators or other vendors. They should be aware of the vulnerability handling policy of the vendor. Timelines for remediation and disclosure should be negotiated as soon as possible.
Table 1: Example of Multi-Party Coordination
Sub-function 184.108.40.206 Report Receipt
PSIRT vendor acknowledges receipt of the vulnerability report from vendor or coordinator.
Sub-function 220.127.116.11 Affected Vendor Identification
PSIRT vendor or coordinator may need to identify those vendors that are impacted by the vulnerability report.
Sub-function 18.104.22.168 Vulnerability Information Sharing
PSIRT vendor or coordinator shares vulnerability information across the different vendors.
Sub-function 22.214.171.124 Remedy Release Planning
PSIRT vendor or coordinator partners with vendors on the timing and availability of remediations, and how the downstream vendors may receive the remedy.
Sub-function 126.96.36.199 Remedy Validation
PSIRT vendor or coordinator validates with vendors that the security remediation addresses the vulnerability.
Sub-function 188.8.131.52 Disclosure Coordination
PSIRT vendor or coordinator negotiates across all vendors to agree on both how the vulnerability will be disclosed and the timing of when the disclosure will be publicly released.
When a security remediation is released, there should be appropriate disclosures to ensure that stakeholders and vendors are properly notified of the remedy. For each, the audience needs to be well defined (there may be different audiences for different types of notices).
Release notes, including readme and change history, should include CVE reference(s) for the remedy. Release notes should clearly communicate how the vulnerability was addressed.
Sub-function 184.108.40.206 Release Note Disclosure
Define what vulnerabilities should be disclosed in the release notes.
Sub-function 220.127.116.11 Release Note Review
Define the review process.
Sub-function 18.104.22.168 Release Note Approvals
Conduct review and approval of disclosure.
Vendors should have a mechanism by which to release security advisories to stakeholders on a public web page, and disclose vulnerabilities that have been remediated.
Sub-function 22.214.171.124 Advisory Template
Define a standardized security advisory template. Include title, summary, CVE(s), supported product impact and status, acknowledgement, references and revision history.
Sub-function 126.96.36.199 Advisory Delivery Method
Determine the mechanism to deliver the security advisory including, but not limited to web document, RSS feed, or subscription.
Sub-function 188.8.131.52 Advisory Formatting
In order for stakeholders and constituents to consume advisories using automation tools, consider publishing advisories in a machine-readable format such as the Common Security Advisory Framework ^13.
Sub-function 184.108.40.206 Advisory Triggers
Define the set of conditions which would trigger the release of a security advisory. For example, if action needs to be taken to notify stakeholders that a hosted environment has been remedied (breach scenario).
Sub-function 220.127.116.11 CVE Assignment
Determine the process for assigning a CVE ID to the vulnerability.
Sub-function 18.104.22.168 Finder Acknowledgement
Determine if the finder would like public acknowledgment or credit.
Sub-function 22.214.171.124 Disclosure Planning
Define review process such as who are the stakeholders and when should the disclosure be crafted.
Sub-function 126.96.36.199 Advisory Review Process
Conduct review process with defined stakeholders.
Vendor should have a mechanism to publish knowledge base articles which may accompany certain security remedies which are deemed a lower severity, or alternatively may be used as a means to communicate why specific reported vulnerabilities were rejected as false positives.
Sub-function 188.8.131.52 Knowledge Base Article Disclosure
Define what vulnerabilities should be disclosed in a knowledge base article.
Sub-function 184.108.40.206 Knowledge Base Article Review
Define the review process.
Sub-function 220.127.116.11 Knowledge Base Article Approvals
Conduct review and approval of disclosure.
In addition to executive business owners who should be notified of vulnerability communication plans, there are many employees who are on the front lines working with stakeholders face-to-face and over the phone every day. Providing advanced, confidential notification and FAQs for coming advisories prepares those who may be asked about them upon publishing.
Sub-function 18.104.22.168 Internal Stakeholder Engagement
Collaborate with internal stakeholders to craft and/or review language for their teams to use when customers ask about the vulnerability issue.
Data to be collected should include, but is not limited to, issue volume, classification, remediation timeline, affected products or services.
Figure 20: Vulnerability Metrics Process
Operational reports may provide additional information on the volume of disclosures posted as well as the number of page views. These reports should be published on a regular basis internally within the PSIRT as well as with internal stakeholders.
Sub-function 22.214.171.124 Number of Security Advisories Posted
The number of different disclosures can be reported and broken down by product. This may help drive the team to have a technical resource assigned.
Sub-function 126.96.36.199 Number of CVEs Posted to NVD
The number of CVEs assigned can be used to promote your status to a CVE Numbering Authority (CNA).
Sub-function 188.8.131.52 Page Views on Security Advisories
This may drive your strategy towards proactive notification if the volume of stakeholders viewing your advisory is low.
The world of product security is in a constant state of flux as new technologies, services, and integration make continuing training and education a top priority for security professionals. As software penetrates every aspect of the world we live in, from our cars to our refrigerators, keeping up with the needs of securing products has never been more critical. PSIRTs play a key role in supporting a strong curriculum in educating all stakeholders on the intricacies of developing, validating, and shipping products/services that meet the standards of today’s connected world.
Training and education needs can vary significantly across a corporation. The concerns of a firmware developer vs. a software services developer are very different and often require very specific and unique types of training. For the sake of this document we will break down training needs into four stakeholder groups: PSIRT, product development, product validation, and other stakeholders involved in the PSIRT process.
PSIRT training is unique since PSIRT members must be plugged into many aspects such as legal, communications, and development.
Product Development (Internal Engineering and Development): Developers require training in their specific areas and thus need training that is just as focused. Developing secure firmware that is very difficult to update in the field has very different requirements than that of a desktop application engineer.
Product Validation (Internal Engineering and Development): Validators require constant training to become familiar with the latest tools and techniques for aspects such as pen-testing, vulnerability scanning, and early design reviews to catch issues before they have to be fixed.
All other Stakeholders: This group represents a less technical audience that requires a solid foundation in understanding the basics of developing, validating, and shipping secure products, as well as in reacting when a shipped product has a vulnerability.
Secure development training is not considered part of PSIRT program and is handled outside the PSIRT process. However, it is important that PSIRTs be champions of all aspects involved in bringing secure products to market, and as such should partner with various development teams to make sure the appropriate training is in place. In many smaller organizations, there may not be a separate group that takes responsibility for making sure products are developed with a security focus. In those cases, PSIRT may be involved in bridging the gap (this is outside the scope of this document).
In each section, we will identify various stakeholder groups and summarize some focus areas that may help a PSIRT engage in meaningful discussions about training and educating their stakeholders. PSIRTs may create all the training material in-house, use external material, or use external training resources to train their stakeholders.
PSIRT staff need to be at the forefront of what is happening in the world of security including, but not limited to, what’s trending, new exploits, and industry activities. This broad level of knowledge starts with requiring a solid foundation in general security-world topics as demonstrated by the leading security certifications. But certifications only provide a base that needs to be constantly updated via activities like security-focused conferences, industry-consortium involvement, and keen awareness of the industry as a whole by being an avid consumer of blogs, industry press, consortium publications, etc. PSIRT members also need to be aware of the constantly evolving world of security and privacy legislation.
It is important that PSIRT staff have a solid understanding of basic security concepts and knowledge of the products that are being supported. The training material must be regularly reviewed to ensure that, as the security landscape changes, new vulnerability techniques are being included in the training material.
Security-concepts training would vary depending on the type of products that are being supported by a vendor (e.g. hardware, firmware, software, networking, cloud products, or all of the above). At a very high level, the training must cover basic security topics like common attacks, cryptography, confidentiality, integrity, availability, authentication, authorization, access control models, multi-tenancy, relevant compliance, and regulations among others. This training should also include any industry-specific regulations that may impact PSIRT activities, such as HIPAA for healthcare verticals and PCI DSS for payment-card vendors and banking. Some level of product training must also be covered for PSIRT staff so that they are able to understand the reported issues.
Since the external finders report issues to the PSIRT, it is important that PSIRT staff are trained on the communication policies and soft skills that cover how to handle communications with external finders and internal stakeholders in a timely manner.
There should be process guidelines that define how the reported issues will be tracked, managed and measured. The roles of the various stakeholders involved in the process of resolution of reported issues should be defined. The process should cover responding to finders in a timely fashion and sending periodic updates to them for all open issues. There should also be a well-defined and secure means of communicating information between an external finder and the vendor.
Sub-function 184.108.40.206 Bug Tracking and Other Management Tools for PSIRT and the Engineering Staff
A formally acknowledged bug tracking tool should be identified for each product (preferably the same for all products) in a given organization. All bugs should be identified in this tool, and security bugs should be uniformly identified as such. Only those that have a need to know should be able to view and access the information related to security vulnerabilities in a product. In addition, the tool should include the capability to support program metrics requirements with both manual and automated reporting capabilities.
Sub-function 220.127.116.11 Third-Party Tracking Tools
Most products include multiple third-party components (including open source) that are shipped with it. The customers will often not know about the third-party software shipped within the product and hence would rely on the vendor to provide fixes or information on the remedy. It is important that internal third-party tracking tools be identified to cover the dependencies of vendor’s products on various third-party components. The National Vulnerability Database (NVD), third-party vendors’ security advisories and other external sites must be monitored to track the vulnerabilities, and fixes for the third-party components so these fixes can be provided to the customer.
PSIRTs will need to track all the trainings that are available to various stakeholders. The team will need to ensure that all these trainings are delivered at a certain frequency as the security landscape changes very rapidly, and hence the trainings and processes will need to be continually redefined.
Secure Development refers to the methodologies and steps taken throughout the development process which are specifically designed to reduce the number and severity of vulnerabilities in software-related products and services. By having a strong curriculum and a focus on secure development methodologies, vulnerabilities can be greatly reduced prior to product release which is much less expensive than dealing with them after the products are already released to the marketplace.
Secure development starts with product requirements and architecture. In addition, secure design reviews are key to spotting possible vulnerabilities before the product even goes into development.
There are numerous activities that are involved in a secure development program, the details of which are well outside the scope of this document. It is strongly recommended that a separate program exists to manage an appropriate secure development lifecycle effort. This program should follow an accepted industry-standard program model. An example of a secure development lifecycle is the [^14]Microsoft Secure Development Lifecycle model.
Secure development training is not always considered as part of PSIRT constituency, and may be handled outside the PSIRT process. In any case, it is an important step that must be considered by any vendor concerned about the secure posture of its products.
Each member of the development process needs to comprehend why the PSIRT process exists, how it works, and what they need to do to develop products to support it. Often after a product is released, development teams move on to different projects and sustaining efforts are minimal. Training the teams and providing them with the appropriate methods to store key information about the product is critical for PSIRT to fully address a product vulnerability issue. Document information such as who was the security architect, the development lead, and the testing lead so that the PSIRT can go back to those who know the most to assess risks and develop mitigations. This documentation should also include aspects such as: what are the third-party components that are being used, what is the product update process, what logging exists, what security exceptions were allowed and how are stakeholders notified. This information is also critical to PSIRT to close a security vulnerability. As new development team members come and go, refresher training is also critical.
Validators need to be constantly updated on the latest tools and techniques for aspects such as pen-testing, vulnerability scanning, fuzzing, ethical hacking, and others. Training the validators on this falls under SDL and is outside the scope of this document. However, PSIRT should encourage the organization to have a group that focuses on this.
Just like secure development, secure validation training is not considered part of PSIRT constituency and is handled outside the PSIRT process. However, it is an equally important step that must be covered as part of SDL of a product by a vendor.
Some members of the validation team may be involved in testing the fixes that are required to fix product vulnerabilities. These team members need to understand the PSIRT process, how it works, what are the expected timeframes and what their role is in the process. They would need a good understanding of the product life cycle so they know the supported versions that need to be tested for vulnerability fixes. They would also need to test the workarounds, if there are any. It will be important for them to also test for regressions.
All stakeholders will require some level of training and awareness of the PSIRT program. There are many stakeholders that are involved in the end-to-end PSIRT process. Therefore, it is important to identify various stakeholder groups and develop training specific to their needs.
This group is typically involved in initial sign-off on the company’s communication, vulnerability protection, and other policies. Management approval may also be required for creating security advisories. Also, executive management’s approval is often required for critical situations that create high risk, are highly visible or create high liability. Management may also want periodic status checks on the security posture of all products. Thus, it is important to inform management of the PSIRT processes.
Legal is involved in setting up the initial corporate policies. Some finder-reported issues may have liability issues and may require assistance from legal groups, so it is important to identify points of contact beforehand.
Government affairs personnel are involved in regulatory compliance issues. Therefore, it is important to identify points of contact beforehand.
Marketing is often involved when there is a risk to brand name. Also, security advisories may be reviewed by them, and associated marketing information may be released alongside. Marketing teams are also involved in marketing the security aspect of products.
Public Relations (PR) teams may be responsible for responding to external security posts or blogs, or press inquiries related to critical product vulnerabilities. Points of contact should be identified so that PR can be involved if any external posting is required.
Sales teams may be trained on basic security concepts and communications regarding security practices. Also, it is very important for salespeople to know what can and cannot be shared externally. It is recommended that sales employees redirect any concerns about security from stakeholders/prospects to either PSIRT staff or support staff as opposed to addressing them directly.
Support teams must be trained to handle security vulnerability reports from the customer. PSIRT may be involved in some cases for working these issues. Support should publish policies that define the lifetime of every product, the versions supported and whether security advisories will be released. Most vendors only provide security advisories for the versions that are in support. So these policies are critical, and must be published on the vendor’s website to make it easily visible to stakeholders. PSIRTs typically maintain a close relationship with support so they understand the kind of issues that are being reported by customers. Sometimes a finder might also be a customer, so handling the issue may move between support and PSIRT.
Use information gained during the root cause analysis of the incident to educate those involved and prevent similar vulnerabilities in the future.
Table 2:Pros and Cons of PSIRT organizational models
National CSIRT (Computer Security Incident Response Team) - a national CSIRT refers to an entity which is constituted by a National Authority to provide national-level coordination of cybersecurity incidents. Its constituency generally includes all government departments and agencies, law enforcement, and civil society. It is also generally the authority to interact with the national CSIRTs of other countries, as well as with regional and international players.
Critical Infrastructure / Sectoral CSIRT - in charge of monitoring, managing, and responding to cybersecurity incidents related to a specific sector (e.g. energy, telecom, finance).
Enterprise (Organizational) CSIRT - an Enterprise CSIRT generally refers to a team in charge of monitoring, managing, and handling cybersecurity incidents impacting the internal ICT infrastructures and services of a specific organization.
Regional / Multi-Party CSIRT - a Regional / Multi-Party CSIRT refers to a team or matrixed team in charge of monitoring, managing, and responding to cybersecurity incidents related to a specific region, or a number of organizations.
Product Security Incident Response Team (PSIRT) - a Product SIRT is a team within a commercial entity (typically a vendor) that manages the receipt, investigation, and internal or public reporting of security vulnerability information related to products or services commercialized by the organization.
Actions - the description of how something is done at varying levels of detail / maturity.
Capability - a measurable activity that may be performed as part of an organization’s roles and responsibilities. For the purposes of the FIRST services framework, the capabilities can either be defined as the broader services or as the requisite functions, tasks, or actions.
Capacity - the number of simultaneous process-occurrences of a particular capability that an organization can execute before they achieve some form of resource exhaustion.
Common Vulnerability Exposures (CVE) - [^20] is a list of entries containing an identification number, a description, and at least one public reference for publicly known vulnerabilities. It serves as a standard identifier to reference vulnerabilities.
Common Vulnerability Scoring System (CVSS) - [^21] a numerical score that reflects a vulnerability’s severity.
Common Weakness Enumeration (CWE) - [^22] a formal list of software weakness types created to:
Health Insurance Portability and Accountability Act (HIPPA) - [^23] a US law designed to provide privacy standards to protect patients’ medical records and other health information provided to health plans, doctors, hospitals, and other health-care providers.
Key Performance Indicators - [^24] a measurable value that demonstrates how effectively a company is achieving key business objectives. Organizations use KPIs at multiple levels to evaluate their success at reaching targets.
Maturity - how effectively an organization executes a particular capability within the mission and authorities of the organization. It is a level of proficiency attained either in executing specific functions actions or tasks, or in an aggregate of functions or services. The ability of an organization will be determined by the extent, quality of established policies and documentation, and the ability to execute a set process.
Payment Card Industry Data Security Standard (PCI DSS) - [^25] an information-security standard that promotes the safety of cardholder data across the globe.
Tasks - the list of actions that must be performed to complete a specific function.