Effectiveness of Proactive CSIRT Services

Background

For the FIRST 2005 conference we put together a paper researching limitations related to the reactive CSIRT services, mainly the response to low priority incidents. As the PhD research project of Johannes Wiik continued [Wiik et al. 2005], the scope was broaden to study the limitations of other services perceived as mandatory, most importantly the advisory service. The intermediate results related to the advisory service seem suggest very interesting, but also provocative, insights. Therefore we agreed to prepare a proposal for the upcoming FIRST conference in Baltimore.

Proactive Services as Cross-Organizational Learning Process

Almost all authors discussing these teams have suggested that Computer Security Incident Response Teams (CSIRTs) need to deliver new as well as additional proactive services to stay effective, but there are hardly any studies investigating to what extent existing proactive services are indeed effective or how to make them more effective. Indeed the advisory service is one of the core CSIRT services and proactive in scope – already part of the description even in the oldest CERT related documents – which has not changed much over the years. Only some technical development can be seen in regard to system categorization, identification schemes for vulnerabilities or formats for the effective exchange.

We argue that the potential of proactive services should be viewed as cross-organisational learning process. They carry the promise of avoiding incidents and the hope of saving considerable resources. The advisory service instigates the transfer of information between vendors of commercial off-the-shelf-software (COTS) or open source software and users of these products in the CSIRT constituency. Another proactive approach is actively searching for vulnerabilities in networks and organizations. Quite specific information is provided through analysis of systems within the constituency and informing the administrators about much needed patches or changes to the setup. Rather than carrying out this analysis only on demand the networks and systems are routinely surveyed. Thus, it is similar to (and hence we call it) a "neighbourhood watch": your neighbours keep an eye on your assets.

In this paper we evaluate two proactive services:
1. The common advisory service as an example of an existing service, and
2. Neighbourhood watch (NBHW) as a new service that builds on the advisory service.

Based on a case study and organisational learning theory, we build a system dynamics simulation model to test the effectiveness of the two services. Preliminary findings indicate that neighbourhood watch has several significant strengths compared to the traditional advisory service with respect to knowledge acquisition, information distribution, information interpretation and organisational memory.

However, as the advisory service is a community service the aim is to reach out to all constituents and it can therefore make an overall impact, despite its weaknesses. As NBHW is dependent on authorisation to scan the networks of each constituent, its effectiveness in the constituency as a whole is very much dependent on the take-up rate.

We also evaluate the short term impact of using NBHW that typically helps new customers of this service to detect previously unnoticed incidents. Thereafter we look at the long term impact as customers mature their way of using more effectively the information provided by this service to secure their networks and organizations.

This last issue is important to put our observations back into the broader picture. It stresses again [Wiik et al. 2005] that all CSIRT related activities are impacting each other and cannot be seen as separate activities. As current management approaches do not consider this aspect, we recommend to all CSIRTs to revisit their services and interdependencies not yet addressed in their current setup.

References

[Wiik et al. 2005] Limits to Effectiveness in CSIRTs / Johannes Wiik; Jose J. Gonzalez; Klaus-Peter Kossakowski. - [Paper for the FIRST 2005 Conference, Conference Proceedings. Also available from www.cert.org/csirts/]