How effective is your security?

Steve Wright discusses how you can measure the effectiveness of security using ISO 27001.

While the intentions and objectives behind ISO 27001 are not dramatically different to those in BS 7799:2002, one of the changes with the biggest potential impact on organisations is the requirement to measure the effectiveness of selected controls - or groups of controls - within the new standard (for more details see ISO/IEC 27001:2005 4.2.2 d).

This new requirement not only demands that businesses specify how these measurements are to be used to assess 'control' effectiveness (there are 133 controls in the new standard), but also how they are comparable and reproducible, so they can be used time and time again, for example.

You could be forgiven for thinking this ought to be a reasonably straightforward task. After all, most IT departments have been working within a measurement (for example service level agreements - SLAs, KPIs, IT Infrastructure Library - ITIL) framework since the mid-1990s and should, therefore, have been considering how to measure IT effectiveness, as well as providing value for money for their stakeholders and shareholders.

But I am afraid it is not that simple when it comes to security. As many a consultant would tell you, it is a specialist area and, of course, you need an expert to help you. Well, you do need an expert, but that does not mean you cannot find one within your organisation or your current service provider's.

The whole area of what is good and effective security is often misconstrued, mis-communicated and, worse, mismanaged. That said, there are usually pockets of good practice within most businesses. This does not diminish, however, the fact that we need to think hard about what to measure, how to measure, and when to measure.

As mentioned above, there is often evidence of good security controls already in place in organisations (especially those who have already implemented CoBit, ITIL or BS 7799). Examples include good anti-virus procedures or good physical security. But, when you start to dig deeper into what, how, and why these controls were selected and are now being measured, you start to come unstuck.

The biggest gaps are usually found in the documentation that should state the relationships between the identified risks, what countermeasures were implemented and why. This gap often originates from a misunderstanding of what the essence of the original BS 7799:1999 standard was all about - risk management.

In fact, this new control is not even new - it has been in existence for years. It is just that previous wording talked about 'methods to monitor and maintain the effectiveness of the information security policy'. So, naturally, most organisations took this to mean security awareness training and so on.

Often what has happened in practice is that businesses have focused on ensuring the control objectives (formerly 127 controls) were implemented and documented in the service-oriented architecture (SOA) and forgot to appreciate, or map, the relationship between their organisational risks and the information security management system (ISMS), which is to say, the framework in which the management of security should be defined, documented and understood by all employees.

But herein lies the problem. Some companies may have chosen to implement, say, an expensive intrusion detection system (IDS), but do not always know why or how they came to choose its implementation in the first place. More significantly, the special relationship between risk and cost will not necessarily have been fully thought through, and so measuring the effectiveness of the IDS may not have even been considered.

Worse still, very few appear to have done any work on understanding the return on investment. In fact, if fewer organisations were being scared into buying security solutions, and concentrated instead on what is fundamentally a risk issue, security-related decisions would only be based on costed evidence and we would not be having this discussion. Each control selected would have to define how it was going to be effective, and the measurements required would, of course, be documented.

So, how can we select which controls should be measured and, equally importantly, how can they be used to provide assurance to stakeholders that security controls are operating effectively? This is an area that is either ignored (for some of the reasons stated above) or is supervised but is not being reported on.

As you may have concluded, this is not a topic that is as straightforward as you might believe. But although it is difficult, it needs to be fully understood in order to prevent it stifling the anticipated growth in security.

Another reason why this is a particularly hot topic is because industry and leading standards bodies, such as the ISO and BSI, have also struggled to get a grip on it. These organisations are all striving to achieve synergies between ISO 20000, ITIL and ISO 9001, and could really do with finding a set of measurements that can easily be linked in with existing or new management systems (for example, quality and service delivery).

This, coupled with an age of increasing identity fraud, escalating IT costs and frequent corporate scandals, is where regulatory bodies are starting to ask awkward questions about how the CIO, CEO and senior management are managing risk and, therefore, the effectiveness of parts of their risk mitigation strategy, including security.

Where to start?

It would be easy to tell you to implement the set of controls contained within ISO 27001 - based on your risks - and to ensure they are used to help mitigate, transfer or simply remove identified risks.

That said, please do not make the mistake of ignoring Clauses 4 to 8 (of the standard), as you will also end up missing out on the fundamentals that make up the information security management system (ISMS - see Annex A: Expected ISMS Documentation Set), which has now become a mandatory section of the new standard.

In fact, it is often these missing foundation stones that make the task of monitoring effectiveness that much harder to implement. The ISMS is like the house built on solid foundations - when the rain fell, it stood firm. An ISMS should be implemented by experts - people who understand risk as well as the importance of using the right controls.

An example of a good ISMS might include well defined roles and responsibilities, captured within the ISMS operational policy. This document should also dictate how and where relationships interact. An effective ISMS should have clear and unambiguous management support as well as clear and demonstrable risk management - with the ability to link identified risks to risk treatment plans, including recorded preventative and corrective actions.

The solid foundations of a well-built ISMS should also have fully documented, tried and tested incident management procedures, full auditing and logging procedures and, above all, a clear strategy of what measurements should be used to measure the effectiveness of security.

Why measure security?

The objectives of measuring security are:

- to show ongoing improvement

- to show compliance (with standards, contracts, SLAs, operating level agreements, etc)

- to justify any future expenditure (new security software, training, people)

- to meet requirements - ISO 27001 requires it, and so do other standards like ISO 9001 and ISO 20000

- to identify where implemented controls are not effective in meeting their objectives

- to provide confidence to senior management and stakeholders that controls are effective.

So, which of the 133 potentially applicable controls (within ISO 27001) can be used to measure security? Well, arguably, all of them. In practice, though, this would invariably be too onerous a task and would cause an already overworked IT department to crumble under the weight of bureaucracy.

Before attempting to answer this question in a more practical way, we should understand the requirement for such clarity. Why are you being asked to provide such information? What is the driver? Where does the requirement come from?

Other drivers may exist, too. It could be that the company has just realised that you can get more from ISO 27001, or perhaps its operational risk management such as BASEL II, SOX, Turnbull or other regulatory requirements and legislation are driving your business.

Either way, you are not alone. Many organisations (but not all) misunderstand the fundamental concepts behind BS 7799 and ISO 27001 and have treated it as a marketing exercise, as opposed to trying to achieve real business benefit and ROI.

ISO 27001 provides much more clarity and goes further into what should be measured for its effectiveness. As such, the much anticipated ISO 27004 (guidelines on how to measure effectiveness) in 2007 should finally put an end to this grey area and should shed much needed light on the types of controls to be measured and what results we should expect (for example industry baseline).

What are the benefits?

Measuring security provides the following benefits:

- It actually eases the process of monitoring the effectiveness of the ISMS (this is less labour intensive, for example, if using tools, and provides a means of self checking).

- Active tools for measuring can prevent problems arising at a later date (for example network bottlenecks, disk clutter, development of poor human practices);.

- It reduces incidents.

- It motivates staff when senior management set targets.

- You can provide tangible evidence to auditors, and assurance to senior management that you are in control - ie corporate information assurance, and a top down approach to information assurance.

Whatever the driver for implementing ISO 27001, it should no longer be just about identifying the controls to be implemented (based on the risk), but also about how each control will be measured. After all, if you cannot measure it, how do you know it is working effectively?

This essentially means that all organisations will soon be able to demand OLAs and SLAs for security - based on real measurements - and will be able to treat security as a measurable business unit (with targets based on industry best practice or ISO 27004).

What should be measured?

For ease of explanation, the measurements have been broken down into the following categories:

- management controls: security policy, IT policies, security procedures, business continuity plans, security improvement plans, business objectives, management reviews

- business processes: risk assessment and risk treatment management process, human resource process, SOA selection process, media handling process

- operational controls: operational procedures, change control, problem management, capacity management, release management, back up, secure disposal, equipment off site

- technical controls: patch management, anti-virus controls, IDS, firewall, content filtering.

Ultimately the risk assessment will confirm the relevance of the most applicable controls that should be measured. Therefore measurement can be achieved against:

- a particular security control or objective

- a group of controls

- ain controls within a standard

- or against the examples shown in the tables. These have been mapped to their nearest ISO 27001 control reference or group of controls (but bear in mind that not all of the controls map easily).

How do you decide?

So what is the process for deciding which of these controls (or groups of controls) should be used?

First, you need to

- confirm relevance of controls through risk assessment

- define objectives, ensuring they map back to the business

- use existing indicators wherever possible, for example in ITIL terms, KPIs. A KPI helps a business define and measure progress towards a particular goal. KPIs are quantifiable measurements of the improvement in performing the activity that is critical to the success of the business

- within the ISMS audit framework, identify controls which can be continuously monitored, using your chosen technique

- before using any tools, agree the objectives with senior managers as well as staff. Agree this contractually where external third parties are concerned, or through SLAs/OLAs where internal third parties are concerned, for example ISO15000 (ITIL)

- establish a baseline, against which all future measurements can be contrasted or compared

- provide periodic reports to the appropriate management forum or ISMS owners (show graphs, pictures paint a thousand words)

- identify review input - agreed recommendations, corrective actions

- implement improvements within your integrated management systems (IMS), for example merged ISOs 9001, 14000, 27001, 20000

- establish or agree new baseline, review the output, apply the PDCA approach (Plan - Do - Check - Act).

Challenges

Measurement for the operation must reflect your business goals, be critical to the success of the operation, be reproducible and contrastable, and facilitate corrective action. Each measurement requires definition and therefore you should ensure the following list is used for each definition:

- title

- scope of the metric

- purpose and objectives

- measurement method

- measurement frequency

- data source and data collection procedures

- chosen indicators

- date of measurement and person responsible

- level of effectiveness achieved (or level of maturity, in case of a maturity metric for controls)

- causes for non-achievement.

As each business will probably have its own measurements already in place (for example KPIs), the challenge is to set a 'measurement', which is measurable and contrastable for all respondents.

In summary

Any large ISMS needs to supplement formal audits with self-checking mechanisms aligned with the equivalent of a performance indicator.

Nevertheless, for any ISO 27001 certified ISMS, no matter how small, some basic measurement is both expected and required. Otherwise, it will be impossible to demonstrate that any improvements have either been made or are required for corrective purposes. Consider using tools that will help assess the effectiveness of a particular security control; examples might include capacity management, where often software-based analysis techniques can be used to supplement any human effort.

Senior management and possibly auditors are more likely to want to see the big picture. Therefore, consider monitoring the effectiveness of a group of controls, for example incident management, plus others. Encourage senior managers to set realistic goals, thereby ensuring that there is demonstrable evidence of good corporate governance.

In a changing environment, new baselines need to be set each time a major change within the ISMS occurs, so this is just the beginning.

You can see from some of the examples in the tables, that measuring the effectiveness of an ISMS is a complex task. It requires time, honesty and a realistic approach to agreeing which measurements can be effective in the business and, significantly, how you intend to measure that effectiveness to ensure results are comparable and reproducible year on year.

It is important to ensure all measurements are recorded into tables (ie spreadsheets) so that the controls to be measured are captured and comparable. This will also make sure that results from these measurements can be added into the table at different dates and with various results. This will help to assure the integrity of data and can help when statistical analyses are being prepared (for senior management presentations/reports and for calculating expected loss, ROI, etc).

Any organisation that already holds certification to BS 7799 or ISO 27001 should start to work towards establishing a set of clearly defined measurements. Begin with agreement or commitment from the sponsor (for example the business/operations) on defining 'what and how' security should be measured.

In addition, some of the measurements can be used as potential KPIs and could help form part of an OLA or SLA with internal or external third parties. Either way, you need to be clear on what benefits can be demonstrated (or not) by providing such transparency.

That said, by creating such transparency in security, you will only be judged on your performance. This will help avoid future misunderstandings about what security is and how important it is to the organisation. It could also, of course, serve as your Achilles heel should things not go to plan. Above all though, it should start to dispel rumours that security is a black art and that it is unmeasurable. In fact, we should start to see tangible benefits from measuring and improving the ISMS.

Finally, do not forget that all existing BS 7799 and ISO 17799 documentation must have been transferred to the new numbering convention of ISO 27001 (including ISO17799) by 15 April 2007 and that, in addition to legal and regulatory requirements, the updated standard now places extra emphasis on contractual obligations at all stages of the ISMS, including risk assessment, risk treatment, selection of controls, control of records, resources, monitoring and reviewing of the ISMS and in the documentation requirements.

Steve Wright is a senior consultant and heads the security management (ISO 27001/BS 7799) team at Insight Consulting, part of Siemens Communications, www.insight.co.uk EXPECTED ISMS DOCUMENTATION SET

Information security policy (one page document)

ISMS scope

ISMS/BS7799 project initiation document (PID - project plan)

ISMS operational manual (organisation chart and reporting structure/objectives)

ISMS roles and responsibilities (project sponsor and business owners)

Information security forum (terms of reference)

Information asset register (all critical business information assets)

Risk methodology and risk treatment (policy and procedure)

Security improvement programme (incorporating risk assessment results, pen test results, audit results, etc)

IT technical security policies (eg change management policy, malicious code policy, software development policy)

Security attributes of IT systems (eg ISMS scope only, access policy, RAS policy)

Security network topology, BS2000 mainframe security access control policy