Monitoring, Evaluation and Learning Frameworks for Cyber Security Strategies

Cyber security is an increasingly critical component of Australia’s national security as technology evolves at a rapid pace and opportunities grow for malicious cyber actors who have become better resourced.

– by Hannah McKenzie and Sara Wilson

It has been estimated that cyber security incidents cost Australian businesses up to $29 billion per year – almost 2% of Australia’s GDP[1]. Recent incidents in Australia, such as the attacks on Victorian hospitals, the Australian Parliamentary network, and on ANU, demonstrated the extent to which cyber criminals have been able to infiltrate our critical community services[2][3][4]. Cyber security incidents such as these have had, and will continue to have huge consequences for our national security and strategic interests.

Australia’s federal and state governments are developing and implementing cyber security strategies to address increasing public concern over cyber threats to government, businesses and our communities. These strategies aim to reduce the frequency and cost of cyber security incidents. For example, Australia’s 2016 Cyber Security Strategy established the foundations for Australia’s action against cyber security threats and was accompanied by a $230 million investment.

Monitoring, Evaluation and Learning (MEL) frameworks can be employed to test and adjust the effectiveness of cyber security strategies in achieving their planned outcomes. Effective MEL provides three key objectives, which have corresponding benefits to organisations as illustrated in Figure 1.

Figure 1: Objectives and benefits of MEL frameworks

MEL frameworks for cyber security strategies have had limited success, both in Australia and internationally, at checking and adjusting the effectiveness of actions and tasks against planned outcomes. While progress indicators are used to track the progression of action tasks, to achieve success it is critical that MEL approaches are expanded to measure the intended outcomes of these action tasks.Only by measuring our current state against these outcomes can governments be confident that they are truly confronting these threats.

Noetic employs a tried and trusted methodology for developing MEL frameworks that break down and simplify what has traditionally been a complex and confusing process. Noetic’s process provides clarity to those responsible for implementing the MEL framework as to what data to collect and what it will be used to measure. For the MEL framework to be most effective, it is critical that the following design and implementation stages are completed concurrently with the design and implementation stages of the strategy.

Figure 2: How to design and implement a MEL framework

Building Effective MEL in a Cyber Security Context

Due to the complex nature of cyber security, designing an effective MEL framework for cyber security strategies can be challenging and requires broad consultation with MEL experts.

The dynamic nature of cyber security threats and responses means that cyber security strategies need to be reviewed and updated regularly to remain relevant. MEL frameworks therefore need to be reviewed and refreshed more frequently than these strategies, while still providing continuity of measurements. Achieving this requires a flexible and adaptable approach to be used during design to meet the challenges of the fast-paced technological environment.

There is currently limited experience and expertise in managing cyber threats from a business perspective with small numbers of tried, tested and proven metrics and no standard MEL methodology that can be applied in this emerging area. Broad and deep collaboration under governmental direction is critical to develop relevant metrics in this area to build resilience. Monitoring, evaluation and learning must be recognised as a strategic priority that requires appropriate resourcing and investment in both design and implementation.

MEL frameworks need to be tailored to be effective in their environments – there is no ‘one-size-fits-all’ solution. The unique cyber security environment would benefit from a principles-based model that is flexible and adaptable to the diversity of cyber threats, as well as the varied levels of vulnerabilities across sectors and across both governmental and non-governmental organisations. This approach was exemplified through “A call for views” on Australia’s 2020 Cyber Strategy[5].

Opportunities in MEL for Cyber Security

The cyber security domain also presents unique opportunities that are advantageous for designing and implementing effective MEL frameworks.

The technological nature of cyber security typically allows for high quality data sources to be generated that can be used for measurement. Costs for data collection are also low due to the relative ease of automating data collection and analysis.

Developing a MEL framework for cyber security will provide widespread benefits as the framework can be leveraged and applied to other industries that relate to national security more broadly that lack a tried and tested MEL methodology.

In conclusion

Cyber security incidents are growing in frequency and severity. It is critical that governments respond by designing and implementing strategies and MEL frameworks to support the community in combatting this problem. Whilst developing and implementing effective MEL frameworks in the cyber security domain has unique challenges, the potential benefits in better protecting Australia cannot be overstated.

[1] Swetha Das, ‘Direct costs associated with cybersecurity incidents costs Australian businesses $29 billion per annum’

[2] Isabella Kwai, ‘Australian Parliament Reports Cyberattack on Its Computer Network’

[3] Stephanie Borys, ‘The ANU hack came down to a single email – here’s what we know’

[4] Melissa Cunningham, Noel Towell & Zach Hope, ‘Regional Victorian hospitals hit by cyber attack’

[5] ‘Australia’s 2020 Cyber Security Strategy – A call for views’