Measuring the health of your security awareness program can be tricky. There are many methods you can use to measure the effectiveness of your program and there are virtually infinite metrics you can pull out and interpret in different ways. Let’s explore 3 method / metric combinations that can set a benchmark for your program’s health and increase the confidence your company has in its effectiveness.
Now, phishing training can be a delicate subject with many employees. Phishing training programs can make employees feel entrapped or tricked when they are not properly run or communicated. This is definitely not the goal of a phishing training program and can actually unravel progress your security awareness program has already made! For the sake of your time, we will save this topic for a later discussion, and assume your phishing training program is already in place and mature.
What is Overall Phish Percentage? Overall Phish Percentage is your company’s susceptibility to phishing attacks (vishing and smishing included). Usually, this statistic is tracked and updated yearly, but it also makes sense to update this more frequently as a topic for strategic business reviews. Simply put, if you have 100 employees and 6 of them are successfully phished by your phishing training program in a given month, then you have a 6% Overall Phish Percentage for that month. If you extend this out over 3 months or a year, then you just add up the number of unique employees that were phished and you can get the overall phish percentage for a specific quarter or year.
Why is this stat important and what are some pitfalls when measuring this? This stat is important because it is a simple way of measuring the effectiveness of your phishing training for your employees. Ideally, you would see the month over month Overall Phish Percentage trend downward to a small amount, but you can’t improve what you don’t measure! When your phishing training program is done right, it should test employees against phishing threats they may realistically face at work (and at home) and train those that failed the tests to recognize phishing threats moving forward. It is normal to have spikes in Overall Phish Percentage around holidays, big current events, and if your employees see a big change in their workplace or in the way they communicate with each other. So long as your Overall Phish Percentage is trending downward over time or is already at a nominal percentage, your program is functioning effectively.
Overall Phish Percentage is great but be careful not to misinterpret results to read the outcome you want! I have talked with many companies and service providers that advocate their clients “4% Phish Rate,” but when we dig into their phishing training program it becomes clear their program is better at getting that result than training employees to recognize phishing threats. I will let you be the judge of your phishing training program’s effectiveness, but be aware of the following pitfalls:
Easy to detect phishing: If every employee is getting a phishing attempt at the same time, it is very easy for them to send it around or have others identify the phishing threat for them. In this case, employees are not identifying the threat because it is a threat, they are identifying it because of the way it was administered. Almost all realistic threats won’t be emails or texts blasted to every employee at the same time, malicious actors usually select groups of people and try different threats. Remember, the goal of your phishing training program is to train your employees against phishing threats they will likely encounter while working or at home not for them to recognize a test.
One size fits all phishing tests: Every human being is unique, and your employees are no exception to this. Every employee is uniquely vulnerable to social engineering attacks. For one employee it could be an urgent text from “Amazon” saying their package might not be delivered and they should log in now. For another it might be an email from their “boss” late on a Friday saying they need $500 in pre-paid gift cards for a client. Employee’s vulnerabilities are unique and your phishing training program should be able to take that into account and adjust.
Poorly interpreted results: There are many ways to interpret results to kind of see the outcome you want. We can’t possibly go over every way this could happen but be careful not to do the following. When you are measuring the Overall Phish Percentage of your employees month over month, you may see a 4% phish rate for 3 months in a row and immediately celebrate! However, if in every month a different 4% of your employees fell for a phishing threat, in reality you have a 12% phishing rate for that quarter. Be careful of this and make sure to inspect your data. Many things can be automated, but sometimes a human touch is necessary. This brings us to our next metric: repeat offenders.
- Assessments – Repeat Offenders
In the majority of Security Awareness Programs, assessments show up in two places: Phishing Tests and Security Awareness Training. Repeat offenders for phishing Tests are pretty straightforward to understand: Employees that continue to fall for phishing threats even after being trained. What does it mean to be a repeat offender for Security Awareness Training?
Security Awareness Training is most commonly a combination of training videos followed by quiz questions on the video content. Repeat offenders in this area can loosely be defined as employees who are lagging behind! People who either don’t complete their training before communicated deadlines or fail multiple training assessments. It is very important to know who these individuals are but be careful not to make your repeat offender problem worse.
When working with these individuals on fixing their behavior, it is very important they feel supported and understood, not punished and alone. There are few ways to work positively with those that need a helping hand, but for the purposes of this article, we just need to be aware of who they are.
Tracking these individuals, both how many repeat offenders there are and how long they stay a repeat offender is a valuable statistic for your company. It is very true that in cybersecurity we are only as strong as our weakest link. It takes one slip of from one person to unravel a great Security Program and being able to improve the behavior of repeat offenders is strong evidence of a highly successful program.
This brings us directly to the third point. A crucial feature of a Security Program is buy-in from the employees. One of the best ways to get their buy-in is to go talk to them! I know this can feel like a radical opinion, but your employees need to feel heard and supported. If they feel like their company’s security program doesn’t work for them, at the end of the day it will become a self-fulfilling prophecy.
- Surveys – Positive Responses
Surveys are not only a great opportunity to get honest feedback from your employees, but also a great opportunity to continue to educate your employees. By asking your employees the right questions you can solicit honest opinions and gain insight into your own Security Programs.
Additionally, by asking employees thought-provoking questions about their behavior you create another opportunity to reinforce good habits. Questions like “When you receive a weird email from a coworker, how do you confirm its authenticity?” are a great way to continue to reinforce behavioral patterns that keep everyone safe.
What exactly do we want to measure about these surveys? There are two specific areas to measure: participation (be careful when making these mandatory) and response. Simply put, if you ask employees to give you their feedback, it is very valuable to know how many employees are willing to give their feedback. It should be a pillar of your Security Program to make sure every employee feels involved and contributes. Measuring the response rate of non-mandatory surveys is a great way to understand this.
Secondly, what are the responses actually saying? Be careful of Positive and Negative Response Bias here (people who feel strongly about something are more likely to volunteer their opinion.), but genuinely measure the sentiment employees have of your program. Ideally, your employees have a great opinion of the program, but in case they don’t, you absolutely need to know about it now before it gets worse. Also, be careful you don’t just measure the “likability” of your security program. While it is important employees generally like your program, it is more important they feel it is effective and protects them. A program that is well liked by employees does not necessarily represent a program that protects them as well, it is important to measure responses in both areas.
Users: How and What we should Measure?
by: Connor Swalm, Phin Security
Reprinted with permission: https://lifecycleinsights.io/2021/08/15/users-how-and-what-we-should-measure/
Connor Swalm is CEO of Phin Security. They help channel partners improve their client's cyber hygiene with our custom training programs.
Red Sky Alliance is a Cyber Threat Analysis and Intelligence Service organization and is in close partnership with Phin Security. For questions, comments or assistance, please contact the office directly at 1-844-492-7225, or feedback@wapacklabs.com
Weekly Cyber Intelligence Briefings:
- Reporting: https://www.redskyalliance.org/
- Website: https://www.wapacklabs.com/
- LinkedIn: https://www.linkedin.com/company/64265941
Weekly Cyber Intelligence Briefings:
REDSHORTS - Weekly Cyber Intelligence Briefings
https://attendee.gotowebinar.com/register/3702558539639477516
Comments