Evaluating Technology and Security Leaders

Great organizations have great leaders. Leaders who surround themselves with top talent. Their teams work well together, using the best technology and efficient processes.

Leaders who inspire others, who motivate, who lead by example. Those who properly reward staff and who get things done the right way. Leaders who get results, with projects delivered on-time and on-budget, but without constant staff burnout to get there.

Leaders with skill, honesty and integrity. Men and women who are kind and yet relentlessly pursue excellence. People who are thrilled when their own staff members surpass them in their career.

Leaders who leave a positive legacy long after moving to another role or another organization or even retiring.

Good Questions to Ponder

But you may be thinking, is such leadership even possible? How do you really know who will stand the test of time?

No doubt, this discourse raises many questions: Who’s really good at security and/or technology leadership?   

Also, inquiring minds ask introspective questions of themselves. Questions like: How am I doing in my leadership role? Am I seeing progress over time? Where should I focus to improve my organizational people/process/technology skills?

These topics typically come up when I am mentoring a mid-career chief information security officer (CISO), or talking with a group of trusted executive friends at a reception during a national or international event, or speaking to a top journalist who is looking for someone to interview for a cyberstory.

Also, sometimes a top-level elected official or career government leader is conducting a national search and wants to find the best and brightest, the age-old question comes up: Who would I recommend?

With so many tech and cyberawards being given out and with the rapid turnover in public- and private-sector leaders, are kudos on LinkedIn profiles even trustworthy? Aren’t there award-winners who seem to receive their recognition through office politics or from less-than-average work or are known to have poor ethics?

Despite confusion, one thing is absolutely clear. Study after study has shown that the overall performance of leaders such as CIOs, CISOs and CTOs is vital to organizational success. It fact, most experts say an organization’s leadership is the No. 1 predictor of future organizational success.

I really like John Maxwell’s books on leadership, and here are some of Maxwell’s inspiring quotes:

• "Great leaders always seem to embody two seemingly disparate qualities. They are both highly visionary and highly practical."
• "The higher you want to climb, the more you need leadership. The greater the impact you want to make, the greater your influence needs to be."
• "If you can't influence people, then they will not follow you. And if people won't follow, you are not a leader. That's the Law of Influence."
• "Anyone can steer the ship, but it takes a leader to chart the course. Leaders who are good navigators are capable of taking their people just about anywhere."
• "If you are a leader, the true measure of your success is not getting people to work. It's not getting people to work hard. It is getting people to work hard together. That takes commitment."

Note: While we can make similar points for most CxO roles, I will focus on the CISO role in this piece.

For good articles on the need for CISOs, see:

• Security Intelligence said this on why the role of the CISO is vital.
• CIO magazine described why you need a CSO.
• McAfee described the changing role of a CISO.
• Wall Street Journal (WSJ) on the changing reporting ties of a CISO.
• My blog on why CISOs need a mentor — and how to get one in government.

How Do You Measure? Typical Responses

OK, so enough background on the vital importance of security and technology leadership. How can we evaluate CISO or chief security officer (CSO) performance?

Typical responses I hear fall into several categories.

First, there are many “hard” metrics used to measure security leaders such as audit findings addressed, trouble tickets opened or closed, breaches reported, budget items managed well, staff vacancies filled (or other staff growth or decline numbers), degrees or certifications obtained, specific project success, outcome focus on top executive initiatives or even bottom-line company returns (may include legislative budgets approved for government entities).

Many CISOs spend the majority of their time putting out cyberfires or dealing with never-ending security incidents. These leaders often struggle with their wider CISO responsibilities or perhaps fail to get the needed resources or support to be effective in the long run. They never seem to gain more executive support and/or improved their security culture.

And while these (and other) metrics are important to consider and certainly help in the evaluation over time, I also see operational metrics being misused on a regular basis.

How?

Some CISOs coast on a past leader’s success in these areas for months and even years. For example, a current CISO's lack of breaches on their watch may be the result of years of work by the former team. Conversely, I have seen a few outstanding CISOs hit by a data breach that has more to do with the inaction of their predecessor.

Closing audit findings is necessary — even essential for most leaders. But in what order? Are these CISOs leading the organization, or just responding to audit reports or regulations issued by others?

Also, are you closing the barn door after the horses have escaped? What about hot new cyberthreats that are not in those internal or external audits? Bottom line: Good metric baselines are hard to obtain and rarely are put in place for new CISOs.

Benchmarking security or technology against other best practice organizations in your industry can help in many important ways, but with CISOs turning over an average of every 17 months (this number seems too low to me for government), it is very hard to gauge the true impact of a single leader on an organization using this data alone.

In some cases, whatever numbers are used to measure performance are clear-cut and directly impact bonuses or incentive plans and are way too simplistic — such as a single number on incidents reported or resolved. Many of these “one and done” metrics are actually not in the control of the CISO. (For example: You can’t control how often you are attacked.)

On the sad side of things, I have seen CISOs fired because of a breach that was largely outside of their control, as they were outmatched by the outside organization that hacked them — or an employee clicked on a link somewhere.

Second, there are many lists of “soft” performance measures that are used to judge performance. These soft skills include items like this list from Inc.:

• Good leaders have a mission and inspire others to join them.
• Good leaders create strong organizations.
• Good leaders have strong interpersonal skills.
• Good leaders are good motivators.

Here are some other good lists of important skills and success factors to help from Habott.com and Kinhave.org. 

There are hundreds of such character trait and leadership lists that are used by management to evaluate people in role-specific situations. In many circumstances security leaders are judged by the perceptions from someone higher in management (often the CIO or department director in government.) Seldom do metrics surrounding these soft-skill topics include a 360-degree feedback evaluation that is really fair, in my experience. When management does get input from others, it often includes only colleagues who provide a similar viewpoint as the evaluator. 

I have also seen 360-degree performance evaluations go painfully wrong based on popularity contests or office politics, but that discussion is for another day. On the other hand, 360-degree evaluations implemented well, can certainly be an effective way to measure performance.

Lohrmann's CISO Grading Tool: Are You a Trusted Adviser to the Enterprise?


Which brings me to a third approach that I use. My analysis uses elements of the first two methods and a few twists. It focuses on becoming a trusted adviser within your specific situation.

I generally evaluate CISO effectiveness in five core areas (Note: sometimes I add a sixth area, which I will describe at the end).

These relationship areas listed below reflect the level of trust, respect, project results, communications skills and overall competence with the different groups that CISOs generally interact with on a regular basis. It also reflects an ability to lead and inspire greatness in others.

• Relationships with your internal security team. Staff under your control who report to you.
• Relationships with internal organizational peers. Similar level of business and technology professionals throughout your organization. This includes internal customers who you are working with and protecting.
• Relationships with your management. This includes your boss(es) and other senior executives that are higher up (your boss’s peers or above.) Of course, many view your boss as the most important evaluator. Certainly, achieving stated duties and core functions is essential. And yet, I know some hands-off tech bosses that do not provide much guidance or direction to their CISO.
• Vendor relationships. How do you work effectively with security providers, managing contracts, contract staff acquisition processes, technology providers, new technology acquisition and effectiveness, etc.
• External customers. This includes relationships with your wider organizational clients. People using your business partner’s products and services. For example, in Michigan government, I would meet with the Treasury Department’s and the Office of Retirement’s (and many other organizations') customers and partners.

I realize that CISOs have differing levels of involvement and relationships in each of these five areas, so the importance of each item can vary depending on a variety of factors. (These are not equally weighted in every organization.) Also, how you grade each area (as a yes or no) can vary based on the items above under hard and soft metrics and skill sets. Nevertheless, I maintain that almost all CISOs have some responsibility in each of these five areas. See below for more on this.

How to Grade to Stand the Test of Time


The main goal is to keep grading simple in my view. This is meant to be a high-level assessment and not a detailed performance review with organizational specifics or required cybersecurity objectives.

Answer this question: Does the CISO have "good" (or even "very good" or better yet "great") relationship with this group. I’m not talking about personal likes and dislikes, but professional rapport. Does this group respect and trust the CISO as their security adviser?

(Note: Sometimes this trust shows as fear of breaches, but they still keep following and abiding-by policies, procedures and supporting the security organization.) No doubt, this trust will be influenced by security events, incidents, data breaches, project success, audits, security competencies, communication skills, etc.

Here’s how to evaluate a security leader (or yourself) over time:

• Only one (1) group says CISO is doing a very good job and trusts CISO — Won’t last long unless your boss loves and protects you. Basically the CISO is in trouble.
• Any two (2) groups show trust and support and respect — Basic competence — but average CISO at best
• Any three (3) groups — Doing well — Keep striving for more
• Any four (4) groups — Above average CISO
• All five groups trust CISO, respect you. They will follow you through cyber good times and bad — Truly exceptional CISO

One Caution: There is a wide variance in CISO roles in both the public and private sectors. Some have many staff, some have none. Some are board-level execs, others are buried in the org chart. Authority and budgets vary widely, and governance models are all over the map. Some leaders have operational security authority and accountability 24/7, while others have matrix control and others are advisory-only roles.

Also, if you are a one-person CISO team on security in your small government, your internal team may also be measured by your own technical skills to program a firewall or some other security task.

This grading approach can work for all types of CISOs, but it requires flexibility in implementation. At first, just ask, how is he/she doing with these five groups? Be sure to get a diverse range of inputs in each category. Sometimes, a single person in a group can throw off the evaluation. A special effort may need to be made to obtain balanced input.

For example, if 20 business-side execs trust the CISO, but still one business person (who doesn’t get along with anyone) doesn’t respect the CISO and opposes you, don’t count that person/situation. However, I have also seen one vocal executive critic — who has the ear of the wider group and substantial influence — sink a CISO.

This simple, high level approach provides a sound foundation for a viable working assessment, strengthened further to stand the test of time by considering the priorities among the relationship areas for the leadership candidate within her or his current employment context; unique strengths and weaknesses; longer term needs and challenges within the future employment arena; and most importantly flexibility balanced by rigor and resilience, the capacity for change, growth.

Final Thoughts

There is a sixth category that I sometimes use for evaluating security and technology leaders that relates to marketing and external media, blogging, industry thought-leadership, speaking at conferences, etc., for CxOs. Are you influencing your wider industry peers and/or moving the bar nationwide or globally? Are you helping improve your client’s business with their customers? Note: all security and technology leaders should be evangelists for their organizational causes and demonstrating communications skills with various internal and external audiences.

But this larger industry influence step usually comes later for CISOs, and can be a part of the external client/customer piece and industry vendor relationship piece (or not). However, I see some people doing this external cyberevangelism (and or hanging out with industry peer CISOs too much), while at the same time failing in many of the other internal relationships. These security leaders are heading for an internal reckoning — maybe a pink slip. 

Also, we all need more development to become more effective cyberleaders. This is a journey, so if you are only trusted by two groups, try to move to three and so on. No doubt, building a good reputation and respect takes hard-earned CISO skills and time-tested cyberoperations results that keep working each day — for years. The hard part is often earning the internal respect of geek-level hackers on your team, and at the same time impressing executives with business acumen and communication skills in the wider community.

I will return to this: “How to evaluate CISOs” topic in a later blog, but I am happy to answer questions on my approach now. My premise is that different audiences have different needs, and you ignore them at your own peril. I have seen CISOs that are fabulous with vendors, spend 90 percent of their time on cutting–edge technology, and fail in other relationship areas.

The vast majority of CISOs can manage one or two groups well in order to become a CISO (or other security leader), but struggle to grow their circle or business influence and knowledge to be trusted by all the important stakeholders involved to achieve lasting success and grow in influence.    

On leadership, Ronald Reagan once said: “The greatest leader is not necessarily the one who does the greatest things. He is the one that gets the people to do the greatest things.”