Know thyself!

When evaluating security controls, it is common to use self-certification as a way to strike a balance between cost and value. For example, whilst you could pay your auditor to flip every stone in your organisation (thereby funding their progeny through medical school), it makes much more financial sense to focus their time on the areas of greatest risk, or least foreknowledge. So how are these areas generally chosen? Typically, through the answers provided in a questionnaire.

Now, whilst using a questionnaire for the quantitative evaluation of security controls is quite straight forward (you count things that are there, or otherwise) the qualitative evaluation is much more subtle. Mostly because it is difficult to separate the answers from both personal and contextual bias.

My own experience of this has been best informed through interviewing several thousand candidates for consultancy roles. As part of this, I have always used a brief telephone interview as the first step in filtering out any mismatches. And whilst the main purpose of the call is to evaluate the psychology of the candidate, the general format will follow a questionnaire targeted at exploring knowledge in several technical domains, along with detecting any affinity towards a particular Disney Princess.

As part of this interview, each technical domain is preceded with a request for the candidate to rate their knowledge on a scale of zero to five, where zero is no knowledge and five is they know everything. In my experience, the answers to these questions really only fall into three broad buckets: those who consistently answer three, those that consistently answer four, and those that alternate between answering one and four.

In practice, it is a rarity for anyone to answer zero or five, just as it is equally rare for anyone to rate theirself accurately: those with weak knowledge consistently over state, whilst those with strong knowledge consistently under state (if only as a form of professional modesty).

So what do I personally take away from this?

In my experience, a qualitative questionnaire is almost worthless to leave with someone to fill in later. In fact, even if you go through it interactively with someone, the answers themselves are rarely useful. For me, the real value lies in reading the interviewee’s body language (or aural cribs) as you take them through the questions.

Once complete, you will probably still not have a reasonable qualitative evaluation of any controls, but if you are paying attention, you will know exactly which areas your interviewee is worried about, or doesn’t understand. No matter what answer they actually gave.

There is no spoon. ;)