If you haven’t had the chance to read Jeff Fry’s treatise on why the CCIE written should be dropped, do it now. He raises some very valid points about relevancy and continuing education and how the written exam is approaching irrelvancy as a prerequisite for lab candidates. I’d like to approach another aspect of this whole puzzle, namely the growing need to get that extra edge to pass the cut score.
Every standardized IT test has a cut score, or the minimum necessary score required to pass. There is a surprising amount of work that goes into calculating a cut score for a standardized test. Too low and you end up with unqualified candidates being certified. Too high and you have a certification level that no one can attain.
The average cut score for a given exam level tends to rise as time goes on. This has a lot to do with the increasing depth of potential candidates as well as the growing average of scores from those candidates. Raising the score with each revision of the test guarantees you have the best possible group representing that certification. It’s like having your entire group be members of the honor roll.
A high cut score ensures that unqualified personnel are washed out of the program quickly. If you take a test with a cut score of 800 and you score a 500, you quickly know that you need to study quite a bit more before you’re ready to attempt the exam again. You might even say to yourself that you don’t know the material in any kind of depth to continue your quest for certification.
What happens if you’re just below the cut score? If you miss the mark by one question or less? How much more studying can you do? What else do you need to know? Sure, you can look at the exam and realize that there are many, many more questions you can answer correctly to hit the right score. But what if the passing score is absurdly high?
I believe the largest consumer of purloined test questions is not the candidate that is completely clueless about a subject. Instead, the largest market of these types of services is the professional that has either missed the mark by a small margin or is afraid they will not pass even after hours of exhaustive study.
Rising cut scores lead to panic during exams. Why was a 790 good enough last time but now I need an 850 to pass? It’s easy to start worrying that your final score may fall in between that gray area that will leave lacking on the latest attempt. What happens if you miss the mark with all of the knowlege that you have obtained?
Those are the kinds of outcomes that drive people to invest in “test aids”. The lure is very compelling. Given the choice between failing an exam that costs $400 or spending a quarter of that to have a peek at what might be on the test, what is stopping the average test taker besides morality? What if your job depended on passing that exam? Now that multi-hundred dollar exam becomes a multi-thousand dollar decision.
Now we’re not talking about a particular candidate’s desire to fleece a potential employer or customer about knowledge. We’re talking about good old fashioned fear. Fear of failure. Fear of embarassement. Fear of losing your livelyhood because of a test. And that fear is what drives people to break the rules to ensure success.
The solution to this issue is complicated. How can you ensure the integrity of a testing program? Worse yet, how can you stem the rising tide of improper behavior when it comes to testing?
The first thing is to realize what drives this behavior. Should a test like the CCIE written have higher and higher cut scores to eliminate illicit behavior? Is that really the issue here? Or is it more about the rising cut score itself causing a feedback loop that drives the behavior?
Companies need to take a hard look at their testing programs to understand what is going on with candidates. Are people missing the mark widely? Or are they coming very close without passing? Are the passing scores in the 99th percentile? Or barely above the mark? Adjustments in the cut score should happen both up and down.
It’s easy to look at testing groups and say, “If you just stuided a bit harder, you’d hit this impossibly high mark.” It’s also very easy to look at scores and say, “We see that many of you are missing the mark by less than ten points. Let’s lower that and see how things go from here.”
Certification programs are very worried about diluting the pool of certified candidates. But is having more members of the group with scores within a question or two of passing preferable to having a group with absurdly high passing scores thanks to outside help?
I’ve taken exams with a 100% cut score. It’s infuriating to think that even a single wrong answer could cost you an entire exam. It’s even worse if you are financing the cost of your exam on your own. Fear of missing that mark can drive people to do lots of crazy things.
I’m not going to say that companies like Cisco need to lower the cut scores of exams to unrealistically low levels. That would cheapen the certifications that people have already earned. What needs to happen is that Cisco and other certification bodies need to learn what they are trying to accomplish with their programs and adjust all the parameters of the tests to accomplish those goals.
Perhaps raising the cut scores to more than 900 points isn’t the answer. Maybe instead more complex questions or more hands-on simulations are required to better test the knowledge of the candidates. These are better solutiosn that take time and research. They aren’t the false panacea of raising the passing score. The rising tide can’t be fixed by making the buoys float just a little higher.
It used to be that a data breach was a singular event that caused massive shock and concern. Today, data breaches happen regularly and, while still shocking in scope, are starting to dull the senses. Credit card numbers, security clearances, and even illicit dating profiles have been harvested, coallated, and provided for everyone to expose. It seems to be an insurmountable problem. But why?
Data is a tantalizing thing. Collecting it makes life easier for customers and providers as well. Having your ordering history allows Amazon to suggest products you might like to buy. Having your address on file allows the pizza place to pull it up without you needing to read your address again. Creating a user account on a site lets you set preferences. All of this leads to a custom experience and lets us feel special and unique.
But, data is just like that slice of cheesecake you think you want for dessert. It looks so delicious and tempting. But you know it’s bad for you. It has calories and sugar and very little nutritional value. In the same manner, all that data you collect is a time bomb waiting to be exposed. The more data you collect, the larger the blowback for your eventual exposure.
Yes, we’ve crossed the line from “I might get hacked one day” to “I will absolutely be hacked in the next 24 months”. The amount of data being stored has increased a hundred fold in the past few years. Every website wants you to sign up. Every department store and restaurant has a preferred customer program. Everyone has a mobile app. And every one of these respositories has data that you don’t want anyone else to have. Hackers used to have to sift through garbage to find sensitive information. Now it can be stolen with a few redirects and no smelly excursions.
Even if a website or app claims they aren’t collecting your data for “nefarious” purposes, you can be sure it will eventually be used against you. And those are the “good guys”. What about sites that won’t let you delete your account? Or worse: the sites that claim they will and then don’t do it?
Having your data laying around in a dormant database is like putting money under a mattress. It doesn’t do the holding company any good. Just like the siren song of the above mentioned cheesecake, if you leave the data laying around long enough a company will decide to do analysis on it. It’s a slippery slope that a company will fall down given enough time.
How do we fix the problems of widespread data sprawl? Given that every app and website login is now an attack surface, how can we minimize the amount of leakage even in the event of disaster? Given that the best solution of not collecting the data in the first place isn’t likely to happen, we need a new solution.
Interestingly enough, wireless companies have stumbled onto the solution in the past year. They are using social media sites as a login for wireless access. Sites like Facebook contain all of the information you would need for a user account on a site. Facebook even offers a login option for many sites, tying their database entry to yours.
What needs to happen is that a site like Facebook or OpenID needs to become the de facto repository for identity information. By containing it all in one place and forcing sites to create links to your identity store, the amount of sensitive data being stored is minimized. Apps can still collect custom information above and beyond that which is provided by the identity store, but it would be paired with a GUID value pointing to an external login with very little identifing information. Like having a phone number but no name or address.
I know exactly what you’re thinking right now: What happens if the identity store gets compromised? Won’t they have all of my data? And my associations with other sites? The answer is “yes”. A compromise of the identity store would cause a lot of headache. At the same time, the keeper of the identity store knows that. When is the last time you’ve heard about Facebook having a data breach? Just like banks are serious about security due to public perception of them being robbed, so too would an identity service suffer if it were to be compromised.
Ask yourself this question: would you rather trust your data security to a company like Facebook, who has everything to lose if they are compromised? Or some app developer in a hurry to create a login profile that forgets to use encrypted passwords or salted hashes?
The amount of times I’ve had replacement credit cards issued because of potential data hacks is becoming annoying. I’m almost certain my data was in the OPM hack even though it was something from over a decade ago. It’s becoming irritating to know that my information is just a brute force attack away from being exposed and I have no control over it.
Making Facebook or similar identity brokers the authoritative database for identity is an imperfect solution. But it’s also the best solution we have today to the escalating problem of data breaches. I’d rather trust that Facebook is doing as much as possible to safeguard my data than believe for an instant that Home Depot cares about their preferred customer database over the credit card repository. Facebook may not be the most upstanding organization when it comes to data analytics, but I trust them not to treat my identity like a steaming pile of toxic waste.