In my 11 year career as an IT instructor, I’ve had to pass a lot of certification exams. In many cases not on the first try. Sometimes for fair reasons, and sometimes, it feels, for unfair reasons. Recently I had to take the venerable Cisco CCNA R&S exam again. For various reasons I’d allowed it to expire, and hadn’t taken many exams for a while. But recently I needed to re-certify with it which reminded me of the whole process.
Having taken so many exams (50+ in the past 11 years) I’ve developed some opinions on the style and content of exams.
In particular, I’ve identified some types of questions I utterly loath for their lack of aptitude measurement, uselessness, and overall jackassery. Plus, a couple of styles that I like.
This criticisms is for all certification exams, from various vendors, and not limited to even IT.
To Certify, Or Not To Certify
The question of the usefulness of certification is not new.
One one hand, you have a need to weed out the know-its from the know-it-nots, a way to effectively measure a person’s aptitude in a given subject. A certification exam, in its purest form, is meant to probe the knowledge of the applicant.
On the other hand, you have an army of test-dumping dullards, passing exams and unable to explain even basic concepts. That results in a cat-and-mouse game between the exam creators and the dump sites.
And mixed in, you have a barrage of badly formed questions that are more appropriate to your local pub’s trivia night than it is a professional aptitude measurement.
So in this article I’m going to discuss the type of questions I despise. Not just because they’re hard, but because I can’t see how they accurately or fairly judge a person’s aptitude.
Note: I made all of these questions up. As far as I know, they do not appear on any certification exam from any vendor. This is not a test-dump.
The story goes that Albert Einstein was once asked how many feet are in a mile. His response was this: “I don’t know, why should I fill my brain with facts I can find in two minutes in any standard reference book?”
I really relate to Einstein here (we’re practically twinsies). So many exam questions I’ve sat through were pure pedantic trivia. The knowledge of the answer had no bearing on the aptitude of the applicant.
Here’s an example, similar to ones I recall on various exams:
What is the order of ink cartridges in your printer? Choose one.
A: Black, Magenta, Cyan, Yellow
B: Yellow, Cyan, Magenta, Black
C: Magenta, Cyan, Black, Yellow
Assuming you have a printer with color cartridges, can you remember the order they go in? Do you care? Does it matter? Chances are there’s a diagram to tell you were to put them.
Some facts are so obscure they’re not worth knowing. That’s why reference sources are there.
I can even make the argument about certain details about regularly used aspects of your job. Take VRRP for example. For network administrators, VRRP and similar are a way to have two or more routers available to answer to a single IP address, increasing availability. This is a fundamental networking concept, one that any network administrator should know.
VRRP uses a concept known as a vMAC. This is a MAC address that sits with the floating IP address, together making a virtual router that can move between physical routers.
So far, everything about what I’ve described about VRRP (and much more that I haven’t) would be fair game for test questions. But a question that I think is useless is the following:
The vMAC for VRRP is (where XX is the virtual router ID):
I’m willing to bet that if you ask 10 good CCIEs what the vMAC address of a VRRP is, none would be able to recite. Knowledge of this address has no bearing on your ability to administer a network. How VRRP works is important to understand, but this minutia is useless.
I have two theories where these questions come from.
Theory 1: I’ve written test questions (for chapter review, I don’t think I’ve written actual certification questions) and I know it’s difficult to come up with good questions. Test banks are often in the hundreds, and it can be a slog to make enough. Trivia questions are easy to come up with and easy to verify.
Theory 2: Test dumpers. In the cat and mouse game between test writers and test dumpers, vendors might feel the need to up the difficulty level because pass rates get too high (which I think only hurts the honest people).
Another one I really despise is when a question asks you for the exact command to do something. For example:
Which command will send the contents of one directory to a remote server using SSH?
A: tar -cvf – directory | ssh email@example.com “cd /home/user/; tar -xvf -”
B: tar -xvf – directory | ssh firstname.lastname@example.org “cd /home/user/; tar -xvf -”
C: tar -cvf – directory > ssh email@example.com “cd /home/user/; tar -cvf -”
D: ssh firstname.lastname@example.org “cd /home/user/ tar -xvf -” > tar -xvf directory
For common tasks, such as deleting files, that’s probably fair game (though not terribly useful). Most CLIs (IOS, Bash, PowerShell) has tab completions, help, etc., so that any command syntax can be looked up. Complex pipes like the former are the kind I use with some regularity, but I often have to look it up.
The Unclear Questions
I see these in certification tests all the time. It’ll be a question like the following:
What are some of the benefits of a pleasant, warm, sunny day? (Choose Three)
- A: Vitamin D from sunlight
- B: Ability to have a picnic in a park
- C: No need for adverse weather clothing
- D: Generally improves most people’s disposition
Look at those answers. You could make an argument for any of the four, though the question is looking for three. They’re all pretty correct. Reasonable people, even intelligent, experienced people, can disagree on that correct answer is.
Questions I Do Like
I try not to complain about something if I don’t have something positive to contribute. So here’s my contribution: These are test questions that I think are more than fair. If I don’t know the answers to these types of questions, I deserve, in every sense of fairness, to get the question wrong.
A scenario question is something like this: “Given X, what would happen”.
For example, if a BDPU was received on portfast enabled interface, what would happen?
If a host with an IP netmask combo of 192.168.1.10/24 was to try to communicate with a host configured on the same Layer 2 segment with an IP address of 192.168.1.119/25, would they be able to communicate?
I like those types of questions because they test your understanding of how things work. That’s far more important for determining competency I think.
There are some network basics, that might seem like trivia, but knowing would be important to know. For example:
What is the order of a TCP handshake?
A: ACK, SYN/ACK, SYN
B: SYN, SYN/ACK, ACK
C: SYN, ACK/SYN, SYN
D: ACK, ACK/SYN, SYN
This question is fundamental to the operations of networks, and I would hope any respectable network engineer would know this. This would be important for TCP dump analysis, and other fundamental troubleshooting.
If you write test questions, ask yourself: Would the best people doing what this question tests get this answer right? Is it overly pedantic? Is there a clear answer?
This was mostly written as a frustration piece. But I think I’m not alone in this frustration.