• Part 1 (this one) Part 2 Part 3 Part 4

Memory Triggered

I started a great new job a month or so back (more about that another time), but noticed a date go by that I recognised. It took me a little while to cotton on, then I realised it was my CCIE anniversary. 

You tend to sit up and take notice of these anniversary dates as an active CCIE if they coincide with having to do (yet) a(nother) re-certification, but this time around: all-clear – it was 21 years.

I’d noticed someone post up about their 20^th CCIE anniversary recently too with a few memories, so I thought – hey – it’s my 21^st, I don’t have to do a recert, so let’s roll out the reminiscing and take a peek backwards. I hope I remember most things vaguely right! It’s taken a little while to put this together – sorry it’s not the actual date, but instead it’s going out about 6 weeks after my 21^st Anniversary!

In the UK (and elsewhere), a 21^st birthday is often marked as your “key to the door” although it’s a bit of an old-fashioned saying.  The phrase appears to have a couple of possible origins: based on the story that if you are still living at home as you reach your 21^st birthday then you’d get a literal key to come and go from the house, or that it’s your ‘coming of age’ where you gain your final key adult rights (like voting (up until 1970 in the UK it was 21 not 18), and in the US it still marks your legal drinking age).  In Britain, the only thing now that now seems to happen on your 21^st is that your minimum wage goes up and you might get (non-explicitly) binned from your job for someone younger and cheaper (depending on your industry (& employer!)).

Anyway, I digress.

I was talking about keys. Getting my CCIE at the time that I did was probably the key to my whole career… 

Why do a CCIE?

I’ll talk another time about recollections of the networking industry before the turn of the millennium, but long story short is that I was working at AT&T and Cisco kit was beginning to get really popular with our customers. It offered great functionality, and both the engineers and customers liked that, so we ended up hacking lots of Cisco networks together.

The CCIE certification had been around a few years (since 1993), but as Cisco market share grew it became well recognised as a solid-gold recognition of someone’s expertise in the Cisco portfolio. 

That CCIE reputation was powerful, you could onward sell it to your customers showing off the skills of your technical staff, and for the engineers themselves, they’d often (but not always) relish the challenge and the opportunities it then opened up…

In recognition of this, a small cohort of the UK AT&T staff volunteered to do our CCIE certs around the 1999 / 2000 timeframe, and away we went. Multiple weeks of expensive residential training, doing CCNA & CCNP in short order, and you’d duly set up a stack of routers/switches at home to lab on.  One of my training sessions (in a hotel in Watford I think – oh the glamour!) was blessed with a solar eclipse, so we put down our console cables and went outside with ‘Heath-Robinson’ pinhole cameras and sunglasses to see it, rather than wait on the UniverCD to load 😉.

Booking the Exam – setting the lab date in stone

Getting the CCIE lab booked was one of the trickiest parts.  The lab testing was only held in three or four locations around the world at the time and was a full 2 days long, which really limited the number of people who could make attempts.  The waiting list was months, so you had to build up your preparation based on the test dates that you could book in advance. You’d want to make sure you could get onto a CCIE ‘bootcamp’ in the few weeks before your test too.  As with any large company this was a bloody nightmare to arrange and coordinate, but I got set for my late February 2000 trip to a grey Brussels suburb.

As well as contending with the waiting list, the CCIE practical lab was also notoriously hard (and still is to this day). This was another real reason for the bootcamp – to maximise your chances of coming away with a pass and avoiding another 4 months of waiting to have another go, losing all your momentum.

Rumours abounded that the pass-rate was between 10 and 25% depending on what you read and who you spoke with – and Cisco didn’t appear to deny this or give a concrete response.  You knew you had to prepare well to minimise the risk of going round the merry-go-round for another significant chunk of a year.    

Luckily for me the rest of my CCIE cohort at AT&T were slightly ahead due to some of the customers I had been working on, so this helped as I got some insight as to what was coming.  Martin Shortland’s two-week residential bootcamp was excellent, but lots of people knew that so there was a waiting list there too!  On it we fed, we labbed, we fed, we labbed and the cycle continued until late into the evenings even though there was a fully stocked bar at a nice hotel in the country.  The bootcamp was taken pretty seriously, without the raucousness of a normal residential course. I didn’t pay much attention to the venue at the time but having just looked up where I think it was: www.newplacehotel.co.uk we probably missed out.  We were fully immersed in our own little San-Franciscan world, trying to learn literally everything there was to.

Preparing for the Exam

The CCIE was (and still is) a two-part affair – the written part (which some of us old-timers still call the ‘Drake’, even though that part of the testing firm’s name was already long gone when we took it, and the practical lab.  Passing the Drake would start a 12-month eligibility timer in which you needed to pass the lab (the part that put the fear of god into most of us (see that pass-rate above)). The written part expiring before getting a lab booking due to the backlog was a real problem for some, especially for re-takers.

For the written part – this was a couple of hours in a testing centre on a very, very poor CRT screen running what appeared to be CGA resolution.  The only thing that’s changed is that the very, very poor CRT has now become a very, very poor LCD. It’s always a good idea to sanitize on the way out after using those keyboards <shudder>.

The practical lab itself, when I did it, was in the concrete Belgium jungle and ran over two-days.  This format changed the year afterwards to a 1-day affair which went some way to reducing the wait-time, more than doubling capacity, but growing popularity soon put paid to that and they had to expand the numbers of test rooms and locations to cope with demand. 

• 

I’ll admit it – there’s a bit of a snobbery about people who did the two-day test versus those who ‘only’ did the one-day version, even though the thing is still as difficult.  There’s an additional set of snobbery too – based on ‘4-digit’ CCIEs.  As everyone on the CCIE path knows, Cisco started numbering people passing the certification from 2¹⁰+1, so nobody has fewer than four digits, but most now have five and it looks like it’ll break into the sixes in this decade.

Me? I’m a snob two times over… and I have a 4-digit 2-day CCIE too 😉 (sorry for the spoiler).

More coming up in Part II, this is getting a bit long!

• Part 1 Part 2 (this one) Part 3 Part 4

Hunkering Down

People often ask how I prepped for the practical lab, as it still exists in this form today. As well as the training and bootcamp (this was before the days of INE etc.), I had dedicated 6 weeks off work before taking the exam. This was immediately after the turn of the millennium, so one of the last things I did at work before was to be on-call for the Asia-Pacific region for Y2K, and their non-calamitous dip into the 21^st century.

In true procrastination style for the final-push a revision timetable was created to prepare a programme of work to lab through.

As mentioned in the previous section I had a stack of kit running for about 5 months, which I managed to fit onto the top shelf of a built-in wardrobe – leaving the door open for ventilation when it was running.  It was important to be able to ‘switch off’, so powering-down and closing the door temporarily gave some respite. It was always on your mind though. I was living in a shared house at the time, so I’d like to take this opportunity to apologise to Kath whose room backed onto the other side of the wardrobe!

The stack comprised a load of 2500 series devices, including the seminal 2511 which is still doing the business today as a console server, all borrowed from stores, maxed with RAM and flash upgrades.

Interconnection was via Ethernet, Token-Ring, X21, V35 and RS232 (although no FDDI!). I think there was a switch, but to be honest switching wasn’t an enormous part of the test – there were many more exotic things to do on routers like learning Banyan Vines (although I don’t remember that being on the syllabus if I’m honest!). I went through all the testing specification and the documentation CD, learning and labbing pretty much all of it.  You could be tested on almost anything that was documented in here. 

One of the key characteristics needed to pass CCIE, as it is today, is the speed of your work.  I saw plenty of really good people through the years fail to get over the line as they weren’t able to move accurately at pace.  The restriction of the speed of the documentation CD at the time was a real-limiting factor too, and it was drummed into you that you should memorise everything, with this only to be used in the lab as a last resort.

Travelling to the Test

I arrived in Brussels the day before the test, staying in the same hotel that most of the test takers used, and I did a recce to the Cisco office so that I’d be ready for the next day (no Google Maps in those days, nor phones that could even have run it!).  Then a bit of light reading/revision, something to eat, setting a couple alarms and enjoying a broken night of sleep before turning up to the lab the next morning.  My memory from here starts to go a bit hazy – it was quite a stressful experience.

Format of the Lab

The way the 2-day lab would work was that on the morning of Day 1 you’d firstly sit down with your test paper and do a network design with an IP addressing plan etc.  I can’t remember if this was on paper or electronic, but this would be assessed/marked (worth 1 point) with the rest of the questions.  All of day 1 was IP (and setting up the underlying transport), whereas day 2’s morning was designated ‘multi-protocol’ and the afternoon ‘troubleshooting’.

The score was out of 100 as I recall, with a passing score of 80. If you dropped 20+ marks on day 1 or in aggregate with the morning of day 2, making it impossible to score 80 or above, you’d be binned. Only those who had the possibility of passing would start the troubleshooting section on the afternoon of day 2 so attendance thinned out.  The exams were hand-marked, section by section by the proctor at the end of each session, so you’d have to wait to find out how you had done (and if you were coming back).    

Day 1

Everyone taking their CCIE lab congregated in the break-out / canteen awaiting our start.  The bootcamp was pretty good in making sure that the environment wasn’t completely alien, so I was happy for that. However, I did seem to remember having a worry that we’d have IP phones ringing and people test talking into handsets – but thankfully this didn’t happen – perhaps that wasn’t on the syllabus by then.

There ended up being 6 people in my lab, and I think (told you about the haziness!) there was another lab of 5 – so eleven in total starting their CCIE lab that day.  My abiding memory here was that it was true that you could just help yourself to cans of Coke, which was important to keep a non-tea/coffee drinker sharp!

Day 1 morning went well, I was able to set up the network and get a good number of the questions ticked off. I was on track.  We all sat around the breakout area with sandwiches and buffet food for a common lunch, not saying much.

Panic

About a third of the way into the afternoon of Day 1 I knew I had a big problem. I could not for the life of me get my OSPF routing protocols fully up and running across my Frame Relay network. Everyone who did the lab around this time will instinctively know what the issue is from what I’ve just said.  This is due to it being drummed into everyone incessantly – about making sure you add the ‘broadcast’ keyword where necessary, depending on the OSPF link types configured at each end and the medium.  Except I didn’t remember it in that session, and I watched the time drain away. Also draining away was the colour on my face.  I started breathing heavily and getting dizzy. I informed the proctor I wasn’t feeling great, and went and had a seat next to my friend, the free drinks fridge.  My CCIE, and all the preparation that had gone into it felt like it was slipping away.

I managed to pull myself together, got back into the lab – did some other questions and took another look and still couldn’t work it out.  I checked it dozens of times and just got nowhere.  I stuck my hand up again, the proctor came over and I said I thought I was experiencing a bug. He ran a few commands and told me there was no bug.  I pored over the command history to try and work out what he’d checked to point me in the right direction, but I’d blanked the fundamental advice which had been drilled into me.  It got to the end of day 1. I felt certain that I’d failed at the first hurdle. I was gutted.

How did it turn out? Tune in to Part III…

• Part 1 Part 2 Part 3 (this one) Part 4

Day 1 Verdict – Half Way Stage

I awaited my fate at the end of day 1; I was late in the marking list and downbeat (see part 2 for why). Eventually I got to sit down with the proctor as he ran over the checks and marked the questions, and the missing part of the map command was laid out bare for me. I was gutted, all-over again. But… even though I’d dropped a fair few marks on this question and its dependencies, I’d nailed everything else, so I was still in the game.  Three in my lab were not.

It was a quiet evening in the hotel.

Day 2 Morning –  Multiprotocol

A quieter lab, only 3 of us were still there – the others hadn’t made it past day 1.  Brutal. The first thing I did was to instate the missing ‘broadcast’ keyword in the relevant places in my configs, and ping, ping, ping – everything was properly up and running. For the rest of the morning it went really well, with the only thing I couldn’t get working being an SNMPget for an undocumented MIB entry, which was irritating (it’s 21 years ago – so I’m not giving away any lab secrets with that!). 

Afterwards, again, we sat and waited for the proctor to mark our work and deliver his verdict. I’d lost a couple more marks due to the non-working SNMP stuff and something else very particularly worded in the questions (the nuance is very important and the Cisco questioners are renowned for their trickiness here). The results were that I remained in the running, albeit I’d need a pretty damn good final afternoon’s troubleshoot to pull it off.

It was then that I found out that one of the guys in our morning lab had actually failed his first day. He’d been allowed, at the proctor’s discretion, to have a go at the 2^nd day’s morning session for unmarked practice. He therefore bowed out and went back to the hotel (the other three hadn’t even come back to try).  The only other remaining candidate in the centre after lunch worked for Cisco, so it was just us two who remained going into the final session.

Day 2 Afternoon – Troubleshooting

I knew I couldn’t afford to drop many more marks given the ones already confirmed gone. The proctor in advance had loaded the ‘faulty’ configs onto the devices in the lab and set us to work. There were all sorts of issues here, log messaging going bananas, incorrect/duplicate IP addressing, OSPF mismatches (not going to get me again!), serial cables forced in upside-down, password recoveries required etc. Basically, everything that you’d done over the last 2 days had been broken, and you had a couple of hours to fix it all. 

I saw panic both in the eyes and actions of the Cisco candidate quite soon after the start.  You have to work really quickly to resolve the myriad of problems inserted into the network, but he kept going through to the lab equipment room and messing, costing him time. I tried to ignore it.

Again, it was a pretty good session, settling things down overall in the first hour and then working on the details piece-by-piece.  I thought I’d pretty much got everything up and running again, except for a pesky ISDN ppp call-back that just didn’t seem to want to play. I thought it’d be borderline with those marks lost, plus anything else I’d not spotted.

We again waited in the canteen to be called in for marking – and I was able to have a chat with the Cisco candidate.  He confirmed what I’d seen him do and he already knew his fate.  I was honestly gutted for him – we’d gone into battle together and he’d been doing really well up until that point – but the troubleshooting had thrown him completely off. It seemed common at the time for more ‘theoretical’ candidates with less practical hands-on day-to-day troubleshooting experience to struggle here.

Results Time

I was called into the lab and sat with the proctor as he went through the devices.  I remember asking if I’d wasted time reinstating my original IP addressing scheme from my original design.  He confirmed I had.

And then – the ISDN call-back worked straight off the bat.

I think I ended up with about 84 marks. And, safe-to-say, I was overjoyed.

Swag & My Number

The other members of the UK AT&T lot who had passed their CCIE came back with a cap.  The proctor applied for your CCIE number when submitting your marks, and then wrote that inside the hat – so you were able to tell your boss, your better half, your mum, the UN etc. your ‘CCIE number’.  But they’d run out of hats.  I got my number on a bit of paper, and a small holdall printed with the laurel leaf on it. 

I was really elated with the pass and my having recovered from the day 1 mess. So I attempted to stagger back to the hotel to make sense of it.

How to celebrate? Beers for one? A final wrap experience in Part IV…

• Part 1 Part 2 Part 3 Part 4 (this one)

Pass – Back to the Hotel

I wandered back towards the hotel with the realisation that I was the only person who had passed in those 2-days.  That’s when I came back to earth. Here I was – on top of the world, genuinely bursting with the joy of passing something so tricky, going back to an identikit hotel on an industrial estate knowing there was no one there to share it with – all the others having failed.

In the end I only ever saw one of the other people from the labs again. I bumped into the French chap who had come back on the 2^nd morning for the practice, and I shared my good news with him.  We sat at the bar in a massive, airy hotel lobby clinking beers, experiencing a weird awkward mix of feelings as he hadn’t had the same success. As we chatted he shared that he’d largely come to terms with it, and was genuinely pleased for me.

I don’t know if he knows how much of a friend he was to me right there and then, it would have been a very lonely hotel that night otherwise. I wish we’d stayed in touch… Did he go back and try again and get through?  We ate together that night – I remember having fish.

Return to Work

My CCIE certification number meant I was probably one of about 4,000 or fewer people at the time who held the active certification.  Looking for stats there were websites (including Brad Reece’s and the ‘CCIE Hall of Fame’) that gave rundowns on where in the world CCIEs were based to give some sort of flavour.  At the time I think around 10% of CCIEs were based in the UK, so I was one in a few hundred. Given my relative youth I also figured I’d be one of the youngest.

Work took care of us, we got genuinely chunky pay rises and a bonus to say congratulations on passing. They got to use our CCIE status to get their kit discounts and marketing messages out, so it kind of balanced out. I also received a Montblanc pen, which I was too scared to use as it was about £200 or something (for a pen!).  I’ve no idea where this is now – one day I’ll open the right box and it’d be interesting to see if it still works after 21 years of stasis, but given it’s price, I say it should!

Key to the Door

Achieving my CCIE undoubtedly played a pivotal role in the opportunities and doors that opened to me, and it has been key to a significant part of my career.  I stayed with AT&T for another 5 years before spreading my wings, and since then I’ve spent time in many sectors contracting, in R&D and in MSP and Enterprise IT leadership roles. Even 21 years on it can still get me towards the top of the list when I’m seeking a new role – it still holds that cachet.  To this day I see why CCIE is still attractive and highly sought after.

Recertification across so many years has been a bit of a chore, but overall more than worth it.  I decided not to go for emeritus, but deliberately stay active on the CCIE track. Rather than make this 4-parter even longer I’ll do another blog on recertification, and perhaps my thoughts on the CCIE as a whole and its future.

In my new role I’ll be drifting away from the core of CCIE, and as I mentioned I’ll give some more thoughts to this new world, putting fingers to the keyboard to talk about it future postings too – I look forward to writing about this as there’s genuinely a seismic shift for network engineering going on.

Postscript

I still have one thing about CCIE after all these years that still needs closure. I never did get a personalised CCIE leather bomber jacket, with my name and number sewn into it.  One day, one day – maybe I’ll treat myself on my 25^th anniversary 😊