Monday, March 3, 2008

The Power of Skunkworks

When you think of the long and gloomy history of man, you will find more hideous crimes have been committed in the name of obedience than have ever been committed in the name of rebellion. -- C. P. Snow

Or what about the definition of insanity? Doing the same thing over and over, but expecting a different result.

Sometimes underhanded measures are the best. I would never recommend making a habit of it, but there are times where obeying the letter of your job description (or contract) is the worst thing you can do for all involved - yourself and your client. As a programmer, architect, or manager of programmers, it is your job to create software that fulfills a business purpose. But sometimes the forest gets lost for the trees. Sometimes the people in charge have been given bad advice, sometimes they have pre-conceived notions about how things must be, or sometimes they are just plain incompetent. At times like these, desperate measures are called for. Fork an alternate reality where software makes sense. Skunkworks.

Some Valid Reasons for Skunkworks

The existing technology would be far more complex to change than to just rewrite. For example, Boss won't allow it, since he thinks because it took his 16 year old "genius" nephew 6 months to write that PHP script, it will take you just as long to rewrite it. Ignore him and crank your own code in a couple days.

Management wants to purchase an expensive system, where you know a perfect FOSS replacement that is easier to manage and free. You asked, but they claim the OSS system isn't "enterprisey" enough - you point out that the Brazilian Healthcare System runs millions of hits a day, and you're only looking at hundreds. Boss still refuses. Try it out, prove it works, and make the move. When you encounter resistance (and you will) keep escalating the issue. I've done this - moving up to the CIO before my change was accepted. Upper management is always happy to save money. I even let my boss claim it was his idea, just to save face.

An out-of-touch former engineer who designed the system (and happens to now be in charge) irrationally insists the system he wrote 10 years ago was perfect and nothing needs changed. You have inherited said system and all headaches contained therein. This is a tough sell, but again: rewrite, prove it works, escalate until you get the answer you want. The average sale requires seven contacts before something is actually sold. The average salesman stops at five. Persistence is shockingly effective.

Methodology; Least intrusive to most

Add new code
Inject alternate chunks of code, and some sort of switch to decide which will be executed. It's simple enough, allows a lot of reuse, but only really works if no-one is paying attention to the codebase.

Fork the code
Create an alternate trunk and make changes in that. If you use the same SCM it may make merging the code easier in the future.

Create a new project
Obviously the biggest endeavor, since this is effectively starting from scratch. But sometimes you can create a new project in a few days, where it could take weeks or months to hammer their square changes into the existing round system.

While at LPA me and a guy from Amentra were told to make the Blaze Rules Engine to work in ways it was not built to be used. After banging our heads on it for a month, countless conference calls with Fair Isaac, and general disgust with the complexity - I spent a weekend learning Drools and replaced it. It was perfect. Just what we needed. I switched their coffee with mine, and they never noticed. I told them a week later what happened. Told my boss. Then his boss. Several conference calls later my system was officially in - Blaze was out, saving our company $250k/year in license and support fees.

Fire people
This isn't really skunkworks, except for the fact that you may have to hide the real reason for firing someone. So in that case it isn't "official". This is damn intrusive. Also only works if you're a project manager. Sometimes the problem isn't the system - it's the people. Start over with "smarter" people. Probably won't work. Last resort.


Huzzah, hurray, congratulations! You've succeeded in keeping your project on the down-low, proved it worked, and moved up the chain until you got the greenlight to use your elegant brain-child. Now you have an easier life, and your clients get their features that much faster. Epic win. Everyone's happy, right? Fat chance. Someone's toes got stepped on - someone is is going to be pissed off.

The idea of skunkwork'd projects lies in the fact that it's easier to ask forgiveness than permission. If you ask beforehand, you are likely to be turned down. But if you do it and get caught, you can always beg forgiveness. Downplay your plan. Act meek. When your architect finds out what you did behind his back, tell him that his system was great, you're just more comfortable working with yours - you used it as a learning experience. When your project manager is angry at the hours you spent, say it took only a couple hours, and you worked on it over the weekend.

The truth is, unsolicited projects are dangerous. Only attempt them if you are confident in your 1337 h4Xing skillz. If you do it, you WILL be noticed. Eventually. This can be a good or bad thing. This may be a feather in your promotion-cap, or you may be signing your own pink-slip. But to quote Ferris Bueller: Only the meek get pinched. The bold survive.


Thomas said...

Sounds like fun times.

Mark Proctor said...

hehe, you made my day.


PS don't forget your Drools subscription, so we can keep up the good work :)

Mark Proctor said...

Btw blogged you here:

You might want to shed a bit more light on your problem, so that people don't assume it is your lack of understanding of Blaze. The BRMS/BRE guys are a bit elitist ;)


Eric said...


Mark, that is entirely correct. It isn't that there was something wrong with Blaze, it was just an example of overkill for our needs. Our CIO bought into the sales pitch and threw a huge budget at this rules engine that was far more than what we needed it to do. We needed the engine to be a component of our system, not the other way around which is exactly how BRE is set up - to be built into, not around.

We had several conference calls with the developers, and the components we required to hook into were not available as an API. When they started suggesting we access unpublished internals to get the job done - that's when I started looking elsewhere.

Again, nothing wrong with BRE - it was just far more than we needed, which was the real point of my post. Also, this was 3 years ago - things may be different now.