Scott Hanselman yesterday posted a piece on how easy it is to become what I call a Search Engine Developer. His post really hit home with me as I've done a lot of research for various development challenges in my last few work weeks where Google was a mainstay. I don't know about you, but I find myself using an ever increasing amount of intellectual content by searching for solutions online, which is a change from how I used to work not all that long ago. That doesn't just mean finding code and cutting and copying, but also searching for ideas and problems solved and applying that knowledge to my particular programming problems. Nevertheless, when I compare how I used to work in the days prior to the Internet and the mountains of shared programming knowledge today, it sometimes feels like I've become a slacker at best and a plagiarizer at worst as I don't have to stretch as far to solve common and not-so common problems today by using a search engine and the many developer resources available on the Internet.

To put this into context, here's a good one to ask yourself: If you had a job interview tomorrow and had to answer a bunch of heavy duty technical questions without Internet access, could you do it? To be honest, I'm not so sure I could - I've come to rely on being able to look up information online whether it's in search engines, code repositories or even my own blog. The concepts and ideas are all in my head, but many of the details and implementation issues often are not. I commit very little detail information these days into permanent memory it seems. My actual memory retention rate is probably very low outside of the commonly used stuff I use day in day out.

I suspect I'm not the only one…

How did we get here?

But it's not always been this way though at least not for me.

I've been doing software development for nearly 30 years now (scary thought that!) and I can remember the times back then when I didn't have any online resources. The only way I learned stuff was by going to school, reading a book or magazine article (lots of that for me) or going to a user group meeting to discuss programming in person with a few other like-minded souls. In those days there was a lot of 'discovery' on my own, even of small and common things, simply because there was no easy way to look things up. The 'offline', off-memory storage simply wasn't there - you either had it in your head or in a book or magazine buried in a pile at the back of a closet somewhere. :-) Sounds like fun dunn'it?

In a way looking back at it, it was fun. You really *had to* learn stuff and figure it out on your own very often - there was no easy cheating. No going online to snake a RegEx expression for validating a phone number, or finding an easy to use SMTP library in FoxPro, or easily do something as simple as base64 encoding in C. You had to sit down and figure it out on your own.

This went on for a long time too - it wasn't really until the end of the 90's that more development content than documentation started coming online. Most of it came from big vendors with product documentation, published articles from classic print magazines as well as the messy content from various developer forums that was difficult to wade through.

Then in the early to mid 2000's things changed as blogging started getting popular, and that's when online programming content really started taking off. It was free form user participation and the ability to pick your own topics to write in detail, about often complex technical content, that really got massive amounts of quality content online and started driving the Search Engine Developer paradigm.

Then in the late 2000's we started seeing more collaborative sites like StackOverflow for question and answer style conversations, and the proliferation of source code sharing sites like GitHub, CodePlex, BitBucket etc. that really brought code sharing to the general masses.

Today it's easy to find solutions to a fair percentage of programming problems that help both for troubleshooting and the learning process. If you look back and think about how far we've come especially in the last 10 years, it's pretty amazing how much of an impact the amount of online programming information and the ability to search it has brought us.

Hail the Search Engine Developer

One of Scott's points in his piece is that it's easy to get lazy and lose some of your edge by being purely a search engine developer. There's definitely something to be said for creating something from scratch, learning from the experience, pushing your knowledge limits, the thrill of discovery and for working through an idea from concept to completion on one's own.

But, this sort of coding seems to get less and less common, as there are fewer solutions that have to be worked out from scratch like that. I know it's true for me - I used to build lots of components and utilities from scratch. I still do to some extent but not nearly as in the past. Today a lot of those kinds of things are much more easily picked up through some utility code in a shared library or code snippet found online and modified and directly integrated into code.

Now I'm not suggesting that pure cut and paste or library integration is always great idea. It's always a good idea to understand the code you're integrating to some extent.

But to be honest - especially with libraries that's not always the case. When I look for a QR reader library to integrate into an application, I'm not going to ask too many questions on how it works for example. OTOH, if I find a short code snippet for integration I usually spend a bit of time experimenting around with that code and usually end up modifying, customizing and abstracting it before integrating to mitigate the 'external code' aspect.

The obvious advantage is that in that process you understand the concepts involved and the code you're working with isn't just a black box. I think that's useful when already so much of the code we use from vendors and tools/libraries that we have very little control over, even if source code is provided. While the QR library I used recently might be open source, I have no innate desire to dig into that source since I have no particular interest or even the background to deal with that sort of interface.

We may joke about the Search Engine Developer and Cut and Paste Development, but the truth is that we are much, much better off NOT having to reinvent the wheel for all these little programming problems. While it may not be very complex to build some string translation routines that extract text easily with a few parameters, even simple code like that that is reliable and works in many different scenarios does take some time to create if you had to do it from scratch. Is it be better to create it from scratch or use somebody's (hopefully) tested and already written solution? I think we all know what the answer to that is, ideals be damned. It's perfectly acceptable to not reinvent that sort of wheel. Getting some of the trivial things taken care of by searching and *adapting* code found online lets us focus on things that really matter in an application rather than the mundane plumbing code that is necessary, but not necessarily a key piece of the application.

With all this shared code available we have more time to build new solutions, come up with new ideas and expand the wealth of knowledge that is already out there. As they say, we can stand on the shoulder of giants to extend the reach of our skills even further.

It's fun to wax nostalgic about 'back in the day' stories, but I don't miss those days one bit. I don't think anybody does - having more information, at least for our jobs as software developers is definitely beneficial and it allows us to focus on doing the work in our problem domain and leaving the little things to solutions that have been solved a million times over.

As one commenter on Scott's post - Nathan - pointed out:

"Never commit to memory what can be easily looked up in books"
   - Albert Einstein

In today's terms, Einstein's books are the Internet. It's much easier to 'store' and access information there. Whether you write it down for later retrieval - in a blog post perhaps - or whether you search, there's no shame in using the Internet as a retrieval source and an extension of our somewhat limited memory store.

'Already been done' Syndrome

These days I find myself doing much less development truly from scratch. I'd like to, but the reality is that it's getting much harder to justify and build something truly unique. The Internet is a vast place, and there's so much stuff out there already that's already been done. If you think of some sort of problem you need to solve, chances are pretty good that somebody's already thought of it and has tackled that problem and solved it.

It's also very easy to fall into the trap that that that's already been done. In some cases that's a great thing - if it's something trivial or non-relevant to your main problem domain then that's already been done is a godsend.

In some cases however, it can also lead to giving up on good ideas. If you have some smart new idea and you check around on the Internet to see if anybody else has done this already, you may find a previous implementation and simply decide it's not worth to build your great new idea out. The problem with this thinking can be that some good ideas are not getting the benefit of alternate and possibly much better implementations. An opportunity lost.

I know there've been a few things I've wanted to do in the past that I didn't, simply because there were already similar solutions out there. Even if those other solutions weren't perfect and I felt that maybe I could do better, it's hard to get motivated with existing solutions out there and then playing catch-up and probably ending up having to compete with an incumbent. I suspect a lot of new development never happens because of this.

But sometimes it definitely makes sense to reinvent a wheel and do a better job. We've all seen really bad libraries or applications out there that could benefit from somebody with a different point of view and maybe more dedication taking a stab at it.

How are your Search Engine Skills?

If you are a good developer today, you have to be good at using search engines to make yourself productive in your work. Being good at searching and finding answers to development questions is one of many critical skills these days. Today's software development involves so many technologies, tools and environments that there are very few - if any - people around who can keep it all in their heads. Information overload is a problem and you probably have a core set of that you use regularly that are always 'just there', but for the rest of it you might as well use The Google to help you jog your memory.

I'm always amazed however that a lot of developers are not very good at coaxing useful information out of search engines. I often feel like sending people off to Let me Google that for you. In this day and age, especially with the wealth of cumulative Q&A knowledge available in StackOverflow there's no reason not be searching efficiently and finding answers to a lot of developer information. Yet I still find developers who are not actually managing to get useful information out of search queries.

There are several things that are critical here. You need to be able to:

  • Find the right keywords to search on - get familiar with advance search options
  • Narrow down search results
  • Differentiate the good from the bad results
  • Use what you find responsibly - learn from the code

One important point about any code you find is that cutting and pasting code blindly without actually understanding it is a recipe for disaster. Always play with code you find and learn from it, then integrate. I find it's often a good idea to review the code then implement it by typing it in (preferably without peeking at the original) instead of cutting and pasting. This helps understanding and also retention of the code that was just snatched and integrated into an application. For more complex pieces like full libraries that's not always an option or even desirable, but especially with shorter solutions like stuff you find on StackOverflow this is good advice.

The big problem with sites like StackOverflow or tons of open source code available to plug in, it's not easy to avoid the simple solution of just using cut and paste or plunking in a library and forgetting about it. And unfortunately that temptation often results in untested or misunderstood code getting integrated into solutions. Bottom line is: make an effort to understand what you're integrating.

Best of both Worlds

We live in an information rich world - it's not 1995 any more and the Internet and the ability to search its vast resources are here to stay. We can choose to stay sharp and build our skillset the old fashioned way and as Scott so frequently suggests we should Sharpen the Saw to keep learning and improving on what we already know. Search engines are another powerful tool in our arsenal and we can and should let them help  us do our job and make that job easier. But at the same time we shouldn't let them lull us into a false sense of security - into a sense of thinking that all we need is information at our fingertips. Mad Skillz is still a desirable feature for the modern developer and keeping up with those skills is an important part of being a developer.