When Data Mining Goes Too Far

Data mining is commonly used to find otherwise undetectable patterns in related sets of data.  However, sometimes these patterns provide absolutely useless information.  This story on FoxNews.com explains a recent correlation between teens who skip breakfast and those that lose their virginity earlier in life.  Interesting information, yes, but what exactly does a parent or other concerned party do with this?  Is it really valuable?

In the business world, I’m sure we could find examples of data mining “hits” that are just as obscure.  Many of them have value, but the real intelligence – I mean human knowledge, not some component of software – is being able to determine which patterns are actually useful.  Delivering bits of information such as the example above could lead to information overload and defeat the business purpose of data mining.

Now I’m off to evaluate whether I should force Cocoa Crispies into my kids to keep them abstinent :)

SQL Quiz: Two Mistakes

I was tagged by Gail Shaw to post two big mistakes made during my professional career.  The only challenge here was to narrow the list down to two :)

The first one is very easy.  During my early days working with SQL Server, I wore a lot of hats, including that of web developer.  One of my first major web projects was to create a student assessment system, which would allow instructors to create online exams.  Students would then be able to use the web interface to take the exams, and the manual paper grading process would be no longer.  Now I’m not ashamed to confess that I was underqualified at the time to effectively complete this process; as the sole developer/DBA, the entire project from spec to support rested on my inexperienced shoulders.  Nevertheless, I forged on and delivered the application on time, albeit untested.  The magic hour was the following morning, when a dozen educators were to begin entering exams on the new application.  I should also mention that I was still in college at the time, and was in class – over an hour’s drive away – during the critical go-live.

It probably goes without saying that my phone started ringing shortly after the first staff members arrived.  Problems were rampant, and I ended up leaving class to go address the issues.  I dodged the angry mob at the front door and managed to get in and take care of the most pressing issues so the test building could commence.  In the end, the application was made usable and found a niche where it worked pretty well.  However, I can’t help but wonder if this tool wouldn’t have gained more widespread acceptance if I had been more experienced at the time and had done a better job during development and deployment.

Lesson learned:  Admit when you’re in over your head, and insist upon a thorough testing cycle.

The second one caused me a good deal of embarrassment and cost me the better part of a day.  After receiving a report from an end user that a critical report had not been run on one of our main databases, I got with our hardware guys to arrange for a restore of the database backup file from tape.  Since this is a large database, it takes a few hours to copy over, but our backup guru agreed to copy the file directly to the development server to save another copy operation from live to dev.  I got the call a few hours later that the copy was complete, but I found only old files on the target directory (and deleted some of them, as part of a periodic manual cleanup).  I called our backup guy again and told him something had gone wrong and the file hadn’t been copied.  Always a good sport, he kicked off the file restore again.  When the call came that the process was again completed, I checked the backup directory and still found only old files, including one I had deleted earlier.  I made another call to our backup guy to find out what was wrong with the backup software, and simultaneously opened a window to the live database server backup folder.  As I was explaining to our backup engineer that he had made a mistake, I saw the filenames in the backup directory on the live server – which looked curiously like the files I had deleted!  The database backups on the live server had a different naming convention than those on the dev side, and I had recklessly deleted the restored file the first time.  A quick RESTORE HEADERONLY confirmed that I had just wasted a good part of my day, as well as that of one of our best hardware guys.

Lesson learned:  Before you assume someone/something else is at fault, make sure you’re not doing something silly to cause the problem in the first place.

SQL Quiz: Career Challenges

I’ve read a number of responses from Chris Shaw’s first DBA networking quiz.  I missed out on the first one, but I have been tagged by Grant Fritchey for the second round.

The Questions for this quiz…
What are the largest challenges that you have faced in your career and how did you overcome those?

1) The first one of these, I still laugh at when I remember it.  I got involved in the IT industry later in life than most (mid-20s) and found quickly that I had a knack for learning and applying new things quickly.  I was doing tech and sysadmin work and there was an acute shortage of those skills, so I probably received more praise and recognition than I really deserved at the time.  During those early days I started to imagine myself as the alpha ubergeek, and believed that I could be an expert at all things technical.  I started to learn programming, jumping from C++ to Java and Perl to PHP, then onto non-Windows system administration – Linux/UNIX and even a little OS/2, and finally database administration in MySQL, Oracle, and of course SQL Server.  I remember at the time thinking that I would be able to set myself apart as an expert on all these disciplines.  Need an enterprise application built?  I’m your guy.  It’ll be a web app?  That’s still me.  I’m also the database guy (architect, dev, and DBA), and I’ll do the sysadmin as well.  Oh, and I maintain the hardware too.  I actually created a schedule that encompassed about two years, and included time for me to self-train in each of these topics.  I wish I still had that schedule, which would now be good for a hardy laugh, but I can remember that I had allocated a mere three months to teach myself everything about both PHP and MySQL.  This story does have a happy ending, in that I realized the absurdity of my intentions before I got myself in over my head.  My youthful inexperience allowed me to convince myself that I could learn everything about everything, and could maintain this knowledge as the technologies changes.  Another positive result is that my study in these other disciplines gave me a cursory understanding of other technologies to which I might not have otherwise been exposed.

Lesson Learned: Don’t try to be an expert in everything.  Identify a few things that you enjoy and do well, and maximize your time in those areas.

2) This challenge is ongoing, but I’ve gotten much better at this, particularly in the past year.  I’m a big believer in hard work, and I have seen that a person who learns a craft that is in demand and puts his or her nose to the grindstone will do well.  However, when I think about the people that I perceive as successful, these are not people that simply work hard (although most of them do work very hard).  Those who are exceptional are people-persons as well.  They work to know their constituency, including executives, end users, and fellow technical staff, and are comfortable at explaining difficult concepts to all groups.  They are good enough at office politics so that they are rarely blindsided.  In short, these successful people have soft skills to accompany their technical prowess.  One of my favorite lines used to be “I’m not a salesman, and I don’t play office politics”.  However, I’ve learned that everyone has to be a salesman to some extent, even if you don’t sell anything, if for no other reason to do enough self-promotion to ensure that you don’t become an office wallflower.  Office politics is not necessarily an evil thing – at its root, it’s about knowing people and understanding interpersonal dynamics.

Lesson Learned:  Keep working hard, but you’ll do even better if you also spend some time talking to and getting to know the people you work with/for.

To keep this thing moving, I’m now tagging Tim Costello and Devin Knight.

The Next Generation DBA

DBA, database developer, analyst, SQL grunt, or whatever your title may be, there is no question that your role will be evolving in the next few years.

I read a couple of posts from Jason Massie about the Death of the DBA (part 1 and part 2) earlier today, in which he predicts a diminishing market for database administrators, and SQL DBAs in particular.  These posts are not the first references to the “cloud computing” initiative stealing away market share from hard-working DBAs, and to some extent I can agree with this.  Opportunities for the “typical” install-backup-restore-defrag DBA will be reduced in the future, though it could be argued that this has already begun.  The stereotypical introverted techie who works best while locked away in an isolated server room can make a good living right now, but it is this type of position that could be in jeopardy due to cloud computing or similar automation.

Tomorrow’s database professional must know not only the technology but the business that requires it.  DBAs cannot afford to be agnostic about the data stored on their systems; he or she must understand not only the technical objects and methods, but has to grasp the big picture of what the data and metadata collectively represent.  This new generation of database professionals must have an understanding of the organization’s objectives, and must have at least a familiarity with – and just as important, an empathy for – users at every level, from data entry clerks to CXOs.  Top-notch database gurus will have to perform as well in a boardroom as they do in the server room.

I think you’ll find that the role of the DBA is not dying, but will certainly be forced to evolve.  There will not be a shortage of opportunity for those who understand the business as well as the technology and can translate (sometimes ill-defined) business needs into intelligent system objects and functionality.