Everyday Solutions
“No other art is a better fit than Cubism for representing software and databases, as they are a digital (the discrete) means of representing the real world (the continuous). In his piece “The Open Window”, the Spanish Cubist, Juan Gris, invites you to look at your everyday world in a slightly different way, as if somehow moving through time and space.”
— Mordechai Danielov
CEO, Bitwise
Database work isn’t always exciting.
Sometimes it’s just plain boring, but someone has to do it and that someone is us – DBAs. On the other hand, our lives are filled with emergencies. So much so, that we seem to be putting out fires all the time. A prickly conundrum indeed.
Well, for every sticky situation BitWise has a logical solution.
After analyzing dozens — if not hundreds — of production environments, we came up with a process that stabilizes the fire fighting and establishes a creative routine.
Our Process
This is production
Proceed with caution! Plan ahead and practice. Importance of forward thinking can not be overstated.
There should be a record of just about everything
Logging is a must. There is a reason why most environments don’t have good logging, because it can get complicated very fast. No worries, we have strategies for log rotation, retention, digestion, etc.
Set up alerts
Because things will go wrong, be prepared for any eventuality. This task is probably the most recognized one, but it often turns into a logistical nightmare. We can help you to think it through and to plan a good response strategy. Even participate in your on-call schedule if needed.
Backups
Yes, those. Make sure there is a backup for everything. It is amazing that this needs to be written, but believe it or not we’ve seen databases without backups.
Monitoring for any budget
You have to know what’s going on in your database. Use a commercial tool, a script, a shoe-string, whatever works, but you have to have a way.
Test environments
You can’t test things in production. Period. No, not even a few silly queries. Create a test environment and have at it without reservation.
Reporting and archiving
While we’re on the subject of doing silly things in production, can we please reduce the size of that OLTP database (i.e. archive your data) and off-load read only queries to secondary servers? Thanks.
Automate everything
Really, there is no excuse. We’ve seen environments with over 100 servers that had 0 automation. How is that even possible? We automate on all platforms with whatever scripting language you like, etc. Just don’t do things manually.
