Flush your systems
Published on 11/6/2018 by Damian Jakubczyk
I was flushing the cooling system on my Jeep and it got me thinking about how important it is to keep systems clean.
Regardless of whether it’s a truck engine, a database, or a suite of integrated software systems, a clean setup keeps the system from overheating, producing unexpected results, or leaving you stranded on the side of the road.
For a Jeep, a cleaning is simple and produces visible results. Drain coolant, flush the heater core, radiator, and engine block. Then rinse and repeat.
Databases and data structures don't need antifreeze but most data systems I encounter could benefit from a deep cleaning:
-
Understand where your data lives. You will be surprised how many additional data sources you will find. Spreadsheets, extracts, cloud-based systems will contain the manipulated data your company actually uses. These manipulations should not be underestimated as additional business rules and decisions are applied by your users as the data transforms. Just ask your business how many members or customers you have and enjoy the variety of answers you will get.
-
Don’t hoard your data. With data growing at an ever-increasing pace, it’s easy to keep accumulating it in case you may need it in the future. I deliberately distinguish “official” data that went through a cleansing process and was deemed ready for production. I may keep some data of value to be used in the future tucked far away from production systems. Using the 80/20 rule, you will find that you use 20% of your data 80% of the time. Unused data I can easily gather again or data that refreshes frequently moves straight to the trash bin.
-
Deep clean your data. With the run to the data dumpster, this tends to be the most time consuming and tedious effort, but it produces the most valuable results. The remaining data now needs to be brought into tip-top shape:
- Remove duplicates and variations. Consolidate multiple conflicting records into ones that make sense. Multiple names for the same customer? Agree on one. Multiple ways to spell the city? Agree on one.
- Fill in the blanks. Capture missing information such as names, addresses, contact info, transaction data. Focus on important and frequently used data you rely on to make business decisions.
- Put in the right place. Validate and correct misplaced content. Name in address field? Move it. Phone number instead of Zip code? Move it. Missing transactions from Wednesday? Recapture.
- Right the wrong. Correct invalid data that goes against your business rules. Email addresses without an @ sign? Fix it. Dollar amounts outside of an acceptable range? Fix it. Birth dates in the future? Fix it.
-
Don’t contaminate. After all these efforts, put a process in place to keep your data clean. Treat new data before it mixes with your existing data. Monitor for anomalies. Repeat a deep clean when necessary.
I’m off to push my Jeep up the next high-country trail with confidence that my cooled inline 6 engine will get me there. How confident are you in your next data adventure?
