Log File Management


This project just started out as research. What is a log file management system? Would it be a useful to implement one at the enterprise level. The answer was a resounding "yes" and this small research project turned into something that consumed most of my time for the next 2 years. The first task was to architect a system that would be highly available, disaster resilient and not impact the production systems while collecting logs. Then I rolled out the offering to teams across the enterprise. I put together a demo and onboarding procedure and grew from collecting just a few gigabytes a day to over 200, almost 80 terabytes a year.

I enjoyed getting to architect another production system but I’m most proud of the impact my little research project had across the enterprise. Using Splunk, teams reduce the amount of time and effort needed to research issues. Even more importantly teams were able to implement monitoring and alerting so that they could resolve issues before clients were impacted.