Log Aggregation & Management
The top question asked by customers and students is how to effectlvely manage Windows logs in an enterprise. Read on for a free (as in beer) solution to this problem!
The reason that everyone’s concerned about aggregating logs is that there are suddenly a number of legal and industry requirements that deal with log management and alerting. Honestly it doesn’t seem that managing logs in a Windows environment should be very difficult. Afterall, Windows has some pretty powerful logging features in it and is fairly granular when it comes to things like privilege and file access auditing.
While this is all true, until the advent of Vista and 2008 there was no easy way to subscribe to or aggregate the events from several systems into one location. Even if you surmount this problem there are other issues, most notably how to correlate events for users across the enterprise since there is no consistent marker in the events from one machine to another that will distinguish users. We’ll leave that problem for another day. Let’s see what we can do about getting the logs into one place and reporting on them.
Of course, there are a fair number of commercial solutions available to solve this problem. The most common comment from people in the trenches who are using these tools, though, is that they all looked good on paper but they don’t seem to perform quite as reliably as the salesman said that they would. Personally, I stay clear of these tools because of the work that I do developing our log management product. We need to be sure that we can never be accused of stealing someone’s interface or concept in our tool.
For several years I’ve been telling one of my customers that they could just, “write some scripts.” After hearing this for years and struggling with Microsoft Operations Manager they finally said to me, “Put your money where your mouth is!” Thus, DAD was born.
DAD (Distributed Aggregation for Data analysis) is a free open source solution to this log aggregation and reporting problem. The entire project is based on open source products and is released under the General Public License. You can find the project here.
The really beautiful thing about this particular solution is that we have abstracted the data so that we are not concerned about how the events are formatted. The real power of this is that we can digest absolutely any kind of text based log format including syslog and web logs very easily. The way that this is handled is by parsing the events out into individual words which are then uniquely inserted into the database. Not only does this give us tremendous speed when searching for events but it also means that we can store far more events than might otherwise be possible because we never duplicate data.
We do encourage you to consider looking at the DAD project as a possible solution but we also want to make sure that you are aware that this product falls somewhere between an Alpha and Beta status. This means that while it is feature complete there are definitely bugs that need to be ironed out. If you do decide to try it out, please drop us a line and let us know how you like it!
While this is all true, until the advent of Vista and 2008 there was no easy way to subscribe to or aggregate the events from several systems into one location. Even if you surmount this problem there are other issues, most notably how to correlate events for users across the enterprise since there is no consistent marker in the events from one machine to another that will distinguish users. We’ll leave that problem for another day. Let’s see what we can do about getting the logs into one place and reporting on them.
Of course, there are a fair number of commercial solutions available to solve this problem. The most common comment from people in the trenches who are using these tools, though, is that they all looked good on paper but they don’t seem to perform quite as reliably as the salesman said that they would. Personally, I stay clear of these tools because of the work that I do developing our log management product. We need to be sure that we can never be accused of stealing someone’s interface or concept in our tool.
For several years I’ve been telling one of my customers that they could just, “write some scripts.” After hearing this for years and struggling with Microsoft Operations Manager they finally said to me, “Put your money where your mouth is!” Thus, DAD was born.
DAD (Distributed Aggregation for Data analysis) is a free open source solution to this log aggregation and reporting problem. The entire project is based on open source products and is released under the General Public License. You can find the project here.
The really beautiful thing about this particular solution is that we have abstracted the data so that we are not concerned about how the events are formatted. The real power of this is that we can digest absolutely any kind of text based log format including syslog and web logs very easily. The way that this is handled is by parsing the events out into individual words which are then uniquely inserted into the database. Not only does this give us tremendous speed when searching for events but it also means that we can store far more events than might otherwise be possible because we never duplicate data.
We do encourage you to consider looking at the DAD project as a possible solution but we also want to make sure that you are aware that this product falls somewhere between an Alpha and Beta status. This means that while it is feature complete there are definitely bugs that need to be ironed out. If you do decide to try it out, please drop us a line and let us know how you like it!