#WorstPractice – Enabling config churn by discovering properties that are significantly dynamic

This post is a kind of new thing for this blog. Recently I have announced a series of articles with common name “7 Worst Practices for SCOM Management Pack design”. I really intended to author all the content myself. But new things tend to happen suddenly. It was Craig Pero, who squeezed into my blog and suggested some more although well-known, yet still important points. I asked Craig if he wants to write a guest blog post and Craig agreed.

So, please meet Craig, an IT Pro who is managing large-scale infrastructures for more than 15 year, observed the evolution of System Center Operations Manager since NetIQ times and today is sharing a very small part of his experience as a guest author at olegkapustin.com.

Discovering Properties That Are Significantly Dynamic and At Excessively Frequent Intervals.

By Craig Pero

I’m not the first to bring this subject up… probably won’t be the last.  This is a more advanced level of management pack development but very important if you love your customer.  Ok… perhaps love is too strong a word, “like keeping their business” or “want to remain on your manager’s good side”.

In short, discovering data that changes frequently and is not static can lead to “config churn”. If you are a seasoned SCOM administrator you shake your head and shutter at the same time since you know the pain that this causes.  Kevin Holman has a good blog explaining config churn.

Kevin Holman’s Article: http://blogs.technet.com/b/kevinholman/archive/2009/10/05/what-is-config-churn.aspx

Free disk space, free DB space, or number of connected users are all BAD values to use as properties on an object since they continually change.  It is even worse if you run your discovery more than once daily.  I’ll give you an example.  We used a SQL management pack and it discovered free space in the database as a property of the database object.  On an active database… one byte change means the database free space changed.  If you have 100 databases and you discover every 24 hours… then this is not so bad.

Now to the point of what excessively frequent can mean.  Let’s look at a case where you have an environment and there are 16,000 databases (I have seen it personally) discovering data several times per day (let’s go with every 6 hours or 4 times per day).  Every 6 hours, you could potentially have 16,000 (probably less but let’s assume the worst case) Config Changes reported to the RMS.  So now you are collecting 64,000 configuration changes every 24 hours for ONE class.  This is in addition to everything else that is being collected.  Imaging if this was every hour!

The configuration service on the RMS has to process ALL of the configuration changes made.  The configuration service calculates the configuration for every agent and the objects hosted by the server where the agent resides on.  This is a large burden on the configuration service.  When agents request an updated configuration because of changes incurred, it may take an excessive amount of time to get the configuration change or not get it at all.  Even though an agent “knows” a change has occurred, the Configuration Service has to rebuild the configuration from the collected data in the operations manager database and send it back out to the agent to use. This then pounds on the database when you have hundreds or 1000’s of servers and objects.  The agent uses the update configuration file to see if anything has changed since the last time a discovery was run. If something has changed, we repeat the cycle over and over and over… (you get the point)

In the case of the 16,000 databases, a custom discovery was written to populate the free space property with 0 (zero) and the original discovery was disabled.  Granted this value of zero is useless… but on the other hand the RMS was useless because it could not process all the changes continually coming in.

If you write something for yourself and you know the scope, it’s up to you.  If you are writing for someone else (and you don’t know who that someone else will be or their environment) then you need to consider the potential worst case scenario when you choose properties to discover.

What about you? Tell your story in comments!
More worst practices.

Leave a Comment