I’m going to be migrating to using ZooKeeper within Spinn3r for a myriad of reasons but this one is especially powerful.
ZooKeeper enables this with its support for ephemeral files.
If you have an external process like a webserver, database, robot, etc you can have it create a ephemeral file which registers its services and presence in the cluster.
Would represent a machine named www32.mydomain.com.
You can then have munin connect to ZooKeeper and enumerate files in /services/www and have a cron script continually regenerate a munin.conf file.
The great thing is that if you shutdown Apache your munin config will be automatically reflected.
This of course implies that you have ZooKeeper integration in your init scripts.
This is becoming easier with the presence of HTTP gateways for ZK. I haven’t looked at them too much but as long as PUT and GET are supported this is about 80% of the functionality needed to implement the above.
One issue is enumerating files in directories over HTTP. I assume a proprietary XML protocol is used.