Splunk Ninja & Processing Distributed Logs

Early this morning, Ilya Grigorik, founder of AideRSS, sent me a short note via Twitter to tell me about his latest blog post.

In the post, he described his use of a single instance of Splunk to process application log files from several dozen Amazon EC2 instances. He also included a bit of Ruby code which illustrates the process of logging data to Splunk over socket connection.

Splunk is a very cool analysis tool for system and application log files. It indexes the logs, makes it easy to search them, lets you create alerts, and even generates some spiffy-looking reports, among other things.

Minutes later, one of my colleagues sent me another blog post related to Splunk. In that post, the Splunk Ninja (motto: "All batbelt. No tights.") demonstrates (in video form) his use of EC2 and S3 to demonstrate Splunk and its log processing tools. The Ninja likes the fact that EC2 offers quick provisioning and scaling, and that he doesn't have to buy anything or to wait for it to be delivered. He does complain that there's no pretty GUI for EC2, so I'll have to tell him about ElasticFox.

Update: The Splunk Ninja has posted a new and longer video! This one covers ElasticFox, RightScale, CloudStatus, and two very cool Splunk add-ins: Splunk Replay and Splunk Globe.

-- Jeff;

Comments

[botchagalupe]

Why not use Hadoop instead.

[Michael Wilde]

Thanks for the notice about my video post on Splunkin' with EC2. I just finished up a post which includes a longer form video called "Splunk Ninja - Inside the Cloud" I cover everything from CLI, Elasticfox, Rightscale, CloudStatus.com and a few new "Applications" on top of Splunk. Check it out and keep the conversation going -- I'm game!

http://blogs.splunk.com/thewilde/2008/06/24/splunk-ninja-inside-the-cloud/