March 10th, 2009
Just found out the hard way that if your Rails app can’t contact New Relic, it blocks your mongrels and brings down your app. Teenormous t-shirts is hosted on Slice Host and I believe there were network connectivity issues between my slice and New Relic and I was not the only one.
First Rule Of Monitoring: Do not bring down the application!
Needless to say, New Relic will most likely no longer be a part of my monitoring solution.
UPDATE: The issue appears to be related to DNS resolution at New Relic and there is a work around to prevent DNS lookup by hard-coding the IP (http://support.newrelic.com/discussions/support/480-why-didnt-you-announce-the-problems-with-rpm-this-morning). The folks at New Relic saw this post and setup a conference call to discuss the issues. I am impressed with this level of customer service. They believe New Relic will no longer cause issues for an application now that the DNS issues are resolved and they informed me they are actively working on improving the way this works in the application. Thanks to their impressive support and follow-through, I may give New Relic a try again in the future. For now, I am going to sit on the sidelines for a bit and only use it in development though.