Warning: New Relic May Bring Down Your Rails App
March 10th, 2009
Just found out the hard way that if your Rails app can’t contact New Relic, it blocks your mongrels and brings down your app. Teenormous t-shirts is hosted on Slice Host and I believe there were network connectivity issues between my slice and New Relic and I was not the only one.
First Rule Of Monitoring: Do not bring down the application!
Needless to say, New Relic will most likely no longer be a part of my monitoring solution.
UPDATE: The issue appears to be related to DNS resolution at New Relic and there is a work around to prevent DNS lookup by hard-coding the IP (http://support.newrelic.com/discussions/support/480-why-didnt-you-announce-the-problems-with-rpm-this-morning). The folks at New Relic saw this post and setup a conference call to discuss the issues. I am impressed with this level of customer service. They believe New Relic will no longer cause issues for an application now that the DNS issues are resolved and they informed me they are actively working on improving the way this works in the application. Thanks to their impressive support and follow-through, I may give New Relic a try again in the future. For now, I am going to sit on the sidelines for a bit and only use it in development though.
Subscribe to the feed!
March 11th, 2009 at 1:43 pm
Hi Tom,
thank’s for the post. I considered New Relic for a project and I’m pretty glad that what happend to you will not happen to me, because of this warning !
March 11th, 2009 at 9:54 pm
Tom
Thanks for taking time to let us explain our fix to the DNS issues. We hope you let RPM monitor Teenormous again soon. Good luck with the site.
Mike at New Relic
March 11th, 2009 at 9:56 pm
Hi Mike,
I updated this post with your fix and a bit more information. Thanks for setting up the call. It was nice to speak with you and nice to see a company care enough to reach out when things go wrong.
Tom