atom thoughts

I recently ran into some difficulties while fetching random html pages and images using ruby so I thought others may benefit from my experience.

Ruby has two (and probably many more) great libraries for fetching http data:

• Net::Http
• OpenURI

OpenURI has a much nicer API and takes care of things like http redirects for you. However, if you try to fetch data over HTTPS, you will see a message like Certificate Verify Failed.

To get around this limitation I’ve written this simple method for fetching data that will enable ssl when necessary and follow all redirects:

def self.get_data(url, limit = 10)
  raise "Too Many Redirects" if limit == 0
  uri = URI.parse url
  server = Net::HTTP.new uri.host, uri.port
  server.use_ssl = uri.scheme == 'https'
  server.verify_mode = OpenSSL::SSL::VERIFY_NONE
  response = server.get uri.request_uri

  case response
    when Net::HTTPSuccess then response['host'] = uri.host; response['path'] = uri.path; response
    when Net::HTTPRedirection then self.get_data(response['location'], limit - 1)
    else
      response.error!
  end
end

Also, you will notice that I add the host and path into the response since in my case I use that information for building the full path to the images.

Enjoy!