Why you should ban Amazon's Cloud IPs
Thursday, 27th December 2012, 14:50

Life used to be simple, you made a website, submitted it to Yahoo and maybe a load of other search engines via a tool or website. These days, in some respects things are simpler still, you submit it to Google if it hasn't found it already, and Bing if you are feeling generous, then you sit back and get scraped by sites like Pinterest who take your images and serve them as their own.

But Pinterest isn't the only problem, there are loads of other similar start-ups out there doing the same, many you won't have heard of, many you'll never hear of. And Amazon's cloud services appear to be the system of choice for them to use.

Added to this, there will be a number of unscrupulous... if you don't consider the above as such... people who will abuse these same cloud services to scrape your content and generally try to free-load off your output whilst eating up your bandwidth.

The solution? Just block Amazon's IP ranges from your websites. Don't use a firewall, this stops you accessing services on AWS, just block them from reading your websites, then you only have to worry about Russian and Chinese scrapers. :)

Blocking Amazon Cloud IPs in nginx

Firstly, you need to create a new file to put the block list in, let's call this blocklist.conf, and we need to make sure we include it in the http section of our nginx.conf file. It shouldn't matter where, so if you use a wild-card include that picks up *.conf in a sub-directory, you can just add the blocklist.conf there and not do the next bit.

    http {
        # all your usual generic stuff here
        include blocklist.conf;
    }

And now for what we'll be putting in our blocklist.conf file, which will use the IPs from the official sticky forum thread on the Amazon AWS forums:

    # US East (Northern Virginia)
    deny 72.44.32.0/19;
    deny 67.202.0.0/18;
    deny 75.101.128.0/17;
    deny 174.129.0.0/16;
    deny 204.236.192.0/18;
    deny 184.73.0.0/16;
    deny 184.72.128.0/17;
    deny 184.72.64.0/18;
    deny 50.16.0.0/15;
    deny 50.19.0.0/16;
    deny 107.20.0.0/14;
    deny 23.20.0.0/14;
    deny 54.242.0.0/15;
    deny 54.234.0.0/15;
    deny 54.236.0.0/15;
    deny 54.224.0.0/15;
    deny 54.226.0.0/15;
    
    # US West (Oregon)
    deny 50.112.0.0/16;
    deny 54.245.0.0/16;
    deny 54.244.0.0/16;
    
    # US West (Northern California)
    deny 204.236.128.0/18;
    deny 184.72.0.0/18;
    deny 50.18.0.0/16;
    deny 184.169.128.0/17;
    deny 54.241.0.0/16;
    
    # EU (Ireland)
    deny 79.125.0.0/17;
    deny 46.51.128.0/18;
    deny 46.51.192.0/20;
    deny 46.137.0.0/17;
    deny 46.137.128.0/18;
    deny 176.34.128.0/17;
    deny 176.34.64.0/18;
    deny 54.247.0.0/16;
    deny 54.246.0.0/16;
    deny 54.228.0.0/16;
    
    # Asia Pacific (Singapore)
    deny 175.41.128.0/18;
    deny 122.248.192.0/18;
    deny 46.137.192.0/18;
    deny 46.51.216.0/21;
    deny 54.251.0.0/16;
    
    # Asia Pacific (Sydney)
    deny 54.252.0.0/16;
    
    # Asia Pacific (Tokyo)
    deny 175.41.192.0/18;
    deny 46.51.224.0/19;
    deny 176.32.64.0/19;
    deny 103.4.8.0/21;
    deny 176.34.0.0/18;
    deny 54.248.0.0/15;
    
    # South America (Sao Paulo)
    deny 177.71.128.0/17;
    deny 54.232.0.0/16;

Now, this is accurate as of the time of posting, but don't assume this IP address range list to be static, you'll need to monitor the forum post I linked above!

Important Note

After doing any major change to your webserver like this, ALWAYS carefully monitor your webstats, Google Webmaster Tools, Bing Webmaster Tools and anything else which shows the continued health of your website. If it looks likely to be impacting your traffic and/or search positions badly, back it out immediately and consider if this is the cause.

Hope you had a nice Christmas, and have a happy new year. :)