Best SEO for Your CDN

By Jonas Krummenacher
Updated on April 25, 2016
Best SEO for Your CDN

Search engine crawlers (also known as bots or spiders) scan your website whether you like it or not. They scan pretty much everything that’s available, which is normally a good thing. Why is SEO with CDNs so important? As you start using a CDN, your content can appear from different domains. Nothing wrong about that as long as the search engines have clarity about your content. If the content is not clearly declared, they will penalize you for duplicate content.

We offer two options to reach the best SEO. Both options fulfill the goal of not having duplicate content and be aligned with search engines. Let’s take a closer look on both solutions.

1. Canonical URLs

An extra HTTP header added to your Zone lets the crawler know, that the content from the CDN is only a copy. Once we add rel=“canonical” to the HTTP header, we’re on the safe side. Crawlers are aware that this is only a copy.

HTTP header with rel=canonical

The rel=“canonical” header will be applied to the whole Zone. If you already send a canonical header from your origin server, there’s no need to enable it in the dashboard.

The canonical header can be enabled in the KeyCDN dashboard. Go to your Zone settings and enable the feature as shown here:

2. Robots.txt

Search engines check for a robots.txt file at the root of a site. If the file is present, they will follow the instructions but if no file is present, they will scan everything. We will not enable the file by default, meaning that everything will be indexed.

Here’s the robots.txt file that we will add to your site if you enable the Robots feature in the dashboard:

User-agent: *
  1. The first line defines the crawler the rule applies to. In the example above, the robots.txt applies to every crawler. “User-agent: Google” would only apply for Google bots.
  2. The next line defines what path can be indexed. “Disallow: /” tells the search engine to not index anything.

You can still create your own robots.txt with more detailed instructions. Several instructions can be added in the same files.

Robots.txt can also be enabled in the Zone settings:

Supercharge your content delivery 🚀

Try KeyCDN with a free 14 day trial, no credit card required.

Get started


KeyCDN uses cookies to make its website easier to use. Learn more about cookies.