Best SEO for Your CDN

best seo

Search engine crawlers (also known as bots or spiders) scan your website whether you like it or not. They scan pretty much everything that’s available, which is normally a good thing. Why is SEO with CDNs so important? As you start using a CDN, your content can appear from different domains. Nothing wrong about that as long as the search engines have clarity about your content. If the content is not clearly declared, they will penalize you for duplicate content.

We offer two options to reach the best SEO. Both options fulfill the goal of not having duplicate content and be aligned with search engines. Let’s take a closer look on both solutions.

1. Option: Canonical URLs

An extra HTTP header added to your zone lets the crawler know, that the content from the CDN is only a copy. Once we add rel=”canonical” to the HTTP header, we’re on the safe side. Crawlers are aware that this is only a copy.

SEO improvement - HTTP header with rel="canonical"

HTTP header with rel=”canonical”

The rel=”canonical” header will be applied to the whole zone. If you already send a canonical header from your origin server, there’s no need to enable it in the dashboard.

The canonical header can be enabled in the KeyCDN dashboard. Go to your zone settings and enable the feature as shown here: 

2. Option: Robots.txt

Search engines check for a robots.txt file at the root of a site. If the file is present, they will follow the instructions but if no file is present, they will scan everything. We will not enable the file by default, meaning that everything will be indexed.

Here’s the robots.txt file that we will add to your site if you enable the Robots feature in the dashboard:

User-agent: *   
Disallow: /
  1. The first line defines the crawler the rule applies to. In the example above, the robots.txt applies to every crawler. “User-agent: Google” would only apply for Google bots.
  2. The next line defines what path can be indexed. “Disallow: /” tells the search engine to not index anything.

You can still create your own robots.txt with more detailed instructions. Several instructions can be added in the same files.

Robots.txt can also be enabled in the zone settings: 

Related Articles

Best SEO for Your CDN was last modified: April 25th, 2016 by Jonas Krummenacher
  • Digital MKT Trends


    I can’t find the Zone Settings in my KeyCDN dashboard. Where is it exactly?. Neither the canonical header enable option.

    Where is it?


    • Hi. In the KeyCDN dashboard, go to “Zones” (right menu) and then go to “Manage” –> “Edit”.

      • Digital MKT Trends

        Yes!. Thanks!

  • How Google will know how to find the original content ?

    • Hey Angristan! Great question. Above in the post where it talks about the canonical tag, that is the most important part. Google uses the canonical tag as a pointer back to your origin server. So when Google’s robot is crawling and sees your image/files with a canonical tag attached, it then knows it isn’t duplicate content and it originates on your origin server, so it will still index it.

      • Thanks ! So I have to use the canonical header and robots.txt ?

        • Correct. The canonical header on your KeyCDN zones should be enabled by default. You can also enable/disable it in your dashboard. And the robots.txt is not needed unless you want to block something from being crawled. We don’t enable the file by default, meaning that everything will be indexed.

          • Thanks, what is the URL of the CDN’s robots.txt ? [ZoneURL]/robots.txt shows my website’s one

          • Yes, it’s your ZoneURL/robots.txt. I have attached a screenshot. You can see I enabled my robots.txt file in KeyCDN dashboard and added some custom text. Make sure to purge your zones cache/or individual file cache after enabling/tweaking robots.txt file. And the file might take a few moments to distribute against all the POPs.

            If robots.txt is not enabled in dashboard it will pull from origin server.

  • Vapeman

    Why is there a robots.txt file. I already have one running on my site. Are you saying I need to duplicate that here?

  • Could I have problems if I use the canonical header on KeyCDN zones when my server is also sending a canonical header?

    The two canonical header are the same?

    I don’t want to be indexed as

  • Nico

    Step 1: Add a header to files.
    Step 2: Block google from seeing that header.
    I don’t get the logic in this.

Share This