Website Crawl Settings in Conductor Intelligence

The Crawling activity in Conductor Intelligence lets you fine-tune Conductor's behavior as it crawls your website for technical SEO information in Pages and Page Details.

For websites you manage in Conductor Intelligence (that you do not manage with our Conductor Monitoring product—see the note below), you can configure a number of different settings that determine how Conductor crawls your sites.

Note!

If you also use the Conductor Monitoring product to manage any of your tracked web properties, you will configure those settings in Conductor Monitoring's Website settings screen.

Any web properties you track with Conductor Intelligence but do not manage in CWM can be configured here in Conductor Intelligence's Crawling activity.

 

Monitoring Settings

The Crawling tab in Conductor Intelligence's crawl settings activity allows you to configure basic aspects of your site crawl.

Device Type

You can choose the device type for which you want Conductor to use as it crawls your website. You can choose either:

• Mobile, which uses a 360px width by 640px height viewport.
• Desktop, which uses 1,350px width by 940px height viewport

Changing the Device Type

You can configure which Device Type Conductor Intelligence should use when monitoring your website in the Monitoring section of the Website settings.

You can pick from Desktop or Mobile.

User-Agent String

What is a user-agent string?

Browsers and web crawlers send a special identification, called the user-agent string, to the website they're accessing. This user-agent can be used to tailor the reply, for example: when the user-agent string makes it clear that the website is being accessed on a phone the mobile variant of the website may be returned. If the web server software keeps logs of all visitors to the website it usually also stores the user-agent string to allow identification of the different traffic.

Default user-agent string

By default, Conductor sends this user-agent string when making requests:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)

This may change in the future, so check this page regularly for updates.

Changing the User-Agent

You can pick a predefined user-agent string from the list. Note that if you want to use a fully custom user-agent string, you'll need to manage the website in the Conductor Monitoring product.

Predefined user-agents:

• Google Chrome (desktop)
• Google Chrome (mobile)
• Googlebot (desktop)
• Googlebot (mobile)
• BingBot (desktop)
• BingBot (mobile)
• Slurp
• YandexBot
• Baiduspider
• Screaming Frog
• DuckDuckGo
• SeznamBot
• ContentKing

Monitoring speed and frequency

The speed with which Conductor monitors and the overhead (or impact) on your website depends on your website’s size, your webserver’s capacity and the monitoring speed setting.

The relation between web server capacity and monitoring speed

In order to avoid performance issues on your website and to align with your business needs, before using Conductor it is essential that you verify what crawl speed your infrastructure can support and adjust your settings accordingly.

Although it is important to assess webserver capacity before using the application, Conductor helps predict server overload by continuously assessing your website’s performance. It does this through its throttling algorithm, which looks at the time it takes to get a response from the webserver and at the response itself. If the algorithm detects an issue (for example, slower speeds, increased errors, etc.), it may slow down crawling automatically until the impact on the webserver has been remedied.

The relation between website size and monitoring speed

Once you have verified webserver capacity, for larger websites you may decide to crawl at a faster speed in order to keep up to date on your website's changes as well as possible. For example, if your website has 200,000 pages, you may wish to configure Conductor to request a page every second.

For smaller websites, you may decide to crawl at a slower speed. So, if your website is only a couple of thousand pages, the optimal crawl frequency may be very low. For example, if your website has 2,000 pages, you may only need Conductor to request a page once every 10 seconds.

Setting a different monitoring speed in Conductor Intelligence

You can speed up or slow down Conductor's crawling by setting a different monitoring speed in the website settings in Conductor Monitoring.

You can also choose to set different speeds depending on the time of day to account for peak and off-peak periods. To do so, click Set peak / off-peak speeds.

From there, you can configure the specific times you consider Peak and the speeds associated with Peak and Off-Peak.

Advanced Settings

The Advanced tab in Conductor Intelligence's crawl settings activity allows you to configure more aspects of your site crawl.

Monitoring location

Out of the box, Conductor monitors websites from the EU, the UK, or the US:

• If you create your account from the US or from Canada, your websites will be monitored from the US by default.
• If you create your account from the UK, your websites will be monitored from the UK by default.
• In all other cases, your websites will be monitored from the EU by default.

When does the monitoring location matter?

Geo redirects

When you operate globally or in multiple countries, you might have multiple domains with different language versions of your website (example.com, example.co.uk, example.de, etc.) and a geo redirect setup in place that redirects visitors to the relevant domain version based on their location.

So when you want to monitor example.com but the default monitoring location is set to the UK, Conductor will get redirected to example.co.uk and it will not be able to monitor example.com.

If this is the case, you can change the default monitoring location to the US to ensure that Conductor doesn’t get redirected to a different domain and will be able to monitor example.com.

GeoIP blocking

Some websites are only available for visitors located in a certain region, and visitors outside of this region are not able to access the website.

So for example, when your website is only available in the US, but your default monitoring location is set to the EU, Conductor will not be able to access and monitor the website.

If this is the case, you can change the monitoring location for the website to the US, which will allow Conductor to access the website and monitor it properly.

Changing Monitoring Location in Conductor Intelligence

1. Follow the path Settings > Crawling, and go to the Advanced tab.
2. Update the Monitoring From menu to determine your preferred monitoring location.

HTTP Auth

Staging and testing environments

Conductor provides great solutions for monitoring your staging and testing environments, allowing you to do comparisons with the production environment before green lighting a release or migration.

Staging and testing environments are usually protected from all outside visitors as well as search engines.

If you’re wondering why you should be protecting your staging environment or you want to learn about different protection methods, be sure to read our Academy article on this topic.

Allowing Conductor access to the staging environment

There are several ways in which you can give Conductor Monitoring access to your staging environment based on what protection method you are using.

HTTP Auth is the best way to protect your staging environment and you can read more about it here.

Conductor allows you to set up HTTP Auth directly in its apps, which means that you can easily enable us to monitor your staging environment.

Changing Advanced Settings in Conductor Intelligence

1. Follow the path Settings > Crawling, and go to the Advanced tab.
2. Use the HTTP Auth toggle to enable a form to enter a Username and Password for the site.

Other ways to whitelist Conductor's monitoring

You can also give Conductor access to your staging environment based on a unique identifier of our crawler.

You can whitelist Conductor based on:

• Monitoring IP addresses
• User-agent string
• Custom HTTP header

Cookies

By default Conductor doesn’t accept or transmit cookies when requesting URLs on websites.

However, some websites can require cookies to load certain elements which Conductor needs to monitor the website properly and report accurate data.

For example, some websites have a cookie consent which is used to load analytics tracking. In such cases Conductor wouldn’t be able to discover the analytics tracking, as it doesn’t transmit cookies.

This is why you can set custom cookies in Conductor.

Changing Advanced Settings in Conductor Intelligence

1. Follow the path Settings > Crawling, and go to the Advanced tab.
2. Use the Cookies to allow you to add a Cookie Name and Value for one or more cookies.

HTTP Headers

HTTP headers serve for transferring additional (not displayed) information between Conductor and your website’s server.

Based on HTTP headers you can identify our monitoring as well as whitelist Conductor in case our crawler is being blocked by your web server.

Changing Advanced Settings in Conductor Intelligence

1. Follow the path Settings > Crawling, and go to the Advanced tab.
2. Use the HTTP Headers section to allow you to enter one or more HTTP Headers, which may be:
   • Default HTTP headers. By default, Conductor sends the following HTTP headers with the monitoring requests:
     • Accept: /
     • Accept-Encoding: gzip,deflate
   • Custom HTTP headers. Beyond the default HTTP headers, you can also set custom HTTP headers that will be part of Conductor’s monitoring requests. Out of the box, Conductor supports the following custom HTTP headers:
     • Accept
     • Accept-Language
     • Authorization
     • Cache-control
     • Forwarded
     • Proxy-Authorization
     • Referer
     • X-*

If you want to set a different custom header, don’t hesitate to contact us and we will be happy to set it up for you.

More Miscellaneous Settings

The More tab in Conductor Intelligence's crawl settings activity allows you to configure aspects of your site crawl even further.

Relevant Search Engine

This setting affects how Conductor reviews the robots.txt and meta robots rules on your website. Because search engines have different rules and regulations for how they crawl websites, you can specify whether Conductor should look at directives for certain search engines.

Also track H2-H6 changes

Conductor can include or exclude tracked changes to H2–H6 tags on your website. Keep in mind that setting this toggle to Yes may greatly increase the total number of tracked changes for each URL—and Conductor does limit the total number of tracked changes to 70,000.

Also track number-only changes

Conductor can include or exclude tracked changes to tags where only a number value has changed. This might be helpful for your site if you want ignore changes to numbers that frequently change, such as quantities, prices, or dates. Turning this setting off can decrease the overall number of tracked changes on your website.

FAQs