Log File Analysis

With Conductor Monitoring’s Log File Analysis feature, you can easily understand how search engine crawlers traverse the content on your website. Discover when Google, Bing, OpenAI, or Perplexity last visited each page, and compare visit frequencies between pages.

What is Log File Analysis

A log file is a text file containing records of all the requests a server has received, from both humans and crawlers, and its responses to the requesters.

Through log file analysis, SEOs aim to get a better understanding of what search engines are actually doing on their websites, in order to improve their SEO performance.

Log Sources

Currently you can feed your logs to Conductor Monitoring using a Cloudflare Worker, Cloudflare Logpush, the Akamai DataStream, the CloudFront Standard logging or the Fastly Real-Time Log Streaming.

Traffic that Conductor Monitoring detects and tracks

Conductor Monitoring detects and tracks visits of the following search engines to your websites:

• Google Desktop
• Google Mobile
• Bing Desktop
• Bing Mobile
• OpenAI (OAI-SearchBot, ChatGPT-User, and GPTBot)
• Perplexity (PerplexityBot and Perplexity-User)

You can learn more about these engines' crawlers in our Log File Analysis FAQs article.

Setting up the Log File Analysis

The process of configuring the Log File Analysis differs based on the log sources that are used to feed them to Conductor Monitoring. Refer to the relevant support article based on your website's log source:

1. Setting up the Cloudflare Worker integration
2. Setting up the Cloudflare Logpush integration
3. Setting up the Akamai integration
4. Setting up CloudFront Standard logging
5. Setting up Fastly Real-Time Log Streaming

Working with the Log File Analysis data in Conductor Monitoring

Using the Log File Analysis data in Pages

Search engines will have two columns each on the Pages screen: Last Visited and Visit Frequency, which you can use for filtering:

When filtering on the columns, you can easily create segments which can be very useful to do spot checks of your least visited pages.

On the Page Detail screen, you can zoom in on how often crawlers are visiting a specific page. With a neat graph you can identify any trends in the search engine traffic on that page:

And in the Tracked Changes tab of the Page Detail screen, you will be able to see whether search engines saw the latest important change that you’ve made.

In combination with our Real-Time IndexNow feature you can see if Bing visited the page that was submitted to IndexNow:

Log File Analysis data in the Platform section

If you have made some changes to the robots.txt file or the XML sitemaps and you want to check if search engines visited them since.

Just head to the Platform section in Conductor Monitoring and easily check if the search engines visited the robots.txt:

Or the XML sitemaps:

Disabling Log File Analysis

To learn how to disable Log File Analysis please refer to the relevant support article based on your website’s log source:

• Akamai DataStream
• Cloudflare Worker
• Cloudflare Logpush
• CloudFront Standard logging
• Fastly Real-Time Log Streaming

Using Log File Analysis

Curious why (and how) you can use Log File Analysis to inform your content strategy in the world of AI search? Check our AI Crawler Activity in Conductor Monitoring guide to learn!