Setting Up the Akamai Integration

Setting up Log File Analysis with the Akamai DataStream in Conductor Monitoring is very straightforward! Follow the steps outlined below to enable Log File Analysis in Conductor Monitoring, and set up the Akamai integration.

Setting up the Log File Analysis

The process of configuring the Log File Analysis consists of two phases:

• Enabling the Log File Analysis feature and creating an AWS S3 bucket in Conductor Monitoring
• Configuring the Akamai DataStream in the Akamai UI and associating the DataStream to the corresponding property

1. Enabling the Log File Analysis feature and creating an AWS S3 bucket in Conductor Monitoring

If you navigate to Account and then Websites section in Conductor Monitoring, you can easily filter and see which of the websites you are monitoring in Conductor Monitoring are on Akamai:

From there, just follow the steps outlined below to enable Log File Analysis on the wanted website:

1. Click any website that is running on Akamai in the Websites section of Conductor Monitoring.
2. Click the Log File Analysis tab in Settings.
3. Enable the Log File Analysis toggle.

After this is done, follow the next steps to create an AWS S3 bucket in Conductor Monitoring:

1. In the same Log File Analysis section, click the How to install link under Log Sources.
   
2. Specify in which region should the AWS S3 bucket be created (EU or US) and click on Create bucket.
   
3. After this, Conductor Monitoring will automatically generate the AWS credentials and the AWS S3 bucket.
   
4. The credentials will be used to associate the AWS S3 bucket with the Akamai DataStream in the next section.

2. Configuring the Akamai DataStream

The Akamai DataStream needs to be manually configured in the Akamai UI for every website. The configuration of the Akamai DataStream consists of two steps:

• Creating a new Akamai DataStream
• Associating the DataStream to the corresponding property (domain)

Creating a new Akamai DataStream

Follow these steps to create a new Akamai DataStream:

1. Log in to your Akamai Control Center and click on DataStream in the navigation bar:
   
2. On the DataStream screen, click Create stream:
   
3. Configure the DataStream: Specify the stream's name (e.g. "Conductor Monitoring LFA example.com") and pick the website for which you want to configure the Akamai DataStream in the Properties list:
   
4. Configure the Data Sets:
   Important: The log format needs to be JSON.
   Choose the following required data sets to be sent to the AWS S3 bucket. All the fields listed below need to be selected for the Log File Analysis feature to work properly. You can also add other data sets, but the ones listed above are required.
   • Log information
     • Request ID
     • Request time
   • Message exchange data
     • Client IP
     • HTTP status code
     • Protocol type
     • Request host
     • Request method
     • Request path
     • Request port
     • Response-Content-Type
     • User-Agent
     • SSL version
     • Query string

5. Choose Amazon S3 as the destination where the data will be streamed:
   
   • Fill in the Display name (e.g. contentking-lfa-s3-example).
   • Enter the Bucket name generated by Conductor Monitoring to the Bucket field.
   • Path (default suggested by Akamai): logs/{%Y}/{%m}/{%d}/{%H}
   • Enter the Region that you selected in Conductor Monitoring:
     • If you've selected US: enter us-east-2 (located in Ohio)
     • If you've selected EU: enter eu-central-1 (located in Frankfurt)
   • Enter the Access key ID and Secret access key from Conductor Monitoring.
   • Once done, click Validate & Save.
6. Finish the setup and click Save stream:
   

Associate the created DataStream to the corresponding property

After creating the new DataStream, you need to associate it with the corresponding property in your Akamai account.

1. Log in to your Akamai Control Center and click on Properties in the navigation bar:
   
2. Click the property which you want to associate with the created DataStream:
   
3. Click the Property Version being monitored by Conductor Monitoring (this can be either the Staging Version or the Production Version):
   
4. Scroll to the DataStream section in Property Configuration Settings. If the DataStream section is missing, follow the steps below to add it. If not, skip to the step 5.
   
   1. Scroll all the way down, and click Edit new version.
      
   2. Scroll down to Property Configuration Settings, and click Add Behavior.
      
   3. Search for DataStream in the modal that appears and select it.
      
   4. Click Insert Behavior.
      
5. In the DataStream box in the Behaviors list, choose DataStream2 as the Stream version.
   
6. In the DataStream2 section, enable the toggle to On, choose the Stream name of the new DataStream you created, and then click the Save button:
   

Ensure Logging of the User Agent header

1. Go to Property Configuration Settings following steps 1 through 3 in the section Associate the created DataStream to the corresponding property.
2. Scroll down to the Log Request Details section in Property Configuration Settings:
   In case the Log Request Details section is not there, follow the same steps as within step 4. in the section Associate the created DataStream to the corresponding property to add it.
3. Enable the "Log User-Agent header" and click the Save button:
   

Filter away non-search engine traffic

1. Go to Property Configuration Settings following steps 1 through 3 in the section Associate the created DataStream to the corresponding property, and click Add Rule:
   
2. Insert the rule for specific User Agents: choose a Blank rule template, then give it a name (e.g. "Enable DataStream for specific user agents", and click Insert rule:
   
3. Click Add Behavior and add DataStream behavior following steps 4.3 and 4.4. in Associate the created DataStream to the corresponding property:
   
4. Click Add Match and set up the criteria for the User Agent to match:
   
   Example:
   
   User agent is one of:
   • Googlebot
   • bingbot
   • PerplexityBot
   • Perplexity-User
   • OAI-SearchBot
   • ChatGPT-User
   • GPTBot
5. Finally, click Save:
   

Reinstalling the Akamai DataStream

If you want to change the AWS S3 bucket region, or reinstall the Akamai DataStream, you can do it in the following way:

1. Click on the website on which you want to reinstall the Akamai DataStream in the Websites section of Conductor Monitoring.
2. Click Log File Analysis in Settings, and then on the reinstall link under Log Sources.
3. If needed change the region, and click Create bucket.
4. Create a new DataStream by following the steps above.

Disabling Log File Analysis

Same as with enabling Log File Analysis, you need to disable the feature in Conductor Monitoring and then remove the Akamai DataStream in the Akamai UI.

Disabling Log File Analysis

1. Click on the website on which you want to disable Log File Analysis in the Websites section of Conductor Monitoring.
2. Click the Log File Analysis in the Settings section.
3. Disable the Log File Analysis toggle.

Once this is done, Conductor Monitoring will automatically disable Akamai DataStream access to the AWS S3 bucket where the data was streamed.

Removing the Akamai DataStream

If you have disabled the Log File Analysis feature, you still need to remove the Akamai DataStream in your Akamai account.

This needs to be done manually, as Conductor Monitoring doesn't have access to your Akamai account.

Security FAQs

For the most common security-related questions about Conductor Monitoring's Log File Analysis, refer to the FAQ section in the Log File Analysis FAQs article