Monitoring Configuration

The Fastly/Magento Log Shipping module is used to ingest user requests, allowing Netacea to monitor and detect bot activity.

Prerequisites

Before Netacea can monitor your platform successfully, please ensure you have:

  1. Provided a list of all known, safe and trusted list users, partners & 3rd party apps. This includes a list of trusted bots & automated tools. Any information on known bad actors is also valuable.

  2. Provided a list of protection use cases and the pathways that are vulnerable to each.

  3. Had the relevant log shipping endpoints & credentials shared via the Netacea portal.

Log Streaming Implementation Steps

  • Log in to the Fastly/Magento web interface.

  • Navigate to Stores → Configuration → Advanced → System → Full Page Cache → Fastly Configuration → Tools → Real Time Log Streaming.

  • Create a new Real-Time Log Streaming job by clicking "Create" on Create Endpoint, selecting Amazon S3 and clicking "Add".

  • Populate the log streaming job with the following information:

FieldValue

Name

Enter a memorable human-readable name for the endpoint, such as Netacea_Logging.

Log Format

Timestamp

Bucket Name

The name provided by Netacea

Access Key

The access key provided by Netacea

Secret Key

The secret key provided by Netacea

Path

The path provided by Netacea

Domain

s3.eu-west-1.amazonaws.com

Log Line Format

Blank

Compression

Gzip

GZIP Level

1

Period

15

If the Shielding is enabled for Fastly Host(s) then please check paragraph below called "Shielding consideration" before you create a logging endpoint.

Log Format

{"@timestamp": "%{%Y-%m-%dT%H:%M:%S%z}t","bc_type": %{if(req.http.netacea_bctype_string, "%u0022" + req.http.netacea_bctype_string + "%u0022", "null")}V,"bytes_sent": "%B","client": "%{json.escape(client.ip)}V","domain": %{if(req.http.host, "%u0022" + req.http.host + "%u0022", "null")}V,"integration_type": %{if(req.http.integration_type, "%u0022" + req.http.integration_type + "%u0022", "null")}V,"integration_version": %{if(req.http.integration_version, "%u0022" + req.http.integration_version + "%u0022", "null")}V,"method": "%{json.escape(req.method)}V","path": "%{json.escape(req.url.path)}V","protocol": "%{json.escape(req.proto)}V","query": "%{json.escape(req.url.qs)}V","referrer": %{if(req.http.referer, "%u0022" + req.http.referer + "%u0022", "null")}V,"request_time": %{time.elapsed}V,"status": "%{json.escape(resp.status)}V","user_agent": %{if(req.http.user-agent, "%u0022" + json.escape(req.http.user-agent) + "%u0022", "null")}V,"user_id": %{if(req.http.x-netacea-userid, "%u0022" + req.http.x-netacea-userid + "%u0022", "null")}V,"client_ja3_md5": %{if(tls.client.ja3_md5, "%u0022" + tls.client.ja3_md5 + "%u0022", "null")}V}

Timestamp

%Y-%m-%dT%H:%M:%S.000

Shielding consideration

Fastly allows users to enable Host's feature called "Shielding" that has some benefits like reducing origin load, improving cache hit ratio, etc. Enabling this feature in a Fastly service that uses our Integration results in a duplication of logs that are being send by it through Netacea logging. We do not expect this. In order to solve this issue we need to attach a Condition to the Netacea Logging.

  • Click on a gear icon next to the logging endpoint

  • In the popup window please find "Condition" row and click on "Attach a condition"

  • After that you can hit "Create a new response condition"

  • A popup window will open where you need to put these details:

FieldValue

Name

e.g. Netacea_Log_Visit

Apply if

(req.url !~ "/media/" && req.url !~ "/static/") && fastly.ff.visits_this_service == 0

Priority

10 (default)

  • Click on "Create". You should now see that the Condition has been attached to the Logging stream

  • Now you can click on "Create" again and this will update Logging appropriately

Finishing Up

Check you have completed the following steps:

  • Configured log streaming to the S3 bucket provided

  • Ensure no error warnings are appearing

You are now ready to deploy by clicking "Save Config"

A Cache flush can sometimes be required. To do this go to System -> Cache Management and click "Flush Magento Cache".

Monitoring will now be deployed on the Fastly/Magento environment. You can verify the deployment is active by:

  • Reviewing the active version for the new log shipping job.

  • Requesting Netacea review internal ingest metrics.

  • Requesting Netacea validate the data content & format.

Last updated

Copyright Netacea 2023