Monitoring Configuration

The Fastly/Magento Log Shipping module is used to ingest user requests, allowing Netacea to monitor and detect bot activity.

Prerequisites

Before Netacea can monitor your platform successfully, please ensure you have:

1. Provided a list of all known, safe and trusted list users, partners & 3rd party apps. This includes a list of trusted bots & automated tools. Any information on known bad actors is also valuable.

2. Provided a list of protection use cases and the pathways that are vulnerable to each.

3. Had the relevant log shipping endpoints & credentials shared via the Netacea portal.

Log Streaming Implementation Steps

• Log in to the Fastly/Magento web interface.

• Navigate to Stores → Configuration → Advanced → System → Full Page Cache → Fastly Configuration → Tools → Real Time Log Streaming.

• Create a new Real-Time Log Streaming job by clicking "Create" on Create Endpoint, selecting Amazon S3 and clicking "Add".

• Populate the log streaming job with the following information:

Field	Value
Name	Enter a memorable human-readable name for the endpoint, such as Netacea_Logging.
Log Format	Log Format
Timestamp	Timestamp
Bucket Name	The name provided by Netacea
Access Key	The access key provided by Netacea
Secret Key	The secret key provided by Netacea
Path	The path provided by Netacea
Domain	s3.eu-west-1.amazonaws.com
Log Line Format	Blank
Compression	Gzip
GZIP Level	1
Period	15

If the Shielding is enabled for Fastly Host(s) then please check paragraph below called "Shielding consideration" before you create a logging endpoint.

Log Format

{"@timestamp": "%{%Y-%m-%dT%H:%M:%S%z}t","bc_type": %{if(req.http.netacea_bctype_string, "%u0022" + json.escape(req.http.netacea_bctype_string) + "%u0022", "null")}V,"bytes_sent": "%B","cookie_session_status": %{if(req.http.x-netacea:cookie_session_status, "%u0022" + json.escape(req.http.x-netacea:cookie_session_status) + "%u0022", "null")}V,"client": "%{json.escape(client.ip)}V","domain": %{if(req.http.host, "%u0022" + json.escape(req.http.host) + "%u0022", "null")}V,"integration_mode": %{"%u0022" + json.escape(req.http.x-netacea:integration_mode) + "%u0022"}V,"integration_type": %{if(req.http.integration_type, "%u0022" + json.escape(req.http.integration_type) + "%u0022", "null")}V,"integration_version": %{if(req.http.integration_version, "%u0022" + json.escape(req.http.integration_version) + "%u0022", "null")}V,"method": "%{json.escape(req.method)}V","path": "%{json.escape(req.url.path)}V","mit_svc_latency":%{if(req.http.x-netacea:mit_svc_latency, "%u0022" + json.escape(req.http.x-netacea:mit_svc_latency) + "%u0022", "0")}V,"mit_status":%{if(req.http.x-netacea:mit_status, "%u0022" + json.escape(req.http.x-netacea:mit_status) + "%u0022", "0")}V,"protocol": "%{json.escape(req.proto)}V","query": "%{json.escape(req.url.qs)}V","referrer": %{if(req.http.referer, "%u0022" + json.escape(req.http.referer) + "%u0022", "null")}V,"request_time": %{time.elapsed}V,"status": "%{json.escape(resp.status)}V","user_agent": %{if(req.http.user-agent, "%u0022" + json.escape(req.http.user-agent) + "%u0022", "null")}V,"user_id": %{if(req.http.x-netacea-userid, "%u0022" + json.escape(req.http.x-netacea-userid) + "%u0022", "null")}V,"client_ja3_md5": %{if(tls.client.ja3_md5, "%u0022" + json.escape(tls.client.ja3_md5) + "%u0022", "null")}V,"x_forwarded_for": %{if(req.http.X-Forwarded-For, "%u0022" + json.escape(req.http.X-Forwarded-For) + "%u0022", "null")}V}

Timestamp

%Y-%m-%dT%H:%M:%S.000

Shielding consideration

Fastly allows users to enable Host's feature called "Shielding" that has some benefits like reducing origin load, improving cache hit ratio, etc. Enabling this feature in a Fastly service that uses our Integration results in a duplication of logs that are being send by it through Netacea logging. We do not expect this. In order to solve this issue we need to attach a Condition to the Netacea Logging.

• Click on a gear icon next to the logging endpoint

• In the popup window please find "Condition" row and click on "Attach a condition"

• After that you can hit "Create a new response condition"

• A popup window will open where you need to put these details:

Field	Value
Name	e.g. Netacea_Log_Visit
Apply if	(req.url !~ "/media/" && req.url !~ "/static/") && fastly.ff.visits_this_service == 0
Priority	10 (default)

• Click on "Create". You should now see that the Condition has been attached to the Logging stream

• Now you can click on "Create" again and this will update Logging appropriately

Finishing Up

Check you have completed the following steps:

• Configured log streaming to the S3 bucket provided

• Ensure no error warnings are appearing

You are now ready to deploy by clicking "Save Config"

A Cache flush can sometimes be required. To do this go to System -> Cache Management and click "Flush Magento Cache".

Monitoring will now be deployed on the Fastly/Magento environment. You can verify the deployment is active by:

• Reviewing the active version for the new log shipping job.

• Requesting Netacea review internal ingest metrics.

• Requesting Netacea validate the data content & format.