# Monitoring Configuration

### Prerequisites

Before Netacea can monitor your platform successfully, please ensure you have:

1. Provided Netacea with your Fastly [IAM Role](https://docs.fastly.com/en/guides/creating-an-aws-iam-role-for-fastly-logging), if requested.
2. Had the relevant log shipping endpoints & credentials shared via the [Netacea portal](https://docs.netacea.com/netacea-plugin-information/accessing-your-integration-settings).
3. Provided a list of all known, safe and trusted list users, partners & 3rd party apps. This includes a list of trusted bots & automated tools. Any information on known bad actors is also valuable.
4. Provided a list of protection use cases and the pathways that are vulnerable to each.

### Minimum Required Dataset

To successfully analyze data, the following data will be monitored:

<table><thead><tr><th width="234">Required Fields</th><th>Description</th></tr></thead><tbody><tr><td>Timestamp</td><td>The time which the request was received</td></tr><tr><td>IP Address</td><td>The IP address from which the request was made</td></tr><tr><td>User Agent</td><td>The user agent string sent in the header by the client</td></tr><tr><td>Method</td><td>The HTTP method of the request</td></tr><tr><td>Path</td><td>The path of the requested resource</td></tr><tr><td>Query</td><td>The query string of the request</td></tr><tr><td>Status</td><td>The HTTP status code returned by the server</td></tr><tr><td>Referrer</td><td>The web page the user followed a link from</td></tr><tr><td>Bytes Sent</td><td>The Bytes sent as part of the user's request</td></tr><tr><td>Protocol</td><td>The protocol of the response-request cycle.</td></tr><tr><td>Request Time</td><td>How long the request took in microseconds.</td></tr><tr><td>Request Host</td><td>The current Host request header.</td></tr><tr><td>JA3 Fingerprint</td><td>JA3 is a method for creating SSL/TLS client fingerprints.</td></tr></tbody></table>

{% hint style="info" %}
In addition to the above standard fields, the Netacea integration will add several custom fields to the log format.\
\
These include:<br>

`bc_type`, `user_id`, `integration_type` and `integration_version`.
{% endhint %}

### Implementation Steps

1. Log in to the Fastly web interface.
2. From the home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click the edit configuration button and then select the option to clone the active version. The Domains page appears.

All configuration changes below will be made to the newly cloned version.

{% hint style="info" %}
Make a note of the currently active version should you need to roll back in the future.
{% endhint %}

### Log Streaming

* Navigate to the Logging area.
* Click the "Create endpoint" button for Amazon S3

<figure><img src="https://3359534748-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8KQH1bDl0sVMvZgHUjkC%2Fuploads%2Fgit-blob-abbcbac1293cdea7b300d226ce1b4b68e72f589b%2Fimage%20(1)%20(1)%20(1)%20(1)%20(1)%20(1)%20(1)%20(1).png?alt=media" alt=""><figcaption></figcaption></figure>

* Enter the following details:

<table><thead><tr><th width="186">Field</th><th>Value</th></tr></thead><tbody><tr><td>Name</td><td>Enter a memorable human-readable name for the endpoint, such as Netacea_Logging.</td></tr><tr><td>Placement</td><td>Format Version Default</td></tr><tr><td>Log Format</td><td><a href="#undefined">Log Format</a></td></tr><tr><td>Timestamp</td><td><a href="#timestamp">Timestamp</a></td></tr><tr><td>Bucket Name</td><td>The name of the Amazon S3 bucket provided by <a href="../accessing-your-integration-settings">Netacea portal</a>.</td></tr><tr><td>Access Method</td><td>The IAM Role ARN or User Credentials provided by <a href="../accessing-your-integration-settings">Netacea portal</a>.</td></tr><tr><td>Period</td><td>15</td></tr><tr><td>Path</td><td>/</td></tr><tr><td>Domain</td><td><p>The domain of the Amazon S3 endpoint that has been provided by Netacea.</p><p>If not specified, please use <code>s3.eu-west-1.amazonaws.com</code>.</p></td></tr><tr><td>Select a log line format</td><td>Blank</td></tr><tr><td>Compression</td><td>Gzip</td></tr></tbody></table>

{% hint style="info" %}
Any option not specified above should be left as the default value.
{% endhint %}

#### Log Format

{% code overflow="wrap" fullWidth="false" %}

```
{"@timestamp": "%{%Y-%m-%dT%H:%M:%S%z}t","bc_type": %{if(req.http.netacea_bctype_string, "%u0022" + json.escape(req.http.netacea_bctype_string) + "%u0022", "null")}V,"bytes_sent": "%B","cookie_session_status": %{if(req.http.x-netacea:cookie_session_status, "%u0022" + json.escape(req.http.x-netacea:cookie_session_status) + "%u0022", "null")}V,"client": "%{json.escape(client.ip)}V","domain": %{if(req.http.host, "%u0022" + json.escape(req.http.host) + "%u0022", "null")}V,"integration_mode": %{"%u0022" + json.escape(req.http.x-netacea:integration_mode) + "%u0022"}V,"integration_type": %{if(req.http.integration_type, "%u0022" + json.escape(req.http.integration_type) + "%u0022", "null")}V,"integration_version": %{if(req.http.integration_version, "%u0022" + json.escape(req.http.integration_version) + "%u0022", "null")}V,"method": "%{json.escape(req.method)}V","path": "%{json.escape(req.url.path)}V","mit_svc_latency":%{if(req.http.x-netacea:mit_svc_latency, "%u0022" + json.escape(req.http.x-netacea:mit_svc_latency) + "%u0022", "0")}V,"mit_status":%{if(req.http.x-netacea:mit_status, "%u0022" + json.escape(req.http.x-netacea:mit_status) + "%u0022", "0")}V,"protocol": "%{json.escape(req.proto)}V","query": "%{json.escape(req.url.qs)}V","referrer": %{if(req.http.referer, "%u0022" + json.escape(req.http.referer) + "%u0022", "null")}V,"request_time": %{time.elapsed}V,"status": "%{json.escape(resp.status)}V","user_agent": %{if(req.http.user-agent, "%u0022" + json.escape(req.http.user-agent) + "%u0022", "null")}V,"user_id": %{if(req.http.x-netacea-userid, "%u0022" + json.escape(req.http.x-netacea-userid) + "%u0022", "null")}V,"client_ja3_md5": %{if(tls.client.ja3_md5, "%u0022" + json.escape(tls.client.ja3_md5) + "%u0022", "null")}V,"x_forwarded_for": %{if(req.http.X-Forwarded-For, "%u0022" + json.escape(req.http.X-Forwarded-For) + "%u0022", "null")}V}
```

{% endcode %}

#### Timestamp

```
%Y-%m-%dT%H:%M:%S.000
```

### Logging considerations

Fastly allows users to enable the Host's feature called "[Shielding](https://developer.fastly.com/learning/concepts/shielding/)" which has some benefits like reducing origin load, improving cache hit ratio, etc. Enabling this feature in a Fastly service that uses our Integration results in a duplication of logs.

When configuring Fastly log shipping by default, it will send all requests, including static & media content that is not required by Netacea.\
\
In order to solve this issue we need to add a Condition to the service and attach it to Netacea Logging.

* Navigate to Conditions area
* Click "Create Condition"

<figure><img src="https://3359534748-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8KQH1bDl0sVMvZgHUjkC%2Fuploads%2Fgit-blob-01f9ffacc8cd724d6e509de7ddae8d0ecf403a4a%2Fimage%20(1)%20(1)%20(1)%20(1)%20(1).png?alt=media" alt=""><figcaption></figcaption></figure>

* Enter the following details in the popup window:

<table><thead><tr><th width="186">Field</th><th>Value</th></tr></thead><tbody><tr><td>Type</td><td>Response</td></tr><tr><td>Name</td><td>e.g. Netacea_Log_Visit</td></tr><tr><td>Apply if</td><td><code>(req.url !~ "/media/" &#x26;&#x26; req.url !~ "/static/") &#x26;&#x26; fastly.ff.visits_this_service == 0</code></td></tr><tr><td>Priority</td><td>10 (default)</td></tr></tbody></table>

* Click "Save" and navigate to Logging area
* Find Netacea Logging and click on the "Attach a condition"

<figure><img src="https://3359534748-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8KQH1bDl0sVMvZgHUjkC%2Fuploads%2Fgit-blob-2eff7aca41eb2212627ed8279214562586b2c974%2Fimage%20(2)%20(1).png?alt=media" alt=""><figcaption></figcaption></figure>

* In the popup window, select the newly created condition

<figure><img src="https://3359534748-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8KQH1bDl0sVMvZgHUjkC%2Fuploads%2Fgit-blob-f4ce847ed63fb722d1bfdb5f38d0f269ce5b686b%2Fimage%20(3).png?alt=media" alt=""><figcaption></figcaption></figure>

### Finishing Up

Check you have completed the following steps:

* Ensure no error warnings are appearing

You are now ready to deploy the version you have been editing by clicking "Activate".

<figure><img src="https://3359534748-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F8KQH1bDl0sVMvZgHUjkC%2Fuploads%2Fgit-blob-d8a873b16f5962dbf29435aaa23055f7cca33664%2Fimage%20(12)%20(2).png?alt=media" alt=""><figcaption></figcaption></figure>

Log shipping will now be deployed on the service it was configured against. You can verify the deployment is correct by:

* Reviewing the active version for the new log shipping job.
* Requesting Netacea review internal ingest metrics.
* Requesting Netacea validate the data content & format.