# Fastly

### **Fastly log streaming to S3**

To send real-time access logs from Fastly to an Amazon S3 bucket, you can configure **Fastly log streaming** using Amazon S3 as the destination. Below are the documented steps required to set up real-time streaming to a Netacea-hosted S3 bucket, using Fastly’s logging functionality and a custom log format that meets Netacea’s requirements.

### **Prerequisites**

To successfully stream web traffic logs to Netacea, please ensure the following:

* You have an active Fastly account with access to the Fastly Control Panel.
* A Fastly service is available and configured to handle your production traffic.
* You have received the necessary AWS S3 bucket details (bucket name, region, path, and credentials) from Netacea. These can be found [here](https://netacea.gitbook.io/portal-docs/user-guides-and-videos/portal-setup-and-log-shipping-guide).
* You are aware of the [minimum required dataset](#netacea-minimum-dataset) fields needed by Netacea for log analysis.

### Sending Data: Real-Time Streaming

This option logs every request and delivers logs in batches. This is configurable to suit the requirements of the POV.

### **Implementation Steps**

1. Log in to the [Fastly control panel](https://manage.fastly.com/).
2. From the [Home](https://manage.fastly.com/home) page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click **Edit configuration** and then select the option to clone the active version.

<figure><img src="/files/lVSQIt2uyhGQq8l9cVTC" alt=""><figcaption></figcaption></figure>

<figure><img src="/files/RwGcx3rRiSx4jTDZpJT4" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
All configuration changes below will be made to the **newly cloned version.**
{% endhint %}

### **Log Streaming Setup**

1. Navigate to the **Logging** area.
2. Click the '**Create endpoint'** button for Amazon S3

<figure><img src="/files/zkzCcKJ7bm4eka9DwlP0" alt=""><figcaption></figcaption></figure>

3. Enter the details from the table below:

<table><thead><tr><th width="196">Field</th><th>Value</th></tr></thead><tbody><tr><td>Name</td><td>Enter a memorable human-readable name for the endpoint, such as <strong>Netacea_Logging</strong></td></tr><tr><td>Placement</td><td>Format Version Default</td></tr><tr><td>Log Format</td><td>Paste the <strong>Log Format</strong> found <a href="#log-format">here</a></td></tr><tr><td>Timestamp</td><td>Paste the <strong>Timestamp Format</strong> found <a href="#timestamp-format">here</a></td></tr><tr><td>Bucket Name</td><td>The name of the Amazon S3 bucket provided by Netacea within the <a href="https://portal.netacea.com/">admin portal</a></td></tr><tr><td>Domain</td><td>s3.eu-west-1.amazonaws.com</td></tr><tr><td>Access Method</td><td>Select <strong>'User Credentials'</strong></td></tr><tr><td>Access Key</td><td>Copy from the Netacea <a href="https://portal.netacea.com/">admin portal</a></td></tr><tr><td>Secret Key</td><td>Copy from the Netacea <a href="https://portal.netacea.com/">admin portal</a></td></tr><tr><td>Period</td><td>15</td></tr></tbody></table>

4. Expand the '**Advanced options'** section and apply the specific values listed in the table below:

| Option                 | Value    |
| ---------------------- | -------- |
| Path                   | /        |
| PGP Public Key         | Blank    |
| Select Log line Format | Blank    |
| Compression            | Gzip     |
| Redundancy Level       | Standard |
| ACL                    | None     |
| Server Side Encryption | None     |
| Maximum Bytes          | 0        |

5. Click **'Create'** to save the logging endpoint.

### Log Format

{% code overflow="wrap" %}

```
{"@timestamp": "%{%Y-%m-%dT%H:%M:%S%z}t","bc_type": %{if(req.http.netacea_bctype_string, "%u0022" + json.escape(req.http.netacea_bctype_string) + "%u0022", "null")}V,"bytes_sent": "%B","cookie_session_status": %{if(req.http.x-netacea:cookie_session_status, "%u0022" + json.escape(req.http.x-netacea:cookie_session_status) + "%u0022", "null")}V,"client": "%{json.escape(client.ip)}V","domain": %{if(req.http.host, "%u0022" + json.escape(req.http.host) + "%u0022", "null")}V,"integration_mode": %{"%u0022" + json.escape(req.http.x-netacea:integration_mode) + "%u0022"}V,"integration_type": %{if(req.http.integration_type, "%u0022" + json.escape(req.http.integration_type) + "%u0022", "null")}V,"integration_version": %{if(req.http.integration_version, "%u0022" + json.escape(req.http.integration_version) + "%u0022", "null")}V,"method": "%{json.escape(req.method)}V","path": "%{json.escape(req.url.path)}V","mit_svc_latency":%{if(req.http.x-netacea:mit_svc_latency, "%u0022" + json.escape(req.http.x-netacea:mit_svc_latency) + "%u0022", "0")}V,"mit_status":%{if(req.http.x-netacea:mit_status, "%u0022" + json.escape(req.http.x-netacea:mit_status) + "%u0022", "0")}V,"protocol": "%{json.escape(req.proto)}V","query": "%{json.escape(req.url.qs)}V","referrer": %{if(req.http.referer, "%u0022" + json.escape(req.http.referer) + "%u0022", "null")}V,"request_time": %{time.elapsed}V,"status": "%{json.escape(resp.status)}V","user_agent": %{if(req.http.user-agent, "%u0022" + json.escape(req.http.user-agent) + "%u0022", "null")}V,"user_id": %{if(req.http.x-netacea-userid, "%u0022" + json.escape(req.http.x-netacea-userid) + "%u0022", "null")}V,"client_ja3_md5": %{if(tls.client.ja3_md5, "%u0022" + json.escape(tls.client.ja3_md5) + "%u0022", "null")}V,"x_forwarded_for": %{if(req.http.X-Forwarded-For, "%u0022" + json.escape(req.http.X-Forwarded-For) + "%u0022", "null")}V}
```

{% endcode %}

### Timestamp Format

```
%Y-%m-%dT%H:%M:%S.000
```

### Netacea Minimum Dataset

The above log format will allow Netacea to collect the following minimum dataset for analysis

<table><thead><tr><th width="215">Required Fields</th><th>Description</th></tr></thead><tbody><tr><td>Timestamp</td><td>The time at which the request was received</td></tr><tr><td>IP Address</td><td>The IP address from which the request was made</td></tr><tr><td>User Agent</td><td>The user agent string sent in the header by the client</td></tr><tr><td>Method</td><td>The HTTP method of the request</td></tr><tr><td>Path</td><td>The path of the requested resource</td></tr><tr><td>Query</td><td>The query string of the request</td></tr><tr><td>Status</td><td>The HTTP status code returned by the server</td></tr><tr><td>Referrer</td><td>The web page the user followed a link from</td></tr><tr><td>Bytes Sent</td><td>The Bytes sent as part of the servers response</td></tr><tr><td>Client JA3</td><td>Clients JA3 fingerprint</td></tr><tr><td>X-Forwarded-For*</td><td>Original IP address of a client request</td></tr><tr><td>Host</td><td>The destination host of the request</td></tr><tr><td>Protocol</td><td>The protocol of the request</td></tr><tr><td>Request Time</td><td>The complete amount of time it took to process the request</td></tr></tbody></table>

\* Useful for when proxies are in the line of traffic from client > origin

### Logging Considerations

When configuring Fastly log shipping, by default it will send all requests, including static & media content to the S3 bucket. This data is not required by Netacea. In order to solve this, we need to add a logging condition to the service and attach it to the logging service that has been created following the steps above.

#### Add a Logging Condition

1. Navigate to the newly created logging endpoint:

<figure><img src="/files/NcNBrzSZ5rk2zmS8jzhm" alt=""><figcaption></figcaption></figure>

2. Click **'Create a new response condition'**

<figure><img src="/files/SDhD7WCwbRwSxptXuSCU" alt=""><figcaption></figcaption></figure>

3. Enter the following details into the popup window

| Field    | Value                                                                                     |
| -------- | ----------------------------------------------------------------------------------------- |
| Name     | e.g. Netacea\_Log\_Visit                                                                  |
| Apply If | (req.url !\~ "/media/" && req.url !\~ "/static/") && fastly.ff.visits\_this\_service == 0 |
| Priority | 10                                                                                        |

{% hint style="warning" %}
Expand the **'Advanced option'** section to set the Priority
{% endhint %}

<figure><img src="/files/X1NYkF1OWiTxyoOa3uge" alt=""><figcaption></figcaption></figure>

4. Click **'Save and apply'**

### Finishing Up

* Check the configuration for any error messages or warnings.
* Once confirmed, deploy your changes by clicking **Activate**.

<figure><img src="/files/ax7QFgQ4d9zGRjjTZifJ" alt=""><figcaption></figcaption></figure>

Your Fastly log streaming configuration is now live and should be sending data to the Netacea S3 bucket.

### Final Steps

Once log delivery is active, **notify your Netacea Solutions Engineer**. This allows us to:

* Confirm receipt of data
* Validate the dataset format and completeness
* Begin analysis once a sufficient volume of data has been collected

If you require support during setup, the Netacea SE team is here to help — don’t hesitate to reach out.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.netacea.com/fastly.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.