Fastly

Real-time data integration from Fastly to Netacea using S3 streaming

Fastly log streaming to S3

To send real-time access logs from Fastly to an Amazon S3 bucket, you can configure Fastly log streaming using Amazon S3 as the destination. Below are the documented steps required to set up real-time streaming to a Netacea-hosted S3 bucket, using Fastly’s logging functionality and a custom log format that meets Netacea’s requirements.

Prerequisites

To successfully stream web traffic logs to Netacea, please ensure the following:

• You have an active Fastly account with access to the Fastly Control Panel.

• A Fastly service is available and configured to handle your production traffic.

• You have received the necessary AWS S3 bucket details (bucket name, region, path, and credentials) from Netacea. These can be found here.

• You are aware of the minimum required dataset fields needed by Netacea for log analysis.

Sending Data: Real-Time Streaming

This option logs every request and delivers logs in batches. This is configurable to suit the requirements of the POV.

Implementation Steps

1. Log in to the Fastly control panel.

2. From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.

3. Click Edit configuration and then select the option to clone the active version.

All configuration changes below will be made to the newly cloned version.

Log Streaming Setup

1. Navigate to the Logging area.

2. Click the 'Create endpoint' button for Amazon S3

1. Enter the details from the table below:

Field	Value
Name	Enter a memorable human-readable name for the endpoint, such as Netacea_Logging
Placement	Format Version Default
Log Format	Paste the Log Format found here
Timestamp	Paste the Timestamp Format found here
Bucket Name	The name of the Amazon S3 bucket provided by Netacea within the admin portal
Domain	s3.eu-west-1.amazonaws.com
Access Method	Select 'User Credentials'
Access Key	Copy from the Netacea admin portal
Secret Key	Copy from the Netacea admin portal
Period	15

1. Expand the 'Advanced options' section and apply the specific values listed in the table below:

Option	Value
Path	/
PGP Public Key	Blank
Select Log line Format	Blank
Compression	Gzip
Redundancy Level	Standard
ACL	None
Server Side Encryption	None
Maximum Bytes	0

1. Click 'Create' to save the logging endpoint.

Log Format

{"@timestamp": "%{%Y-%m-%dT%H:%M:%S%z}t","bc_type": %{if(req.http.netacea_bctype_string, "%u0022" + json.escape(req.http.netacea_bctype_string) + "%u0022", "null")}V,"bytes_sent": "%B","cookie_session_status": %{if(req.http.x-netacea:cookie_session_status, "%u0022" + json.escape(req.http.x-netacea:cookie_session_status) + "%u0022", "null")}V,"client": "%{json.escape(client.ip)}V","domain": %{if(req.http.host, "%u0022" + json.escape(req.http.host) + "%u0022", "null")}V,"integration_mode": %{"%u0022" + json.escape(req.http.x-netacea:integration_mode) + "%u0022"}V,"integration_type": %{if(req.http.integration_type, "%u0022" + json.escape(req.http.integration_type) + "%u0022", "null")}V,"integration_version": %{if(req.http.integration_version, "%u0022" + json.escape(req.http.integration_version) + "%u0022", "null")}V,"method": "%{json.escape(req.method)}V","path": "%{json.escape(req.url.path)}V","mit_svc_latency":%{if(req.http.x-netacea:mit_svc_latency, "%u0022" + json.escape(req.http.x-netacea:mit_svc_latency) + "%u0022", "0")}V,"mit_status":%{if(req.http.x-netacea:mit_status, "%u0022" + json.escape(req.http.x-netacea:mit_status) + "%u0022", "0")}V,"protocol": "%{json.escape(req.proto)}V","query": "%{json.escape(req.url.qs)}V","referrer": %{if(req.http.referer, "%u0022" + json.escape(req.http.referer) + "%u0022", "null")}V,"request_time": %{time.elapsed}V,"status": "%{json.escape(resp.status)}V","user_agent": %{if(req.http.user-agent, "%u0022" + json.escape(req.http.user-agent) + "%u0022", "null")}V,"user_id": %{if(req.http.x-netacea-userid, "%u0022" + json.escape(req.http.x-netacea-userid) + "%u0022", "null")}V,"client_ja3_md5": %{if(tls.client.ja3_md5, "%u0022" + json.escape(tls.client.ja3_md5) + "%u0022", "null")}V,"x_forwarded_for": %{if(req.http.X-Forwarded-For, "%u0022" + json.escape(req.http.X-Forwarded-For) + "%u0022", "null")}V}

Timestamp Format

%Y-%m-%dT%H:%M:%S.000

Netacea Minimum Dataset

The above log format will allow Netacea to collect the following minimum dataset for analysis

Required Fields	Description
Timestamp	The time at which the request was received
IP Address	The IP address from which the request was made
User Agent	The user agent string sent in the header by the client
Method	The HTTP method of the request
Path	The path of the requested resource
Query	The query string of the request
Status	The HTTP status code returned by the server
Referrer	The web page the user followed a link from
Bytes Sent	The Bytes sent as part of the servers response
Client JA3	Clients JA3 fingerprint
X-Forwarded-For*	Original IP address of a client request
Host	The destination host of the request
Protocol	The protocol of the request
Request Time	The complete amount of time it took to process the request

* Useful for when proxies are in the line of traffic from client > origin

Logging Considerations

When configuring Fastly log shipping, by default it will send all requests, including static & media content to the S3 bucket. This data is not required by Netacea. In order to solve this, we need to add a logging condition to the service and attach it to the logging service that has been created following the steps above.

Add a Logging Condition

1. Navigate to the newly created logging endpoint:

1. Click 'Create a new response condition'

1. Enter the following details into the popup window

Field	Value
Name	e.g. Netacea_Log_Visit
Apply If	(req.url !~ "/media/" && req.url !~ "/static/") && fastly.ff.visits_this_service == 0
Priority	10

Expand the 'Advanced option' section to set the Priority

1. Click 'Save and apply'

Finishing Up

• Check the configuration for any error messages or warnings.

• Once confirmed, deploy your changes by clicking Activate.

Your Fastly log streaming configuration is now live and should be sending data to the Netacea S3 bucket.

Final Steps

Once log delivery is active, notify your Netacea Solutions Engineer. This allows us to:

• Confirm receipt of data

• Validate the dataset format and completeness

• Begin analysis once a sufficient volume of data has been collected

If you require support during setup, the Netacea SE team is here to help — don’t hesitate to reach out.