Skip to main content

Export Teleport Audit Events with Datadog

Report an Issue

Teleport's Event Handler plugin receives audit events from the Teleport Auth Service and forwards them to your log management solution, letting you perform historical analysis, detect unusual behavior, and form a better understanding of how users interact with your Teleport cluster.

Datadog is a SAAS monitoring and security platform. In this guide, we'll explain how to forward Teleport audit events to Datadog using Fluentd.

How it works​

The Teleport Event Handler authenticates to the Teleport Auth Service to receive audit events over a gRPC stream, then sends those events to Fluentd as JSON payloads over a secure channel established via mutual TLS:

Since the Datadog Agent can only receive logs from remote sources as JSON-encoded bytes over a TCP or UDP connection, the Teleport Event Handler needs to send its HTTPS payloads without using the Datadog Agent. Fluentd handles authentication to the Datadog API.

Prerequisites​

  • A Datadog account.
  • Fluentd version v1.12.4 or greater. The Teleport Event Handler will create a new fluent.conf file you can integrate into an existing Fluentd system, or use with a fresh setup.
  • A server, virtual machine, Kubernetes cluster, or Docker environment to run the Teleport Event Handler plugin.

This guide requires you to have completed one of the Event Handler setup guides:

The instructions below demonstrate a local test of the Event Handler plugin on your workstation. You will need to adjust paths, ports, and domains for other environments.

Step 1/2. Install the Fluentd output plugin for Datadog​

In order for Fluentd to communicate with Datadog, it requires the Fluentd output plugin for Datadog. Install the plugin on your Fluentd host using either gem or the td-agent, if installed:

Using Gem
gem install fluent-plugin-datadog

Using td-agent
/usr/sbin/td-agent-gem install fluent-plugin-datadog
Testing Locally?

If you're running Fluentd in a local Docker container for testing, you can adjust the entrypoint to an interactive shell as the root user, so you can install the plugin before starting Fluentd:

docker run -u $(id -u root):$(id -g root) -p 8888:8888 -v $(pwd):/keys -v \$(pwd)/fluent.conf:/fluentd/etc/fluent.conf --entrypoint=/bin/sh -i --tty  fluent/fluentd:edge
From the container shell:
gem install fluent-plugin-datadog
fluentd -c /fluentd/etc/fluent.conf

Configure Fluentd​

We will modify the fluent.conf file generated in the prerequisite setup guide.

  1. Visit Datadog and generate an API key for Fluentd by following the Datadog documentation.

  2. Copy the API key and use it to add a new <match> block to fluent.conf:

    <match test.log>

  @type datadog
  @id awesome_agent
  api_key abcd123-insecure-do-not-use-this

  host http-intake.logs.us5.datadoghq.com

  # Optional parameters
  dd_source teleport

</match>

  3. Edit your configuration as follows:

    • Add your API key to the api_key field.
    • Adjust the host value to match your Datadog site. See the Datadog Log Collection and Integrations guide to determine the correct value.
    • dd_source is an optional field you can use to filter these logs in the Datadog UI.
    • Adjust ca_path, cert_path and private_key_path to point to the credential files generated in the prerequisite setup guide. If you're testing locally, the Docker command above already mounted the current working directory to keys/ in the container.

  4. Restart Fluentd after saving the changes to fluent.conf.

Step 2/2. Run the Event Handler plugin​

In this section, you will modify the Event Handler configuration you generated and run the Event Handler to test your configuration.

Configure the Event Handler​

Edit the configuration for the Event Handler, depending on your installation method.

Earlier, we generated a file called teleport-event-handler.toml to configure the Teleport Event Handler. This file includes setting similar to the following:

storage = "./storage"
timeout = "10s"
batch = 20
# concurrency is the number of concurrent sessions to process. By default, this is set to 5.
concurrency = 5
# The window size configures the duration of the time window for the event handler
# to request events from Teleport. By default, this is set to 24 hours.
# Reduce the window size if the events backend cannot manage the event volume
# for the default window size.
# The window size should be specified as a duration string, parsed by Go's time.ParseDuration.
window-size = "24h"
# types is a comma-separated list of event types to search when forwarding audit
# events. For example, to limit forwarded events to user logins
# and new Access Requests, you can assign this field to
# "user.login,access_request.create".
types = ""
# skip-event-types is a comma-separated list of audit log event types to skip.
# For example, to forward all audit events except for new app deletion events,
# you can include the following assignment:
# skip-event-types = ["app.delete"]
skip-event-types = []
# skip-session-types is a comma-separated list of session recording event types to skip.
# For example, to forward all session events except for malformed SQL packet
# events, you can include the following assignment:
# skip-session-types = ["db.session.malformed_packet"]
skip-session-types = []

[forward.fluentd]
ca = /home/bob/event-handler/ca.crt
cert = /home/bob/event-handler/client.crt
key = /home/bob/event-handler/client.key
url = "https://fluentd.example.com:8888/test.log"
session-url = "https://fluentd.example.com:8888/session"

[teleport]
addr = teleport.example.com:443
identity = "identity"

Update the following fields.

[teleport]

addr: Include the hostname and HTTPS port of your Teleport Proxy Service or Teleport Enterprise Cloud account: teleport.example.com:443

identity: Fill this in with the path to the identity file you exported earlier.

If you are providing credentials to the Event Handler using a tbot binary that runs on a Linux server, make sure the value of identity in the Event Handler configuration is the same as the path of the identity file you configured tbot to generate, /opt/machine-id/identity.

[forward.fluentd]

ca: Include the path to the CA certificate: /home/bob/event-handler/ca.crt

cert: Include the path to the Fluentd client certificate. /home/bob/event-handler/client.crt

key: Include the path to the Fluentd client key. /home/bob/event-handler/client.key

url: Include the Fluentd URL where the audit event logs will be sent.

session-url: Include the Fluentd URL where the session logs will be sent.

Start the Event Handler​

Start the Teleport Event Handler by following the instructions below.

Copy the teleport-event-handler.toml file to /etc on your Linux server. Update the settings within the toml file to match your environment. Make sure to use absolute paths on settings such as identity and storage. Files and directories in use should only be accessible to the system user executing the teleport-event-handler service such as /var/lib/teleport-event-handler.

Next, create a systemd service definition at the path /usr/lib/systemd/system/teleport-event-handler.service with the following content:

[Unit]
Description=Teleport Event Handler
After=network.target

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/teleport-event-handler start --config=/etc/teleport-event-handler.toml --teleport-refresh-enabled=true
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport-event-handler.pid

[Install]
WantedBy=multi-user.target

If you are not using Machine & Workload Identity to provide short-lived credentials to the Event Handler, you can remove the --teleport-refresh-enabled true flag.

Enable and start the plugin:

sudo systemctl enable teleport-event-handler
sudo systemctl start teleport-event-handler
Choose when to start exporting events

You can configure when you would like the Teleport Event Handler to begin exporting events when you run the start command. This example will start exporting from May 5th, 2021:

teleport-event-handler start --config /etc/teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"

You can only determine the start time once, when first running the Teleport Event Handler. If you want to change the time frame later, remove the plugin state directory that you specified in the storage field of the handler's configuration file.

Once the Teleport Event Handler starts, you will see notifications about scanned and forwarded events:

sudo journalctl -u teleport-event-handler
DEBU   Event sent id:f19cf375-4da6-4338-bfdc-e38334c60fd1 index:0 ts:2022-09-2118:51:04.849 +0000 UTC type:cert.create event-handler/app.go:140...

The Logs view in Datadog should now report your Teleport cluster events:

Troubleshooting connection issues​

If the Teleport Event Handler is displaying error logs while connecting to your Teleport Cluster, ensure that:

  • The certificate the Teleport Event Handler is using to connect to your Teleport cluster is not past its expiration date. This is the value of the --ttl flag in the tctl auth sign command, which is 12 hours by default.
  • In your Teleport Event Handler configuration file, you have provided the correct host and port for the Teleport Proxy Service.
  • Start the Fluentd container prior to starting the Teleport Event Handler. The Event Handler will attempt to connect to Fluentd immediately upon startup.

Next steps​

  • Review the Fluentd output plugin for Datadog README file to learn how to customize the log format entering Datadog.
  • To see all of the options you can set in the values file for the teleport-plugin-event-handler Helm chart, consult our reference guide.