Funnel

Last updated on November 19, 2015


  1. Home
  2. Overview
  3. Getting Started
  4. Modules
  5. Services

Introduction

Funnel is a distributed monitoring system based on a lightweight streaming protocol. To get started, please read the most appropriate documentation:

Funnel allows systems to push, pull, mirror, and aggregate metrics from disparate systems, all with a unified API. High-level dashboards can then be created by downstream consumers. Here is a screenshot of a Kibana dashboard visualising telemetry data collected by Funnel into ElasticSearch:

image

Design Overview

This section outlines some of the specific design choices in Funnel that make it different from other monitoring systems. Above all else, Funnel is a stream processor and all metrics are modeled as streams internally. At a high-level, the design can be split into four discrete parts:

design

Each part of this pipeline has a distinct function:

This is the view of the Funnel system from the perspective of a single machine, but this design composes accross the network to create a large distributed system. Viewed at a larger scale, the system can be visualized like this:

design

The specifics of this diagram are covered in the operatations guide section.

Several design choices were made specifically with the goal of giving Funnel the ability to scale linearly. The most obvious feature is that the relatioship between Agent and Flask is pull-based. By downsampling at the level of individual agents, the aggregating Flask systems never see load spikes that might cause cascading failures. The load on a Flask depends on the number of metric keys it has been asked to consume, and not on the volume of changes to the metrics under those keys. If the data rate on a given host increases, only the overhead on that host is affected since the downstream system pulls samples at a semi-fixed rate.

Time Periods

As with all monitoring systems, understanding the way time periods are handled is really important. A time period represents a time in which metrics are collected, and resets occur at the period boundaries. Funnel supports three different time window types, as illustrated by the diagram below:

image

In this diagram, T represents time moving left to right. A, B and C illustrate time period boundaries. The duration of time periods is arbitrary, but the default setting in Funnel is 1 minute. In this frame:

Units

A given metric is denoted in specific Units. For example, the value 18544 for, say, memory usage would not be especially useful. Funnel will be explicit about whether that is 18544 bytes or 18544 megabytes. At the time of writing the library supports the following Units:

This is not something that users generally will ever need to extend, but when using the gauge instrument you will need to specify the units for that gauge.