ES Grafana Sample

[Allan Hardy]

Create an ES Generic Web Grafana dashboard as working with the InfluxDB reporter

[Allan Hardy]

@jenyayel started work on a grafana dashboard based on what I've done with InfluxDB. I'm having issues when group on tags i.e. I get the following error from ES. Any ideas? Do we need to index tags differently to support grouping?

{
    "root_cause": [
        {
            "type": "illegal_argument_exception",
            "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [tags.route] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
        }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
        {
            "shard": 0,
            "index": "appmetricssandbox",
            "node": "YUPWdZMPRiO1Vw14D73ICg",
            "reason": {
                "type": "illegal_argument_exception",
                "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [tags.route] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
            }
        }
    ],
    "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [tags.route] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
    }
}

Here's the Grafana JSON panel

{
  "annotations": {
    "list": []
  },
  "editMode": false,
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "hideControls": false,
  "id": 1,
  "links": [],
  "refresh": "5s",
  "rows": [
    {
      "collapse": false,
      "height": "250px",
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "datasource": "AppMetricsSandboxES",
          "fill": 1,
          "id": 1,
          "interval": "10s",
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "span": 12,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "bucketAggs": [
                {
                  "fake": true,
                  "field": "tags.route",
                  "id": "3",
                  "settings": {
                    "min_doc_count": 0,
                    "order": "asc",
                    "orderBy": "1",
                    "size": "5"
                  },
                  "type": "terms"
                },
                {
                  "field": "@timestamp",
                  "id": "2",
                  "settings": {
                    "interval": "auto",
                    "min_doc_count": 0,
                    "trimEdges": 0
                  },
                  "type": "date_histogram"
                }
              ],
              "dsType": "elasticsearch",
              "metrics": [
                {
                  "field": "fields.rate1m",
                  "id": "1",
                  "meta": {},
                  "settings": {},
                  "type": "avg"
                }
              ],
              "query": "name:application.httprequests__transactions_per_endpoint",
              "refId": "A",
              "timeField": "@timestamp"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Panel Title",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    }
  ],
  "schemaVersion": 14,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-5m",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "browser",
  "title": "AppMetricsSandbox_ES",
  "version": 0
}

[Jenya Y.]

You will need to define mappings for those fields, otherwise they are only indexed. You add fieldata to an existing index (which will allow sorting and grouping/aggregating) by calling:

PUT YOUR_INDEX_NAME/_mapping/timer?update_all_types
{
  "properties": {
    "tags.route": { 
      "type":     "text",
      "fielddata": true
    }
  }
}

Because each metric is a separate type, you will need to do it for each one (the example above is for timer).

[Allan Hardy]

@jenyayel cheers.

Added es grafana dashboards and configured in the main repo's sandbox project and included the requierd the mappings, see https://github.com/alhardy/AppMetrics/tree/1.1.0-alpha/sandbox

Also, the duplicated code in terms of reporters has been refactored, see https://github.com/alhardy/AppMetrics.Reporters/blob/1.1.0-alpha/src/App.Metrics.Extensions.Reporting.ElasticSearch/ElasticSearchReporterProvider.cs#L41