Elastic Data stream

Elastic Data stream

Elasticsearch

Time Series Data (TSD) Time Series Data

Data stream

Time Series Data?

TSD

ES

Time Series Data

  • +

         - ...

 

Time Series Data

TSD

close/frozen 

ILM

 

Data stream

Data stream  Elastic Stack 7.9 Data stream data stream

data stream stream ILM ILM ILM

 

Backing indices

hidden Backing indices

@timestamp date date_nanos @timestamp Elasticsearch @timestamp

 

 

 

Rollover

Elasticsearch rollover

ILM rollover

 

rollover 000001

.ds-<data-stream>-<generation>

generation web-server-logs generation  34 .ds-web-server-logs-000034

shrink restore

 

 update by query   delete by query 

ILM

ILM data stream

 

Elastic Stack

  Elasticsearch Index node Elasticsearch

Elasticsearch shard filtering node cluster Elasticsearch terminal

./bin/elasticsearch -E node.name=node1 -E node.attr.data=hot -Enode.max_local_storage_nodes=2

node1 terminal

./bin/elasticsearch -E node.name=node2 -E node.attr.data=warm -Enode.max_local_storage_nodes=2

node2

GET _cat/nodes?v

node

GET _cat/nodeattrs?v&s=name

node hot warm

Elasticsearch

 

  data stream

ILM rollover

  • Lifecycle Policy
  • data stream index template
  • data stream
  • Lifecycle

ILM 4 Hot, Warm, Cold, Delete Cold

 

Index Lifecycle Policy

Policy demo Hot phase rollover

  • 1G
  • 5
  • 30

rollover

Warm phase:

Warm phase phase warm warm number of replica 0 replica read Shrink index warm phase primary shard primary shard warm phase hot

Delete phase

Delete phase  Warm 3 Save as new policy API Policy:

GET _ilm/policy/demo
{ "demo" : { "version" : 3, "modified_date" : "2020-12-03T14:33:30.508Z", "policy" : { "phases" : { "warm" : { "min_age" : "0ms", "actions" : { "allocate" : { "number_of_replicas" : 0, "include" : { }, "exclude" : { }, "require" : { "data" : "warm" } }, "shrink" : { "number_of_shards" : 1 }, "set_priority" : { "priority" : 50 } } }, "hot" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "1gb", "max_age" : "30d", "max_docs" : 5 }, "set_priority" : { "priority" : 100 } } }, "delete" : { "min_age" : "3m", "actions" : { "delete" : { "delete_searchable_snapshot" : true } } } } } } }

7.9.1 7.9.3 7.10 delete API

PUT _ilm/policy/demo { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_age": "30d", "max_size": "1gb", "max_docs": 5 }, "set_priority": { "priority": 100 } } }, "warm": { "actions": { "allocate": { "number_of_replicas": 0, "include": {}, "exclude": {}, "require": { "data": "warm" } }, "shrink": { "number_of_shards": 1 }, "set_priority": { "priority": 50 } } }, "delete": { "min_age": "3m", "actions": { "delete" : {} } } } } }

delete actions "delete": {}

 

 index template

Kibana console

# Create tje template to apply the policy to every new backing index of the data stream PUT _index_template/template_demo { "index_patterns": ["demo-*"], "data_stream": {}, "priority": 200, "template": { "settings": { "number_of_shards": 2, "index.lifecycle.name": "demo", "index.routing.allocation.require.data": "hot" } } }

template_demo index template data_stream object hot warm tag data primary shards

 

data stream

  data stream

# Create a data stream PUT _data_stream/demo-ds

 demo-* index_pattern index template

PUT _data_stream/demo

index template

# Check the shards allocation GET _cat/shards/demo-ds?v

index shard prirep state docs store ip node .ds-demo-ds-000001 1 p STARTED 0 208b 127.0.0.1 node1 .ds-demo-ds-000001 1 r UNASSIGNED .ds-demo-ds-000001 0 p STARTED 0 208b 127.0.0.1 node1 .ds-demo-ds-000001 0 r UNASSIGNED

primary shards hot replica shards

data stream

# Verify data stream indexes GET _data_stream/demo-ds

{ "data_streams" : [ { "name" : "demo-ds", "timestamp_field" : { "name" : "@timestamp" }, "indices" : [ { "index_name" : ".ds-demo-ds-000001", "index_uuid" : "sPN5JEW8SVuTFNh4UqK9zw" } ], "generation" : 1, "status" : "YELLOW", "template" : "template_demo", "ilm_policy" : "demo" } ] }

 .ds-demo-ds-000001

GET .ds-demo-ds-000001/_settings

{ ".ds-demo-ds-000001" : { "settings" : { "index" : { "lifecycle" : { "name" : "demo" }, "routing" : { "allocation" : { "require" : { "data" : "hot" } } }, "hidden" : "true", "number_of_shards" : "2", "provided_name" : ".ds-demo-ds-000001", "creation_date" : "1606989315969", "priority" : "100", "number_of_replicas" : "1", "uuid" : "sPN5JEW8SVuTFNh4UqK9zw", "version" : { "created" : "7100099" } } } }

hot hidden true

 

data stream

PUT _ingest/pipeline/add-timestamp { "processors": [ { "set": { "field": "@timestamp", "value": "{{_ingest.timestamp}}" } } ] }

 add-timestamp pipeline timestamp pipeline

POST demo-ds/_doc?pipeline=add-timestamp { "user": { "id": "liuxg" }, "message": "This is so cool!" }

{ "_index" : ".ds-demo-ds-000001", "_type" : "_doc", "_id" : "3JsVKHYBtwZVzHJGZXRc", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 0, "_primary_term" : 1 }

 .ds-demo-ds-000001

# Search the data stream GET demo-ds/_search

{ "took" : 351, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 1, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : ".ds-demo-ds-000001", "_type" : "_doc", "_id" : "3JsVKHYBtwZVzHJGZXRc", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T10:10:59.541518Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } } ] } }

 .ds-demo-ds-000001

4

POST demo-ds/_doc?pipeline=add-timestamp { "user": { "id": "liuxg" }, "message": "This is so cool!" }

{ "_index" : ".ds-demo-ds-000001", "_type" : "_doc", "_id" : "4JueKHYBtwZVzHJGfnRv", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 2, "_primary_term" : 1 }

5  .ds-demo-ds-000001 ILM Policy 5 rollover, warm

POST demo-ds/_doc?pipeline=add-timestamp { "user": { "id": "liuxg" }, "message": "This is so cool!" }

6 ILM

# Check ILM status per demo-ds data stream GET demo-ds/_ilm/explain

{ "indices" : { ".ds-demo-ds-000001" : { "index" : ".ds-demo-ds-000001", "managed" : true, "policy" : "demo", "lifecycle_date_millis" : 1606989315969, "age" : "2.86h", "phase" : "hot", "phase_time_millis" : 1606989317057, "action" : "rollover", "action_time_millis" : 1606989760360, "step" : "check-rollover-ready", "step_time_millis" : 1606989760360, "phase_execution" : { "policy" : "demo", "phase_definition" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "1gb", "max_age" : "30d", "max_docs" : 5 }, "set_priority" : { "priority" : 100 } } }, "version" : 1, "modified_date_in_millis" : 1606988416663 } } } }

action rollover rollover

indices.lifecycle.poll_interval

10

PUT _cluster/settings { "transient": { "indices.lifecycle.poll_interval": "10s" } }

Elasticsearch 10 ILM policy

GET demo-ds/_ilm/explain

{ "indices" : { ".ds-demo-ds-000002" : { "index" : ".ds-demo-ds-000002", "managed" : true, "policy" : "demo", "lifecycle_date_millis" : 1606999959439, "age" : "5.22m", "phase" : "hot", "phase_time_millis" : 1606999961410, "action" : "rollover", "action_time_millis" : 1607000204426, "step" : "check-rollover-ready", "step_time_millis" : 1607000204426, "phase_execution" : { "policy" : "demo", "phase_definition" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "1gb", "max_age" : "30d", "max_docs" : 5 }, "set_priority" : { "priority" : 100 } } }, "version" : 1, "modified_date_in_millis" : 1606988416663 } }, ".ds-demo-ds-000001" : { "index" : ".ds-demo-ds-000001", "managed" : true, "policy" : "demo", "lifecycle_date_millis" : 1606999959458, "age" : "5.22m", "phase" : "warm", "phase_time_millis" : 1606999962564, "action" : "shrink", "action_time_millis" : 1607000207450, "step" : "shrink", "step_time_millis" : 1607000271877, "phase_execution" : { "policy" : "demo", "phase_definition" : { "min_age" : "0ms", "actions" : { "allocate" : { "number_of_replicas" : 0, "include" : { }, "exclude" : { }, "require" : { "data" : "warm" } }, "shrink" : { "number_of_shards" : 1 }, "set_priority" : { "priority" : 50 } } }, "version" : 1, "modified_date_in_millis" : 1606988416663 } } } }

Warm phase, shrink action

GET demo-ds/_search

{ "took" : 2, "timed_out" : false, "_shards" : { "total" : 3, "successful" : 3, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 8, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "3ZueKHYBtwZVzHJGdHRU", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:40:41.812609Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "3pueKHYBtwZVzHJGd3Sd", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:40:42.653342Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "4JueKHYBtwZVzHJGfnRv", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:40:44.399382Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "4puiKHYBtwZVzHJGeHT3", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:45:05.143514Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "45unKHYBtwZVzHJGK3RB", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:50:12.929209Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "3JsVKHYBtwZVzHJGZXRc", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T10:10:59.541518Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "35ueKHYBtwZVzHJGe3S4", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:40:43.704291Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "4ZuhKHYBtwZVzHJGb3Si", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:43:57.217932Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } } ] } }

8  shrink-.ds-demo-ds-000001 Warm phase shrink primary

POST demo-ds/_doc?pipeline=add-timestamp { "user": { "id": "liuxg" }, "message": "This is so cool!" }

{ "_index" : ".ds-demo-ds-000002", "_type" : "_doc", "_id" : "C5uzKHYBtwZVzHJGDHW6", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 0, "_primary_term" : 1 }

 .ds-demo-ds-000002  .ds-demo-ds-000001

# Search the data stream GET demo-ds/_search

9

{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 3, "successful" : 3, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 9, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "3ZueKHYBtwZVzHJGdHRU", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:40:41.812609Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "3pueKHYBtwZVzHJGd3Sd", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:40:42.653342Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "4JueKHYBtwZVzHJGfnRv", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:40:44.399382Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "4puiKHYBtwZVzHJGeHT3", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:45:05.143514Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "45unKHYBtwZVzHJGK3RB", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:50:12.929209Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "3JsVKHYBtwZVzHJGZXRc", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T10:10:59.541518Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "35ueKHYBtwZVzHJGe3S4", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:40:43.704291Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000001", "_type" : "_doc", "_id" : "4ZuhKHYBtwZVzHJGb3Si", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T12:43:57.217932Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : ".ds-demo-ds-000002", "_type" : "_doc", "_id" : "C5uzKHYBtwZVzHJGDHW6", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T13:03:11.546240Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } } ] } }

 shrink-.ds-demo-ds-000001  .ds-demo-ds-000002

data stream

GET _data_stream/demo-ds
{ "data_streams" : [ { "name" : "demo-ds", "timestamp_field" : { "name" : "@timestamp" }, "indices" : [ { "index_name" : "shrink-.ds-demo-ds-000001", "index_uuid" : "SMlpBdzdTSala5hMt0XmpQ" }, { "index_name" : ".ds-demo-ds-000002", "index_uuid" : "1uZk3ug0SfmD-1UUgaeqDw" } ], "generation" : 2, "status" : "YELLOW", "template" : "template_demo", "ilm_policy" : "demo" } ] }

shrink-.ds-demo-ds-000001  .ds-demo-ds-000002

GET demo-ds/_ilm/explain

{ "indices" : { "shrink-.ds-demo-ds-000001" : { "index" : "shrink-.ds-demo-ds-000001", "managed" : true, "policy" : "demo", "lifecycle_date_millis" : 1606999959458, "age" : "13.83m", "phase" : "delete", "phase_time_millis" : 1607000275318, "action" : "complete", "action_time_millis" : 1607000275157, "step" : "complete", "step_time_millis" : 1607000275318, "phase_execution" : { "policy" : "demo", "phase_definition" : { "min_age" : "3m", "actions" : { } }, "version" : 1, "modified_date_in_millis" : 1606988416663 } }, ".ds-demo-ds-000002" : { "index" : ".ds-demo-ds-000002", "managed" : true, "policy" : "demo", "lifecycle_date_millis" : 1606999959439, "age" : "13.83m", "phase" : "hot", "phase_time_millis" : 1606999961410, "action" : "rollover", "action_time_millis" : 1607000204426, "step" : "check-rollover-ready", "step_time_millis" : 1607000204426, "phase_execution" : { "policy" : "demo", "phase_definition" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "1gb", "max_age" : "30d", "max_docs" : 5 }, "set_priority" : { "priority" : 100 } } }, "version" : 1, "modified_date_in_millis" : 1606988416663 } } } }

 shrink-.ds-demo-ds-000001 delete

ILM policy 3 11

POST demo-ds/_doc?pipeline=add-timestamp { "user": { "id": "liuxg" }, "message": "This is so cool!" }

# Search the data stream GET demo-ds/_search

{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 3, "successful" : 3, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 6, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "shrink-.ds-demo-ds-000002", "_type" : "_doc", "_id" : "PO34KHYB_Lwe3F0spf66", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T14:19:12.698241Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000002", "_type" : "_doc", "_id" : "Pe34KHYB_Lwe3F0sp_7i", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T14:19:13.249898Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000002", "_type" : "_doc", "_id" : "Ou34KHYB_Lwe3F0soP6E", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T14:19:11.364617Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000002", "_type" : "_doc", "_id" : "O-34KHYB_Lwe3F0so_54", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T14:19:12.120209Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000002", "_type" : "_doc", "_id" : "Pu34KHYB_Lwe3F0sqv4S", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T14:19:13.809864Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } }, { "_index" : "shrink-.ds-demo-ds-000002", "_type" : "_doc", "_id" : "P-34KHYB_Lwe3F0srP4N", "_score" : 1.0, "_source" : { "@timestamp" : "2020-12-03T14:19:14.317471Z", "message" : "This is so cool!", "user" : { "id" : "liuxg" } } } ] } }

shrink-.ds-demo-ds-000001 3 data stream

# Get data stream information GET _data_stream/demo-ds/_stats
{ "_shards" : { "total" : 5, "successful" : 3, "failed" : 0 }, "data_stream_count" : 1, "backing_indices" : 2, "total_store_size_bytes" : 29508, "data_streams" : [ { "data_stream" : "demo-ds", "backing_indices" : 2, "store_size_bytes" : 29508, "maximum_timestamp" : 1607000591546 } ] }

data stream

data stream

DELETE _data_stream/demo-ds

Kibana

data stream