elfk的历史索引数据清理

2022年11月11日 148点热度 0人点赞 0条评论

0x1

在elfk日志量很大并且只需要看到实时效果的时候,我们可以对历史数据做及时的清理
工具:curator

0x2

安装之后我们做操作系统级别的调度任务,来定时清理数据。共需要准备两个配置文件,一是连接elasticsearch相关配置,二是清理策略,如下:

# config.yml

client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth: elastic:xxxx
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile: /tmp/cur.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']
# action.yml

actions:
  1:
    action: delete_indices
    description: "delete f5 log indices"
    options:
      ignore_empty_list: True
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-f5-http-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 5
  2:
    action: delete_indices
    description: "delete nginx access log indices"
    options:
      ignore_empty_list: True
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: nginx_access_
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 5
  3:
    action: delete_indices
    description: "delete nginx error log indices"
    options:
      ignore_empty_list: True
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: nginx_error_
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 5

其中action.yml定义了需要清理的索引模式,以及保留策略,需要注意的是这里匹配的索引模式是年月日,如果elasticsearch中创建的索引是年月,即使有大于5天的数据也不会被清理

yuc

这个人很懒,什么都没留下

error: Content is protected !!