Chia sẻ code deployment ELK single sử dụng Kubernetes (K8S)

Keywords: Share code deployment elk single use kubernetes (k8s) to build elk single

Việc xây dựng hệ thống đọc logs cũng như cài đặt thủ công ELK đã xuất hiện rất nhiều trên các trang mạng rồi đúng không nào vì thế mình sẻ không giới thiệu cho các bạn thêm về nó nữa, mà hôm nay mình sẻ hướng dẫn cho các bạn xây dựng hệ thống ELK Single sử dụng trên K8S, do nhu cầu build ELK đòi hỏi Ram cao mà máy mình yếu quá nên không dựng lab được thành ra mình chỉ share code và nói sơ qua dựa trên mô hình thực tế mình đã build tại công ty cho các bạn xem thôi nhé.

Hôm nay mình sẻ không tâm sự cùng các bạn về các vấn đề đi làm công việc cá nhân nữa vì tâm trạng mình khi làm chổ mới này không cảm thấy vui vẻ cho lắm vì có nhiều vấn đề trong công việc thành ra Blog mình cũng ít viết đi, thế nên hôm nay chúng ta cứ đi thẳng tới bài viết chính của hôm nay là ELK luôn nhé, như các bạn hiểu rằng hệ thống ELK là hệ thống cho chúng ta giám sát, filter cũng như phân tích một cách trực quan hơn cho việc phát hiện lỗi trên hệ thống, chứ thời nay ai đâu mà mò vô folder log của từng service rồi ngồi coi nữa mất thời gian chết đi được mà còn tìm chữ ERROR muốn lòi con mắt ra, thành ra thằng ELK đã giải quyết rất nhiều cho chúng ta vấn đề đó, nó sẻ có các bước đi như sau:

• Filebeat (hoặc một script đẩy dữ liệu nào đó chúng ta có thể tự tạo cũng được, thực hiện việc lấy dữ liệu đưa đến nơi tiếp nhận) => Logstash (tiếp nhận và xử lý log sau đó đẩy lên nơi lưu trữ) => Elasticsearch (nơi lưu trữ và tìm kiếm logs) => Kibana (giao diện trực quan để chúng ta dễ dàng làm việc với nó hơn)

Thường thì mọi người hay gọi là ELK nhưng mình thấy thứ tự LEK mới đúng haha, cơ mà thằng Elasticsearch cũng có khả năng lấy logs đó nha, ở một số môi trường mọi người vẫn có thể sử dụng Elasticsearch và loại bỏ luôn thằng Logstash vấn đề tại sao thì mọi người có thể search trên mạng vì hiện tại mình đang chia sẻ theo mô hình ELK

Chúng ta thực hiện tạo file elk-k8s.yaml sau và chạy nó bằng lệnh:

kubectl apply -f elk-k8s.yaml

Code:

apiVersion: v1
kind: Namespace
metadata:
  name: elk
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-cm
  namespace: elk
  labels:
    environment: production
    app: elasticsearch
    release: elk
data:
  elasticsearch.yml: |
    http.port: 9200
    path.data: /usr/share/elasticsearch/data
    path.logs: /usr/share/elasticsearch/logs
    network.host: 0.0.0.0
    transport.host: localhost
    transport.tcp.port: 9300
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: elk
  labels:
    environment: production
    app: elasticsearch
    release: elk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
      release: elk
  template:
    metadata:
      name: elasticsearch
      labels:
        app: elasticsearch
        release: elk
    spec:
      volumes:
      - name: "elasticsearch-config"
        configMap:
          name: elasticsearch-cm
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
        ports:
          - containerPort: 9200
          - containerPort: 9300
        volumeMounts:
          - name: "elasticsearch-config"
            mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
            subPath: elasticsearch.yml
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-svc
  namespace: elk
  labels:
    environment: production
    app: elasticsearch
    release: elk
spec:
  selector:
    app: elasticsearch
    release: elk
  type: ClusterIP
  ports:
    - name: port-defaults
      port: 9200
      targetPort: 9200
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-cm
  namespace: elk
  labels:
    environment: production
    app: kibana
    release: elk
data:
  kibana.yml: |
    server.host: 0.0.0.0
    elasticsearch.hosts: "http://elasticsearch-svc.elk.svc.cluster.local:9200"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: elk
  labels:
    environment: production
    app: kibana
    release: elk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
      release: elk
  template:
    metadata:
      name: kibana
      labels:
        app: kibana
        release: elk
    spec:
      volumes:
      - name: "kibana-config"
        configMap:
          name: kibana-cm
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.10.1
        ports:
          - containerPort: 5601
        volumeMounts:
          - name: "kibana-config"
            mountPath: /usr/share/kibana/config/kibana.yml
            subPath: kibana.yml
---
apiVersion: v1
kind: Service
metadata:
  name: kibana-svc
  namespace: elk
  labels:
    environment: production
    app: kibana
    release: elk
spec:
  selector:
    app: kibana
    release: elk
  type: NodePort
  ports:
    - name: port-defaults
      port: 5601
      targetPort: 5601
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-cm
  namespace: elk
  labels:
    environment: production
    app: logstash
    release: elk
data:
  logstash.yml: |
    #http.host: "0.0.0.0"
    #xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch-svc.elk.svc.cluster.local:9200" ]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-pipeline-cm
  namespace: elk
  labels:
    environment: production
    app: logstash
    release: elk
data:
  logstash.conf: |
    input {
      beats {
        host => "0.0.0.0"
        port => 5044
      }
    }
    output {
      elasticsearch {
        hosts => ["elasticsearch-svc.elk.svc.cluster.local:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      }
    }
    filter {
      if [fileset][module] == "system" {
        if [fileset][name] == "auth" {
          grok {
            match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
            pattern_definitions => {
              "GREEDYMULTILINE"=> "(.|\n)*"
            }
            remove_field => "message"
          }
          date {
            match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
          }
          geoip {
            source => "[system][auth][ssh][ip]"
            target => "[system][auth][ssh][geoip]"
          }
        }
        else if [fileset][name] == "syslog" {
          grok {
            match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
            pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
            remove_field => "message"
          }
          date {
            match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
          }
        }
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
  namespace: elk
  labels:
    environment: production
    app: logstash
    release: elk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash
      release: elk
  template:
    metadata:
      name: logstash
      labels:
        app: logstash
        release: elk
    spec:
      volumes:
      - name: "logstash-config"
        configMap:
          name: logstash-cm
      - name: "logstash-cf-pipeline"
        configMap:
          name: logstash-pipeline-cm
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:7.10.1
        ports:
          - containerPort: 5044
          - containerPort: 9600
        volumeMounts:
          - name: "logstash-config"
            mountPath: /usr/share/logstash/config/logstash.yml
            subPath: logstash.yml
          - name: "logstash-cf-pipeline"
            mountPath: /usr/share/logstash/pipeline/logstash.conf
            subPath: logstash.conf
---
apiVersion: v1
kind: Service
metadata:
  name: logstash-svc
  namespace: elk
  labels:
    environment: production
    app: logstash
    release: elk
spec:
  selector:
    app: logstash
    release: elk
  type: NodePort
  ports:
    - name: port-get-logs
      port: 5044
      targetPort: 5044
    - name: port-logstash-api
      port: 9600
      targetPort: 9600

Sau khi các bạn chạy xong trên hệ thống k8s sẻ có một namespace tên là elk

Các Port sẻ nằm ở svc nghĩa là service của k8s của chúng ta nhưng thuộc ns elk nhé

Những ai sử dụng k8s rồi thì họ sẻ hiểu được những cái mình nói, còn những ai chưa sử dụng qua k8s thì mình nghĩ bài viết này họ sẻ hoàn toàn khó hiểu, sau khi đã hoàn thành việc cài đặt elk trên k8s bằng file yaml rồi giờ chỉ còn việc cài đặt filebeat trên máy mà chúng ta muốn lấy logs và cấu hình config để cho nó lấy, mình sẻ để sẵn cấu hình filebeat của mình ở phía dưới để cho các bạn có thể dễ dàng đọc hơn nha, nhớ chú ý lại phần port ở svc k8s mà mình đã get info ở trên nhé

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /var/log/containers/*.log
    #- c:\programdata\elasticsearch\logs\*
  
  scan_frequency: 10s
  close_inactive: 10m
  clean_removed: true

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

# filestream is an experimental input. It is going to replace log input in the future.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["51.222.44.17:31288"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

Các bạn cần chú ý những chỗ mình đã tô màu xanh lên vì những nơi đó sẻ thay đổi, thực hiện ẩn elasticsearch và open thằng logstash lên sau đó trỏ đến port của logstash mà chúng ta thực hiện NodePort từ K8S ở phần logs thì các bạn trỏ đến đường dẫn mà các bạn muốn lấy file logs nhé, còn lại các config kia chủ yếu là hẹn thời gian clear logs thôi, khi các bạn chưa cài filebeat thì tại index của Elasticsearch trên Kibana sẻ chẳng có gì cả, khi filebeat được cài đặt và trỏ về thì lúc này trên index mới xuất hiện ra như hình nhé

Còn khi không có dữ liệu đẩy về thì tại đây sẻ là zero không có gì cả hihi, sau khi đã có index gửi về từ logstash rồi mình sẻ thực hiện tiếp tạo index patterns cho kibana bằng cách click vào như hình dưới

Thực hiện bấm Create và tạo như hình

Do đầu index của mình được tạo bằng filebeat ở cấu hình logstash phía trên trong k8s cm nên thành ra mình sẻ filter những index có đầu giá trị là filebeat còn lại thì mình lấy hết nên đặt giá trị là "*",  sau đó thực hiện chọn @timestamp, ok như vậy ta đã hoàn thành xong việc nạp dữ liệu từ filebeat rồi giờ chỉ cần vào phần Discover của Kibana để filter các logs theo ý của chúng ta nữa là coi như xong, phần này thì các bạn tự tìm hiểu cách filter nhé, khá đơn giản nên mình sẻ không hướng dẫn ở đây

Mình xin chia sẻ thêm một số config của mình, mong rằng những config này có thể giúp ích cho các bạn

• elk-k8s.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: elk
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: elasticsearch-pv
  namespace: elk
  labels:
    disk: pvc
    environment: production
    app: elasticsearch
    release: elk
spec:
  storageClassName: elasticsearch-pv
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/data/k8s/elasticsearch"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elasticsearch-pvc
  namespace: elk
  labels:
    disk: pvc
    environment: production
    app: elasticsearch
    release: elk
spec:
  storageClassName: elasticsearch-pv
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-cm
  namespace: elk
  labels:
    environment: production
    app: elasticsearch
    release: elk
data:
  elasticsearch.yml: |
    http.port: 9200
    path.data: /usr/share/elasticsearch/data
    path.logs: /usr/share/elasticsearch/logs
    network.host: 0.0.0.0
    transport.host: localhost
    transport.tcp.port: 9300
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: elk
  labels:
    environment: production
    app: elasticsearch
    release: elk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
      release: elk
  template:
    metadata:
      name: elasticsearch
      labels:
        app: elasticsearch
        release: elk
    spec:
      volumes:
      - name: "elasticsearch-config"
        configMap:
          name: elasticsearch-cm
      - name: "elasticsearch-data"
        persistentVolumeClaim:
          claimName: elasticsearch-pvc
      nodeSelector:
        disk: pvc
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
        ports:
          - containerPort: 9200
          - containerPort: 9300
        volumeMounts:
          - name: "elasticsearch-config"
            mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
            subPath: elasticsearch.yml
          - name: "elasticsearch-data"
            mountPath: /usr/share/elasticsearch/data
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-svc
  namespace: elk
  labels:
    environment: production
    app: elasticsearch
    release: elk
spec:
  selector:
    app: elasticsearch
    release: elk
  type: ClusterIP
  ports:
    - name: port-defaults
      port: 9200
      targetPort: 9200
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-cm
  namespace: elk
  labels:
    environment: production
    app: kibana
    release: elk
data:
  kibana.yml: |
    server.host: 0.0.0.0
    elasticsearch.hosts: "http://elasticsearch-svc.elk.svc.cluster.local:9200"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: elk
  labels:
    environment: production
    app: kibana
    release: elk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
      release: elk
  template:
    metadata:
      name: kibana
      labels:
        app: kibana
        release: elk
    spec:
      volumes:
      - name: "kibana-config"
        configMap:
          name: kibana-cm
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.10.1
        ports:
          - containerPort: 5601
        volumeMounts:
          - name: "kibana-config"
            mountPath: /usr/share/kibana/config/kibana.yml
            subPath: kibana.yml
---
apiVersion: v1
kind: Service
metadata:
  name: kibana-svc
  namespace: elk
  labels:
    environment: production
    app: kibana
    release: elk
spec:
  selector:
    app: kibana
    release: elk
  type: NodePort
  ports:
    - name: port-defaults
      port: 5601
      targetPort: 5601
      nodePort: 31605
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-cm
  namespace: elk
  labels:
    environment: production
    app: logstash
    release: elk
data:
  logstash.yml: |
    #http.host: "0.0.0.0"
    #xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch-svc.elk.svc.cluster.local:9200" ]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-pipeline-cm
  namespace: elk
  labels:
    environment: production
    app: logstash
    release: elk
data:
  logstash.conf: |
    input {
      beats {
        host => "0.0.0.0"
        port => 5044
      }
    }
    output {
      elasticsearch {
        hosts => ["elasticsearch-svc.elk.svc.cluster.local:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      }
    }
    filter {
      if [fileset][module] == "system" {
        if [fileset][name] == "auth" {
          grok {
            match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
            pattern_definitions => {
              "GREEDYMULTILINE"=> "(.|\n)*"
            }
            remove_field => "message"
          }
          date {
            match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
          }
          geoip {
            source => "[system][auth][ssh][ip]"
            target => "[system][auth][ssh][geoip]"
          }
        }
        else if [fileset][name] == "syslog" {
          grok {
            match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
            pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
            remove_field => "message"
          }
          date {
            match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
          }
        }
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
  namespace: elk
  labels:
    environment: production
    app: logstash
    release: elk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash
      release: elk
  template:
    metadata:
      name: logstash
      labels:
        app: logstash
        release: elk
    spec:
      volumes:
      - name: "logstash-config"
        configMap:
          name: logstash-cm
      - name: "logstash-cf-pipeline"
        configMap:
          name: logstash-pipeline-cm
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:7.10.1
        ports:
          - containerPort: 5044
          - containerPort: 9600
        volumeMounts:
          - name: "logstash-config"
            mountPath: /usr/share/logstash/config/logstash.yml
            subPath: logstash.yml
          - name: "logstash-cf-pipeline"
            mountPath: /usr/share/logstash/pipeline/logstash.conf
            subPath: logstash.conf
---
apiVersion: v1
kind: Service
metadata:
  name: logstash-svc
  namespace: elk
  labels:
    environment: production
    app: logstash
    release: elk
spec:
  selector:
    app: logstash
    release: elk
  type: NodePort
  ports:
    - name: port-get-logs
      port: 5044
      targetPort: 5044
      nodePort: 31288
    - name: port-logstash-api
      port: 9600
      targetPort: 9600
      nodePort: 31658

• logstash-pipeline-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-pipeline-cm
  namespace: elk
  labels:
    environment: production
    app: logstash
    release: elk
data:
  logstash.conf: |
    input {
      tcp {
        port => 9600
        codec => json
        type => "ynm-services"
      }
      beats {
        host => "0.0.0.0"
        port => 5044
        type => "filebeat"
      }
    }
    filter {
      if [fileset][module] == "system" {
        if [fileset][name] == "auth" {
          grok {
            match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
                      "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
            pattern_definitions => {
              "GREEDYMULTILINE"=> "(.|\n)*"
            }
            remove_field => "message"
          }
          date {
            match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
          }
          geoip {
            source => "[system][auth][ssh][ip]"
            target => "[system][auth][ssh][geoip]"
          }
        }
        else if [fileset][name] == "syslog" {
          grok {
            match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
            pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
            remove_field => "message"
          }
          date {
            match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
          }
        }
      }
    }
    output {
      elasticsearch {
        hosts => ["elasticsearch-svc.elk.svc.cluster.local:9200"]
        manage_template => false
        index => "%{type}-%{+YYYY.MM.dd}"
      } 
    }

Chúc các bạn thành công nhé, mọi vấn đề không thực hiện được liên hệ hoặc cmt bên dưới nhé !

Nguồn: itblognote.com