fluent-bit收集nginx日志到es
来源:原创
时间:2020-11-21
作者:脚本小站
分类:Linux
官方安装文档:
https://docs.fluentbit.io/manual/installation/linux/redhat-centos
安装fluent-bit:
cat > /etc/yum.repos.d/td-agent-bit.repo <<EOF [td-agent-bit] name = TD Agent Bit baseurl = https://packages.fluentbit.io/centos/7/$basearch/ gpgcheck=0 gpgkey=https://packages.fluentbit.io/fluentbit.key enabled=1 EOF wget https://packages.fluentbit.io/centos/7/x86_64/td-agent-bit-1.6.4-1.x86_64.rpm yum install td-agent-bit-1.6.4-1.x86_64.rpm -y systemctl start td-agent-bit.service systemctl status td-agent-bit.service
配置文件:
cd /etc/td-agent-bit
解析配置:parsers.conf,里面有很多常见的日志解析规则
插件配置:plugins.conf
收集配置:td-agent-bit.conf
配置收集nginx日志:
vim /etc/td-agent-bit/td-agent-bit.conf [INPUT] Name tail Tag nginx.* Path /var/log/nginx/*.log [OUTPUT] name es match * Host 192.168.0.26 Port 9200 Logstash_Format On Logstash_Prefix fluent Logstash_DateFormat %Y.%m.%d Retry_Limit 5 [FILTER] Name parser Match ** Parser nginx Key_Name log
一些常用配置:
[SERVICE] Flush 1 #buffer里的数据每隔1秒写到output插件里,这里写到ES里。 Log_Level info #fluent-bit的日志级别 Daemon off Parsers_File parsers.conf #指向了另外一个配置文件,里面配置所有的parser。 HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 [INPUT] Name tail #指定了input插件的类型,这里是tail类型 Tag {{$Host}}_{{$ESIndex}}_{{.ContainerName}}_{{$i}}给采集的日志打个标签,后面filter和output根据这个标签选择数据源 Path /var/log/pods/${POD_UID}/{{.ContainerName}}/*.log Path_Key filename Parser docker DB /var/log/pods/${POD_UID}/{{.ContainerName}}/flb_std.db#记录哪个文件采集到哪一行 DB.Sync Full #internal SQLite engine用哪种方法同步数据到磁盘,full为安全优先 #Key log Mem_Buf_Limit 5MB #一旦buffer里的数据超过Mem_buf_limit,tail就会暂停采集,直到buffer数据被flush到output。 Skip_Long_Lines On #跳过长度大于Buffer_Max_Size的行 Buffer_Chunk_Size 32k #tail命令的buffer初始大小,具体作用和对性能的影响还需进一步研究 Buffer_Max_Size 32k #tail命令的buffer最大值,具体作用和对性能的影响还需进一步研究 Refresh_Interval 10 #定时扫描磁盘上的新文件的间隔。 Rotate_Wait 5 #文件rotate后的等待一段时间后再继续监控这个文件,以防flush一些pending data, 具体作用和对性能的影响还需进一步研究 Ignore_Older 10d #忽略近十天来未更改的文件 [FILTER] Name modify #插件的类型 Match * #匹配到任何数据源 Add node_name ${NODE_NAME} Add node_ip ${NODE_IP} Add pod_name ${POD_NAME} [OUTPUT] Name es #插件的类型 Match {{.Host}}_{{.ESIndex}}* #匹配到tag为{{.Host}}_{{.ESIndex}}*的数据源 Host {{.HostName}} #es的hostname 可以是域名和ip Port {{.Port}} #es的端口 Index {{.ESIndex}} HTTP_User {{.UserName}} HTTP_Passwd {{.Password}} Pipeline #不要用 Trace_Error On # 问题追踪,日志里输出问题 Logstash_Format On #是否采用类似logstash的index,可以根据时间设置index名字 Logstash_Prefix logstash #索引名称的前缀 Logstash_DateFormat %Y.%m.%d #名称后缀格式 Time_Key tail-time #Logstash_Format enabled的时候,每条记录会新产生一个时间戳 Time_Key_Format %Y-%m-%dT%H:%M:%S #新时间戳的格式 Generate_ID On #对记录去重,可能有性能消耗。 Trace_Output Off #打印elasticsearch API calls 调试的时候用。 Logstash_Prefix_Key ttt # Retry_Limit 5 #传输失败后重试次数,默认为2,设置为False时,无限次重试
常见的PARSER:前面留的四个空格可以直接粘贴到configmap里。
[PARSER] Name apache Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache2 Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache_error Format regex Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$ [PARSER] Name nginx Format regex Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] # http://rubular.com/r/tjUt3Awgg4 Name cri Format regex Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$ Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z [PARSER] Name syslog Format regex Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S
使用fluent-bit以sidecar的方式收集应用的多个日志:
真实的生产环境中一个应用往往有多个日志,使用daemonset的方式固然省事,但是不适用上述的场景,所以需要使用日志收集工具以sidecar的方式来收集日志。
configmap:
apiVersion: v1 data: filter.conf: | [FILTER] fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 @INCLUDE input.conf @INCLUDE filter.conf @INCLUDE output-elasticsearch.conf input.conf: | [INPUT] Name tail Tag ${SERVER_NAME}-info Path /data/logs/${SERVER_NAME}/info.log DB /data/logs/${SERVER_NAME}/flb.db # 用来记录日志读取到第几行的数据库,放在日志目录下,删除Pod后新Pod挂载到当前目录时会接着读取日志,而不是重头开始读 Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 [INPUT] Name tail Tag ${SERVER_NAME}-error Path /data/logs/${SERVER_NAME}/error.log DB /data/logs/${SERVER_NAME}/flb.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 [INPUT] Name tail Tag ${SERVER_NAME}-api Path /data/logs/${SERVER_NAME}/api.log DB /data/logs/${SERVER_NAME}/flb.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 [INPUT] Name tail Tag ${SERVER_NAME}-api_client Path /data/logs/${SERVER_NAME}/api_client.log DB /data/logs/${SERVER_NAME}/flb.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 output-elasticsearch.conf: | [OUTPUT] Name es Match ${SERVER_NAME}-info Host ${FLUENT_ELASTICSEARCH_HOST} Port ${FLUENT_ELASTICSEARCH_PORT} Logstash_Format On Logstash_Prefix sit-${SERVER_NAME}-info Replace_Dots On Trace_Error On Retry_Limit False [OUTPUT] Name es Match ${SERVER_NAME}-error Host ${FLUENT_ELASTICSEARCH_HOST} Port ${FLUENT_ELASTICSEARCH_PORT} Logstash_Format On Logstash_Prefix sit-${SERVER_NAME}-error Replace_Dots On Trace_Error On Retry_Limit False [OUTPUT] Name es Match ${SERVER_NAME}-api # 匹配上面的tag Host ${FLUENT_ELASTICSEARCH_HOST} Port ${FLUENT_ELASTICSEARCH_PORT} Logstash_Format On Logstash_Prefix sit-${SERVER_NAME}-api # 每个日志都有特定的索引名称,方便查询 Replace_Dots On Trace_Error On Retry_Limit False [OUTPUT] Name es Match ${SERVER_NAME}-api_client Host ${FLUENT_ELASTICSEARCH_HOST} Port ${FLUENT_ELASTICSEARCH_PORT} Logstash_Format On Logstash_Prefix sit-${SERVER_NAME}-api_client Replace_Dots On Trace_Error On Retry_Limit False parsers.conf: | [PARSER] # 这些个parser用不到可以删了 Name apache Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache2 Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache_error Format regex Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$ [PARSER] Name nginx Format regex Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] # http://rubular.com/r/tjUt3Awgg4 Name cri Format regex Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$ Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z [PARSER] Name syslog Format regex Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S kind: ConfigMap metadata: name: fluent-bit-config namespace: default
deployment:
kind: Deployment apiVersion: apps/v1 metadata: name: pmml-processor namespace: default labels: app: pmml-processor spec: replicas: 2 selector: matchLabels: app: pmml-processor template: metadata: labels: app: pmml-processor spec: containers: - name: pmml-processor image: 192.168.199.240/test-registry/pmml-processor:123456 ports: - name: 8089tcp00 containerPort: 8089 protocol: TCP volumeMounts: - name: pmml-processor mountPath: /data/logs imagePullPolicy: IfNotPresent - name: fluent-bit image: fluent/fluent-bit:1.6 # 这里测试使用的es是6.3.2,不同的es版本可能不匹配会报错 ports: - containerPort: 2020 protocol: TCP env: - name: FLUENT_ELASTICSEARCH_HOST value: 192.168.199.39 - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: SERVER_NAME # es索引名称,一般就使用dm控制器的名称 value: pmml-processor resources: limits: memory: 128Mi requests: memory: 64Mi volumeMounts: - name: fluent-bit-config # fluent-bit的configmap的名称 mountPath: /fluent-bit/etc/ # 挂载路径 - name: pmml-processor mountPath: /data/logs # fluent-bit也要挂载到日志目录才能读取到日志 imagePullPolicy: IfNotPresent volumes: - name: pmml-processor hostPath: path: /trust/logs # 日志挂载在本地节点的目录中 type: "" - name: fluent-bit-config configMap: name: fluent-bit-config defaultMode: 420