iLogtail

1769
下载
iLogtail 是阿里云日志服务(Log Service)中的一款高性能、可扩展的日志采集工具。它被设计用于实时收集、解析和传输各类日志数据,支持多种数据源和复杂的日志处理场景,广泛应用于分布式系统、Web 服务器、应用程序等环境中的日志管理。

Kubernetes使用


使用前提

已部署Kubernetes集群或minikube

具备访问Kubernetes集群的kubectl

采集Kubernetes容器日志

  1. 创建部署iLogtail的命名空间 将下面内容保存为ilogtail-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ilogtail

您也可以直接从下面的地址下载示例配置。

Terminal window
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-ns.yaml

应用上述配置

Terminal window
kubectl apply -f ilogtail-ns.yaml
  1. 创建配置iLogtail的ConfigMap和Secret 将下面内容保存为ilogtail-user-configmap.yaml。该ConfigMap后续将作为配置目录挂载到iLogtail容器中,因此可包含多个采集配置。
apiVersion: v1
kind: ConfigMap
metadata:
name: ilogtail-user-cm
namespace: ilogtail
data:
nginx_stdout.yaml: |
enable: true
inputs:
- Type: service_docker_stdout
Stderr: false
Stdout: true # only collect stdout
IncludeK8sLabel:
app: nginx # choose containers with this label
processors:
- Type: processor_regex # structure log
SourceKey: content
Regex: '([\d\.:]+) - (\S+) \[(\S+) \S+\] \"(\S+) (\S+) ([^\\"]+)\" (\d+) (\d+) \"([^\\"]*)\" \"([^\\"]*)\" \"([^\\"]*)\"'
Keys:
- remote_addr
- remote_user
- time_local
- method
- url
- protocol
- status
- body_bytes_sent
- http_referer
- http_user_agent
- http_x_forwarded_for
flushers:
- Type: flusher_stdout
OnlyStdout: true

将下面内容保存为ilogtail-secret.yaml。该Secret为可选,当需要将日志写入SLS时会用到。

apiVersion: v1
kind: Secret
metadata:
name: ilogtail-secret
namespace: ilogtail
type: Opaque
data:
access_key_id: # base64 accesskey id if you want to flush to SLS
access_key: # base64 accesskey secret if you want to flush to SLS

您也可以直接从下面的地址下载示例配置。

Terminal window
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-user-configmap.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-secret.yaml

应用上述配置

Terminal window
kubectl apply -f ilogtail-user-configmap.yaml
kubectl apply -f ilogtail-secret.yaml
  1. 创建iLogtail DaemonSet 将下面内容保存为ilogtail-daemonset.yaml。
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ilogtail-ds
namespace: ilogtail
labels:
k8s-app: logtail-ds
spec:
selector:
matchLabels:
k8s-app: logtail-ds
template:
metadata:
labels:
k8s-app: logtail-ds
spec:
tolerations:
- operator: Exists # deploy on all nodes
containers:
- name: logtail
env:
- name: ALIYUN_LOG_ENV_TAGS # add log tags from env
value: _node_name_|_node_ip_
- name: _node_name_
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: _node_ip_
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: cpu_usage_limit # iLogtail's self monitor cpu limit
value: "1"
- name: mem_usage_limit # iLogtail's self monitor mem limit
value: "512"
- name: default_access_key_id # accesskey id if you want to flush to SLS
valueFrom:
secretKeyRef:
name: ilogtail-secret
key: access_key_id
optional: true
- name: default_access_key # accesskey secret if you want to flush to SLS
valueFrom:
secretKeyRef:
name: ilogtail-secret
key: access_key
optional: true
image: >-
sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:latest
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 400m
memory: 384Mi
volumeMounts:
- mountPath: /var/run # for container runtime socket
name: run
- mountPath: /logtail_host # for log access on the node
mountPropagation: HostToContainer
name: root
readOnly: true
- mountPath: /usr/local/ilogtail/checkpoint # for checkpoint between container restart
name: checkpoint
- mountPath: /usr/local/ilogtail/config/local # mount config dir
name: user-config
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
volumes:
- hostPath:
path: /var/run
type: Directory
name: run
- hostPath:
path: /
type: Directory
name: root
- hostPath:
path: /etc/ilogtail-ilogtail-ds/checkpoint
type: DirectoryOrCreate
name: checkpoint
- configMap:
defaultMode: 420
name: ilogtail-user-cm
name: user-config

您也可以直接从下面的地址下载示例配置。

Terminal window
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-daemonset.yaml

添加参数示例

- name: logtail
command:
- /usr/local/ilogtail/ilogtail_control.sh
args:
- "start_and_block"
- "-enable_containerd_upper_dir_detect=true"
- "-dirfile_check_interval_ms=5000"
- "-logtail_checkpoint_check_gc_interval_sec=120"

应用上述配置

Terminal window
kubectl apply -f ilogtail-daemonset.yaml
  1. 部署用来测试的nginx 将下面内容保存为nginx-deployment.yaml。
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: default
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: 'nginx:latest'
name: nginx
ports:
- containerPort: 80
name: http
protocol: TCP
resources:
requests:
cpu: 100m
memory: 100Mi

您也可以直接从下面的地址下载示例配置。

Terminal window
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/nginx-deployment.yaml

应用上述配置

Terminal window
kubectl apply -f nginx-deployment.yaml
  1. 发送请求构造示例日志
Terminal window
kubectl exec nginx-<pod-id> -- curl localhost/hello/ilogtail
  1. 查看采集到的测试容器标准输出日志
Terminal window
kubectl logs ilogtail-ds-<pod-id> -n ilogtail

结果为

2022-07-14 16:36:50 {"_time_":"2022-07-15T00:36:48.489153485+08:00","_source_":"stdout","_image_name_":"docker.io/library/nginx:latest","_container_name_":"nginx","_pod_name_":"nginx-76d49876c7-r892w","_namespace_":"default","_pod_uid_":"07f75a79-da69-40ac-ae2b-77a632929cc6","_container_ip_":"10.223.0.154","remote_addr":"::1","remote_user":"-","time_local":"14/Jul/2022:16:36:48","method":"GET","url":"/hello/ilogtail","protocol":"HTTP/1.1","status":"404","body_bytes_sent":"153","http_referer":"-","http_user_agent":"curl/7.74.0","http_x_forwarded_for":"-","__time__":"1657816609"}

采集模版

了解采集配置结构:采集配置

查看更多K8s采集日志模版(从容器中采集文件日等):https://github.com/alibaba/ilogtail/blob/main/k8s_templates

参考更多可组合的日志采集和处理配置样例:https://github.com/alibaba/ilogtail/blob/main/example_config


observability.cn Authors 2024 | Documentation Distributed under CC-BY-4.0
Copyright © 2017-2024, Alibaba. All rights reserved. Alibaba has registered trademarks and uses trademarks.
浙ICP备2021005855号-32