Kubernetes使用
使用前提
已部署Kubernetes集群或minikube
具备访问Kubernetes集群的kubectl
采集Kubernetes容器日志
-
创建部署 LoongCollector 的命名空间
将下面内容保存为
loongcollector-ns.yaml
apiVersion: v1kind: Namespacemetadata:name: loongcollector您也可以直接从下面的地址下载示例配置。
Terminal window wget https://raw.githubusercontent.com/alibaba/loongcollector/main/example_config/start_with_k8s/loongcollector-ns.yaml应用上述配置
Terminal window kubectl apply -f loongcollector-ns.yaml -
创建配置 LoongCollector 的 ConfigMap 和 Secret
将下面内容保存为
loongcollector-user-configmap.yaml
。该ConfigMap后续将作为配置目录挂载到LoongCollector 容器中,因此可包含多个采集配置。apiVersion: v1kind: ConfigMapmetadata:name: loongcollector-user-configmapnamespace: loongcollectordata:nginx_stdout.yaml: |enable: trueinputs:- Type: service_docker_stdoutStderr: falseStdout: true # only collect stdoutIncludeK8sLabel:app: nginx # choose containers with this labelprocessors:- Type: processor_regex # structure logSourceKey: contentRegex: '([\d\.:]+) - (\S+) \[(\S+) \S+\] \"(\S+) (\S+) ([^\\"]+)\" (\d+) (\d+) \"([^\\"]*)\" \"([^\\"]*)\" \"([^\\"]*)\"'Keys:- remote_addr- remote_user- time_local- method- url- protocol- status- body_bytes_sent- http_referer- http_user_agent- http_x_forwarded_forflushers:- Type: flusher_stdoutOnlyStdout: true将下面内容保存为
loongcollector-secret.yaml
。该Secret为可选,当需要将日志写入SLS时会用到。apiVersion: v1kind: Secretmetadata:name: loongcollector-secretnamespace: loongcollectortype: Opaquedata:access_key_id: # base64 accesskey id if you want to flush to SLSaccess_key: # base64 accesskey secret if you want to flush to SLS您也可以直接从下面的地址下载示例配置。
Terminal window wget https://raw.githubusercontent.com/alibaba/loongcollector/main/example_config/start_with_k8s/loongcollector-user-configmap.yamlwget https://raw.githubusercontent.com/alibaba/loongcollector/main/example_config/start_with_k8s/loongcollector-secret.yaml应用上述配置
Terminal window kubectl apply -f loongcollector-user-configmap.yamlkubectl apply -f loongcollector-secret.yaml -
创建 LoongCollector DaemonSet
将下面内容保存为
loongcollector-daemonset.yaml
。apiVersion: apps/v1kind: DaemonSetmetadata:name: loongcollector-dsnamespace: loongcollectorlabels:k8s-app: loongcollector-dsspec:selector:matchLabels:k8s-app: loongcollector-dstemplate:metadata:labels:k8s-app: loongcollector-dsspec:tolerations:- operator: Exists # deploy on all nodescontainers:- name: loongcollectorenv:- name: ALIYUN_LOG_ENV_TAGS # add log tags from envvalue: _node_name_|_node_ip_- name: _node_name_valueFrom:fieldRef:apiVersion: v1fieldPath: spec.nodeName- name: _node_ip_valueFrom:fieldRef:apiVersion: v1fieldPath: status.hostIP- name: cpu_usage_limit # loongcollector's self monitor cpu limitvalue: "1"- name: mem_usage_limit # loongcollector's self monitor mem limitvalue: "512"- name: default_access_key_id # accesskey id if you want to flush to SLSvalueFrom:secretKeyRef:name: loongcollector-secretkey: access_key_idoptional: true- name: default_access_key # accesskey secret if you want to flush to SLSvalueFrom:secretKeyRef:name: loongcollector-secretkey: access_keyoptional: trueimage: >-sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/loongcollector-community-edition/loongcollector:0.2.0imagePullPolicy: IfNotPresentresources:limits:cpu: 1000mmemory: 1Girequests:cpu: 400mmemory: 384MivolumeMounts:- mountPath: /var/run # for container runtime socketname: run- mountPath: /logtail_host # for log access on the nodemountPropagation: HostToContainername: rootreadOnly: true- mountPath: /usr/local/loongcollector/data # for checkpoint between container restartname: checkpoint- mountPath: /usr/local/loongcollector/conf/continuous_pipeline_config/local # mount config dirname: user-configreadOnly: truednsPolicy: ClusterFirstWithHostNethostNetwork: truevolumes:- hostPath:path: /var/runtype: Directoryname: run- hostPath:path: /type: Directoryname: root- hostPath:path: /etc/loongcollector-loongcollector-ds/checkpointtype: DirectoryOrCreatename: checkpoint- configMap:defaultMode: 420name: loongcollector-user-configmapname: user-config您也可以直接从下面的地址下载示例配置。
Terminal window wget https://raw.githubusercontent.com/alibaba/loongcollector/main/example_config/start_with_k8s/loongcollector-daemonset.yaml添加参数示例
- name: loongcollectorcommand:- /usr/local/loongcollector/loongcollector_control.shargs:- "start_and_block"- "-dirfile_check_interval_ms=5000"- "-logtail_checkpoint_check_gc_interval_sec=120"env:- name: enable_containerd_upper_dir_detectvalue: "true"应用上述配置
Terminal window kubectl apply -f loongcollector-daemonset.yaml -
部署用来测试的 nginx
将下面内容保存为
nginx-deployment.yaml
。apiVersion: apps/v1kind: Deploymentmetadata:name: nginxnamespace: defaultlabels:app: nginxspec:replicas: 1selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:containers:- image: 'nginx:latest'name: nginxports:- containerPort: 80name: httpprotocol: TCPresources:requests:cpu: 100mmemory: 100Mi您也可以直接从下面的地址下载示例配置。
Terminal window wget https://raw.githubusercontent.com/alibaba/loongcollector/main/example_config/start_with_k8s/nginx-deployment.yaml应用上述配置
Terminal window kubectl apply -f nginx-deployment.yaml -
发送请求构造示例日志
Terminal window kubectl exec nginx-<pod-id> -- curl localhost/hello/loongcollector -
查看采集到的测试容器标准输出日志
Terminal window kubectl logs loongcollector-ds-<pod-id> -n loongcollector结果为
2024-12-05 08:53:45 {"_time_":"2024-12-05T16:53:42.165362808+08:00","_source_":"stdout","_pod_name_":"nginx-b9cf9cc78-8tzpx","_namespace_":"default","_pod_uid_":"6deb87e1-02b3-4002-a556-32a3f668cd6f","_container_ip_":"192.168.0.65","_image_name_":"docker.io/library/nginx:latest","_container_name_":"nginx","remote_addr":"127.0.0.1","remote_user":"-","time_local":"05/Dec/2024:08:53:42","method":"GET","url":"/hello/loongcollector","protocol":"HTTP/1.1","status":"404","body_bytes_sent":"153","http_referer":"-","http_user_agent":"curl/7.88.1","http_x_forwarded_for":"-","__time__":"1733388822"}
采集模版
了解采集配置结构:采集配置
查看更多K8s采集日志模版(从容器中采集文件日等):https://github.com/alibaba/loongcollector/blob/main/k8s_templates
参考更多可组合的日志采集和处理配置样例:https://github.com/alibaba/loongcollector/blob/main/example_config