Home (kube-prometheus-stack) Prometheus 기반 모니터링 스택 Thanos 연동
Post
Cancel

(kube-prometheus-stack) Prometheus 기반 모니터링 스택 Thanos 연동

By woobni
Posted May 24 Updated Jul 17 17 min read

본 글은 ‘(kube-prometheus-stack) Prometheus 기반 모니터링 스택 구축’ 포스팅 이후 진행됨

Background

  • K8S 클러스터에 구성된 Prometheus 기반 모니터링 스택의 Prometheus HA, 메트릭 장기 저장, global view를 구현하기 위해 Thanos 스택 연동
  • 각 Thanos 컴포넌트에 ServiceMonitor를 생성하여 Thanos 컴포넌트 메트릭 추가 수집

Prerequisite

  1. Helm v3.14.4
  2. kubectl v1.28.2
  3. Monitoring Stack Setup
    • kube-prometheus-stack helm chart v58.2.1
  4. Object Storage

Prerequisite

  1. Helm v3.14.4
  2. kubectl v1.28.2
  3. kube-prometheus-stack helm chart v58.2.1
  4. S3 Bucket

Thanos Sidecar

Create Object Storage Secret

https://thanos.io/tip/thanos/storage.md/#s3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

type: S3
config:
  bucket: "kubenetes-thanos"
  endpoint: "s3.ap-northeast-2.amazonaws.com"
  region: "ap-northeast-2"
  aws_sdk_auth: false
  access_key: ""
  insecure: false
  signature_version2: false
  secret_key: ""
  session_token: ""
  put_user_metadata: {}
  http_config:
    idle_conn_timeout: 1m30s
    response_header_timeout: 2m
    insecure_skip_verify: false
    tls_handshake_timeout: 10s
    expect_continue_timeout: 1s
    max_idle_conns: 100
    max_idle_conns_per_host: 100
    max_conns_per_host: 0
    tls_config:
      ca_file: ""
      cert_file: ""
      key_file: ""
      server_name: ""
      insecure_skip_verify: false
    disable_compression: false
  trace:
    enable: false
  list_objects_version: ""
  bucket_lookup_type: auto
  send_content_md5: true
  part_size: 67108864
  sse_config:
    type: ""
    kms_key_id: ""
    kms_encryption_context: {}
    encryption_key: ""
  sts_endpoint: ""
prefix: ""

1

kubectl create secret generic thanos-objstore --from-file=thanos-objstore.yaml --namespace monitoring
  • S3의 access_key, secret_key 설정

Customizing the Chart values.yaml 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# -------------------Promtetheus-------------------
prometheus:
  prometheusSpec:
    # Prometheus의 데이터 압축을 비활성화
    # Thanos compactor가 작업할 때 업로드된 데이터가 손상되지 않도록 하는 데 필요
    # true로 설정하면 --storage.tsdb.max-block-duration=2h 옵션을 Prometheus에 전달
    # thanos 설정 시 자동으로 true로 설정됨
	  disableCompaction: true

	  # Object Storage Secret 설정
    thanos:
      objectStorageConfig:
        existingSecret:
          key: thanos-objstore.yaml
          name: thanos-objstore

	# Thanos Sidecar에서 Thanos 서비스 발견을 위한 서비스
	# 활성화하면 Thanos Query가 --store=dnssrv+_grpc._tcp.${kube-prometheus-stack.fullname}-thanos-discovery.${namespace}.svc.cluster.local를 사용하여 Prometheus 노드에서 Thanos 사이드카를 발견할 수 있음
	thanosService:
		enabled: true
		clusterIP: ""
		type: ClusterIP
		portName: grpc
		port: 10901
		targetPort: "grpc"
		httpPortName: http
		httpPort: 10902
		targetHttpPort: "http"
		clusterIP: "" # None에서 변경
		nodePort: 30901
		httpNodePort: 30902

	# Thanos Sidecar의 메트릭을 수집하기 위한 서비스 모니터 설정
	thanosServiceMonitor:
	    enabled: true

# Thanos 이미지 버전 설정
prometheusOperator:
	prometheusConfigReloader:
	  thanosImage:
	    registry: quay.io
	    repository: thanos/thanos
	    tag: v0.34.1

1

helm upgrade -f custom-values.yaml kube-prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring

Thanos Sidecar 확인 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

kubectl get pod prometheus-kube-prometheus-stack-prometheus-0 -n monitoring -o yaml

...
status:
  containerStatuses:
  - containerID: containerd://4d837c419903774587ef0dfc35c5869782f61ed9d5413e11996c7ef9ab0292ac
    image: quay.io/thanos/thanos:v0.34.1
    imageID: quay.io/thanos/thanos@sha256:567346c3f6ff2927c2c6c0daad977b2213f62d45eca54d48afd19e6deb902181
    lastState: {}
    name: thanos-sidecar
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-05-17T06:10:07Z"
  hostIP: 192.168.25.221
  hostIPs:
  - ip: 192.168.25.221
...

prometheus pod에 statefulset으로 Thanos Sidecar가 배포된 것을 확인할 수 있음

1
2
3
4

kubectl get svc -n monitoring

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
kube-prometheus-stack-thanos-discovery           ClusterIP   10.97.125.31     <none>        10901/TCP,10902/TCP             54m

Thanos Sidecar 엔드포인트를 위한 service도 생성됨을 확인

1
2
3
4
5
6
7
8
9
10
11

kubectl describe pod prometheus-kube-prometheus-stack-prometheus-0 -n monitoring

...
thanos-sidecar:
  State:          Running
    Started:      Fri, 17 May 2024 17:33:12 +0900
  Ready:          True
  Restart Count:  0
  Environment:
    OBJSTORE_CONFIG:  <set to the key 'thanos-objstore.yaml' in secret 'thanos-objstore'>
...

Environment에 OBJSTORE_CONFIG 가 설정되어 있는지 확인

S3 확인

Untitled

2시간 이후부터 메트릭이 적재됨. 2시간 동안의 메트릭은 Prometheus에 존재

참고) Secret config 변경 

1
2
3

kubectl delete secret thanos-objstore -n monitoring
kubectl create secret generic thanos-objstore --from-file=thanos-objstore.yaml -n monitoring
kubectl rollout restart statefulset prometheus-kube-prometheus-stack-prometheus -n monitoring

Thanos Querier

https://thanos.io/tip/thanos/quick-tutorial.md/#querierquery

Thanos Querier는 Prometheus 쿼리를 실행하고 여러 데이터 소스에서 데이터를 집계하여 제공하는 API 게이트웨이

ServiceAccount 

1
2
3
4
5
6
7
8
9
10
11

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations: {}
  labels:
    app.kubernetes.io/component: query-layer
    app.kubernetes.io/instance: thanos-query
    app.kubernetes.io/name: thanos-query
    app.kubernetes.io/version: v0.34.1
  name: thanos-query
  namespace: monitoring
  • Kubernetes에서는 RBAC(Role-Based Access Control)를 사용하여 클러스터 내의 리소스에 대한 접근 권한을 관리. ServiceAccount는 Pod가 클러스터 내에서 특정 리소스에 접근할 수 있도록 권한을 부여하는 데 사용. 서로 다른 Thanos 컴포넌트(thanos-compact, thanos-query, …)에 대해 개별적인 ServiceAccount를 사용하여 서비스 간의 역할을 분리하고, 각 서비스가 필요한 권한만 부여받도록 하고자 함

Service 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: query-layer
    app.kubernetes.io/instance: thanos-query
    app.kubernetes.io/name: thanos-query
    app.kubernetes.io/version: v0.34.1
  name: thanos-query
  namespace: monitoring
spec:
  ports:
  - name: grpc
    port: 10901
    targetPort: 10901
  - name: http
    port: 9090
    targetPort: 9090
    nodePort: 32090
  selector:
    app.kubernetes.io/component: query-layer
    app.kubernetes.io/instance: thanos-query
    app.kubernetes.io/name: thanos-query
  type: NodePort

Deployment 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: query-layer
    app.kubernetes.io/instance: thanos-query
    app.kubernetes.io/name: thanos-query
    app.kubernetes.io/version: v0.34.1
  name: thanos-query
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: query-layer
      app.kubernetes.io/instance: thanos-query
      app.kubernetes.io/name: thanos-query
  template:
    metadata:
      labels:
        app.kubernetes.io/component: query-layer
        app.kubernetes.io/instance: thanos-query
        app.kubernetes.io/name: thanos-query
        app.kubernetes.io/version: v0.34.1
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - thanos-query
              namespaces:
              - monitoring
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - args:
        - query
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:9090
        - --log.level=debug
        - --log.format=logfmt
        - --query.replica-label=prometheus_replica
        - --endpoint=dnssrv+_grpc._tcp.kube-prometheus-stack-thanos-discovery.monitoring.svc.cluster.local
        - --query.timeout=5m
        - --query.lookback-delta=15m
        - |-
          --tracing.config="config":
            "sampler_param": 2
            "sampler_type": "ratelimiting"
            "service_name": "thanos-query"
          "type": "JAEGER"
        - --query.auto-downsampling
        - --web.external-prefix=/thanos/query
        #- --web.route-prefix=/thanos/query
        env:
        - name: HOST_IP_ADDRESS
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        image: quay.io/thanos/thanos:v0.34.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 4
          httpGet:
            path: /-/healthy
            port: 9090
            scheme: HTTP
          periodSeconds: 30
        name: thanos-query
        ports:
        - containerPort: 10901
          name: grpc
        - containerPort: 9090
          name: http
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 9090
            scheme: HTTP
          periodSeconds: 5
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsGroup: 65532
          runAsNonRoot: true
          runAsUser: 65534
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePolicy: FallbackToLogsOnError
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        fsGroup: 65534
        runAsGroup: 65532
        runAsNonRoot: true
        runAsUser: 65534
        seccompProfile:
          type: RuntimeDefault
      serviceAccountName: thanos-query
      terminationGracePeriodSeconds: 120
  • --endpoint=dnssrv+_grpc._tcp.kube-prometheus-stack-thanos-discovery.monitoring.svc.cluster.local: thanos-sidecar의 svc 지정

ServiceMonitor 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: query-layer
    app.kubernetes.io/instance: thanos-query
    app.kubernetes.io/name: thanos-query
    app.kubernetes.io/version: v0.34.1
    tenant: monitoring
  name: thanos-query
  namespace: monitoring
spec:
  endpoints:
  - port: http
    relabelings:
    - action: replace
      separator: /
      sourceLabels:
      - namespace
      - pod
      targetLabel: instance
  selector:
    matchLabels:
      app.kubernetes.io/component: query-layer
      app.kubernetes.io/instance: thanos-query
      app.kubernetes.io/name: thanos-query

1

kubectl apply -f thanos-query-serviceAccount.yaml -f thanos-query-service.yaml -f thanos-query-deployment.yaml -f thanos-query-serviceMonitor.yaml

확인

Untitled

NodePort로 Thanos Query에 접속 후 위와 같이 sidecar가 store 설정 되어 있음을 확인

Untitled

Thanos Query의 ServiceMonitor도 Target에 설정 되어 있음을 확인

Thanos Compactor

https://thanos.io/tip/components/compact.md

Thanos Compactor는 Prometheus TSDB(시계열 데이터베이스) 블록을 관리하고 최적화하는 구성 요소. 다음과 같은 작업을 수행:

  1. 블록 컴팩션
    • 여러 개의 작은 블록을 큰 블록으로 병합하여 디스크 사용량을 줄이고 쿼리 성능을 향상시킴.
    • 컴팩션 과정에서 데이터가 삭제될 수도 있으므로 데이터 무결성을 유지해야 합니다.
  2. 다운샘플링
    • 장기 저장소의 데이터 크기를 줄이기 위해 시간 시계열 데이터를 다운샘플링하여 저장.
    • 이를 통해 장기 저장된 데이터에 대한 쿼리 성능을 개선.
  3. 블록 삭제
    • 오래된 블록을 삭제하여 디스크 공간을 확보.
    • 특정 보존 정책에 따라 데이터를 관리.

ServiceAccount 

1
2
3
4
5
6
7
8
9
10
11

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations: {}
  labels:
    app.kubernetes.io/component: database-compactor
    app.kubernetes.io/instance: thanos-compact
    app.kubernetes.io/name: thanos-compact
    app.kubernetes.io/version: v0.34.1
  name: thanos-compact
  namespace: monitoring

Service 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: database-compactor
    app.kubernetes.io/instance: thanos-compact
    app.kubernetes.io/name: thanos-compact
    app.kubernetes.io/version: v0.34.1
  name: thanos-compact
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 10902
    targetPort: 10902
    nodePort: 32091
  selector:
    app.kubernetes.io/component: database-compactor
    app.kubernetes.io/instance: thanos-compact
    app.kubernetes.io/name: thanos-compact
  type: NodePort

StatefultSet 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/component: database-compactor
    app.kubernetes.io/instance: thanos-compact
    app.kubernetes.io/name: thanos-compact
    app.kubernetes.io/version: v0.34.1
  name: thanos-compact
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: database-compactor
      app.kubernetes.io/instance: thanos-compact
      app.kubernetes.io/name: thanos-compact
  serviceName: thanos-compact
  template:
    metadata:
      labels:
        app.kubernetes.io/component: database-compactor
        app.kubernetes.io/instance: thanos-compact
        app.kubernetes.io/name: thanos-compact
        app.kubernetes.io/version: v0.35.1
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - thanos-compact
                - key: app.kubernetes.io/instance
                  operator: In
                  values:
                  - thanos-compact
              namespaces:
              - monitoring
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - args:
        - compact
        - --web.disable
        - --web.disable-cors
        - --wait
        - --log.level=warn
        - --log.format=logfmt
        - --objstore.config=$(OBJSTORE_CONFIG)
        - --data-dir=/var/thanos/compact
        - --debug.accept-malformed-index
        - --retention.resolution-raw=0d
        - --retention.resolution-5m=0d
        - --retention.resolution-1h=0d
        - --delete-delay=12h
        - --compact.concurrency=1
        - --downsample.concurrency=1
        - --downsampling.disable
        - --deduplication.replica-label=prometheus_replica
        - --deduplication.replica-label=rule_replica
        - |-
          --tracing.config="config":
            "sampler_param": 2
            "sampler_type": "ratelimiting"
            "service_name": "thanos-compact"
          "type": "JAEGER"
        env:
        - name: OBJSTORE_CONFIG
          valueFrom:
            secretKeyRef:
              key: thanos-objstore.yaml
              name: thanos-objstore
        - name: HOST_IP_ADDRESS
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        image: quay.io/thanos/thanos:v0.34.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 4
          httpGet:
            path: /-/healthy
            port: 10902
            scheme: HTTP
          periodSeconds: 30
        name: thanos-compact
        ports:
        - containerPort: 10902
          name: http
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 10902
            scheme: HTTP
          periodSeconds: 5
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /var/thanos/compact
          name: data
          readOnly: false
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        fsGroup: 65534
        runAsGroup: 65532
        runAsNonRoot: true
        runAsUser: 65534
        seccompProfile:
          type: RuntimeDefault
      serviceAccountName: thanos-compact
      terminationGracePeriodSeconds: 120
      volumes: []
  volumeClaimTemplates:
  - metadata:
      labels:
        app.kubernetes.io/component: database-compactor
        app.kubernetes.io/instance: thanos-compact
        app.kubernetes.io/name: thanos-compact
      name: data
    spec:
      storageClassName: "nfs-client"
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
  • 사전에 구성된 Object Storage Secret 설정 ```yaml
    • name: OBJSTORE_CONFIG valueFrom: secretKeyRef: key: thanos-objstore.yaml name: thanos-objstore ```

ServiceMonitor 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: database-compactor
    app.kubernetes.io/instance: thanos-compact
    app.kubernetes.io/name: thanos-compact
    app.kubernetes.io/version: v0.34.1
    tenant: monitoring
  name: thanos-compact
  namespace: monitoring
spec:
  endpoints:
  - port: http
    relabelings:
    - action: replace
      separator: /
      sourceLabels:
      - namespace
      - pod
      targetLabel: instance
  selector:
    matchLabels:
      app.kubernetes.io/component: database-compactor
      app.kubernetes.io/instance: thanos-compact
      app.kubernetes.io/name: thanos-compact

1

kubectl apply -f thanos-compact-serviceAccount.yaml -f thanos-compact-service.yaml -f thanos-compact-statefulSet.yaml -f thanos-compact-serviceMonitor.yaml

Prometheus Target 확인

Untitled

Thanos Compactor의 ServiceMonitor도 Target에 설정 되어 있음을 확인

Grafana에서 Thanos Query를 통해 메트릭 조인

Untitled

Grafana에 접속하여 Thanos Querier를 데이터소스로 추가. Connection 섹션의 URL 필드에 다음 URL을 입력

1

http://thanos-query.monitoring:9090/

thanos-query.monitoring: 네임스페이스는 monitoring이고 서비스 이름은 thanos-query

Untitled

Prometheus type을 Thanos로 설정

Untitled

Thanos Querier를 데이터소스로 하여 메트릭 조회. 아래와 같은 과정을 Thanos가 수행:

  • 2시간 내 메트릭은 Prometheus에서 조회
  • Prometheus의 보존 기간(일반적으로 2시간)보다 오래된 모든 메트릭이 Object Storage에서 조회

일반적인 설정에서는 Prometheus 서버가 짧은 보존 기간(예: 2시간)으로 구성됨. 이 보존 기간보다 오래된 모든 데이터는 더 큰 블록으로 압축되어 Thanos Sidecar에 의해 Object Storage로 업로드됨. 히스토리 데이터를 요청할 때 Object Storage에 있는 이러한 블록을 조회.

Thanos Bucket Web

https://thanos.io/tip/components/tools.md/#bucket-web

Thanos Bucket Web은 웹 UI를 통해 Thanos 스토리지 버킷의 상태를 검사하고 주어진 갱신 주기로 뷰를 주기적으로 업데이트

ServiceAccount 

1
2
3
4
5
6
7
8
9
10
11

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations: {}
  labels:
    app.kubernetes.io/component: object-store-bucket-debugging
    app.kubernetes.io/instance: thanos-bucket
    app.kubernetes.io/name: thanos-bucket
    app.kubernetes.io/version: v0.34.1
  name: thanos-bucket
  namespace: monitoring

Service 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: object-store-bucket-debugging
    app.kubernetes.io/instance: thanos-bucket
    app.kubernetes.io/name: thanos-bucket
    app.kubernetes.io/version: v0.34.1
  name: thanos-bucket
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 10902
    targetPort: 10902
  selector:
    app.kubernetes.io/component: object-store-bucket-debugging
    app.kubernetes.io/instance: thanos-bucket
    app.kubernetes.io/name: thanos-bucket
  type: NodePort

Deployment 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: object-store-bucket-debugging
    app.kubernetes.io/instance: thanos-bucket
    app.kubernetes.io/name: thanos-bucket
    app.kubernetes.io/version: v0.34.1
  name: thanos-bucket
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: object-store-bucket-debugging
      app.kubernetes.io/instance: thanos-bucket
      app.kubernetes.io/name: thanos-bucket
  template:
    metadata:
      labels:
        app.kubernetes.io/component: object-store-bucket-debugging
        app.kubernetes.io/instance: thanos-bucket
        app.kubernetes.io/name: thanos-bucket
        app.kubernetes.io/version: v0.34.1
    spec:
      containers:
      - args:
        - tools
        - bucket
        - web
        - --log.level=warn
        - --log.format=logfmt
        - --objstore.config=$(OBJSTORE_CONFIG)
        - --web.external-prefix=/thanos/bucket
        - --web.route-prefix=/thanos/bucket
        - |-
          --tracing.config="config":
            "sampler_param": 2
            "sampler_type": "ratelimiting"
            "service_name": "thanos-bucket"
          "type": "JAEGER"
        - --label=cluster_name
        - --refresh=5m
        env:
        - name: OBJSTORE_CONFIG
          valueFrom:
            secretKeyRef:
              key: thanos-objstore.yaml
              name: thanos-objstore
        - name: HOST_IP_ADDRESS
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        image: quay.io/thanos/thanos:v0.34.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 4
          httpGet:
            path: /-/healthy
            port: 10902
            scheme: HTTP
          periodSeconds: 30
        name: thanos-bucket
        ports:
        - containerPort: 10902
          name: http
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 10902
            scheme: HTTP
          periodSeconds: 5
        resources:
          limits:
            cpu: 0.42
            memory: 420Mi
          requests:
            cpu: 0.123
            memory: 123Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsGroup: 65532
          runAsNonRoot: true
          runAsUser: 65534
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts: []
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        fsGroup: 65534
        runAsGroup: 65532
        runAsNonRoot: true
        runAsUser: 65534
        seccompProfile:
          type: RuntimeDefault
      serviceAccountName: thanos-bucket
      terminationGracePeriodSeconds: 120
      volumes: []
  • 사전에 구성된 Object Storage Secret 설정 ```yaml
    • name: OBJSTORE_CONFIG valueFrom: secretKeyRef: key: thanos-objstore.yaml name: thanos-objstore ```
1

kubectl apply -f thanos-bucket-serviceAccount.yaml -f thanos-bucket-service.yaml -f thanos-bucket-deployment.yaml

Bucket Web 접속 확인

Untitled

Reference

  1. https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/thanos.md
  2. https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#thanosspec
  3. https://github.com/thanos-io/kube-thanos/tree/main/manifests
  4. https://github.com/thanos-io/kube-thanos/tree/main/examples/all/manifests
Kubernetes
Kubernetes Prometheus Thanos
This post is licensed under CC BY 4.0 by the author.
Recently Updated
Contents

© 2024 woobni. Some rights reserved.