KubeEdge chekka tugunlariga javob berish qo'llanmasi

1. fon

So'nggi loyiha davomida dastur ishlatish jarayonida ‘kubectl get nodes’da ‘NotReady’ holatida bo'lishi yoki ‘kubectl exec’ning ishlamasligi, Podning to'satdan Evicted holatiga o'tishi kabi turli muammolarga duch keldim. Shu sababli haqiqiy ishlash davomida yuzaga kelgan to'rtta nosozlik turi bo'yicha misolga asoslangan holda KubeEdge muhitidagi muammolarni hal qilish bo'yicha qo'llanma tayyorladim.

2. Kirish

Kubernetes asosidagi umumiy klasterlardan farqli o'laroq, KubeEdge bulut va chekka tugmalar o'rtasida CloudCore ↔ EdgeCore qo'shimcha tunel mavjud. Buning evaziga tarmoq barqaror bo'lmagan joylarda ham chekka tugmalarni boshqarish mumkin, lekin shu bilan birga muammoning sabablari yanada murakkablashadi.

3. Nosozlik turlari

Nosozlik turi 1	kubectl exec muvaffaqiyati — "kutilmagan EOF"

(1) Belgi 

$ kubectl exec -it my-app-6jf2m -n iot-edge -- /bin/bash
error: Internal error occurred: error sending request:
Post "https://192.168.1.3:10351/exec/...": unexpected EOF

‘kubectl exec’ni ishga tushirdim, ammo ulanish uzilib, hech narsa ishlamay qoldi. O'sha tugma bilan bir xil tugmada boshqa Podlar Running holatida ko'rinmoqda, lekin faqat exec ishlamaydi.

(2) Sabablarni tahlil qilish

KubeEdge’da ‘kubectl exec’ ishlashi uchun EdgeStream funksiyasi yoqilgan bo'lishi kerak. Ushbu funktsiya o'chirilgan bo'lsa, API Server → CloudCore → EdgeCore orqali boradigan exec tuneli o'z-o'zidan mavjud bo'lmaydi.

Oddiy Kubernetes API Server kubeletning 10250 portiga to'g'ridan-to'g'ri murojaat qilib execni bajaradi, lekin KubeEdge bu yo'lga ega emas. EdgeStream bo'lmasa exec, logs, port-forwardning barchasi ishlamaydi.

[kubectl exec flow in KubeEdge]
API Server
    ↓
CloudCore (streamPort: 10003)
    ↓ ← without this tunnel, unexpected EOF occurs
EdgeCore (edgeStream.server)
    ↓
컨테이너

(3) Yechimlar

CloudCore sozlamalarini tekshirish va faollashtirish:

# cloudcore.yaml
cloudStream:
  enable: true
  streamPort: 10003
  tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
  tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
  tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key

EdgeCore sozlamalarini tekshirish va faollashtirish:

# edgecore.yaml
edgeStream:
  enable: true
  handshakeTimeout: 30
  readDeadline: 15
  server: <CloudCore_IP>:10004
  tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
  tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
  tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key
  writeDeadline: 15

Sozlamalar o'zgartirilgandan so'ng, ikkala tomon ham qayta ishga tushirilishi kerak:

# On the master node where CloudCore is running
systemctl restart cloudcore.

# On the edge node
systemctl restart edgecore

Muammo turi 2	DiskPressure oqibatida Pod Eviction

(1) Belgilari 

$ kubectl get po -A -o wide | grep <NodeName>
iot-edge chronograf-xxxxx 0/1 Evicted 0 2m
iot-edge influxdb-xxxxx 0/1 Evicted 0 9m
iot-edge my-app-xxxxx 0/1 Evicted 0 14m

Maxsus tugunlarda joylashgan Podlar bir vaqtning o'zida Evicted holatiga o'tadi. Bir necha emas, balki ushbu tugundagi barcha Podlar chiqariladigan holatlarda tugun darajasidagi resurs muammolarini shubhalanish kerak.

(2) Sababni tahlil qilish

Eviction kubelet (KubeEdge'da edgecore) tugun resurslari kamligini sezganda Podni majburan tugatish mexanizmidir. 'kubectl describe node' komandasi orqali to'g'ridan-to'g'ri sababni tekshirish mumkin:

kubectl describe node <NodeName> | grep -A5 "Conditions"

Amaldagi holatlarda quyidagi xabarlar paydo bo'ldi:

DiskPressure True ...
Message: The node was low on resource: ephemeral-storage.
  Threshold: 37556267476 (~35GB)
  Available: 36674584Ki (~34.9GB)

Mavjud disk chegarasidan pastga tushganda kubelet ushbu tugundagi Podni majburan chiqarib yuboradi

Diskdan foydalanishni tekshirish
Edge tuguniga SSH orqali ulanish va qayerda diskni egallab olganligini aniqlash:

df -h
du -sh /* 2>/dev/null | sort -rh | head -15

Amaliy misolda bir xil dasturiy ta'minotning loglar katalogi butun diskning 89% ni egalladi.

$ du -sh /var/app/logs/*/
166G /var/app/logs/device-001.ASSET/

(3) Yeсhim usuli

Eski loglarni tozalash(O'chirishdan oldin saqlanishi kerak yoki kerak emasligini tekshirish shart!)

# Delete files older than N days
find /var/app/logs -mtime +7 -type f -delete

Ishlatilmayotgan konteyner rasmlarini tozalash

crictl rmi --prune

edgecore-ni qayta ishga tushirish orqali DiskPressure qayta baholash

systemctl restart edgecore

Evicted Podni tozalash va qayta tarqatish

# On the master node
kubectl get pods -A | grep Evicted | awk '{print "kubectl delete pod " $2 " -n " $1}' | sh

Falokat turining 3	Init Container ImagePullBackOff — DNS muammosi

(1) Belgi 

$ kubectl describe pod my-app-qn6r8 -n iot-edge
Init Containers:
  init-snapshot:
    State: Waiting
      Reason: ImagePullBackOff
    Image: registry.company.internal:5000/infra/busybox:latest

Aksincha, Init Container'da rasmni ko‘tarishda xato qaytganida, bu asosan Start Container's izohidir. Agar Init Container muvaffaqiyatsiz bo‘lsa, asosiy konteyner hatto boshlanishi ham mumkin emas.

Bu holatda, asosiy konteynerning avvalgi chiqarilishi шаклida quyidagicha ko‘rinadi:

Last State: Terminated
  Reason: Unknown
  Exit Code: 255

Exit Code 255 konteyner runtime butun jarayoni majburiy to‘xtatganini bildiradi, ko‘pincha tug‘ilgan muammolar (oldin DiskPressure, tarmoq uzilishlari va hk.) natijasidir.

(2) Sabablar tahlili — shaxsiy registr DNS muammosi

ImagePullBackOff sabablari turlicha, ammo ichki shaxsiy registrlardan foydalanayotgan IoT muhitlarida DNS muammosibu ayniqsa tez-tez yuz beradi.

Tekshirish usuli:

# On the edge node
nslookup registry.company.internal

DNS SERVFAIL paydo bo'lsa, domen o'zini tahlil qila olmaydigan holatdir.

server can't find registry.company.internal: SERVFAIL

(3) Hal qilish usuli

Tezkor vaqtinchalik yechim — /etc/hosts ga ro'yxatdan o'tkazish
Boshqa normal tugunlardan IP ni tekshirgandan so'ng ‘/etc/hosts’ ga to'g'ridan-to'g'ri ro'yxatdan o'tkazing:

# Resolve IP from a healthy node
nslookup registry.company.internal
# Add entry to /etc/hosts on the affected node
echo "<IP> registry.company.internal" >> /etc/hosts
# Test image pull
crictl pull registry.company.internal:5000/infra/busybox:latest

DNSning asosiy sababini hal qilish

Vaqtinchalik chora-tadbirlarni ko'rganingizdan so'ng ‘systemd-resolved’ holatini tekshirib, qayta ishga tushiring:

systemctl status systemd-resolved
systemctl restart systemd-resolved

# Verify DNS resolution is restored
nslookup registry.company.internal

DNSni tiklagandan so'ng Podni qayta ishga tushiring

# Force delete and recreate the Pod from the master node
kubectl delete pod my-app-qn6r8 -n iot-edge --force --grace-period=0

# Watch Pod status
kubectl get pod -n iot-edge -w | grep my-app

4. Xulosa

- ‘kubectl exec’ bo'lmasa, muammo kubeletda emas, balki EdgeStream sozlashida bo'lishi mumkin.

- Agar Pod ko'p miqdorda Evicted bo'lsa, tugunning disk foydalanishini tekshiring.

- ImagePullBackOff tasvir muammosi emas, balki DNS muammosi bo'lishi mumkin.

Chekka muhit ma'lumot markazidan farqli o'laroq, maydondagi tarmoqning noma'lumligi, cheklangan disk hajmi va tarqatilgan tugunlarni boshqarish kabi murakkab qiyinchiliklarga ega. Yuqoridagi tekshirish ro'yxatini jamoangizda bo'lishsangiz, kim on-call bo'lishidan qat'i nazar javob vaqtingizni qisqartirishingiz mumkin.

Jsia