深入kubernetes之Pod——一pod多容器
六、深入Pod——一pod多容器
一pod多容器，可以说是kube精华所在，让多个同应用的单一容器可以整合到一个类虚拟机中，使其所有容器共用一个vm的资源，提高耦合度，神来之笔，从而方便副本的复制，提高整体的可用性
接下来会从我自己的学习历程，讲诉一pod多容器，其中历经的困难，此问题有困扰一个月之久。
1、测试过程：
根据文章：http://www.csdn.net/article/2014-12-18/2823196 ，看到pod还有一pod多容器的功能，仅是看了文章便激动不已，一pod多容器，可以说是kube的精华所在，具体优点，可以参考第一章的简介（14项）
①、第一次测试——失败：
使用文件：【nginx_redis_pod.json】
目的：一pod两容器
结论：失败，1个启动成功，1个启动失败，一直在重启，不知道是配置文件的问题，还是什么
提前说明下各image启动的端口：
images | port |
www.perofu.com:7070/centos6.4_ip_nginx | 80、22 |
www.perofu.com:7070/centos6.4_redis | 6379、22 |
[root@www pod_nginx_redis_kube]# vi nginx_redis_pod.json { "id": "nginx-1", "kind": "Pod", "metadata": { "name": "nginx-1" }, "apiVersion": "v1", "desiredState": { "manifest": { "version": "v1", "id": "nginx-1" } }, "spec": { "containers": [ { "name": "nginx-1", "image": "www.perofu.com:7070/centos6.4_ip_nginx", "command": [ "/etc/rc.local" ] }, { "name": "redis", "image": "www.perofu.com:7070/centos6.4_redis", "command": [ "/etc/rc.local" ] } ] }, "labels": { "name": "nginx-1" } } ~ "nginx_redis_pod.json" 36L, 602C written [root@www pod_nginx_redis_kube]# /usr/bin/kubectl create -f nginx_redis_pod.json pods/nginx-1 [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE nginx-1 0/2 Pending 0 4s [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE nginx-1 2/2 Running 1 56s [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE nginx-1 1/2 Running 2 59s |
②、第二次测试——失败：
使用文件：【nginx_redis_pod_test.json】
目的：看到node，都是192.168.16.240，就想，一个pod里的容器，是不是都不能放到一个node中？
结论：失败，1个启动成功，1个启动失败，一直在重启，
加了一个node，一个pod就只能在一个node上，有的容器能起来，有的不行，没有什么规律，看官网的多容器yaml写的，怎么就不行？
提前说明下各image启动的端口：
images | port |
www.perofu.com:7070/centos6.4_ip_nginx | 80、22 |
www.perofu.com:7070/centos6.4_redis | 6379、22 |
[root@www pod_nginx_redis_kube]# cat nginx_redis_pod_test.json --- apiVersion: v1 kind: Pod metadata: name: www spec: containers: - name: nginx image: www.perofu.com:7070/centos6.4_ip_nginx command: - /etc/rc.local - name: redis-1 image: www.perofu.com:7070/centos6.4_redis command: - /etc/rc.local [root@www pod_nginx_redis_kube]# /usr/bin/kubectl describe pod www Name: www Namespace: default Image(s): www.perofu.com:7070/centos6.4_ip_nginx,www.perofu.com:7070/centos6.4_redis Node: 192.168.16.240/192.168.16.240 Labels: <none> Status: Running Reason: Message: IP: 172.22.2.2 Replication Controllers: <none> Containers: nginx: Image: www.perofu.com:7070/centos6.4_ip_nginx State: Running Started: Thu, 21 Jan 2016 16:07:42 +0800 Ready: False Restart Count: 9 redis-1: Image: www.perofu.com:7070/centos6.4_redis State: Running Started: Thu, 21 Jan 2016 16:04:19 +0800 Ready: True Restart Count: 0 Conditions: Type Status Ready False Events: FirstSeen LastSeen Count From SubobjectPath Reason Message Thu, 21 Jan 2016 16:03:34 +0800 Thu, 21 Jan 2016 16:03:34 +0800 1 {scheduler } scheduled Successfully assigned www to 192.168.16.240 Thu, 21 Jan 2016 16:04:16 +0800 Thu, 21 Jan 2016 16:04:16 +0800 1 {kubelet 192.168.16.240} implicitly required container POD pulled Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine Thu, 21 Jan 2016 16:04:17 +0800 Thu, 21 Jan 2016 16:04:17 +0800 1 {kubelet 192.168.16.240} implicitly required container POD created Created with docker id c7884e18b3ff Thu, 21 Jan 2016 16:04:17 +0800 Thu, 21 Jan 2016 16:04:17 +0800 1 {kubelet 192.168.16.240} implicitly required container POD started Started with docker id c7884e18b3ff Thu, 21 Jan 2016 16:04:18 +0800 Thu, 21 Jan 2016 16:04:18 +0800 1 {kubelet 192.168.16.240} spec.containers{nginx} created Created with docker id ce71cdeddab3 Thu, 21 Jan 2016 16:04:18 +0800 Thu, 21 Jan 2016 16:04:18 +0800 1 {kubelet 192.168.16.240} spec.containers{nginx} started Started with docker id ce71cdeddab3 Thu, 21 Jan 2016 16:04:19 +0800 Thu, 21 Jan 2016 16:04:19 +0800 1 {kubelet 192.168.16.240} spec.containers{redis-1} created Created with docker id 15ca805a9e99 Thu, 21 Jan 2016 16:04:19 +0800 Thu, 21 Jan 2016 16:04:19 +0800 1 {kubelet 192.168.16.240} spec.containers{redis-1} started Started with docker id 15ca805a9e99 Thu, 21 Jan 2016 16:04:32 +0800 Thu, 21 Jan 2016 16:04:32 +0800 1 {kubelet 192.168.16.240} spec.containers{nginx} created Created with docker id edfacd145c9b Thu, 21 Jan 2016 16:04:32 +0800 Thu, 21 Jan 2016 16:04:32 +0800 1 {kubelet 192.168.16.240} spec.containers{nginx} started Started with docker id edfacd145c9b
|
③、第三次测试——侥幸成功：
使用文件：【test2_redis_nginx_pod.yaml 】
目的：尝试使用其他image（不带command启动的）进行测试
结论：成功，怀疑是配制文件只能允许一个command的
说明一下，这一次和之前yaml的差别，
成功：两个images，一个有command，一个没有
失败：两个images，两个都有command
再加上网上的文档，包括官网的，都是没有command的，由于我是自己做的images，都是通过command进行启动的，就一直失败。
不是配置文件的问题，而是一个kube的坑，由此可得：
【在一个pod中创建多个容器，只能有一个images可以使用command，否则就会有容器一直在重启】
新发现：看到node，都是192.168.16.240，就想，一个pod里的容器，是不是都不能放到一个node中？
新发现之测试结果：加了一个node，一个pod就只能在一个node上，有的容器能起来，有的不行，没有什么规律，看官网的多容器yaml写的，怎么就不行？
提前说明下各image启动的端口：
images | port |
www.perofu.com:7070/registry | 5000 |
www.perofu.com:7070/centos6.4_redis | 6379、22 |
[root@www pod_nginx_redis_kube]# vi test2_redis_nginx_pod.yaml
name: perofu4 labels: name: perofu4 spec: replicas: 1 selector: name: perofu4 template: metadata: labels: name: perofu4 spec: containers: - name: redis image: www.perofu.com:7070/centos6.4_redis command: - ‘/bin/bash‘ - ‘-c‘ - ‘/etc/rc.local‘ - name: nginx image: www.perofu.com:7070/registry [root@www pod_nginx_redis_kube]# /usr/bin/kubectl create -f test2_redis_nginx_pod.yaml replicationcontrollers/perofu4 [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE perofu-wduar 1/2 Running 1 14s [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE perofu-wduar 2/2 Running 2 2m [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE perofu-wduar 2/2 Running 2 2m [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# /usr/bin/kubectl describe pod perofu-wduar Name: perofu-wduar Namespace: default Image(s): www.perofu.com:7070/centos6.4_redis,www.perofu.com:7070/registry Node: 192.168.16.240/192.168.16.240 Labels: name=perofu4 Status: Running Reason: Message: IP: 172.22.2.9 Replication Controllers: perofu4 (1/1 replicas created) Containers: redis: Image: www.perofu.com:7070/centos6.4_redis State: Running Started: Tue, 16 Feb 2016 16:54:32 +0800 Ready: True Restart Count: 0 nginx: Image: www.perofu.com:7070/registry State: Running Started: Tue, 16 Feb 2016 16:54:50 +0800 Ready: True Restart Count: 2 Conditions: Type Status Ready True
#验证 [root@localhost ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5c6581d919a5 www.perofu.com:7070/registry:latest "docker-registry" 2 minutes ago Up 2 minutes k8s_nginx.65e9d290_perofu-wduar_default_eaa53352-cc1f-11e5-938c-000c29eae008_03b774d2 08315bce234b centos6.4_redis:latest "/bin/bash -c /etc/r 2 minutes ago Up 2 minutes k8s_redis.510ddc4c_perofu-wduar_default_eaa53352-cc1f-11e5-938c-000c29eae008_3e6b7907 15bd3a44a717 gcr.io/google_containers/pause:0.8.0 "/pause" 2 minutes ago Up 2 minutes k8s_POD.e4cc795_perofu-wduar_default_eaa53352-cc1f-11e5-938c-000c29eae008_8c768cca [root@localhost ~]# [root@localhost ~]# [root@localhost ~]# docker exec -it 08315bce234b /bin/bash [root@perofu-wduar /]# [root@perofu-wduar /]# ifconfig eth0 Link encap:Ethernet HWaddr 02:42:AC:16:02:09 inet addr:172.22.2.9 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::42:acff:fe16:209/64 Scope:Link UP BROADCAST RUNNING MTU:1500 Metric:1 RX packets:10 errors:0 dropped:0 overruns:0 frame:0 TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:823 (823.0 b) TX bytes:718 (718.0 b)
lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:518 (518.0 b) TX bytes:518 (518.0 b)
[root@perofu-wduar /]# netstat -anplt Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 26/sshd tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 27/redis-server 0.0 tcp 0 0 :::22 :::* LISTEN 26/sshd [root@perofu-wduar /]# [root@perofu-wduar /]# ps axu USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.2 11348 1288 ? Ss 03:54 0:00 /bin/sh /etc/rc.local root 26 0.0 0.6 64168 2972 ? S 03:54 0:00 /usr/sbin/sshd -D root 27 0.1 0.3 36484 1896 ? Ssl 03:54 0:00 redis-server 0.0.0.0:6379 root 30 0.1 0.3 11484 1816 ? S 03:59 0:00 /bin/bash root 45 0.0 0.2 13364 1044 ? R+ 04:00 0:00 ps axu [root@perofu-wduar /]# [root@perofu-wduar /]# ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 03:54 ? 00:00:00 /bin/sh /etc/rc.local root 26 1 0 03:54 ? 00:00:00 /usr/sbin/sshd -D root 27 1 0 03:54 ? 00:00:00 redis-server 0.0.0.0:6379 root 30 0 0 03:59 ? 00:00:00 /bin/bash root 46 30 0 04:01 ? 00:00:00 ps -ef
#登陆第二个容器： [root@localhost ~]# docker exec -it 5c6581d919a5 /bin/bash root@perofu-wduar:/# root@perofu-wduar:/# ifconfig eth0 Link encap:Ethernet HWaddr 02:42:ac:16:02:09 inet addr:172.22.2.9 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::42:acff:fe16:209/64 Scope:Link UP BROADCAST RUNNING MTU:1500 Metric:1 RX packets:10 errors:0 dropped:0 overruns:0 frame:0 TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:823 (823.0 B) TX bytes:718 (718.0 B)
lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:518 (518.0 B) TX bytes:518 (518.0 B)
root@perofu-wduar:/# root@perofu-wduar:/# ps axu USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.9 55520 4716 ? Ss 08:54 0:00 /usr/bin/python /usr/local/bin/gunicorn --access-logfile - --error-logfile - --max-requests 100 -k gevent --grac root 16 0.9 5.8 104620 28940 ? S 08:54 0:07 /usr/bin/python /usr/local/bin/gunicorn --access-logfile - --error-logfile - --max-requests 100 -k gevent --grac root 17 0.9 6.3 104632 31392 ? S 08:54 0:07 /usr/bin/python /usr/local/bin/gunicorn --access-logfile - --error-logfile - --max-requests 100 -k gevent --grac root 20 0.9 6.3 104644 31356 ? S 08:54 0:07 /usr/bin/python /usr/local/bin/gunicorn --access-logfile - --error-logfile - --max-requests 100 -k gevent --grac root 21 0.9 6.3 104656 31456 ? S 08:54 0:07 /usr/bin/python /usr/local/bin/gunicorn --access-logfile - --error-logfile - --max-requests 100 -k gevent --grac root 28 0.7 0.3 18140 1968 ? S 09:08 0:00 /bin/bash root 44 0.0 0.2 15568 1132 ? R+ 09:08 0:00 ps axu root@perofu-wduar:/# root@perofu-wduar:/# netstat -anptl Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 1/python tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN - tcp6 0 0 :::22 :::* LISTEN - root@perofu-wduar:/#
#在同一个pod的两个容器中，都有ip：172.22.2.9，都出现了redis的6397端口和python端口5000 [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE perofu-wduar 2/2 Running 2 17h |
④、第四次测试——失败：
使用文件：【test6_redis_nginx_pod_faile.yaml】
目的：重新build镜像，将启动脚本放到容器中，再测试是否是command的问题
结论：失败，为何一pod两容器，无command，都还是有一个容器，在一直重启？
提前说明下各image启动的端口：
images | port |
www.perofu.com:7070/centos6.4_ip_nginx_cmd | 80、22 |
www.perofu.com:7070/centos6.4_ip_redis_cmd | 6379、22 |
[root@www pod_nginx_redis_kube]# vi test6_redis_nginx_pod_faile.yaml
apiVersion: v1 kind: ReplicationController metadata: name: perofu1 labels: name: perofu1 spec: replicas: 1 selector: name: perofu1 template: metadata: labels: name: perofu1 spec: containers: - name: nginx5 image: www.perofu.com:7070/centos6.4_ip_nginx_cmd - name: redis5 image: www.perofu.com:7070/centos6.4_ip_redis_cmd ~ "test6_redis_nginx_pod_ok.yaml" 20L, 439C written [root@www pod_nginx_redis_kube]# /usr/bin/kubectl create -f test6_redis_nginx_pod_faile.yaml replicationcontrollers/perofu1 [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE perofu1-9h8of 0/2 Pending 0 3s perofu-wduar 2/2 Running 0 19h [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE perofu1-9h8of 1/2 Running 2 30s perofu-wduar 2/2 Running 0 19h [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE perofu1-9h8of 1/2 Running 10 2m perofu-wduar 2/2 Running 0 19h [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# /usr/bin/kubectl get pod NAME READY STATUS RESTARTS AGE perofu1-9h8of 1/2 Running 8 3m perofu-wduar 2/2 Running 0 19h [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# [root@www pod_nginx_redis_kube]# /usr/bin/kubectl describe pod perofu1-9h8of Name: perofu1-9h8of Namespace: default Image(s): www.perofu.com:7070/centos6.4_ip_nginx_cmd,www.perofu.com:7070/centos6.4_ip_redis_cmd Node: 192.168.16.240/192.168.16.240 Labels: name=perofu1 Status: Running Reason: Message: IP: 172.22.2.14 Replication Controllers: perofu1 (1/1 replicas created) Containers: nginx5: Image: www.perofu.com:7070/centos6.4_ip_nginx_cmd State: Running Started: Wed, 17 Feb 2016 12:12:12 +0800 Ready: True Restart Count: 0 redis5: Image: www.perofu.com:7070/centos6.4_ip_redis_cmd State: Running Started: Wed, 17 Feb 2016 12:16:31 +0800 Ready: False Restart Count: 11 Conditions: Type Status Ready False |
⑤、第五次测试——找到问题（端口冲突）：
使用文件：【test12_redis_nginx_pod_ok.yaml】
突发奇想：自己做的镜像，每个都会启动ssh的，会不会是端口冲突？
目的：验证是否是端口冲突，导致容器一直重启
结论：确实是因为端口冲突，导致容器一直重启（sshd）
提前说明下各image启动的端口：
images | port |
www.perofu.com:7070/centos6.4_ip_nginx_cmd | 80、22 |
www.perofu.com:7070/centos6.4_ip_redis_cmd_without_sshd_v1 | 6379 |
[root@www pod_nginx_redis_kube]# cat test12_redis_nginx_pod_ok.yaml apiVersion: v1 kind: ReplicationController metadata: name: wechatv labels: name: wechatv spec: replicas: 1 selector: name: wechatv template: metadata: labels: name: wechatv spec: containers: - name: nginx image: www.perofu.com:7070/centos6.4_ip_nginx_cmd - name: redis image: www.perofu.com:7070/centos6.4_ip_redis_cmd_without_sshd_v1 [root@www pod_nginx_redis_kube]# |
2、问题总结：
①、看来专门为Docker做的image，都是启动ssh的，无法放到kube的同一个pod中（主要问题所在），需要重新为kube创建新的image
②、由于使用kube管理的容器，无法查到为何会重启，查了kube的日志没看到，容器直接重启，无法看到容器启动日志，所以不是很快的确定问题所在
③、以后使用kube，docker镜像的build，需要做一套有ssh的，和一套没有ssh的，即Dockerfile中，不要在/etc/rc.local中加入ssh启动，即可，利用build缓存的机制，只要先做好一套image，再修改Dockerfile在/etc/rc.local的ssh部分，再build一次就可以了，很快，真的很快
④、在使用kube的同一pod多容器时，最好还是搭配上一个有ssh的，这样方便用户登陆
本文出自 “无咎” 博客，谢绝转载！