ingress nginx 服务部署

ingress 简介

Ingress 公开了从集群外部到集群内服务的 HTTP 和 HTTPS 路由。 流量路由由 Ingress 资源上定义的规则控制。

1
2
3
4
5
internet
|
[ Ingress ]
--|-----|--
[ Services ]

可以将 Ingress 配置为服务提供外部可访问的 URL、负载均衡流量、终止 SSL/TLS,以及提供基于名称的虚拟主机等能力。 Ingress 控制器 通常负责通过负载均衡器来实现 Ingress,尽管它也可以配置边缘路由器或其他前端来帮助处理流量。

ingress控制器借助service的服务发现机制实现配置的动态更新以实现Pod的负载均衡机制实现,由于涉及到Ingress Controller的动态更新,目前社区Ingress Controller大体包含两种类型的控制器:

传统的七层负载均衡如Nginx,HAproxy,开发了适应微服务应用的插件,具有成熟,高性能等优点;
新型微服务负载均衡如Traefik,Envoy,Istio,专门适用于微服务+容器化应用场景,具有动态更新特点;

类型 常见类型 优点 缺点
传统负载均衡 nginx,haproxy 成熟,稳定,高性能 动态更新需reload配置文件
微服务负载均衡 Traefik,Envoy,Istio 天生为微服务而生,动态更新 性能还有待提升

安装

使用 NodePort的方式部署 ingress-nginx-controller

1
2
3
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/baremetal/deploy.yaml
mv deploy.yaml ingress-deploy-hostnetwork.yaml
kubectl apply -f ingress-deploy-hostnetwork.yaml

检查服务启动是否正常(拉镜像需要等待一段时间)

1
kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

检查服务是否正常

1
2
3
4
5
6
7
8
9
10
11
12
POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')

kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v0.35.0
Build: 54ad65e32bcab32791ab18531a838d1c0f0811ef
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.2

-------------------------------------------------------------------------------

配置本地网络启动

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
.....
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
.....
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
.....
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet

修改内容:

  1. kind: DaemonSet 配置服务启动方式为DaemonSet(原来是deployment)。
  2. 配置 spec.hostNetwork: true,本地网络启动。
  3. 配置 dns的策略为: ClusterFirstWithHostNet, 与hostNetwork 配合使用,参考官方文档pod-s-dns-policy

重新同步一下pod文件

1
kubectl apply -f ingress-deploy-hostnetwork.yaml

配置标签,让ingress 只调度到Master节点上

为了让网关的性能更好,我们需要给ingress打标签,让ingress调度到master机器上。同时给master 打上污点,禁止其他pod调度到master节点上。

1
2
3
4
5
6
7
8
9
kubectl get pods -n ingress-nginx  -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-z7sts 0/1 Completed 0 2d16h 172.30.199.211 node7 <none> <none>
ingress-nginx-admission-patch-4qgtz 0/1 Completed 0 2d16h 172.30.139.14 node6 <none> <none>
ingress-nginx-controller-h5fdd 1/1 Running 1 2d16h 192.168.66.16 node6 <none> <none>
ingress-nginx-controller-j6n9z 1/1 Running 0 2d16h 192.168.66.11 node1 <none> <none>
ingress-nginx-controller-kfnxt 1/1 Running 1 2d16h 192.168.66.17 node7 <none> <none>
ingress-nginx-controller-n5pl4 1/1 Running 0 2d16h 192.168.66.12 node2 <none> <none>
ingress-nginx-controller-qzbb5 1/1 Running 0 2d16h 192.168.66.13 node3 <none> <none>

对 node1 node2 打标签

1
2
kubectl label nodes node1 api/ingress-controller=true
kubectl label nodes node2 api/ingress-controller=true

查看标签

1
2
3
4
5
6
7
8
# 为了好看,去掉相同的标签
kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
node1 Ready <none> 3h53m v1.16.6 api/ingress-controller=true,kubernetes.io/hostname=node1
node2 Ready <none> 3h53m v1.16.6 api/ingress-controller=true,kubernetes.io/hostname=node2
node3 Ready <none> 3h53m v1.16.6 kubernetes.io/hostname=node3
node6 Ready <none> 3d4h v1.16.6 kubernetes.io/hostname=node6
node7 Ready <none> 3d4h v1.16.6 kubernetes.io/hostname=node7

增加标签选择,及容忍污点

1
2
3
4
5
6
7
8
9
10
11
# vim ingress-deploy-hostnetwork.yaml
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
nodeSelector:
api/ingress-controller: "true"
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Equal"
value: ""
effect: "NoSchedule"

在刚才修改网络的后面继续增加以上命令

从新同步一下配置

1
kubectl apply -f ingress-deploy-hostnetwork.yaml
1
2
3
4
5
6
[root@node1 ~]# kubectl get pods -n ingress-nginx  -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-z7sts 0/1 Completed 0 2d16h 172.30.199.211 node7 <none> <none>
ingress-nginx-admission-patch-4qgtz 0/1 Completed 0 2d16h 172.30.139.14 node6 <none> <none>
ingress-nginx-controller-6s2lp 1/1 Running 0 57m 192.168.66.12 node2 <none> <none>
ingress-nginx-controller-j6n9z 1/1 Running 0 57m 192.168.66.11 node1 <none> <none>

为了避免master节点会被调度新的pod,需要给master节点打上Taints。
如果一个节点标记为 Taints ,除非 pod 也被标识为可以容忍污点节点,否则该 Taints 节点不会被调度 pod。
影响策略是 NoSchedule,只会影响新的 pod 调度

1
2
kubectl taint nodes node1 node-role.kubernetes.io/master=:NoSchedule
kubectl taint nodes node2 node-role.kubernetes.io/master=:NoSchedule
1
2
3
4
5
[root@node1 ~]# kubectl describe node node1 |grep Taints
Taints: node-role.kubernetes.io/master:NoSchedule
[root@node1 ~]# kubectl describe node node2 |grep Taints
Taints: node-role.kubernetes.io/master:NoSchedule
[root@node1 ~]#
1
2
3
4
5
6
7
[root@node1 ~]# kubectl get pods -n ingress-nginx  -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-z7sts 0/1 Completed 0 2d19h 172.30.199.211 node7 <none> <none>
ingress-nginx-admission-patch-4qgtz 0/1 Completed 0 2d19h 172.30.139.14 node6 <none> <none>
ingress-nginx-controller-6s2lp 1/1 Running 0 3h6m 192.168.66.12 node2 <none> <none>
ingress-nginx-controller-kv4xs 1/1 Running 0 3h5m 192.168.66.11 node1 <none> <none>
[root@node1 ~]#

汇总一下配置文件的修改内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
.....
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
.....
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
.....
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
nodeSelector:
api/ingress-controller: "true"
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Equal"
value: ""
effect: "NoSchedule"

修改参数如下:

  • kind: Deployment #修改为DaemonSet
  • hostNetwork: true #添加该字段让docker使用物理机网络,在物理机暴露服务端口(80),注意物理机80端口提前不能被占用
  • dnsPolicy: ClusterFirstWithHostNet #使用hostNetwork后容器会使用物理机网络包括DNS,会无法解析内部service,使用此参数让容器使用K8S的DNS
  • nodeSelector:api/ingress-controller: “true” #添加节点标签
  • tolerations: 添加对指定节点容忍度

验证ingress 是否正常

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
cat > nginx-test.yaml <<EOF
#deploy
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-test
spec:
selector:
matchLabels:
app: nginx-test
replicas: 1
template:
metadata:
labels:
app: nginx-test
spec:
containers:
- name: nginx-test
image: nginx
ports:
- containerPort: 80
---
#service
apiVersion: v1
kind: Service
metadata:
name: nginx-test
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx-test

---
#ingress
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: nginx-test
spec:
rules:
- host: nginx.biglittleant.cn
http:
paths:
- path: /
backend:
serviceName: nginx-test
servicePort: 80
EOF
1
apply -f nginx-test.yaml

检查执行结果

1
2
3
4
5
6
7
8
9
10
[root@node1 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-test-54789bbd4-tplgz 1/1 Running 0 2m27s
[root@node1 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 18h
nginx-test ClusterIP 10.254.51.46 <none> 80/TCP 3m57s
[root@node1 ~]# kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
nginx-test nginx.biglittleant.cn 192.168.66.11,192.168.66.12 80 4m2s

本地访问确认ingress工作正常

1
2
3
4
5
6
7
8
9
10
11
 curl -I nginx.biglittleant.cn -x 192.168.66.11:80
HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Wed, 30 Sep 2020 03:39:53 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 11 Aug 2020 14:50:35 GMT
ETag: "5f32b03b-264"
Accept-Ranges: bytes

在NGINX容器中新建一个文件,确认访问是否正常。

1
2
3
4
5
6
7
[root@node1 ~]# kubectl exec -it nginx-test-54789bbd4-tplgz /bin/bash
root@nginx-test-54789bbd4-tplgz:/# echo 1 > /usr/share/nginx/html/1.html
root@nginx-test-54789bbd4-tplgz:/# cat /usr/share/nginx/html/1.html
1
root@nginx-test-54789bbd4-tplgz:/# exit
exit
[root@node1 ~]#
1
2
 curl nginx.biglittleant.cn/1.html -x 192.168.66.11:80
1

参考文章

官方仓库
官方文档
k8s配置ingress
nginx ingress的安装配置和使用