部署裸金属 Kubernetes 集群
2021-10-23T19:15:48+08:00
这篇文章还未写完。
很多 IDC 都提供开箱即用的 Kubernetes 服务。但在学习 Kubernetes 时往往都会使用单台或多台 VM，它们可以划为裸金属服务器。
裸金属 Kubernetes 集群缺陷
当需要在自己的裸金属服务器上部署 Kubernetes 集群时，最大的问题是没有 LoadBalancer。
临时解决方案
虽然没有 LoadBalancer，但依然可以使用 NodePort 配合 Nginx 进行反向代理。 但 Kubernetes 默认在 30000-32767 内随机分配 NodePort，你无法预先知道服务最终会被分配到哪个端口，也需要不断改动 Nginx 反向代理的配置。
很显然，这些并不是我们想要的。 后面我会介绍如何使用 MetalLB 自建 LoadBalancer。在那之前，需要先部署一个 Kubernetes 集群。
安装 Kubernetes 组件
我准备了 1 台 2C4G 的 Ubuntu Server 20.04 LTS 服务器，用于部署裸金属单节点 Kubernetes 集群。
请注意满足官方强调的每台机器 2 GB 或更多的 RAM以及2 CPU 核或更多配置要求，并禁用 SWAP。
允许 iptables 检查桥接流量
首先需要加载 br_netfilter 模块。
sudo modprobe br_netfilter
执行以下命令，用于以后自动加载 br_netfilter 模块，并在 sysctl 里设置 net.bridge.bridge-nf-call-iptables 及其 IPv6 对应的值为 1。
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system
安装 Container Runtime
Kubernetes 支持的 Container Runtime 有 Docker、containerd、CRI-O。 顺带一提，在 Docker 18.09 开始，Docker 就附带 containerd，Kubernetes 也会优先选择 Docker。
如果你不知道它们的区别，直接安装 Docker。
sudo apt-get remove docker docker-engine docker.io containerd runc
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
安装 kubelet、kubeadm、kubectl
kubeadm 用于初始化集群的指令；kubelet 用于在集群中的每个节点上启动 Pod 和容器等；kubectl 用于与集群通信的命令行工具。
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
配置 cgroup 驱动程序
注意，这是为 Docker 配置 cgroup 驱动程序的方法，其他 Runtime 请参考官方文档。
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
重新启动 Docker。
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker
配置 CNI
这里我们选择 Calico，如果你需要使用其他 CNI，跳过此部分即可。 只需要注意 MetalLB 完全或部分兼容 Calico、Canal、Cilium、Flannel、Kube-ovn、Kube-router、Weave Net。
sudo kubeadm init --pod-network-cidr=192.168.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml
等待 calico-system 命名空间下所有的 Pod 状态变更为 Running。
watch kubectl get pods -n calico-system
kubectl taint nodes --all node-role.kubernetes.io/master-
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-172-26-3-160 Ready control-plane,master 8m59s v1.22.2 172.26.3.160 <none> Ubuntu 20.04 LTS 5.4.0-1018-aws docker://20.10.9
安装 Nginx Ingress
下载官方的配置文件，对其进行更改。
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/baremetal/deploy.yaml
因为需要让 Nginx Ingress 在 Host 层直接处理请求，所以需要在 Deployment 的 spec > template > spec 内添加 hostNetwork: true。
然后，因为在负载均衡中，最简单的情况下集群内的每一个节点都需要运行 Nginx Ingress，所以把 Deployment 改为 DaemonSet。
最后将名为 ingress-nginx-controller 的 Service 的 type 从 NodePort 改为 LoadBalancer，让 Nginx Ingress 能被分配到外部 IP。
下面是已经改好的配置。
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
data:
allow-snippet-annotations: 'true'
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ''
resources:
- nodes
verbs:
- get
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- namespaces
verbs:
- get
- apiGroups:
- ''
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- configmaps
resourceNames:
- ingress-controller-leader
verbs:
- get
- update
- apiGroups:
- ''
resources:
- configmaps
verbs:
- create
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
type: ClusterIP
ports:
- name: https-webhook
port: 443
targetPort: webhook
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
type: LoadBalancer
ipFamilyPolicy: SingleStack
ipFamilies:
- IPv4
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
spec:
hostNetwork: true
dnsPolicy: ClusterFirst
containers:
- name: controller
image: k8s.gcr.io/ingress-nginx/controller:[email protected]:545cff00370f28363dad31e3b59a94ba377854d3a11f18988f5f9e56841ef9ef
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
runAsUser: 101
allowPrivilegeEscalation: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
volumeMounts:
- name: webhook-cert
mountPath: /usr/local/certificates/
readOnly: true
resources:
requests:
cpu: 100m
memory: 90Mi
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/controller-ingressclass.yaml
# We don't support namespaced ingressClass yet
# So a ClusterRole and a ClusterRoleBinding is required
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: nginx
namespace: ingress-nginx
spec:
controller: k8s.io/ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
name: ingress-nginx-admission
webhooks:
- name: validate.nginx.ingress.kubernetes.io
matchPolicy: Equivalent
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
failurePolicy: Fail
sideEffects: None
admissionReviewVersions:
- v1
clientConfig:
service:
namespace: ingress-nginx
name: ingress-nginx-controller-admission
path: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- ''
resources:
- secrets
verbs:
- get
- create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-create
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-create
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: create
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:[email protected]:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
imagePullPolicy: IfNotPresent
args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-patch
namespace: ingress-nginx
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-patch
labels:
helm.sh/chart: ingress-nginx-4.0.6
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.4
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: patch
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:[email protected]:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
imagePullPolicy: IfNotPresent
args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
部署 ingress-nginx。
kubectl apply -f deploy.yaml
等待 ingress-nginx 启动完成。
watch kubectl get pod -n ingress-nginx
可以看到 ingress-nginx-controller 已经成功运行。
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create--1-bmhr4 0/1 Completed 0 22s
ingress-nginx-admission-patch--1-hkdfw 0/1 Completed 0 22s
ingress-nginx-controller-6f5fbb99bf-wtdc5 1/1 Running 0 22s
kubectl get svc -n ingress-nginx
可以看到，EXTERNAL-IP 一直停留在 pending 状态，这是因为没有 LoadBalancer 为其分配 IP。
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.102.45.177 <pending> 80:31968/TCP,443:32266/TCP 55s
ingress-nginx-controller-admission ClusterIP 10.96.180.202 <none> 443/TCP 55s
MetalLB 自建 LoadBalancer
创建命名空间和相关组件。
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.3/manifests/metallb.yaml
编写配置文件，因为只有单节点，所以直接使用公网 IP/32。
如果是多节点，你可以使用内网 IP 段作为参数，例如 172.26.3.240-172.26.3.250，可供 LoadBalancer 分配 10 个 IP。
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 13.212.2.7/32
等待配置生效，查看分配的 EXTERNAL-IP。
kubectl get svc -n ingress-nginx
可以看到已经把公网 IP 以 LoadBalancer 的形式分配给了 ingress-nginx-controller。
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.102.45.177 13.212.2.7 80:31968/TCP,443:32266/TCP 26m
ingress-nginx-controller-admission ClusterIP 10.96.180.202 <none> 443/TCP 26m
部署示例服务和 Ingress
以下配置会启动 3 个 ealen/echo-server 镜像，该示例服务器会返回请求的详细信息，而 Ingress 则用于反向代理这些示例服务。
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: echo
name: echo
spec:
replicas: 3
selector:
matchLabels:
app: echo
template:
metadata:
labels:
app: echo
spec:
containers:
- image: ealen/echo-server
name: echo
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: echo
spec:
selector:
app: echo
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-echo
spec:
rules:
- host: echo.kallydev.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: echo
port:
number: 80
ingressClassName: nginx
因为配置文件中只写了一行 rule，且 host 为 echo.kallydev.com。将这个域名解析到服务器的公共 IP，等待 DNS 生效。
访问 http://echo.kallydev.com/，返回类似的结果则表示部署完成。
{
"host": {
"hostname": "echo.kallydev.com",
"ip": "::ffff:172.26.3.160",
"ips": []
},
"http": {
"method": "GET",
"baseUrl": "",
"originalUrl": "/",
"protocol": "http"
},
"request": {
"params": {
"0": "/"
},
"query": {},
"cookies": {},
"body": {},
"headers": {
"host": "echo.kallydev.com",
"x-request-id": "4810a69384784d8c3c91a13f5a53c025",
"x-real-ip": "x.x.x.x",
"x-forwarded-for": "x.x.x.x",
"x-forwarded-host": "echo.kallydev.com",
"x-forwarded-port": "80",
"x-forwarded-proto": "http",
"x-forwarded-scheme": "http",
"x-scheme": "http",
"upgrade-insecure-requests": "1",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15",
"accept-language": "en-US,en;q=0.9",
"accept-encoding": "gzip, deflate"
}
},
"environment": {
"PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"HOSTNAME": "echo-6455ff4cbd-h2jrt",
"ECHO_PORT": "tcp://10.99.24.236:80",
"ECHO_PORT_80_TCP": "tcp://10.99.24.236:80",
"ECHO_PORT_80_TCP_PROTO": "tcp",
"ECHO_PORT_80_TCP_ADDR": "10.99.24.236",
"KUBERNETES_SERVICE_PORT_HTTPS": "443",
"KUBERNETES_PORT_443_TCP": "tcp://10.96.0.1:443",
"KUBERNETES_PORT_443_TCP_PORT": "443",
"ECHO_PORT_80_TCP_PORT": "80",
"KUBERNETES_SERVICE_HOST": "10.96.0.1",
"KUBERNETES_SERVICE_PORT": "443",
"KUBERNETES_PORT": "tcp://10.96.0.1:443",
"KUBERNETES_PORT_443_TCP_PROTO": "tcp",
"KUBERNETES_PORT_443_TCP_ADDR": "10.96.0.1",
"ECHO_SERVICE_HOST": "10.99.24.236",
"ECHO_SERVICE_PORT": "80",
"NODE_VERSION": "14.17.1",
"YARN_VERSION": "1.22.5",
"HOME": "/root"
}
}
附一份完整的集群信息。
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-apiserver pod/calico-apiserver-6dd4bc68c6-8cp6z 1/1 Running 0 57m
calico-system pod/calico-kube-controllers-767ddd5576-rfzzp 1/1 Running 0 58m
calico-system pod/calico-node-6pdwd 1/1 Running 0 58m
calico-system pod/calico-typha-6d787488f6-xcmmk 1/1 Running 0 58m
default pod/echo-6455ff4cbd-65wxm 1/1 Running 0 8m32s
default pod/echo-6455ff4cbd-h2jrt 1/1 Running 0 8m32s
default pod/echo-6455ff4cbd-xtgl2 1/1 Running 0 8m32s
ingress-nginx pod/ingress-nginx-admission-create--1-bmhr4 0/1 Completed 0 25m
ingress-nginx pod/ingress-nginx-admission-patch--1-hkdfw 0/1 Completed 0 25m
ingress-nginx pod/ingress-nginx-controller-6f5fbb99bf-wtdc5 1/1 Running 0 25m
kube-system pod/coredns-78fcd69978-9vsrb 1/1 Running 0 65m
kube-system pod/coredns-78fcd69978-lprkh 1/1 Running 0 65m
kube-system pod/etcd-ip-172-26-3-160 1/1 Running 0 65m
kube-system pod/kube-apiserver-ip-172-26-3-160 1/1 Running 0 66m
kube-system pod/kube-controller-manager-ip-172-26-3-160 1/1 Running 0 65m
kube-system pod/kube-proxy-nzdgk 1/1 Running 0 65m
kube-system pod/kube-scheduler-ip-172-26-3-160 1/1 Running 0 65m
metallb-system pod/controller-77c44876d-2nwpv 1/1 Running 0 23m
metallb-system pod/speaker-4ggz6 1/1 Running 0 23m
tigera-operator pod/tigera-operator-59f4845b57-rtnmq 1/1 Running 0 59m
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
calico-apiserver service/calico-api ClusterIP 10.108.142.168 <none> 443/TCP 57m
calico-system service/calico-kube-controllers-metrics ClusterIP 10.110.60.87 <none> 9094/TCP 57m
calico-system service/calico-typha ClusterIP 10.100.112.55 <none> 5473/TCP 58m
default service/echo ClusterIP 10.99.24.236 <none> 80/TCP 8m32s
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 66m
ingress-nginx service/ingress-nginx-controller LoadBalancer 10.102.45.177 13.212.2.7 80:31968/TCP,443:32266/TCP 25m
ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.96.180.202 <none> 443/TCP 25m
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 65m
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
calico-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 58m
kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 65m
metallb-system daemonset.apps/speaker 1 1 1 1 1 kubernetes.io/os=linux 23m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
calico-apiserver deployment.apps/calico-apiserver 1/1 1 1 57m
calico-system deployment.apps/calico-kube-controllers 1/1 1 1 58m
calico-system deployment.apps/calico-typha 1/1 1 1 58m
default deployment.apps/echo 3/3 3 3 8m32s
ingress-nginx deployment.apps/ingress-nginx-controller 1/1 1 1 25m
kube-system deployment.apps/coredns 2/2 2 2 65m
metallb-system deployment.apps/controller 1/1 1 1 23m
tigera-operator deployment.apps/tigera-operator 1/1 1 1 59m
NAMESPACE NAME DESIRED CURRENT READY AGE
calico-apiserver replicaset.apps/calico-apiserver-6dd4bc68c6 1 1 1 57m
calico-apiserver replicaset.apps/calico-apiserver-7df657687c 0 0 0 57m
calico-system replicaset.apps/calico-kube-controllers-767ddd5576 1 1 1 58m
calico-system replicaset.apps/calico-typha-6d787488f6 1 1 1 58m
default replicaset.apps/echo-6455ff4cbd 3 3 3 8m32s
ingress-nginx replicaset.apps/ingress-nginx-controller-6f5fbb99bf 1 1 1 25m
kube-system replicaset.apps/coredns-78fcd69978 2 2 2 65m
metallb-system replicaset.apps/controller-77c44876d 1 1 1 23m
tigera-operator replicaset.apps/tigera-operator-59f4845b57 1 1 1 59m
NAMESPACE NAME COMPLETIONS DURATION AGE
ingress-nginx job.batch/ingress-nginx-admission-create 1/1 2s 25m
ingress-nginx job.batch/ingress-nginx-admission-patch 1/1 2s 25m