Contents

使用Kubespray离线部署kubernetes

这篇文章是记录一下目前homelab中kubernetes集群的部署方式,以及遇到的问题。

2025.06.02更新测试

在设计homelab的时候考虑到比较多的一个场景就是离线部署,是希望能够做到离线部署+离线升级。 将实际的生产网络进行隔离,因此在很多组建部署的时候都会考虑到优先考虑如何进行离线部署。

kubernetes有很多的发行版,当前还是用的kubespray来进行离线部署,在后续的文章中也会介绍其他发行版的离线部署。

这里一共有11个节点,分别是:

nameroledesc
mirrorsN/A离线包制作
nginxGateway离线包发布
harbor镜像仓库离线镜像仓库
infra-master-1master
infra-master-2master
infra-master-3master
infra-worker-1worker
infra-worker-2worker
infra-worker-3worker
infra-worker-4worker
infra-worker-5worker

所有节点均使用debian12。

节点需要配置一下免密:

ssh-keygen

复制到其他节点上:

ssh-copy-id [email protected]

设置内网仓库:

sed -i s/http/https/g /etc/apt/sources.list
sed -i s/10.31.0.2/mirrors.infra.plz.ac/g /etc/apt/sources.list

💡 :以下操作是在可以上外网节点上执行

安装依赖:

sudo apt-get update
sudo apt-get install git wget ansible python3-venv -y
curl -fsSL https://get.docker.com -o get-docker.sh

下载源码:

git clone https://github.com/kubernetes-sigs/kubespray

切换分支,这个也可以根据你实际需求来去使用不同的分支:

cd kubespray
git switch release-2.28

下载二进制文件:

cd contrib/offline
LC_ALL=C.UTF-8 bash generate_list.sh
wget -x -P temp/files -i temp/files.list

保持这个目录结构 放到了nas上面 10.31.0.2,这里是用的一个nginx服务器来做的内容发布,当然你也可以选择自己喜欢的web服务器来去发布。

scp -r  temp/files/* [email protected]:/data/mirrors/k8s

除了二进制之外还需要将镜像也要下载下来,可以是harbor也可以是registry:

我这里准备了一个脚本可以用这个脚本批量将这些镜像搬运到内网的harbor:

#!/bin/bash

# 定义代理映射
declare -A proxies=(
    ["docker.io"]="docker-proxy.plz.ac"
    ["ghcr.io"]="ghcr.plz.ac"
    ["registry.k8s.io"]="k8s-io.plz.ac"
    ["quay.io"]="quay.plz.ac"
)

# 读取镜像列表文件
input_file="$1"
harbor_registry="harbor.infra.plz.ac/k8s"

# 读取镜像列表文件
while IFS= read -r image; do
    # 提取镜像仓库和名称
    registry=$(echo $image | cut -d '/' -f 1)
    name=$(echo $image | cut -d '/' -f 2-)

    # 选择对应的代理
    proxy=${proxies[$registry]}
    if [ -z "$proxy" ]; then
        echo "No proxy found for $registry"
        continue
    fi

    # 构建代理镜像URL
    proxy_image="$proxy/$name"

    # 拉取镜像
    echo "Pulling image: $proxy_image"
    docker pull $proxy_image

    # 重新tag镜像
    echo "Tagging image: $proxy_image as $image"
    docker tag $proxy_image $image

    # 构建目标镜像URL(去掉域名前缀,如果没有仓库,使用library)
    if [[ $name == *"/"* ]]; then
        target_image="$harbor_registry/$name"
    else
        target_image="$harbor_registry/library/$name"
    fi

    # 使用skopeo拷贝镜像到Harbor
    echo "Copying image: $image to $target_image"
    skopeo --insecure-policy copy docker-daemon:$image docker://$target_image

    # 删除中间镜像以节省空间
    docker rmi $proxy_image
    docker rmi $image

done < "$input_file"

这里还用到了一个工具skopeo用于搬镜像

wget -O skopeo https://github.com/lework/skopeo-binary/releases/download/v1.16.1/skopeo-linux-amd64
chmod +x skopeo
mv skopeo /usr/local/bin/skopeo

添加认证:

skopeo login harbor.infra.plz.ac

要注意的是harbor是要先去手动创建仓库的不然会失败!

以下是要创建的仓库:

  • mirantis
  • coreos
  • cilium
  • k8snetworkplumbingwg
  • flannel
  • calico
  • rajchaudhuri
  • kubeovn
  • cloudnativelabs
  • kube-vip
  • coredns
  • dns
  • cpa
  • metrics-server
  • sig-storage
  • rancher
  • ingress-nginx
  • amazon
  • jetstack
  • kubernetesui
  • metallb

执行:

~/tools/images_pull.sh temp/images.list

后续部署的时候发现还有几个镜像不太对,这里先手动push一下了

docker pull registry.k8s.io/kube-apiserver:v1.32.5
docker tag registry.k8s.io/kube-apiserver:v1.32.5 harbor.infra.plz.ac/k8s/kube-apiserver:v1.32.5
docker push harbor.infra.plz.ac/k8s/kube-apiserver:v1.32.5
# 下面的几个镜像也要处理一下
kube-controller-manager:v1.32.5
kube-scheduler:v1.32.5
kube-proxy:v1.32.5

这里已经有了一个sample,我们可以直接复制对应的目录来做自己的环境:

cp -rv  inventory/local inventory/infra

修改对应的配置文件如下 inventory/infra/group_vars/all/offline.yml,这里需要改一下对应的registry_hostfiles_repo还有对应工具地址这些是只需要取消注释即可:

---
registry_host: "harbor.infra.plz.ac/k8s"
files_repo: "https://mirrors.infra.plz.ac/k8s"
kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
github_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"
kubeadm_download_url: "{{ files_repo }}/dl.k8s.io/release/v{{ kube_version }}/bin/linux/{{ image_arch }}/kubeadm"
kubectl_download_url: "{{ files_repo }}/dl.k8s.io/release/v{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl"
kubelet_download_url: "{{ files_repo }}/dl.k8s.io/release/v{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet"
cni_download_url: "{{ files_repo }}/github.com/containernetworking/plugins/releases/download/v{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-v{{ cni_version }}.tgz"
crictl_download_url: "{{ files_repo }}/github.com/kubernetes-sigs/cri-tools/releases/download/v{{ crictl_version }}/crictl-v{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
etcd_download_url: "{{ files_repo }}/github.com/etcd-io/etcd/releases/download/v{{ etcd_version }}/etcd-v{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
calicoctl_download_url: "{{ files_repo }}/github.com/projectcalico/calico/releases/download/v{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
calico_crds_download_url: "{{ files_repo }}/github.com/projectcalico/calico/archive/v{{ calico_version }}.tar.gz"
ciliumcli_download_url: "{{ files_repo }}/github.com/cilium/cilium-cli/releases/download/v{{ cilium_cli_version }}/cilium-linux-{{ image_arch }}.tar.gz"
helm_download_url: "{{ files_repo }}/get.helm.sh/helm-v{{ helm_version }}-linux-{{ image_arch }}.tar.gz"
crun_download_url: "{{ files_repo }}/github.com/containers/crun/releases/download/{{ crun_version }}/crun-{{ crun_version }}-linux-{{ image_arch }}"
kata_containers_download_url: "{{ files_repo }}/github.com/kata-containers/kata-containers/releases/download/{{ kata_containers_version }}/kata-static-{{ kata_containers_version }}-{{ image_arch }}.tar.xz"
cri_dockerd_download_url: "{{ files_repo }}/github.com/Mirantis/cri-dockerd/releases/download/v{{ cri_dockerd_version }}/cri-dockerd-{{ cri_dockerd_version }}.{{ image_arch }}.tgz"
runc_download_url: "{{ files_repo }}/github.com/opencontainers/runc/releases/download/v{{ runc_version }}/runc.{{ image_arch }}"
crio_download_base: "download.opensuse.org/repositories/devel:kubic:libcontainers:stable"
crio_download_crio: "http://{{ crio_download_base }}:/cri-o:/"
crio_download_url: "{{ files_repo }}/storage.googleapis.com/cri-o/artifacts/cri-o.{{ image_arch }}.v{{ crio_version }}.tar.gz"
skopeo_download_url: "{{ files_repo }}/github.com/lework/skopeo-binary/releases/download/v{{ skopeo_version }}/skopeo-linux-{{ image_arch }}"
containerd_download_url: "{{ files_repo }}/github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"
nerdctl_download_url: "{{ files_repo }}/github.com/containerd/nerdctl/releases/download/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
gvisor_runsc_download_url: "{{ files_repo }}/storage.googleapis.com/gvisor/releases/release/{{ gvisor_version }}/{{ ansible_architecture }}/runsc"
gvisor_containerd_shim_runsc_download_url: "{{ files_repo }}/storage.googleapis.com/gvisor/releases/release/{{ gvisor_version }}/{{ ansible_architecture }}/containerd-shim-runsc-v1"

创建虚拟环境:

python3 -m venv ~/kubespray-basic-pyenv

激活:

. ~/kubespray-basic-pyenv/bin/activate

设置国内的加速站点(暂时没内网的同步仓库):

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

安装模块:

pip install -r requirements.txt
pip install ruamel.yaml

设置配置清单

vi inventory/infra/hosts.ini

如下所示:

[all:vars]
# 定义全局变量,适用于所有主机
ansible_user=user
# 假设 local_release_dir 是部署在远程用户家目录下的 releases 目录
# {{ ansible_user }} 会被替换为上面定义的 'user'
local_release_dir=/home/{{ ansible_user }}/releases
# 如果使用 SSH 密钥进行连接,并需要指定私钥文件,可以取消注释并修改以下行:
# ansible_ssh_private_key_file=/home/user/.ssh/id_rsa_k8s  # 替换为实际私钥路径

[kube_control_plane]
# Kubernetes 控制平面节点 (Masters)
# 格式: <主机别名> ansible_host=<IP地址> [其他主机变量]
infra-master-1 ansible_host=10.31.0.110
infra-master-2 ansible_host=10.31.0.111
infra-master-3 ansible_host=10.31.0.112

[etcd]
# etcd 集群节点 (通常与控制平面节点相同)
infra-master-1 ansible_host=10.31.0.110
infra-master-2 ansible_host=10.31.0.111
infra-master-3 ansible_host=10.31.0.112

[kube_node]
# 工作节点
infra-worker-1 ansible_host=10.31.0.113
infra-worker-2 ansible_host=10.31.0.114
infra-worker-3 ansible_host=10.31.0.115
infra-worker-4 ansible_host=10.31.0.116
infra-worker-5 ansible_host=10.31.0.117

连通性测试:

ansible -i inventory/infra/hosts.ini -m ping all

修改额外的插件

这里可以启用类似lb,argo、helm等一系列的插件非常方便

vi inventory/infra/group_vars/k8s_cluster/addons.yml

内容如下:

---
dashboard_enabled: true
helm_enabled: false
registry_enabled: false
metrics_server_enabled: true
local_path_provisioner_enabled: false
local_volume_provisioner_enabled: false
ingress_nginx_enabled: true
ingress_publish_status_address: ""
ingress_alb_enabled: false
cert_manager_enabled: true
cert_manager_namespace: "cert-manager"
metallb_enabled: false
metallb_speaker_enabled: "{{ metallb_enabled }}"
metallb_namespace: "metallb-system"
argocd_enabled: false
kube_vip_enabled: false
node_feature_discovery_enabled: false

修改网络插件

这里支持的插件是有很多可以根据需要去灵活的修改inventory/infra/group_vars/k8s_cluster/k8s-cluster.yml

修改参数:

kube_proxy_mode: ipvs
kube_proxy_strict_arp: true
# 这里是我常用到的两个后续要单独部署lb需要开启arp
cluster_name: infra.homelab # 单独定义了一下集群的名称
enable_nodelocaldns: false # 关掉了nodelocaldns
auto_renew_certificates: true # 集群证书自动renew
auto_renew_certificates_systemd_calendar: "Mon *-*-1,2,3,4,5,6,7 03:{{ groups['kube_control_plane'].index(inventory_hostname) }}0:00"

部署:

ansible-playbook -i inventory/infra/hosts.ini --user=user  --become --become-user=root cluster.yml

如何添加新的计算节点

清空集群:

ansible-playbook -i inventory/infra/hosts.ini --user=user --become --become-user=root reset.yml

目前这套方案已经在homelab里面运行了1年多了还没有出现什么问题,在后期的homelab专题里面我再去更新如何配合外部lb来进行服务访问、以及如何对集群进行升级。

发现下载失败

打开download 的debug

-e “unsafe_show_logs=true”

ansible-playbook -i inventory/infra/hosts.ini  -e "unsafe_show_logs=true" --user=user  --become --become-user=root cluster.yml