Custom Installation and Infrastructure Tools

CUSTOM INSTALLATION AND INFRASTRUCTURE TOOLS

Nutanix Kubernetes Platform (NKP) cluster can be customized for different environments and infrastructures depending on your network technology choices. If you have already installed using Basic Installations by Infrastructure on page 72 instructions, you might find helpful tools in this section. However, if you have not already installed NKP with the Basic installation instructions, find your infrastructure and begin the process of custom NKP installation in this area.

See the following sections for details on custom installation options and infrastructure-specific details.

Universal Configurations for all Infrastructure Providers

Several areas of Nutanix Kubernetes Platform (NKP) configuration are shared amongst all infrastructure providers. Some of the universal configurations related to environment variables, flags for cluster creation, local registries and more are described in this section.

For more information regarding global configurations, customization of specific components, or additional Konvoy customizations, see Additional Konvoy Configurations on page 1025.

Container Engine

• Podman Version 4.0 or later for Linux. For more information, see Podman Installation and host requirements in kind .

Note: Installation of Podman is different for different OS environments. For more information, see Podman
Installation.

Tip:

You will need to restart your machine after following the Host Requirements to add the /etc/systemd/ system/[email protected]/delegate.conf and the /etc/modules-load.d/iptables.conf files. On Linux, this will mean restarting the host. On MacOS, it means restarting the Docker or Podman VM.

Configuring an HTTP or HTTPS Proxy

When creating a Nutanix Kubernetes Platform (NKP) cluster in environments that use an HTTP or HTTPS proxy, you must provide proxy details. The proxy values are strings that list a set of proxy servers, URLs, or wildcard addresses specific to your environment.

When creating a NKP cluster in a proxied environment, you need to specify proxy settings for the following:

• Bootstrap cluster
• CAPI components
• NKP Kommander component

When you create a NKP cluster through the nkp create cluster command, the bootstrap cluster and Cluster API (CAPI) components are created for you automatically, and use the HTTP and HTTPS proxy settings you specify in the nkp create cluster <provider>… command through the flags –http-proxy, –https-proxy, and –no-proxy.

You can also create the bootstrap cluster and CAPI components manually, using the appropriate commands: nkp create bootstrap and nkp create capi-components , respectively, combined with the command line flags as shown above to include your HTTP, HTTPS and no-proxy information.

You can also specify HTTP or HTTPS proxy information in an override file when using Nutanix Image Builder (NIB). .

Without these values provided as part of the relevant nkp create command, NKP cannot create the requisite parts of your new cluster correctly. This is true of both management and managed clusters alike.

Note: For NKP installation, create the bootstrap cluster from within the same network where the new cluster will run.
Using a bootstrap cluster on a laptop with different proxy settings, for example, or residing in a different network, can
cause problems.

To modify HTTP or HTTPS proxy settings after initial deployment, see Adding or Modifying HTTP Proxy Settings After Deployment on page 701.

Bootstrap Cluster HTTP Proxy Settings

The Application Programming Interface (API) server doesn’t exist yet in the bootstrap environment before you install Nutanix Kubernetes Platform (NKP) because the API server is created during cluster creation. To create a bootstrap server in a proxied environment, you need to include the following flags:

• –http-proxy <<http proxy list>>
• –https-proxy <<https proxy list>>
• –no-proxy <<no proxy list>>

The following is an example of the nkp create bootstrap command’s syntax, with the HTTP proxy settings included.

nkp create bootstrap --http-proxy <<http proxy list>> --https-proxy <<https proxy
list>> --no-proxy <<no proxy list>>

Creating a Bootstrap Cluster with HTTP Proxy Settings

Before you begin

If an HTTP proxy is required, locate the values to use for the http_proxy, https_proxy, and no_proxy flags. They will be built into the bootstrap cluster during cluster creation.

About this task

The flags can include a mix of IP addresses and domain names. Note that the delimiter between each proxy value within a flag is a comma (,) with no space character following it.

Procedure

Create a bootstrap cluster and any other flags you need using the command.

nkp create bootstrap --kubeconfig $HOME/.kube/config \
--http-proxy <string> \
--https-proxy <string> \
--no-proxy <string>

Example output shows values for the proxy settings:

nkp create bootstrap \
--http-proxy 10.0.0.15:3128 \
--https-proxy 10.0.0.15:3128 \
--no-proxy
127.0.0.1,192.168.0.0/16,10.0.0.0/16,10.96.0.0/12,169.254.169.254,169.254.0.0/24,localhost,kuberne
operator-logging-fluentd.kommander.svc.cluster.local,elb.amazonaws.com

Creating CAPI Components with HTTP or HTTPS Proxy Settings

About this task

If you created a cluster without using the –self-managed flag, the cluster will not have any Cluster API (CAPI) controllers or the cert-manager component. This means that the cluster will be managed from the context of the cluster from which it was created, such as the bootstrap cluster. However, you can transform the cluster to a self-managed cluster by performing the commands nkp create capi-components – kubeconfig=<newcluster> and nkp move –to- kubeconfig=<newcluster>. This combination of actions is sometimes called a pivot.

When creating the CAPI components for a proxied environment using the Nutanix Kubernetes Platform (NKP) command line interface, you must include the following flags :

• –http-proxy <<http proxy list>>
• –https-proxy <<https proxy list>>
• –no-proxy <<no proxy list>>

The following is an example nkp create capi-components command’s syntax with the HTTP proxy settings included:

Note:

nkp create capi-components --http-proxy <<http proxy list>> --https-proxy
<<https proxy list>> --no-proxy <<no proxy list>>

Create CAPI Components with HTTP Proxy Settings

Note that the delimiter between each proxy value within a flag is a comma ( , ) with no space character following it. The flags can include a mix of IP addresses and domain names.

Procedure

1. If an HTTP proxy is required, locate the values to use for the http_proxy, https_proxy, and no_proxy flags. They will be built into the CAPI components during their creation.
2. Create CAPI components using this command syntax and any other flags you might need.

nkp create capi-components --kubeconfig $HOME/.kube/config \
--http-proxy <string> \
--https-proxy <string> \
--no-proxy <string>

This code sample shows the command with example values for the proxy settings:

nkp create capi-components \
--http-proxy 10.0.0.15:3128 \
--https-proxy 10.0.0.15:3128 \
--no-proxy
127.0.0.1,192.168.0.0/16,10.0.0.0/16,10.96.0.0/12,169.254.169.254,169.254.0.0/24,localhost,kube
operator-logging-fluentd.kommander.svc.cluster.local,elb.amazonaws.com

Cluster Creation with HTTP or HTTPS Proxy

During cluster creation, you might need to configure the control plane and worker nodes to use an HTTP proxy. This can occur during installation of the Konvoy component of Nutanix Kubernetes Platform (NKP) or when creating a managed cluster.

If you require HTTP proxy configurations, you can apply them during the NKP create cluster operation by adding the appropriate flags to the command example below:

Table 62: Nutanix Infrastructure

Table 63: AWS, Azure, GCP, VSphere, and Preprovisoned Infrastructures

Note: Using an HTTP override file, you must apply the same configuration to any custom machine images built with
the Nutanix Image Builder (NIB).

Configure the Control Plane and Worker Nodes to Use HTTP/S Proxy

This method uses environment variables to configure the HTTP proxy values. (You are not required to use this method.)

| Proxy configuration | Flag |

string` |

• HTTPS proxy for control plane machines; --control-plane-https-proxy string
• No Proxy list for control plane machines; --control-plane-no-proxy strings

| HTTP proxy for worker machines | --worker-http-proxy string | | | | | HTTPS proxy for worker machines | --worker-https-proxy string | | | | | No Proxy list for worker machines | --worker-no-proxy strings | | | |

Review this sample code to configure environment variables for the control plane and worker nodes, considering the list of considerations that follow the sample.

export HTTP_PROXY=http://example.org:8080
export HTTPS_PROXY=http://example.org:8080
export
NO_PROXY="example.org,example.com,example.net,localhost,127.0.0.1,10.96.0.0/12,192.168.0.0/16,kuber
export WORKER_HTTP_PROXY=http://example.org:8080
export WORKER_HTTPS_PROXY=http://example.org:8080
export
WORKER_NO_PROXY="example.org,example.com,example.net,localhost,127.0.0.1,10.96.0.0/12,192.168.0.0/1

HTTP proxy configuration considerations to ensure the core components work correctly

• Replace example.org,example.com,example.net with your internal addresses
• localhost and 127.0.0.1 addresses that can be accessed directly, not through the proxy.
• 10.96.0.0/12 is the default Kubernetes service subnet
• 192.168.0.0/16 is the default Kubernetes pod subnet
• kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc. cluster,kubernetes.d are the internal Kubernetes kube-apiserver services
• The entries .svc,.svc.cluster,.svc.cluster.local are the internal Kubernetes services
• Auto-IP addresses 169.254.169.254 for any cloud provider
• 169.254.169.254 is the AWS metadata server
• .elb.amazonaws.com is for the worker nodes to allow them to communicate directly to the kube-apiserver ELB

Example of Creating a Cluster Using the Configured HTTP Proxy Variables

Nutanix Infrastructure: The following is an example of a nkp create cluster… command that uses the values set in the environment variables from the code sample.

nkp create cluster nutanix \
--cluster-name ${CLUSTER_NAME} \
--http-proxy="${HTTP_PROXY}" \
--https-proxy="${HTTPS_PROXY}" \
--no-proxy="${NO_PROXY}"

AWS, Azure, GCP, VSphere, and Preprovisoned Infrastructures: The following is an example of a nkp create cluster… command that uses the values set in the environment variables from the code sample above. Use the appropriate infrastructure provider name in line 1 from the choices listed:

nkp create cluster [aws, azure, gcp, preprovisoned, vsphere] \
--cluster-name ${CLUSTER_NAME} \
--control-plane-http-proxy="${HTTP_PROXY}" \
--control-plane-https-proxy="${HTTPS_PROXY}" \
--control-plane-no-proxy="${NO_PROXY}" \
--worker-http-proxy="${WORKER_HTTP_PROXY}" \
--worker-https-proxy="${WORKER_HTTPS_PROXY}" \
--worker-no-proxy="${WORKER_NO_PROXY}"

HTTP or HTTPS Proxy Configuration for the NKP Kommander Component

After the cluster is running in the Konvoy component; you need to configure the NO_PROXY variable for each provider.

For example, in addition to the values above for Amazon Web Services (AWS), you need the following settings:

• The default VPC Classless Inter-Domain Routing (CIDR) range of 10.0.0.0/16
• kube-apiserver internal or external ELB address

Warning: The NO_PROXY variable contains the Kubernetes Services CIDR. This example uses the default CIDR,
10.96.0.0/12. If your cluster's CIDR differs, update the value in the NO_PROXY field.

Set the httpProxy and httpsProxy environment variables to the address of the HTTP and HTTPS proxy servers, respectively. (Frequently, environments use the same values for both.) Set the noProxy environment variable to the addresses that can be accessed directly and not through the proxy.

For the Kommander component of Nutanix Kubernetes Platform (NKP), refer to more HTTP Proxy information in Configuring HTTP proxy for the Kommander Clusters on page 1019.

Nutanix Image Builder HTTP or HTTPS Proxy

In some networked environments, the machines used for building images can reach the Internet, but only through an HTTP or HTTPS proxy. For Nutanix Kubernetes Platform (NKP) to operate in these networks, you need a way to specify what proxies to use; see Configuring an HTTP or HTTPS Proxy on page \696. You can use an HTTP proxy override file to specify that proxy. When NIB tries installing a particular OS package, it uses that proxy to reach the Internet to download it.

Warning: The proxy setting specified here is NOT "baked into" the image - it is only used while the image is being
built. The settings are removed before the image is finalized.

While it might seem logical to include the proxy information in the image, the reality is that many companies have multiple proxies - one perhaps for each geographical region or maybe even a proxy per datacenter or office datacenter. All network traffic to the Internet goes through the proxy. If you were in Germany, you probably would not want to send all your traffic to a U.S.-based proxy. Doing that slows traffic down and consumes too many network resources. If you bake the proxy settings into the image, you must create a separate image for each region. Creating an image without a proxy makes more sense, but remember that you still need a proxy to access the Internet. Thus, when creating the cluster (and installing the Kommander component of NKP), you must specify the correct proxy settings for the network environment into which you install the cluster. You will use the same base image for that cluster installed in an environment with different proxy settings.

Adding or Modifying HTTP Proxy Settings After Deployment

Before you begin

• Ensure that the proxy server works correctly.
• Identify any addresses or domain suffixes that need to be added to NO_PROXY. See Cluster Creation with HTTP or HTTPS Proxy on page 699.

About this task

NKP clusters support modifying HTTP proxy settings through the Cluster API. To add or modify HTTP proxy settings after deployment, follow these steps:

Procedure

1. Set your new proxy configuration as environment variables.

export HTTP_PROXY=http://example.org:8080
export HTTPS_PROXY=http://example.org:8080
export
NO_PROXY="example.org,example.com,example.net,localhost,127.0.0.1,10.96.0.0/12,192.168.0.0/16,k

1. Apply the new environment variables to the CAPI provider deployments and the git operator.

kubectl set env deployment/caaph-controller-manager -n caaph-system
HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" NO_PROXY="${NO_PROXY}"
kubectl set env deployment/capa-controller-manager -n capa-system
HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" NO_PROXY="${NO_PROXY}"
kubectl set env deployment/capg-controller-manager -n capg-system
HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" NO_PROXY="${NO_PROXY}"
kubectl set env deployment/capi-kubeadm-bootstrap-controller-manager -n capi-
kubeadm-bootstrap-system HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}"
NO_PROXY="${NO_PROXY}"
kubectl set env deployment/capi-kubeadm-control-plane-controller-manager -n capi-
kubeadm-control-plane-system HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}"
NO_PROXY="${NO_PROXY}"
kubectl set env deployment/capi-controller-manager -n capi-system
HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" NO_PROXY="${NO_PROXY}"
kubectl set env deployment/cappp-controller-manager -n cappp-system
HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" NO_PROXY="${NO_PROXY}"
kubectl set env deployment/capv-controller-manager -n capv-system
HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" NO_PROXY="${NO_PROXY}"
kubectl set env deployment/capx-controller-manager -n capx-system
HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" NO_PROXY="${NO_PROXY}"
kubectl set env deployment/capz-controller-manager -n capz-system
HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" NO_PROXY="${NO_PROXY}"
kubectl set env deployment/cluster-api-runtime-extensions-nutanix -n caren-system
HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" NO_PROXY="${NO_PROXY}"
kubectl set env statefulset/git-operator-git -n git-operator-system
HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" NO_PROXY="${NO_PROXY}"
Note: If you are upgrading CAPI components as part of an NKP upgrade, use nkp upgrade capi-
components with the proxy flags instead of applying the environment variables individually:
nkp upgrade capi-components \
--http-proxy "${HTTP_PROXY}" \
--https-proxy "${HTTPS_PROXY}" \
--no-proxy "${NO_PROXY}"

1. Update the gatekeeper-overrides ConfigMap to apply the new proxy settings to Kommander applications.

See Creating Gatekeeper ConfigMap in the Kommander Namespace on page 1020 for detailed steps.

Note: If a gatekeeper-overrides ConfigMap already exists in the kommander namespace, running the
command in that procedure updates it in place.

1. Update proxy configurations for kubelet and containerd by editing the Cluster resource.

Include the following proxy configurations in the spec:

spec:
topology:
variables:
- name: clusterConfig
value:
proxy:
http: http://example.com
https: http://example.com
additionalNo:
- no-proxy-1.example.com

- no-proxy-2.example.com

Note: After applying the new manifests, Cluster API recreates nodes with the new settings.

Load Balancers

In a Kubernetes cluster, depending on the flow of traffic direction, there are two kinds of load balancing:

• Internal load balancing for the traffic within a Kubernetes cluster
• External load balancing for the traffic coming from outside the cluster

Nutanix Kubernetes Platform (NKP) includes both internal and external load balancing solutions for the supported cloud infrastructure providers and pre- provisioned environments. For more information, see Load Balancing on page 637.

MetalLB MetalLB is an external load balancer that NKP deploys by default on supported clusters to allocate virtual IP addresses for Kubernetes services. You can create additional MetalLB configuration as needed.

MetalLB advertises virtual IP addresses using the following protocols:

• Layer 2, with Address Resolution Protocol (ARP)
• Border Gateway Protocol (BGP)

Important:

Do not edit the default MetalLB IPAddressPool and L2Advertisement on a cluster. Modifying these default objects to host additional IP address pools for applications will cause the cluster upgrade process to stall and fail.

When configuring MetalLB for your applications, always create a new IPAddressPool and a new L2Advertisement instead of modifying the default objects.

Select one of the following procedures to create your MetalLB manifest.

Layer 2 Configuration

Layer 2 mode is the simplest to configure. In many cases, you do not need any protocol-specific configuration, only IP addresses.

Layer 2 mode does not require IP addresses to be bound to the network interfaces of your worker nodes. It responds to ARP requests on the local network and gives clients the machine’s MAC address.

Note the following requirements for MetalLB IP address ranges:

• MetalLB IP address ranges or CIDRs must be within the node’s primary network subnet.
• MetalLB IP address ranges, CIDRs, and node subnets must not conflict with the Kubernetes cluster pod and service subnets.

For example, the following configuration gives MetalLB control over IP addresses from 192.168.1.240 to 192.168.1.250 and configures layer 2 mode. Replace the values with values specific to your environment.

cat << EOF > metallb-conf.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: example
namespace: metallb-system
spec:
addresses:
- 192.168.1.240-192.168.1.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: example
namespace: metallb-system
spec:
ipAddressPools:
- example
EOF

After this completes, run the following kubectl command.

kubectl apply -f metallb-conf.yaml

BGP Configuration

For a basic BGP configuration featuring one router and one IP address range, you need the following information:

• The router IP address that MetalLB needs to connect to.
• The router’s autonomous systems (AS) number.
• The MetalLB AS number.
• An IP address range expressed as a Classless Inter-Domain Routing (CIDR) prefix.

For example, if you specify 192.168.10.0/24 as the MetalLB range, 64500 as the MetalLB AS number, 10.0.0.1 as the router IP address, and 64501 as the router AS number, your configuration looks like this:

cat << EOF > metallb-conf.yaml
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
name: example
namespace: metallb-system
spec:
myASN: 64500
peerASN: 64501
peerAddress: 10.0.0.1
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: example
namespace: metallb-system
spec:
addresses:
- 192.168.10.0/24
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: example
namespace: metallb-system
spec:
ipAddressPools:
- example
EOF

After this completes, run the following kubectl command.

kubectl apply -f metallb-conf.yaml

Registry and Registry Mirrors

In an air-gapped environment, you need a local repository to store Helm charts, Docker images, and other artifacts. In an environment with access to the Internet, you can retrieve artifacts from specialized repositories dedicated to them, such as Docker images contained in DockerHub and Helm Charts that come from a dedicated Helm Chart repository.

Nutanix Kubernetes Platform (NKP) supports operation with several local Registry Mirror Tools on page 1028.

• Registry Mirrors are local copies of images from a public registry that follow (or mirror) the file structure of a public registry. If you need to set up a private registry with a registry mirror or details on using the flag(s), see Using a Registry Mirror on page 1030.
• Container registries are collections of container repositories and can also offer API paths and access rules.
• Container repositories are a collection of related container images. The container image has everything the software might need to run, including code, resources, and tools. Container repositories store container images for setup and deployment, and you use the repositories to manage, pull, and push images during cluster operations.

Kubernetes does not natively provide a registry for hosting the container images you will use to run the applications you want to deploy on Kubernetes. Instead, Kubernetes requires you to use an external solution to store and share container images. A variety of Kubernetes-compatible registry options are compatible with NKP.

How the Registry Mirror Works

The first time you request an image from your local registry mirror, it pulls the image from the public registry (such as Docker) and stores it locally before handing it back to you. On subsequent requests, the local registry mirror can serve the image from its storage.

Air-gapped vs. Non-air-gapped Environments

In a non-air-gapped environment, you can access the Internet. You retrieve artifacts from specialized repositories dedicated to them, such as Docker images contained in DockerHub and Helm Charts that come from a dedicated Helm Chart repository. You can also create your local repository to hold the downloaded container images needed or any custom images you have created with the Nutanix Image Builder on page 51 tool.

In an air-gapped environment, you need a local repository to store Helm charts, Docker images, and other artifacts. Private registries provide security and privacy in enterprise container image storage, whether hosted remotely or on- premises locally in an air-gapped environment. NKP in an air-gapped environment requires a local container registry of trusted images to enable production-level Kubernetes cluster management. However, a local registry is also an option in a non-air-gapped environment for speed and security.

If you want to use images from this local registry to deploy applications inside your Kubernetes cluster, you will need to set up a secret for a private registry. The secret contains your login data, which Kubernetes needs to connect to your private repository. It is not required to export any variables for most of the command examples. However, the export, along with an arbitrary variable name, primarily clarifies what values in the commands need to be substituted. Also, that makes it easier to copy and paste the examples. Furthermore, if multiple steps in a procedure need you to specify a variable, you export it once with the following export command and then reuse it in future commands.

For example,

export REGISTRY_URL="<https/http>://<registry-address>:<registry-port>"

To run the create cluster command using that variable created above, use the example command replacing azure with your choice of provider gcp, vsphere, vcd, pre-provisioned, aws:

nkp create cluster azure --registry-mirror-url=${REGISTRY_URL}

A cluster administrator uses NKP CLI commands to upload the image bundle to your registry with the parameters:

nkp push bundle --bundle <bundle> --to-registry=${REGISTRY_URL}

Parameter definitions:

• –bundle <bundle> the group of images. The example below is for the NKP air-gapped environment bundle
• Either use exported variable ${REGISTRY_URL} or –to-registry=<registry- address>/<registry- name> to provide registry location for push

Command example:

nkp push bundle --bundle container-images/konvoy-image-bundle-nkp-version.tar --to-
registry=333000009999.dkr.ecr.us-west-2.amazonaws.com/can-test

Any URL can contain an optional port specification. If no port is specified, then the default port for the protocol is assumed. For example, for HTTPS protocol, port 443 is the default, meaning these two URLs are equivalent:

https://docs.nutanix.com
https://docs.nutanix.com:443

A port specification is only required if the URL target uses a port number other than the default.

Related Information

• Registry Mirror Tools on page 1028
• Using a Registry Mirror on page 1030
• Seeding the Registry for an Air-gapped Cluster on page 1030
• Images Download into Your Registry: Air-gapped Environments on page 982

Cluster Pod and Services Subnets

About this task

A Kubernetes cluster defines subnets for Pods and Services. These subnets must not overlap with subnets used by cluster Nodes, i.e. the machines that form the cluster. In infrastructures where you allocate virtual IPs for LoadBalancer Services, the Pods and Services subnets must not overlap with the virtual IP subnets.

Note:

• If the default Pods or Services subnets overlap with any subnets used by cluster Nodes, you must change the Pods or Services subnets. You must do this at cluster creation.
• The custom Pod network configurations for Grafana Loki are not supported.
• The default subnets used in NKP are:

spec:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12

To change the subnets, perform the following steps:

Procedure

1. Generate the YAML Ain’t Markup Language (YAML) manifests for the cluster using the –dry-run and -o

yaml flags, along with the desired nkp cluster create command. Example:

nkp create cluster preprovisioned --cluster-name ${CLUSTER_NAME} --control-plane-
endpoint-host <control plane endpoint host> --control-plane-endpoint-port <control
plane endpoint port, if different than 6443> --dry-run -o yaml > cluster.yaml

1. To modify the Services subnet, add or edit the spec.clusterNetwork.services.cidrBlocks field of the

Cluster object. Example:

kind: Cluster
spec:
clusterNetwork:
services:
cidrBlocks:
  - 10.0.0.0/12

1. To modify the cluster: Add or edit the spec.clusterNetwork.pods.cidrBlocks field. Example:

kind: Cluster
spec:
clusterNetwork:
pods:
cidrBlocks:
  - 172.16.0.0/16

1. If using Calico CNI, update its configuration to edit the data.“custom-

resources.yaml“.spec.calicoNetwork.ipPools.cidr field with your desired Pod subnet. Example:

apiVersion: v1
data:
custom-resources.yaml: |
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
# Configures Calico networking.
calicoNetwork:
# Note: The ipPools section cannot be modified post-install.
ipPools:
- blockSize: 26
cidr: 172.16.0.0/16
kind: ConfigMap
metadata:
name: calico-cni-<cluster-name>

When you provision the cluster, the configured Pod and Services subnets are applied.

Creating a Bastion Host

About this task

Ensure the items below are installed and the environment matches the requirements below:

• Create a bastion VM host template for the cluster nodes to use within the air-gapped network. This bastion VM host also needs access to a local registry instead of an Internet connection to pull images.
• Find and record the bastion VM’s IP or hostname.
• Download the following required NKP Konvoy binaries and installation bundles are discussed in step 5 below. To access the download bundles, see Downloading NKP on page 16.
• A local registry or Docker version 27.4.0 installed. You must install Docker on the host where the NKP Konvoy CLI runs. For example, if you install Konvoy on your laptop, ensure the computer has a supported version of Docker. On macOS, Docker runs in a virtual machine that you configure with at least 8GB of memory. For information on the local registry, see Registry Mirror Tools on page 1028. For information on Docker, see <https://docs.docker.com/get-docker/>.
• To interact with the running cluster, install kubectl on the host where the NKP Konvoy command line interface (CLI) runs. For more information, see kubectl.
• You must have the following resource requirements: vCPU Count 8, Memory 16 GB, and Disk Volume 80 GB. For more information see Resource requirements

Depending on your OS, various commands exist to set up your bastion host in an air-gapped environment. The vSphere example workflow shows a generic instance for Red Hat Enterprise Linux (RHEL) Bastion nodes using Docker.

Procedure

1. Open an ssh terminal to the bastion host and install the tools and packages using the command sudo yum

install -y yum-utils bzip2 wget. 2. Install kubectl. RHEL example:

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
sudo yum install -y kubectl

1. Install Docker, for example (only on the Bastion Host), and add the repository for upstream Docker using the command.

sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-
ce.repo
Note: Other Docker repository downloads are available on docker.com: https://download.docker.com/
linux/

Docker Install example:

sudo yum install -y docker-ce docker-ce-cli containerd.io

1. Get the required Nutanix Software by downloading the air-gapped bundle. Download nkp-air-gapped-bundle_nkp-version_linux_amd64.tar.gz, extract the tar file to a local directory using the command tar -xzvf nkp-air-gapped- bundle_nkp-version_linux_amd64.tar.gz
2. Set the following environment variables to enable connection to an existing Docker or other registry using the command export REGISTRY_ADDRESS=<https/http>://<registry-address>:<registry-port> export REGISTRY_CA=<path to the CA on the bastion host>.

Note: You must create the VM template with the Nutanix Image Builder to use the registry mirror feature.

Command variables for the export REGISTRY command.

• REGISTRY_ADDRESS: The address of an existing registry accessible in the environment where the new cluster nodes will be configured to use a mirror registry when pulling images.
• REGISTRY_CA: (Optional) path on the bastion host to the registry CA. Konvoy configures the cluster nodes to trust this CA. This value is only needed if the registry is using a self-signed certificate and the VMs are not already configured to trust this CA.

Each infrastructure provider has its own set of bastion host instructions. Refer to your own OS instructions to setup a bastion host like AWS Bastion, Azure, GCP, or vSphere.

Export Registry Variables and Flags for Cluster Creation

If you have a local registry, you must provide additional arguments when creating the cluster. These tell the cluster where to locate the local registry to use by defining the URL. Set the needed environment variable(s) with your registry information:

export REGISTRY_URL="<https/http>://<registry-address>:<registry-port>"
export REGISTRY_CA=<path to the CA on the bastion>
export REGISTRY_USERNAME=<username>
export REGISTRY_USERNAME=<password>

• REGISTRY_URL: the address of an existing container registry accessible in the VPC that the new cluster nodes will be configured to use a mirror registry when pulling images.
• REGISTRY_CA: (optional) the path on the bastion machine to the container registry CA. Konvoy will configure the cluster nodes to trust this CA. This value is only needed if the registry is using a self-signed certificate and the AMIs are not already configured to trust this CA.
• REGISTRY_USERNAME: optional, set to a user that has pull access to this registry.
• REGISTRY_PASSWORD: optional if username is not set.

When creating the cluster, apply the variables you defined above during the nkp create cluster command with the flags needed for your environment:

--registry-mirror-url=${REGISTRY_URL} \
--registry-mirror-cacert=${REGISTRY_CA} \
--registry-mirror-username=${REGISTRY_USERNAME} \
--registry-mirror-password=${REGISTRY_PASSWORD}

Export Variables

Set the environment variable with your registry information.

export REGISTRY_URL="<https/http>://<registry-address>:<registry-port>"
export REGISTRY_USERNAME=<username>
export REGISTRY_PASSWORD=<password>
export REGISTRY_CA=<path to the cacert file on the bastion>

Definitions:

REGISTRY_URL: the address of an existing local registry accessible in the VPC that the new cluster nodes will be
configured to use a mirror registry when pulling images.

• For example, <https://registry.example.com>

Other local registries may use the options below:

• REGISTRY_USERNAME: optional-set to a user with pull access to this registry.
• REGISTRY_PASSWORD: optional if username is not set.
• JFrog - REGISTRY_CA: (optional) the path on the bastion machine to the registry CA. This value is only needed if the registry uses a self-signed certificate and the AMIs are not already configured to trust this CA.
• To increase Docker Hub’s rate limit use your Docker Hub credentials when creating the cluster by setting flags ––registry-mirror- url=<https://registry-1.docker.io> –registry-mirror- username=<your- username> –registry-mirror-password=<your-password> when running nkp create cluster.

FIPS Requirements

To create a cluster in FIPS mode, inform the controllers of the appropriate image repository and version tags of the official Nutanix FIPS builds of Kubernetes by adding those flags to nkp create cluster command:

--kubernetes-version=v<kubernetes-version>+fips.0 \
--etcd-version=etcd-version+fips.0 \
--kubernetes-image-repository=docker.io/mesosphere \

For more information about enabling FIPS in the cluster, see Creating the Nutanix Kubernetes Platform Management Cluster using CLI on page 740.

For more information about the supported Kubernetes version, see Supported Kubernetes Versions section in the NKP Release Notes. For more information about the supported etcd version, see Supported Components section in the NKP Release Notes.

Output Directory Flag

When creating a cluster, you can use the –output-directory flag to organize the cluster configuration into individual files. This is particularly useful for ease of editing and managing the cluster configuration. The flag creates multiple files in the specified directory, which must already exist.

Refer to the Pre-provisioned Cluster Creation Customization Choices on page 770 section for more information on how to use optional flags such as the –output-directory flag.

Example:
--output-directory=<existing-directory>

Cluster creation results:

The output from this command is shortened here for reading clarity, but should start like this:

Generating cluster resources
cluster.cluster.x-k8s.io/cluster_name created
cont.........

Provision Flatcar Linux OS

Flatcar default network interface name might require specifying. It is most likely to be ens192 , which requires passing the parameter –virtual-ip- interface ens192 to the nkp create cluster aws command. Otherwise, the cluster creation might fail because kube-vip can not configure the first control-plane virtual IP.

Flatcar Linux Example

These flags are also shown in context on the Create Cluster page for either air-gapped or non-air-gapped environments:

Amazon Web Services (AWS) Example is shown; replace aws with vsphere if required:

nkp create cluster aws \
--cluster-name ${CLUSTER_NAME} \
--os-hint flatcar
Note: For provisioning Nutanix Kubernetes Platform (NKP) on Flatcar, NKP configures cluster nodes to use Control
Groups (cgroups) version 1. In versions before Flatcar 3033.3.x, a restart is required to apply the changes to the kernel.

Also note that once Ignition runs, it is not available on reboot.

For more information on Flatcar usage, see:

• Flatcar documentation: <<https://www.flatcar.org/docs/latest/container-> runtimes/switching-to-> unified-cgroups/#starting-new-nodes-with-legacy- cgroups
• Control Groups version 1:<<https://www.kernel.org/doc/html/latest/admin-> guide/cgroup-v1/> cgroups.html#what-are-cgroups
• Ignition <<https://www.flatcar.org/docs/latest/provisioning/ignition/#ignition-only-> runs-once>

Inspect Cluster for Issues

You can investigate what is running and what has failed to try to resolve those issues independently.

Investigate Cluster Issues

These commands can provide helpful information for troubleshooting.

• Check your pods to see if anything not running and investigate those pods. You can view your pods by checking their status with the command:

kubectl get pods -A

• You can check the logs in your cluster API pod of your cluster infrastructure provider choice. The example below uses Nutanix infrastructure, so replace it with your infrastructure.

kubectl logs -l cluster.x-k8s.io/provider=infrastructure-nutanix --namespace capx-
system --kubeconfig ${CLUSTER_NAME}.conf

• If you still have your bootstrap cluster running, you can check your CAPI logs from the bootstrap with the command. The example below uses Nutanix infrastructure, so replace it with your CAPI driver and infra name.

kubectl logs -l cluster.x-k8s.io/provider=infrastructure-nutanix --namespace capx-
system --kubeconfig ${CLUSTER_NAME}-bootstrap.conf

Configuring Self-healing

About this task

To configure self-healing, follow these steps:

Procedure

1. Open a command line interface (CLI).
2. View the MachineHealthCheck resources:

kubectl get MachineHealthCheck

The list of MachineHealthCheck resources is displayed.

$ kubectl get MachineHealthCheck
NAME CLUSTER EXPECTEDMACHINES MAXUNHEALTHY CURRENTHEALTHY AGE
nkp-j5h4n nkp 3 40% 3 8h
nkp-md-0-qgxr8 nkp 4 40% 4 8h

In this example, the MachineHealthChecks for a control node pool named nkp-j5h4n and a worker node pool named nkp-md-0-qgxr8 are displayed. 3. Edit the required MachineHealthCheck resource:

» For ClusterClass-based clusters, run the following command:

kubectl edit cluster -n <namespace> <cluster name>

Replace <namespace> and <cluster name> with the appropriate information.

» For non-ClusterClass-based clusters, run the following command:

kubectl edit machinehealthcheck -n <namespace> <mhc-name>

Replace <mhc-name> with the name of the MachineHealthCheck resource of the control pane node pool or the worker pane node pool. Edit the resource under spec.

To configure MachineHealthCheck settings per node pool, edit the Cluster resource under:

spec.topology.workers.machineDeployments[*].machineHealthCheck

For example:

spec:
topology:
workers:
machineDeployments:
- name: <nodepool-name>
class: <md-class&gt;
machineHealthCheck:
maxUnhealthy: 50%

1. Search for spec.maxUnhealthy.

Note: The default value is 40%. 5. Update the value of spec.maxUnhealthy as required For example:

[...]
spec:
clusterName: nkp
maxUnhealthy: 50%
nodeStartupTimeout: 10m0s
[...]

1. Save the changes.
2. Verify this update:

kubectl describe machinehealthcheck <mhc-name>

Certificate Renewal

You can renew the control plane certificate using one of the following automated methods:

Renewing the Control Plane Certificate Using Cluster API Method

Note:

• Nutanix recommends that you renew the control plane certificates in the NKP cluster only using the Cluster API (CAPI) method.
• NKP does not support automated renewal of control plane certificates for attached clusters. Nutanix recommends renewing the control plane certificates for those clusters manually to prevent cluster inaccessibility due to an expired certificate.

Certificate Renewal (2)

About this task

To configure an automated renewal of control plane certificates using CAPI in a new NKP cluster or in an existing control plane node, follow these steps:

Procedure

1. If you are creating a NKP cluster, follow these steps:

a. Create a NKP cluster.

By default, NKP sets the control plane certificate expiry to 180 days.

The supported infrastructure providers are Nutanix, AWS, Azure, GCP, Pre- provisioned, and vSphere.

b. (Optional) Set the automated renewal interval for the control plane certificate.

nkp create cluster infrastructure-provider --control-plane-renew-certificates-
before=50

The control-plane-renew-certificates-before flag indicates the number of days before expiry when the control plane certificate renews automatically.

For example, a control-plane-renew-certificates-before value of 50 indicates that the certificate automatically renews 50 days before expiration.

c. (Optional) Disable the automated renewal of control plane certificate on a newly created NKP cluster.

nkp create cluster infrastructure-provider --control-plane-renew-certificates-
before=0

A control-plane-renew-certificates-before value of 0 disables the automated renewal of control plane certificate.

Note: If you disable the automated renewal of control plane certificate, you must manually renew the certificate
to prevent cluster inaccessibility due to expired certificate. For more information, see KB 19301.

1. To enable automated renewal of control plane certificate on an existing NKP cluster, follow these steps:

a. Update an existing control plane node pool in the NKP cluster.

nkp update controlplane infrastructure-provider --control-plane-renew-
certificates-before=50

The supported infrastructure providers are Nutanix, AWS, Azure, GCP, Pre- provisioned, and vSphere.

The control-plane-renew-certificates-before flag indicates the number of days before expiry when the control plane certificate renews automatically.

For example, a control-plane-renew-certificates-before value of 50 indicates that the certificate automatically renews 50 days before expiration.

To modify the automated renew interval of control plane certificate, update the control-plane-renew- certificates-before flag value as necessary.

b. (Optional) Disable the automated renewal of control plane certificate on an existing NKP cluster.

nkp update controlplane infrastructure-provider --control-plane-renew-
certificates-before=0

A control-plane-renew-certificates-before value of 0 disables the automated renewal of control plane certificate.

Note: If you disable the automated renewal of control plane certificate, you must manually renew the certificate
to prevent cluster inaccessibility due to expired certificate. For more information, see KB 19301.

Viewing Alerts

control plane certificate before it expires. Based on the expiry status and renewal interval, NKP displays alerts with different severity levels, including informational, warning, and critical alerts in the Kommander user interface.

About this task

If you enable automated renewal of the control plane certificate during cluster creation, the Kommander user interface displays the following alerts:

• Renews in (X) day: Triggers between one and seven days before automatic renewal.

(X) indicates one to seven days.

• Renewal In Progress: Triggers on the renewal day.
• Renewal Might Fail: Triggers as the renewal interval advances and one or more nodes fail to complete automated certificate renewal.
• Expired: Triggers when the automated certificate renewal fails and the certificate expires.

If you disable or when NKP does not support automated renewal of the control plane certificate during cluster creation, the Kommander user interface displays the following alerts:

• Auto-Renewal Disabled: Triggers when you disable automated renewal of control plane certificates.
• Unknown: Displays when NKP does not support automated renewal or cannot retrieve the renewal information.

Note: NKP does not support automated renewal of control plane certificates for the attached clusters.

To view the alerts, follow these steps:

Procedure

1. Log in to the NKP user interface.

By default, the Dashboard displays both management and managed clusters. 2. In the left navigation menu, click Clusters.

You can view the alerts on the following pages:

• To filter the cluster widget based on the severity of certificate alert, click All, Errors, Warning, or Notice tab.
• To view all the alerts, click All.
• When the certificate renewal fails, NKP displays the alerts in Errors.
• When the certificate might fail, NKP displays the alerts in Warning.
• When the certificate renewal is in X number of days or in progress, NKP displays the alerts in Notice.
• In the cluster (management or managed) widget, click the Certificate button.

A dialog box displays an appropriate alert message based on the alert.

Note:

• The Certificate button appears in the widget when seven or fewer days remain before renewal.
• The Certificate button is greyed out when you disable or when NKP does not support automated renewal.
• In the Management Cluster or managed cluster widget, click View Details.

In the General Cluster Information page, click the alert message such as:

• Renews in 7 days
• Renewal In Progress
• Renewal Might Fail
• Expired
• Auto-Renewal Disabled
• Unknown

A dialog box displays an appropriate alert message based on the alert.

• To view the expiry and automated renewal information of a control plane certificate, go to Management Cluster or managed cluster widget, click View Details and select the Configuration tab.

Figure 21: Certificate Renewal Alerts

If you disable automated renewal of the control plane certificate, Nutanix recommends that you either enable automated renewal or manually renew the control plane certificate.

If NKP does not support automated renewal of the control plane certificate, it triggers an alert Unknown. Nutanix recommends that you manually renew the control plane certificate.

If the automated control plane certificate renewal fails or passes its expiry date, Nutanix recommends that you manually renew it.

For more information, see KB 19301. 3. (Optional) To view the certificate alerts in the Prometheus user interface, follow these steps:

a. In the Management Cluster widget, click View Details.

b. In the Enabled Applications tab, go to Prometheus widget and click Dashboard. The Prometheus application is launched.

c. Go to the Alerts tab. The Alerts tab displays all the active and inactive AutoRenewalCertificateAlerts. 4. To view the certificate alerts in NKP Insights, see Nutanix Kubernetes Platform Insights Alerts on page 1125.

Install Nutanix Kubernetes Platform on Nutanix Infrastructure

Install Nutanix Kubernetes Platform (NKP) on a Nutanix infrastructure.

This table lists all the steps to configure Nutanix infrastructure and NKP for air-gapped and non-air-gapped environments.

Table 64: Installing NKP on Nutanix

Pre-requisites and Planning

• Nutanix Kubernetes Platform Requirements on page 45
• Nutanix Infrastructure Requirements on page 719
• Control Plane Nodes and Worker Nodes Resource Requirements for Nutanix Kubernetes Platform on page 721

Access & Identity Management - Prism Central

• Prism Central Requirements for Nutanix Kubernetes Platform Installation on page 729
• Prism Central Credential and Role Management on page 729
• Preparing Prism Central Resources for the Nutanix Kubernetes Platform Cluster on page 732

Infrastructure Preparation

• Managing VMs from VLAN Basic Subnets to Network Controller-based VLAN Subnets on page 733
• Creating the Operating System Package Bundle on page 734
• Creating a Nutanix VM Image on page 734
• Preparing a Local Registry Mirror on page 736 (Required for air-gapped, and optional for non- airgapped environments)
• Pushing Images to the Registry on page 737 (Required for air-gapped, and optional for non- airgapped environments)

Installation

• Preflight Checks on page 747
• Deploying Nutanix Kubernetes Platform using UI from Nutanix Marketplace on page 738
• Creating the Nutanix Kubernetes Platform Management Cluster using CLI on page 740
• Creating a Nutanix Cluster With Custom Cilium Configuration on page 748 (Advanced Use-cases)

Post-Installation

• Setting up the Nutanix Kubernetes Platform User Interface Access on page 750
• Updating Prism Central Credentials on Deployed NKP Clusters on page 751

Nutanix Kubernetes Platform Installation Prerequisites

The following requirements apply to installations of NKP on a Nutanix infrastructure:

Nutanix Infrastructure Requirements

• Control Plane Nodes and Worker Nodes Resource Requirements for Nutanix Kubernetes Platform on page 721
• Nutanix Kubernetes Platform Requirements on page 45
• Prism Central Requirements for Nutanix Kubernetes Platform Installation on page 729
• Managing VMs from VLAN Basic Subnets to Network Controller-based VLAN Subnets on page 733
• Creating the Operating System Package Bundle on page 734

Before installing Nutanix Kubernetes Platform (NKP) on a Nutanix infrastructure, verify that your environment meets the following basic requirements:

• Ensure the Prism Central and AOS versions are compatible. For more information, see Compatibility and Interoperability Matrix.
• The Nutanix environment must be either on-premises or hosted on one of the public clouds, such as NC2 Azure, NC2 AWS, or NC2 GCP.
• NKP is supported on Nutanix Cloud Platform with the following external storage integrations:
• Dell PowerFlex:
• For the supported configuration maximums and minimums, see Configuration Maximums and Minimums for NCP with Dell PowerFlex.
• For the recommendations and limitations, see Recommendations and Limitations of NCP with Dell PowerFlex.
• Pure Storage FlashArray:
• For the supported configuration maximums and minimums, see Configuration Maximums and Minimums for NCP with Pure Storage FlashArray.
• For the recommendations and limitations, see Recommendations and Limitations of NCP with Pure Storage FlashArray.

• Configure valid values in the Prism Central instance.

For more information, see Prism Central Settings (Infrastructure).

• Ensure that the DNS servers configured on the Prism Central instance are reachable.
• You must have a pre-designated subnet.
• Ensure that a subnet is available with unused IP addresses.

Configure either Nutanix IP address management (IPAM) or dynamic host configuration protocol (DHCP) on the subnet to automatically assign IP addresses to control plane nodes and worker nodes.

Compute the number of required IP addresses:

• One IP address for each node in the Kubernetes cluster. The default cluster size includes three control plane nodes and four worker nodes, requiring a total of seven IP addresses.
• One IP address within the same classless inter-domain routing (CIDR) as the subnet, but outside the address pool for the Kubernetes API server.
• One IP address in the same CIDR as the subnet, but outside the address pool for the default ingress.
• Additional IP addresses might be required for additional load balancer services, such as Nutanix Data Services for Kubernetes (NDK).

For more information, see Prerequisites and Limitations in the Nutanix Data Services for Kubernetes guide.

Example configuration: If your subnet uses the CIDR 10.0.0.0/24 and the DHCP or Nutanix IPAM IP address pool range is 10.0.0.100 to 10.0.0.200, the system randomly selects the individual node IP addresses of the cluster from that address pool. Outside the IP address pool, reserve one IP address (for example, 10.0.0.90)

for the Kubernetes control plane endpoint. Reserve a contiguous IP address range on the same subnet as the worker nodes for MetalLB load balancer services; for example, 10.0.0.150 to 10.0.0.160.

• Multi subnet configuration (optional)

Use different subnets for control plane nodes and worker nodes only under the following conditions:

• Routing: Establish network routing between the control plane and worker node subnets.
• Nodes on multiple subnets: If you distribute worker and control plane nodes across multiple subnets, ensure that the IP range provided for the load balancer service (MetalLB) belongs to a Layer 2 subnet shared by all those nodes.
• For air-gapped environments, create a bastion VM host template with access to a configured local registry.

Nutanix recommends using a naming pattern such as ../folder-name/NKP-bastion- template. Each infrastructure provider has its own instructions for setting up a bastion host.

For more information, see Creating a Bastion Host on page 707.

• You need access to a bastion VM or other network-connected host running NKP image builder.

Note: Nutanix provides a complete image built on its infrastructure, which eliminates the need to create your
own from a BaseOS image.

• Ensure that you can reach the Nutanix endpoint from where you run NKP CLI.

Note:

• For an air-gapped environment, ensure that you download the bundle and extract the TAR file to a local directory.

For more information, see Downloading NKP on page 16.

• Some commands, such as nkp push bundle, require temporary disk space. These commands write to the temporary directory, which is usually /tmp. To override the directory, export the TMPDIR environment variable before running a command.

For example,

export TMPDIR=/path/to/your/directory

For more information on troubleshooting or additional information, see Nutanix Knowledge Base.

Control Plane Nodes and Worker Nodes Resource Requirements for Nutanix

Kubernetes Platform

Note:

• NKP uses the Nutanix Container Storage Interface (CSI) driver as the default storage provider.

For more information, see Default Storage Providers on page 34.

• The Nutanix CSI driver uses the storage container associated with the Prism Central cluster. Every Prism Central cluster maintains its own set of storage containers, and the CSI driver searches for a storage container with the same name across all Prism Central clusters.

Starter Nutanix Kubernetes Platform Cluster Minimum Requirements

Note the minimum requirements for the Starter Nutanix Kubernetes Platform (NKP) management and managed clusters.

The default configuration is set to three control plane nodes to provide high availability. The exact number of worker nodes required for your environment might vary depending on the workload of your cluster and the size of the nodes.

Note:

• The Starter license is supported exclusively with the Nutanix infrastructure.
• If you have two worker nodes in your Starter NKP management and managed clusters and you apply for a Pro or Ultimate license, the additional Pro or Ultimate features fail because your environment does not have enough cluster resources to deploy them.

To install the Starter NKP management and managed clusters with the minimum amount of resources, review the resource tables before starting the installation.

Table 65: Resource Requirements for Management Cluster

Minimum node requirements

3 2

vCPU count 2 4

Memory 8 GiB 8 GiB

Disk volume Approximately 80 GiB:

Used for /var/lib/kubelet and /var/ lib/containerd

Approximately 80 GiB:

Used for /var/lib/kubelet and /var/ lib/containerd

Root volume Disk usage must be below 85% Disk usage must be below 85%

Non-default flags in the CLI –control-plane-vcpus 2 \ –control-plane-memory 8 \ –worker-replicas 2 \ –worker-vcpus 4 \ –worker-memory 8

Table 66: Resource Requirements for Managed Cluster

Minimum node requirements

3 2

vCPU count 2 3

Memory 6 GiB 6 GiB

Disk volume Approximately 80 GiB:

Used for /var/lib/kubelet and /var/ lib/containerd

Approximately 80 GiB:

Used for /var/lib/kubelet and /var/ lib/containerd

Root volume Disk usage must be below 85% Disk usage must be below 85%

Non-default flags in the CLI –control-plane-vcpus 2 \ –control-plane-memory 6 \ –worker-replicas 2 \ –worker-vcpus 3 \ –worker-memory 6

Caution: You can create a working NKP management cluster with a single control plane node. However, if the single
control plane node faces corruption or failure, it results in the loss of the entire cluster.

Pro and Ultimate Nutanix Kubernetes Platform Cluster Minimum Requirements

Note the minimum requirements for Pro and Ultimate Nutanix Kubernetes Platform (NKP) management cluster.

The exact number of worker nodes required for your environment might vary depending on the workload of your cluster and the size of the nodes. The default configuration is set to three control plane nodes to provide high availability. If you follow the instructions to create a cluster using the default NKP settings without modifying the configuration files or additional flags, you can deploy the cluster with three control plane nodes and four worker nodes, matching the requirements in the General Resource Requirements for Pro and Ultimate Clusters table.

Caution: You can create a working NKP management cluster with a single control plane node. However, if the single
control plane node faces corruption or failure, it results in the loss of the entire cluster.

To install Pro and Ultimate NKP clusters with minimum resources, review the requirements table before starting the installation.

Table 67: General Resource Requirements for Pro and Ultimate Clusters

Optimal nodes recommended

3 4

vCPU count 4 8

Memory 16 GiB 32 GiB

Disk volume Approximately 80 GiB: Used for / var/lib/kubelet and /var/lib/ containerd

Approximately 80 GiB: Used for / var/lib/kubelet and /var/lib/ containerd

Root volume Disk usage must be below 85% Disk usage must be below 85%

Table 68: Resource Requirements for Managed Clusters

Optimal nodes recommended

3 4

vCPU count 4 8

Memory 12 GiB 12 GiB

Disk volume Approximately 80 GiB: Used for / var/lib/kubelet and /var/lib/ containerd

Approximately 80 GiB: Used for / var/lib/kubelet and /var/lib/ containerd

Root volume Disk usage must be below 85% Disk usage must be below 85%

Infrastructure Provider-Specific Requirements

Note the additional requirements that might apply to a certain infrastructure providers.

For example, the Nutanix Kubernetes Platform (NKP) on Azure by default deploys a Standard_D4s_v3 VM with a 128 GiB volume for the operating system and an 80 GiB volume for etcd storage, which meets the specified requirements.

For more information on resource requirements, see the relevant installation options for the respective infrastructure provider:

• Nutanix Installation Options on page 72
• Pre-provisioned Installation Options on page 72
• AWS Installation Options on page 168
• Azure Installation Options on page 250
• vSphere Installation Options on page 189
• EKS Installation Options on page 168
• AKS Installation Options on page 260

Nutanix Kubernetes Platform Configuration Maximums

Note a list of supported configuration maximum for Nutanix Kubernetes Platform (NKP).

You can view the latest list of supported configuration maximums from the Nutanix Support Portal. For more information, see NKP Configuration Maximums. Ensure that you select the required NKP version from the list.

Note: A Nutanix account is required to access the Nutanix Support Portal.

Nutanix Kubernetes Platform Managed Cluster Requirements

Note the requirements for the Nutanix Kubernetes Platform (NKP) managed clusters.

Minimum Recommendation for Managed Clusters

To create additional clusters in your Nutanix environment, ensure that you have at least the following minimum recommended resources:

• Worker Nodes: The default installation includes at least four worker nodes with the following specifications:
• 8 CPU cores each
• 12 GiB of memory
• A storage class and disk mounts that accommodate at least four persistent volumes

Note: You need four worker nodes to support the upgrades to rook-ceph platform application. rook-ceph
supports the logging stack, velero backup tool, and NKP Insights. If you disable rook-ceph platform
application, you need only three worker nodes.

• Control Plane Nodes: The default installation includes at least three control plane nodes with the following specifications:
• 4 CPU cores each
• 12 GiB of memory

Note: While one control plane node is acceptable for a non-critical test environment, any production workload
cluster must have at least three control plane nodes.

• Cluster: Requires a default storage class and three volumes of 10GiB, 100GiB, and 10GiB or the ability to create those volumes based on the storage class.

Create a volume based on the storage class.

$ kubectl get pv -A
NAME CAPACITY ACCESS MODES RECLAIM POLICY
STATUS CLAIM STORAGECLASS
VOLUMEATTRIBUTESCLASS REASON AGE
pvc-01497817-b03a-4572-80e3-37839e29e116 10Gi RWO Delete
Bound york-ws/opencost-pvc ebs-sc
<unset> 151m
pvc-82461b4c-51e6-4f5f-b1c9-04cbf5f5eb40 100Gi RWO Delete
Bound york-ws/db-prometheus-kube-prometheus-stack-prometheus-0 ebs-sc
<unset> 151m
pvc-8a70db2f-101a-4d09-8fd3-2bc0b649df3d 10Gi RWO Delete
Bound york-ws/kube-prometheus-stack-grafana ebs-sc
<unset> 151m

Note: Actual workload clusters might demand more resources based on the usage.

Nutanix Kubernetes Platform Management Cluster Application Requirements

Note the requirements for the Nutanix Kubernetes Platform (NKP) management cluster applications.

The following table lists the workspace platform applications that are specific to a management cluster including their minimum resource requirements, minimum persistent storage requirements, and default PriorityClass values. For more information on a list of all platform applications, see Platform Application Configuration Requirements.

Common Name

Application ID Deployed by Default

Minimum Resources

Minimum Persistent Storage

Default PriorityClass

Centralized Grafana*

centralized- grafana

Yes cpu: 200m

memory: 100Mi

Not applicable NKP Critical (100002000)

Centralized OpenCost*

centralized- openCost

Yes cpu: 10m

memory: 55Mi

of PVs: 1

PV sizes: 10Gi

NKP High (100001000)

Dex dex Yes cpu: 100m

memory: 50Mi

Not applicable NKP Critical (100002000)

Dex Authenticator

dex-k8s- authenticator

Yes cpu: 100m

memory: 128Mi

Not applicable NKP High (100001000)

Karma* karma Yes Not applicable Not applicable NKP Critical (100002000)

Kommander kommander Yes cpu: 1100m

memory: 896Mi

Not applicable NKP Critical (100002000)

Kommander AppManagement

kommander- appmanagement

Yes cpu: 300m

memory: 256Mi

Not applicable NKP Critical (100002000)

Kommander Flux

kommander-flux Yes cpu: 7500m

memory: 5Gi

Not applicable NKP Critical (100002000)

Kommander UI kommander-ui Yes cpu: 100m

memory: 256Mi

Not applicable NKP Critical (100002000)

Kubefed kubefed Yes cpu: 300m

memory: 192Mi

Not applicable NKP Critical (100002000)

Kubetunnel kubetunnel Yes cpu: 200m

memory: 148Mi

Not applicable NKP Critical (100002000)

Thanos* thanos Yes Not applicable Not applicable NKP Critical (100002000)

Traefik ForwardAuth

traefik-forward- auth-mgmt

Yes cpu: 100m

memory: 128Mi

Not applicable NKP Critical (100002000)

Applications with an asterisk (“*”) are only for NKP Ultimate users. If you have an Ultimate license, NKP deploys these applications automatically.

For more information on application versions, see NKP Release Notes on the Nutanix Support Portal.

Workspace Platform Application Defaults and Resource Requirements

Note the resource requirements for the workspace platform applications available within Nutanix Kubernetes Platform (NKP).

This section lists the workspace platform applications that are available within NKP. Some of these applications are deployed by default upon attachment. You need to manually install other applications through the CLI under cluster operations. For more information, see Platform Applications.

Workspace platform applications require more resources than solely deploying or attaching clusters to a workspace. Therefore, your cluster must have sufficient resources available during deployment or attachment to ensure the successful installation of platform services.

The following table describes all the workspace platform applications available to the clusters within a workspace. It includes minimum resource requirements, deployment information, and their default PriorityClasses.

Table 69: Available Workspace Platform Applications

Cert Manager cert-manager Yes cpu: 10m

memory: 32Mi

Not applicable system- cluster-critical (2000000000)

External DNS external-dns No Not applicable NKP High (100001000)

Fluent Bit fluent-bit No cpu: 350m * # of nodes

memory: 250Mi * #
of nodes

Not applicable NKP Critical (100002000)

Gatekeeper gatekeeper Yes cpu: 300m

memory: 768Mi

Not applicable system- cluster-critical (2000000000)

Grafana grafana- logging

No cpu: 200m

memory: 100Mi

Not applicable NKP Critical (100002000)

Loki grafana-loki No Not applicable # of PVs: 8

PV sizes: 10Gi x 8 (total: 80Gi)

NKP Critical (100002000)

Istio-Helm istio-helm No cpu: 1270m

memory: 4500Mi

Not applicable NKP Critical (100002000)

Jaeger jaeger No Not applicable Not applicable NKP High (100001000)

Kiali kiali No cpu: 20m

memory: 128Mi

Not applicable NKP High (100001000)

Knative knative No cpu: 610m

memory: 400Mi

Not applicable NKP High (100001000)

Kube OIDC Proxy

kube-oidc- proxy

Yes Not applicable Not applicable NKP Critical (100002000)

Kube Prometheus Stack

kube- prometheus- stack

Yes cpu: 1300m

memory: 4300Mi

of PVs: 1 (2)

PV sizes: 100Gi

NKP Critical (100002000)

• Common Name; Application ID; Deployed by Default; Minimum Resources; Minimum Persistent Storage; Default PriorityClass

| — | — | — | — | — | — |

OpenCost* OpenCost Yes cpu: 10m

memory: 55Mi

of PVs: 1 (3)

PV sizes: 10Gi

NKP High (100001000)

Kubernetes Dashboard

kubernetes- dashboard

Yes cpu: 250m

memory: 300Mi

Not applicable NKP High (100001000)

Logging Operator

logging- operator

No cpu: 350m * # of nodes + 600m

memory: 228Mi +
350Mi * # of nodes

of PVs: 1 (4)

PV sizes: 10Gi

NKP Critical (100002000)

NVIDIA GPU Operator

nvidia-gpu- operator

No cpu: 100m

memory: 128Mi

Not applicable system- cluster-critical (2000000000)

Prometheus Adapter

prometheus- adapter

Yes cpu: 1000m

memory: 1000Mi

Not applicable NKP Critical (100002000)

Reloader reloader Yes cpu: 100m

memory: 128Mi

Not applicable NKP High (100001000)

Rook Ceph rook-ceph Yes cpu: 100m

memory: 128Mi

Not applicable system- cluster-critical (2000000000)

Rook Ceph Cluster

rook-ceph- cluster

Yes cpu 2500m

mem 8Gi

of PVs: 4

PV sizes: 40Gi

NKP Critical (100002000)

system- cluster-critical (2000000000)

system-node-critical

Traefik traefik Yes cpu: 500m Not applicable NKP Critical (100002000)

Traefik ForwardAuth

traefik- forward-auth

Yes cpu: 100m

memory: 128Mi

Not applicable NKP Critical (100002000)

Velero velero No cpu: 1000m

memory: 1024Mi

Not applicable NKP Critical (100002000)

• Application with an asterisk (“*”) is only for NKP Ultimate users. If you have an Ultimate license, NKP deploys these applications automatically.

• Currently, NKP only supports a single deployment of cert-manager for each cluster. Therefore, you cannot install cert-manager on Konvoy managed clusters or clusters with pre-installed cert-manager.

• NKP supports only a single deployment of traefik per cluster.

• Common Name; Application ID; Deployed by Default; Minimum Resources; Minimum Persistent Storage; Default PriorityClass

| — | — | — | — | — | — |

• NKP automatically manages the deployment of traefik-forward-auth and kube- oidc-proxy when you attach clusters to the workspace.

The NKP user interface does not display these workspace platform applications.

• You need to enable the workspace platform applications in NKP and then deploy them to the attached clusters.

Ensure successful deployment and verification of the applications enabled through the CLI. For more information, see Deployment of Catalog Applications in Workspaces on page 395.

For more information on application versions, see NKP Release Notes on the Nutanix Support Portal.

Prism Central Requirements for Nutanix Kubernetes Platform Installation

Prism Central requirements include setting up the Prism Central credentials, user roles, and Prism Central resources for the NKP management and workload clusters.

Prism Central Credential Requirements

Before updating your Prism Central credentials on a deployed Nutanix Kubernetes Platform (NKP) clusters, ensure that you meet the following requirements:

• Ensure that you have a kubeconfig file for the management cluster.
• Ensure that you have a kubeconfig file for each workload clusters that requires a Prism Central credential update.
• Update the following secrets with a new password for Prism Central instance:

Secrets for Prism Central Credentials

global-nutanix-credentials capx-system Management Cluster

${MANAGEMENT_CLUSTER_NAME}- pc-credentials

${MANAGEMENT_CLUSTER_NAMESPACE} Management Cluster

${MANAGEMENT_CLUSTER_NAME}- pc-credentials-for-csi

${MANAGEMENT_CLUSTER_NAMESPACE} Management Cluster

${MANAGED_CLUSTER_NAME}- pc-credentials

${MANAGED_CLUSTER_NAMESPACE} Management Cluster

${MANAGED_CLUSTER_NAME}- pc-credentials-for-csi

${MANAGED_CLUSTER_NAMESPACE} Management Cluster

nutanix-ccm-credentials kube-system

• Management Cluster
• Managed Cluster

nutanix-csi-credentials ntnx-system

• Management Cluster
• Managed Cluster

Prism Central Credential and Role Management

Define the user roles in Prism Central to manage the Nutanix Kubernetes Platform (NKP) management and workload clusters.

When you provision management and workload clusters with NKP on Nutanix infrastructure, you need an admin role that includes the minimum required permissions for NKP to provide proper access to deploy clusters.

NKP management cluster uses Prism Central credentials for the following components:

• Cluster management: For listing subnets and other infrastructure, and creating VMs in Prism Central for the CAPX infrastructure provider.
• Persistent storage management: For Nutanix Container Storage Interface (CSI) provider.
• Node metadata discovery: For Nutanix Cloud Controller Manager (CCM) provider.

NKP requires Prism Central credentials to authenticate the Prism Central APIs. CAPX currently supports two mechanisms to assign the required credentials:

• Credentials injected into the CAPX manager deployment.
• Managed cluster-specific credentials.

For examples, see Credential Management.

Prism Central Roles and Permissions for Nutanix Kubernetes Platform

Note the Prism Central roles and permissions required for the Kubernetes cluster lifecycle on Nutanix Kubernetes Platform (NKP).

Following are the list of Prism Central roles and permissions required for the Kubernetes cluster lifecycle on NKP:

• AHV VM: You need the following permissions for the AHV VM role:
• Create New VM
• Delete Existing VM
• Power On VM
• View Existing VM
• Update Virtual Machine Custom Attributes
• Category: You need the following permissions for the Category role:
• Create Category
• Delete Category
• Delete Value Category
• View Category
• Cluster: You need the following permissions for the Cluster role:
• View Cluster
• View Cluster Pgpu Profiles
• View Cluster Vgpu Profiles
• Host: You need the View Host permission for the Host role.
• Image: You need the following permissions for the Image role:
• Create Image
• Delete Image
• View Image
• Domain Manager (Prism Central): You need the View Domain Manager permission for the Domain Manager (Prism Central) role.
• Project: You need the View Project permission for the Project role.
• Subnet: You need the View Subnet permission for the Subnet role.
• Volume Group: You need the Detach Volume Group From AHV VM permission for the Volume Group role.
• VPC: You need the View VPC permission for the VPC role.

For information about roles, see Displaying Role Permissions in the Security Guide.

To view the minimum permissions required for Container Storage Interface (CSI) volume driver, see Operations Required to Perform CSI Actions in the CSI Volume Driver Guide.

Note: You must apply the required CSI permissions.

Configuring the Role with an Authorization Policy

Use Prism Central to create an authorization policy to assign the system- defined roles.

About this task

When you provision Kubernetes clusters with Nutanix Kubernetes Platform (NKP) on a Nutanix infrastructure, a pre- defined role that includes the minimum permissions to deploy clusters is also provisioned. For more information, see Configuring an Authorization Policy in the Security Guide.

To configure the role with an authorization policy, follow these steps:

Procedure

1. Log in to Prism Central as an administrator.
2. From the Application Switcher Function, select Admin Center.
3. Click IAM and go to Authorization Policies.
4. Click Create New Authorization Policy. The Create New Authorization Policy window appears.
5. In the Choose Role section, enter a role name in the Select the role to add to this policy field and click Next. You can enter the name of any built-in or custom roles.
6. In the Define Scope section, select one of the following options:

» Full Access: Grants all added users access to all entity types across the entire environment under the associated role.

» Configure Access: Restricts permissions to specific target infrastructure boundaries.

Note: If you select this option for either All or Specific instances of objects, you must configure the following
entity requirements as listed:

• Domain Manager: Set this field to All Domain Manager.
• Cluster: Set this field to All Cluster. This setting is required to ensure the added users can view and interact with the Prism Central instance.
• Category: Set this field to All Category.

1. Click Next.
2. In the Assign Users section, select one of the following options:

» From the dropdown list, select Local User to add a local user or group to the policy.

» From the dropdown list, select the available directory to add a directory user or group.

To search for a user or group, type the first few letters of the name in the text field. 9. Click Save.

For more information on displaying role permissions for any built-in role, see Viewing Role Permissions. in the Security Guide.

Prism Central saves the authorization policy and lists it in the Authorization Policies page.

Preparing Prism Central Resources for the Nutanix Kubernetes Platform

Cluster

Prepare Prism Central resources to create Nutanix Kubernetes Platform (NKP) management cluster.

Before you begin

• Locate the following information in Prism Central instance:
• Prism Central Endpoint with or without the port:

For more information, see Admin Center Overview in the Prism Central Admin Center Guide.

• Name of the Prism Element cluster:

For more information, see Modifying Cluster Details in the Prism Element Web Console Guide.

• Name of the subnet:
• An available control plane endpoint IP not assigned to any VM.
• Name of the operating system image:

For more information, see Creating the Operating System Package Bundle on page 734.

• Docker Hub credentials. If you do not have Docker Hub credentials, you cannot create a NKP management cluster when the Docker Hub rate limit is reached.

About this task

To prepare Prism Central resources for creating the NKP management cluster, follow these steps:

Procedure

1. Log in to Prism Central.
2. From the Application Switcher Function, select the Infrastructure Application.
3. Update the subnet.

Use an IP address in the subnet classless inter-domain routing (CIDR) but outside the IP address pool. For more information, see Updating a Subnet in the Flow Virtual Networking Guide. 4. Create a storage container.

For more information, see Creating a Storage Container in the Prism Central Infrastructure Guide.

Prism Central resources are ready to create an NKP management cluster in the air-gapped or non-air-gapped Nutanix infrastructure.

Managing VMs from VLAN Basic Subnets to Network Controller-based VLAN

Subnets

About this task

• If the node subnet in your Nutanix Kubernetes Platform (NKP) configuration overlaps with Kubernetes-reserved subnets, such as the pod or service subnet, the cluster might fail to deploy. You cannot change the subnets after creating a cluster. Ensure that the node subnet does not overlap with your host subnet.
• The default subnets used in NKP are as follows:

spec:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12

The current setup uses a basic VLAN configuration, while advanced VLAN uses network-controller as the control plane instead of Acropolis.

To migrate a subnet from VLAN basic subnets to network controller-based VLAN subnets, follow these steps:

Procedure

1. Log in to Prism Central.
2. From the Application Switcher Function, select the Infrastructure Application.
3. Click the Settings icon or from the navigation bar, navigate to Prism Central Setting.

For information about the navigation bar, see Application-specific Navigation Bar.

The Settings page appears. 4. Click Network Controller, and select the Configure Network Controller-Managed VLANs checkbox to set it as the default setting. 5. Migrate VMs from VLAN basic subnets.

For more information, see Flow Virtual Networking Network Controller 5.0 - Migrating VMs from VLAN Basic Subnets. 6. To use the new subnet, modify the subnet specification in the control plane and worker nodes .

kubectl edit cluster <clustername>

The cluster API provider for Nutanix cloud infrastructure (CAPX) rolls out the new control plane and worker nodes in the new subnet and destroys the old ones.

You can choose basic or network controller-based VLAN subnets when you create a subnet during cluster creation. If you create the cluster with basic VLAN subnets, you can migrate it to network controller-based VLAN subnets.

For more information on modifying the service subnet, adding or modifying the configmap, see Managing Subnets and Pods.

Creating the Operating System Package Bundle

About this task

You can complete perform this procedure using a machine with access to the internet, or access to the operating system package repositories to download the operating system packages.

To create an operating system package bundle, follow these steps:

Procedure

1. Open a terminal with access to the NKP CLI.
2. Create an operating system package bundle.

Use a sub-directory of the extracted air-gapped bundle to store the operating system package bundle:

For Ubuntu 22.04:

export OS=ubuntu-22.04
nkp create package-bundle --artifacts-directory ${OS_BUNDLE_DIR} ${OS}

For Rocky 9.6:

export OS=rocky-9.6
nkp create package-bundle --artifacts-directory ${OS_BUNDLE_DIR} ${OS}

For RHEL 8.10:

export OS_BUNDLE_DIR=nib/artifacts
export OS=rhel-8.10
export RHSM_ORG_ID=<your-RHSM-id>
export RHSM_ACTIVATION_KEY=<your-RHSM-Activation-key>
nkp create package-bundle --artifacts-directory ${OS_BUNDLE_DIR} ${OS}

Creating a Nutanix VM Image

About this task

To create a Nutanix image, follow these steps:

Procedure

1. Open a terminal with access to the NKP CLI.
2. (Only for an air-gapped environment, and optional for non-air-gapped environments) Load the NKP image builder container image into the local container runtime store:

The NKP CLI image builder uses a container image. In an air-gapped environment, you cannot automatically download the image from the public registry. The image is included in the air-gapped bundle, and you must load the image into the container runtime. 3. Create a Nutanix VM image:

nkp create image nutanix

To create a specialized VM image, the NKP CLI creates a VM from the base VM image.

export NUTANIX_ENDPOINT=Prism Central Endpoint URL
export NUTANIX_CLUSTER=Prism Central Cluster Name
export NUTANIX_USER=Prism Central Username
export NUTANIX_PASSWORD=Prism Central Password
export SUBNET=VLAN Subnet
export BASE_VM_IMAGE=VM image name
export OS=OS name

The NKP CLI also makes changes, such as installing dependencies and configuring services, and finally creates a new VM image with these changes. The following attributes apply in the previous command:

• NUTANIX_ENDPOINT - Prism Central endpoint URL, including the scheme, host, and port.
• NUTANIX_CLUSTER - Prism Central cluster.
• NUTANIX_USER - Prism Central username.
• NUTANIX_PASSWORD - Prism Central password.
• SUBNET - Prism Central subnet to create a temporary VM.
• BASE_VM_IMAGE - Use a VM image uploaded to Prism Central as the base.
• OS - OS name with version. For possible values, run nkp create image nutanix –help.

1. Create the Nutanix VM image with the environment variables exported in the environment.

If you have created a package bundle, use it here:

If you use internal registry when creating an NKP cluster:

export BUNDLE_FLAG="--bundle nkp-${nkp-version}/container-images/konvoy-image-bundle-
${nkp-version}.tar

This bundle contains all the necessary container images required to bootstrap an NKP cluster.

Replace nkp_version with the NKP version at your site.

nkp create image nutanix $OS \
--endpoint ${NUTANIX_ENDPOINT} \
--cluster ${NUTANIX_CLUSTER} \
--subnet ${SUBNET} \

• docker load –input “nkp-; nkp-version; /nkp-image-builder-; nkp-version; .tar”

| — | — | — | — | — |

• export ARTIFACTS DIRECTORY FLAG=“–artifacts-directory=nkp- _ _; nkp version _; /nib/artifacts”

| — | — | — |

--source-image ${BASE_IMAGE} \
"${ARTIFACTS_DIRECTORY_FLAG:-""}" \
--${BUNDLE_FLAG:-""}

1. Note the name of the created VM image.

Use this VM image name when you create the cluster. 6. Verify the image placement policies of the VM image.

The VM image is covered by one or more image placement policies. Ensure that these policies allow the VM image to be copied to all Prism Central clusters that you plan to use with NKP.

The image placement policies are listed in the VM image details view in Prism Central. For more information, see Image Placement Policies in the Prism Central Infrastructure Guide.

What to do next

• You can Prepare a Local Registry Mirror and then Push Images to the Registry (Required for an air-gapped environment and optional for a non-air-gapped environment).
• You can proceed to Deploy NKP Using UI from Nutanix Marketplace, or Create the NKP Management Cluster through CLI.

Preparing a Local Registry Mirror

Before you begin

If your site does not have a local registry set up, ensure that you configure one before proceeding. For more information, see the Local Registry Tools.

About this task

Important: This procedure is required for an air-gapped environment that does not use an internal registry mirror. It
is optional for a non-air-gapped environment, however it can increase stability and security. You can also use a registry
mirror with a self-signed registry CA certificate in the non-air-gapped environment.

In a non-air-gapped environment, Nutanix Kubernetes Platform (NKP) downloads artifacts from sources on the internet. For example, NKP downloads container images from container registries. In an air-gapped environment, NKP downloads these artifacts from sources within the environment. For example, NKP downloads container images from a registry reachable on the local network. The registry must be reachable from the bastion, where the NKP CLI runs, and from every machine in the Kubernetes cluster.

To prepare a local registry, follow these steps:

Procedure

1. Open the terminal with access to the NKP CLI.
2. Provide the registry mirror URL.

The registry address includes the scheme, the host, port, and optional path: 3. (Optional) Provide the registry mirror username and password. If the registry mirror requires a username and password, provide the username and password:

export REGISTRY_MIRROR_USERNAME=<username>

• export REGISTRY MIRROR URL=“https://; registry-mirror-host; :; registry- mirror-port; “

| — | — | — | — | — |

export REGISTRY_MIRROR_PASSWORD=<password>

1. Access the registry without a trusted CA certificate:

a. Export the environment variables:

export REGISTRY_MIRROR_CA=Path to store the downloaded CA certificate
export REGISTRY_MIRROR_ADDRESS=Registry address
export REGISTRY_MIRROR_PORT=Registry port

Replace:

• Path to store the downloaded CA certificate with the path to store the downloaded CA certificate.
• Registry address with the host of the registry URL.
• Registry port with the port of the registry URL.

b. Download the CA certificate:

openssl s_client \
-showcerts \
-connect $REGISTRY_MIRROR_ADDRESS:$REGISTRY_MIRROR_PORT \
</dev/null | \
openssl x509 \
-outform PEM \
> $REGISTRY_MIRROR_CA

If your registry CA certificate is self-signed, it might be a untrusted certificate.

Pushing Images to the Registry

Before you begin

Important: This procedure is required for an air-gapped environment that does not use an internal registry mirror. It
is optional for a non-air-gapped environment, however it can increase stability and security. You can also use a registry
mirror with a self-signed registry CA certificate in the non-air-gapped environment.

Ensure that the registry is prepared and the environment variables are exported before you push images to the registry. For more information, see Preparing a Local Registry Mirror on page 736.

If you do not already have a local registry set up and want to use an external solution, see Local Registry Tools.

About this task

To push images to the registry, follow these steps:

Procedure

1. Open a terminal with access to the NKP CLI.
2. Push all bundles to the registry:

nkp push bundle \
--bundle "nkp-nkp-version/container-images/konvoy-image-bundle-nkp-version.tar" \
--bundle "nkp-nkp-version/container-images/kommander-image-bundle-nkp-version.tar" \
--bundle "nkp-nkp-version/container-images/nutanix-product-catalog-nkp-version-
airgapped.tar" \
--to-registry=${REGISTRY_MIRROR_URL} \
--to-registry-username=${REGISTRY_MIRROR_USERNAME} \
--to-registry-password=${REGISTRY_MIRROR_PASSWORD} \
--to-registry-ca-cert-file=${REGISTRY_MIRROR_CA:-""}

Replace nkp_version with the NKP version at your site. Specify only the required bundle as comma-separated values in the –bundle parameter.

Note: The nutanix-product-catalog-nkp-version-airgapped.tar included in the NKP air-
gapped bundle contains images exclusively for Nutanix Data Services for Kubernetes (NDK). You must specify the
Nutanix product catalog bundle only if you plan to deploy NDK.

What to do next

You can Deploy NKP using UI from Nutanix Marketplace, or Create the NKP Management Cluster using CLI.

Deploying Nutanix Kubernetes Platform using UI from Nutanix Marketplace

Before you begin

• Ensure that you have enabled Nutanix Marketplace.
• Verify the NKP image, and Kubernetes version in the overview section. For more information, see Nutanix Image Builder on page 51 and Supported Infrastructure Operating Systems on page 12.
• You must have the required Prism Central Roles and Permissions for Nutanix Kubernetes Platform on page 730 to create a cluster.
• Ensure Valid Node OS images conforming to the Kubernetes version are available. For details on how to make these images available, see Using an Image provided by Nutanix and Importing Images to Prism Central. Do not rename the image downloaded from the Nutanix Portal.
• In connected environments, please ensure reachability to the nutanix-portal (download.nutanix.com). This will ensure the Bastion VM image can be downloaded during deployment.
• Ensure 1 IP for Kubernetes API VIP.
• Ensure IPs for Kubernetes service load balancer.
• Ensure selected VLAN has a pool of free IPs (required for DHCP allocation).
• Selected subnet should have IPAM/DHCP enabled.
• NKP starter licenses support deployment of Rocky Linux images only. For any other image, including Ubuntu, you must have a NKP Pro or NKP Ultimate License.

Note:

• NKP deployment on Nutanix marketplace is not supported on X-Small Prism Central.
• NKP cluster deployment is not supported on a dual stack enabled Prism Central.

Procedure

1. Log in to Prism Central as an administrator.
2. Select Admin Center in the Application Switcher.
3. Click Marketplace in the navigation bar.
4. In the Nutanix Apps section, click Get for the Nutanix Kubernetes Platform app. The app details page opens.
5. Click Deploy. The deployment page is displayed
6. In the Name & Access tab, fill in the details:

a. Enter the cluster name.

This is the name assigned to the NKP cluster during deployment.

b. Enter the SSH username and SSH Public Key

The SSH username and public key is applied to the bastion VM, as well as the control-plane and worker nodes. If the deployment fails, you can use these credentials to SSH into the bastion VM and extract the support bundle. 7. In the Provider tab, enter the details:

a. Enter the Prism Central username and password of the user creating and managing the NKP cluster.

b. Select the Prism Element name from the dropdown list. This list is based on the Prism Central credentials entered. 8. In the Resources tab, enter the details:

a. Select the OS image.

b. Enter the values for the control plane nodes and worker nodes.

The recommended values are pre-filled. 9. In the Storage tab, enter the details:

a. Select the Nutanix storage container.

b. Select the Reclaim policy.

c. Select the Hypervisor attached volume. 10. In the Network & Proxy tab, enter the details:

a. Select the subnets that are configured with either Nutanix IPAM or DHCP. They should also be connected to the control plane and worker node pools.

b. Enter details of the Control Plane Endpoint IP and Port. The IP details entered here must be unused, and must be outside the scope of Nutanix IPAM and DHCP.

c. Enter details of the K8s load balancer Service IP Range.

d. Enter details of the Kubernetes Pod Network and Service CIDR. Ensure the subnets do not overlap with your host subnet as it cannot be changed after creation of the cluster. If you select a non-default CIDR, you must update the new CIDR value in the no-proxy list.

e. Enter details of the Internet proxy for HTTP, HTTPS or if there is no proxy. See Configuring an HTTP or HTTPS Proxy on page 696 for more details on the flags and format to be used. 11. (Optional) In the Registry tab, enter the details:

• Image Registry Mirror:
• If provided, the cluster uses this external registry mirror to pull and cache required container images.
• If you do not provide a mirror, the system automatically deploys an in#cluster private registry that acts as a pull#through cache for the required images.
• Private Registry:
• If you provide a private registry, the system configures it to the cluster to pull specific application images.
• If you do not provide a private registry, the system skips this configuration.

Note: As both the Image Registry Mirror and Private Registry fields are optional. If you do not provide
details for either field, the system deploys an internal registry mirror to complete the cluster deployment.

1. Click Next. After deployment, the NKP App appears on the My Apps page. You can select the NKP Cluster in the application switcher to open and manage the app.

During deployment of the NKP instance, a bastion host VM tile is temporarily visible. This is cleared on successful deployment. In case of a failure, you can delete the failed NKP application instance which also cleans the bastion VM for this instance.

What to do next

• The dashboard can be launched from the NKP application page after deployment.
• You can download the kubeconfig to connect with or manage the cluster, or to create worker clusters. See Cluster Operations Management on page 284 for more details.
• For license update or activation, you must use the Cluster UUID, and not the Application UUID details.
• Upgrade: To upgrade NKP, see Upgrade NKP.

Note: NKP upgrade through Nutanix marketplace is not supported.

• Refresh: When you upgrade the cluster manually, you can select the Refresh option to view the latest cluster and Kubernetes details as well as the upgraded version details in the widget.
• Delete a cluster: If you have to delete a cluster, to avoid a failure you must ensure this cluster is not managing another cluster or workload. In the App widget, select the delete option to delete the selected cluster.

If there is an error during the delete operation, you can initiate the soft- delete option which ignores the failures and clears the application tiles from Nutanix Marketplace

Creating the Nutanix Kubernetes Platform Management Cluster using CLI

About this task

To create the NKP management cluster, follow these steps:

Note: To create a NKP management cluster when the bootstrap cluster is running on the host, you need a minimum of
2 CPUs and 4GB memory available for successful creation of the management cluster.

Procedure

1. Open the terminal with access to the NKP CLI.
2. (Required for air-gapped environment and optional for non-air-gapped environment) Load the NKP bootstrap cluster container image into the local container runtime store:

docker load --input "nkp-nkp-version/konvoy-bootstrap-image-nkp-version.tar"

The NKP CLI uses a container image to create the bootstrap cluster. In an air- gapped environment, you cannot automatically download the image from the public registry. The image is included in the air-gapped bundle, and you must load the image into the container runtime. 3. Assign a name to your cluster and store it in an environment variable:

Replace Name of the NKP Cluster with the name of the NKP management cluster.

Note: NKP also uses the cluster name for some Kubernetes resources. Ensure that you only use lowercase
alphabets a-z, numbers 0-9, ., and - in the cluster name. For more information, see Kubernetes.

1. Provide the Prism Central endpoint. The NKP CLI uses the Prism Central API to create the NKP cluster, and this API is hosted at the Prism Central endpoint.

The endpoint must be in one of three formats:

• host
• host:port
• A valid URL. For example, https:// host:port

For example:

Replace URL of Prism Central Endpoint with the URL of Prism Central endpoint. 5. Provide the Prism Central credentials:

export NUTANIX_USER=Prism Central Username
export NUTANIX_PASSWORD=Prism Central Password

Replace:

• Prism Central Username with the Prism Central username.
• Prism Central Password with the Prism Central password.

The NKP CLI needs a username and password to access Prism Central.

1. (Optional) Access an HTTPS Prism Central endpoint hosted on a self-signed certificate:

» If the Prism Central endpoint uses a self-signed CA certificate, the CA certificate is untrusted. When creating an NKP cluster, use –additional- trust-bundle to pass only the Prism Central self#signed certificate file. The input must be a Base64-encoded.

Run the following command to encode the plain text SSL certificate file into a Base64-encoded file:

Note: Do not include the full CA bundle, as large files can exceed the 16 KB cloud#init user data limit
and cause node VM creation errors.

For more information, see SSL Certificate Management in Prism Central.

» If you cannot provide a trusted CA certificate that can be verified, allow insecure HTTPS access to the Prism Central endpoint:

export INSECURE=true

This method is not recommended for a production environment, as it skips certificate validation and runs in an insecure mode. 7. Choose the name of a Prism Element cluster:

Replace Name of the Prism Central Cluster with the name of the Prism Element cluster.

The system creates the NKP control plane and worker nodes virtual machines in the Prism Element cluster.

In this step, a single Prism Element cluster is used for both the control plane and worker nodes. However, you can modify the NKP CLI command to use separate Prism Element clusters for the control plane and worker nodes. 8. Choose the name of a Nutanix storage container to use for Kubernetes persistent volumes:

Replace Name of Storage Container with the name of the Nutanix storage container.

By default, Kubernetes persistent volumes are provisioned using Nutanix volumes in a Nutanix storage container. The storage container configuration controls features such as replication, compression, and deduplication. You can dedicate a storage container to your cluster, or share the storage container.

Important: The storage container must be in the Prism Element cluster that you selected in step 4 on
page 741.

1. Choose an IP address for the control plane endpoint:

Replace Dedicated static IP with a static IP address for the control plane endpoint. The NKP cluster control plane consists of multiple nodes, each capable of responding to Kubernetes API requests. However, only one node serves the requests at a time. The control plane endpoint uses a virtual IP

(VIP) address, which is assigned to an active node. This VIP ensures that API requests are always directed to the current active control plane node.

Important: Ensure that the IP address for the control plane endpoint adheres to the following guidelines:

• Routable from the control plane and worker subnets
• A static IP that is not part of a dynamic host configuration protocol (DHCP) or IP address management (IPAM) pool
• Not used for any other purpose

1. (Optional) Configure an external endpoint to access the control plane:

export CONTROLPLANE_EXTERNAL_ENDPOINT_FLAG="
--control-plane-external-endpoint=<IP or DNS name>"

The external endpoint can be used to access the cluster from outside the control plane and worker subnets. The endpoint can be an IP or a fully- qualified domain name.

Note:

• Inside the VPC: To access a cluster in a Virtual Private Cloud (VPC), you can use a floating IP as the external endpoint. Use Prism Central to request a new floating IP, or choose a floating IP that is not associated. Finally, associate the floating IP with the static IP you chose in the previous step.
• To access the cluster in the VPC, use the Bastion VM or any other VM in the same VPC.
• Outside the VPC: To access a cluster from outside the VPC, link the floating IP to an internal IP used as CONTROL_PLANE_ENDPOINT_IP while deploying the cluster. For information on Floating IP, see the topic Request Floating IPs in Flow Virtual Networking.
• Access the cluster in the VPC from outside using updated kubeconfig after creating the cluster.
• To access the UI outside the VPC, you need to request three floating IPs.
• One IP for the bastion
• One IP for passing the –extra-sans flag during cluster creation
• One IP for the UI

1. Choose an IP address range for Kubernetes load balancer services:

Replace IP range in first IP-last IP format with the IP range in first IP-last IP format. For example, 100.0.0.30-100.0.0.56.

The NKP cluster assigns an external IP address from a predefined range to expose a Kubernetes service. The NKP UI uses the first IP address in this range.

Important: Ensure that the IP addresses for the Kubernetes load balancer services adhere to the following
guidelines:

• Routable from the control plane and worker subnets

• A static IP that is not part of a dynamic host configuration protocol (DHCP) or IP address management (IPAM) pool

• Not used for any other purpose

• export SERVICE LB IP RANGE=“ _ _ _; IP range in first IP-last IP format; “

| — | — | — |

1. Choose a subnet for control plane and worker nodes:

Replace Subnet name with the name of the subnet for control plane and worker nodes.

Every control plane and worker node is assigned an IP address from this subnet.

Important: Ensure that the subnet is in the Prism Element cluster that you selected in Step 4 on page 741.

In this step, one subnet is used for both the control plane and the worker nodes. However, you can modify the NKP CLI command to use separate subnets for the control plane and the worker nodes. 13. Choose a Nutanix VM image:

Replace VM Image name with the name of the Nutanix VM image.

If you did not create a Nutanix VM image, you can use a prebuilt VM image. For more information, see Creating a Nutanix VM Image on page 734.

In this step, one Nutanix VM image is used for both the control plane and the worker nodes. However, you can modify the NKP CLI command to use separate Nutanix VM images for the control plane and the worker nodes.

Note: If the VM image was built with FIPS enabled, you also need to enable FIPS in the cluster, as described in
the next step.

1. (Optional) Enable FIPS in the cluster:

export FIPS_FLAG="--fips"

This flag enables FIPS in the cluster.

Note: To enable FIPS in the cluster, the VM image used in the previous step must be built with FIPS enabled. 15. (Required for air-gapped environment and optional for non-air-gapped environment) Use one of the following registry methods to manage the container images:

a. Internal Registry Mirror: If you want NKP to manage the required container images using its internal registry with a bundled image set.

export BUNDLE_FLAGS=" --bundle ./nkp-${nkp-version}/container-images/kommander-
image-bundle-${nkp-version}.tar,./nkp-${nkp-version}/container-images/konvoy-
image-bundle-${nkp-version}.tar,./nkp-${nkp-version}/container-images/nutanix-
product-catalog-${nkp-version}-airgapped.tar"

Replace nkp-version with the NKP version at your site.

NKP provides a bundle that contains all the necessary container images required to bootstrap an NKP cluster. Specify only the required bundle as comma-separated values in the –bundle parameter.

Note: The nutanix-product-catalog-nkp-version-airgapped.tar included in the NKP
air-gapped bundle contains images exclusively for Nutanix Data Services for Kubernetes (NDK). You must
specify the Nutanix product catalog bundle only if you plan to deploy NDK.

b. Local Registry Mirror: If you want to configure a local or user-provided registry mirror manually.

export REGISTRY_MIRROR_FLAGS="
--registry-mirror-url=$REGISTRY_MIRROR_URL \
--registry-mirror-username=$REGISTRY_MIRROR_USERNAME \
--registry-mirror-password=$REGISTRY_MIRROR_PASSWORD \
--registry-mirror-cacert=$REGISTRY_MIRROR_CA \

"

By default, an NKP cluster pulls container images from public registries. If these registries are not available, or you do not use them, NKP can pull the images from a registry mirror. To use a registry mirror, follow the steps in Preparing a Local Registry Mirror on page 736, and export the following environment variables.

For more information about the registry types, see Air-Gapped or Non-Air- Gapped Environment on page 22. 16. (Required for air-gapped environment and optional for non-air-gapped environment) Enable the air-gapped mode:

export AIRGAPPED=true

The air-gapped mode runs the NKP UI, and other NKP applications in an air- gapped environment. 17. (Optional) Use registry credentials:

export REGISTRY_FLAGS="
--registry-url=<Registry URL> \
--registry-username=<Registry username> \
--registry-password=<Registry password> \
--registry-cacert=<Path to registry CA certificate file> \
"

By default, an NKP cluster pulls container images from public registries. However, some registries, such as Docker Hub, limit the number of image pulls unless you provide the credentials.

Tip: If you do not use a local registry mirror or an internal registry, Nutanix recommends that you provide
credentials for Docker Hub.

1. (Optional) Choose networks for the Kubernetes pods and services:

export KUBERNETES_PODS_NETWORK=IP range in CIDR format
export KUBERNETES_SERVICES_NETWORK=IP range in CIDR format

Replace IP range in CIDR format with the IP range in CIDR format. For example, 192.168.0.0/16.

Every NKP cluster has two networks; one for Kubernetes pods, and another for Kubernetes services. These networks are internal to the cluster. Every Kubernetes pod is assigned an IP address from the Kubernetes pods network, and every Kubernetes service is assigned an IP address from the Kubernetes services network. The network size is based on the number of pods and services in a cluster.

Important:

• The Kubernetes pods network and Kubernetes services network must not overlap with each other, or with your control plane and worker nodes subnets.
• After cluster creation, you cannot change the pods and services subnets.

By default, the pods network is 192.168.0.0/16, and the services network is 10.96.0.0/12. These network sizes allow up to approximately 65 thousand pods, and one million services. 19. (Optional) Enable SSH access to nodes:

Replace:

• username with the SSH user name
• path to SSH public key file with the location of SSH public key file

Provide a username and public SSH key to enable SSH access with these credentials to all NKP cluster nodes. NKP creates the user on every cluster node, and adds the public SSH key to the authorized SSH keys of the user. By default, NKP cluster nodes allow SSH public key authentication.

Provide the public SSH key as a path to a file. 20. (Optional) To disable the automatic onboarding of clusters:

export PC_ONBOARDING_FLAGS="
--onboard-to-prism-central=false \
"

This is an optional step to disable the automatic onboarding of clusters, which is enabled by default.

Note: If the konnector agent pod is stuck in Init state, you must enable karbon-core service in PC in version
7.3 and upgrade the karbon-core version to 2.10.2.

1. Configure Network Time Protocol (NTP) servers on Nutanix Kubernetes Platform (NKP) clusters.

Configure Network Time Protocol (NTP) on Nutanix Kubernetes Platform (NKP) clusters to synchronize system time is mandatory across all the nodes in an air-gapped environment.

Configure NTP servers on Nutanix Kubernetes Platform (NKP) clusters if you cannot use the default public NTP servers that are set in the operating system or prefer to use your own custom NTP server in non-air-gapped environments.

Use –ntp-servers flag to configure a comma separated list of NTP servers across all the nodes. Each entry must be a Fully Qualified Domain Name (FQDN) or an IP address (IPv4 or IPv6). This list overrides any default NTP settings preconfigured in the machine image. For example, export NTP_FLAGS=“–ntp- servers=pool.ntp.org,time.google.com \ “. 22. Create a management cluster:

nkp create cluster nutanix \
--self-managed \
--airgapped=${AIRGAPPED:-false} \
\
--cluster-name=${CLUSTER_NAME} \
--endpoint=${NUTANIX_ENDPOINT} \
--insecure=${INSECURE:=false} \
--control-plane-endpoint-ip=${CONTROLPLANE_IP} \
--control-plane-external-endpoint=${CONTROLPLANE_EXTERNAL_ENDPOINT_FLAG} \
--csi-storage-container=${STORAGE_CONTAINER_NAME} \
--kubernetes-pod-network-cidr=${KUBERNETES_PODS_NETWORK:-"192.168.0.0/16"} \
--kubernetes-service-cidr=${KUBERNETES_SERVICES_NETWORK:-"10.96.0.0/12"} \
--kubernetes-service-load-balancer-ip-range=${SERVICE_LB_IP_RANGE} \
\
--control-plane-prism-element-cluster=${NUTANIX_CLUSTER} \
--control-plane-subnets=${SUBNET} \
--control-plane-vm-image=${VM_IMAGE} \
\
--worker-prism-element-cluster=${NUTANIX_CLUSTER} \
--worker-subnets=${SUBNET} \
--worker-vm-image=${VM_IMAGE} \
--ssh-username=${SSH_USERNAME} \
--ssh-public-key-file=${SSH_PUBLIC_KEY} \

\
${NTP_FLAGS} \
${BUNDLE_FLAGS} \
${REGISTRY_MIRROR_FLAGS} \
\
${REGISTRY_FLAGS}
\
${FIPS_FLAG}
Important: Do not use both BUNDLE_FLAG and REGISTRY_MIRROR_FLAG simultaneously. Use these flags
based on how you manage the container images.

To assign values to flags, the NKP CLI command creates a management cluster with the configuration and environment variables that you exported.

If you require the use of an HTTP/S proxy in your environment, add the flags –http-proxy, –https- proxy, and –no-proxy. For more information, see Cluster Creation with HTTP or HTTPS Proxy on page 699.

Preflight Checks

For example, if the VM image name is missing from the worker and control plane nodes, the following error is displayed in the output while creating cluster resources:

You can skip a specific preflight check using a flag with the name of that check. For example,

--skip-preflight-checks=NutanixVMImage

If you want to skip more than one preflight check, specify the flag for each check individually. For example,

--skip-preflight-checks=NutanixVMImage --skip-preflight-
checks=NutanixStorageContainer

• Verifies that Prism Central is reachable and the credentials provided are valid.

The credentials are stored in their secret.

• Verifies that the VM image is present in Prism Central.
• Verifies that the Kubernetes cluster version is part of the VM image name. For example, Kubernetes version 1.35.x.
• If you configure Container Storage Interface (CSI), NKP verifies that the storage containers specified in the configuration exist in Prism Central.

The Nutanix CSI driver uses storage containers to provision persistent volumes (PV).

• Validates your Pod and Service CIDR configurations. This check ensures that the Pod and Service CIDRs are large enough to prevent IP exhaustion and that they do not overlap each other, the Node subnets, or the control plane endpoint to prevent conflicts.

Preflight checks on the local registry:

• Verifies that the local registry is reachable and the credentials provided are valid.

The credentials are stored in their secret.

Creating a Nutanix Cluster With Custom Cilium Configuration

About this task

Warning: For this procedure, you must modify .yaml files directly. Therefore, this task is recommended only for
advanced users.

By default, Nutanix Kubernetes Platform (NKP) installs the Cilium add-on with the default configuration while creating a cluster. However, you can override the default configuration with a custom configuration for advanced use- cases. For example, you can create a Nutanix cluster using NKP with transparent encryption (Wireguard).

You can create a NKP cluster with a custom Cilium configuration in both air- gapped and non-air-gapped environments.

Before you begin

• You must customize the configuration before you create a cluster. Only the custom configuration that you provide while creating the cluster is applied. If you update the cluster specification or the Cilium CNI configmap after you create a cluster, the updates are not applied to an installed Cilium add-on.
• This task must be performed only by advanced users as you are required to modify YAML files directly.

Procedure

1. Generate Cluster Objects.

You must set the target namespace with the name of the workspace you are creating the cluster in, using the nkp create cluster nutanix … –dry-run –output=yaml > cluster.yaml command.

Depending upon your infrastructure, this command generates a set of cluster objects that can be customized for advanced use-cases. 2. Create a file named values.yaml with the following helm values. You must edit the values as required:

cni:
exclusive: false
hubble:
enabled: true
tls:
auto:
enabled: true # enable automatic TLS certificate generation
method: cronJob # auto generate certificates using cronJob method
certValidityDuration: 60 # certificates validity duration in days (default 2
months)
schedule: "0 0 1 * *" # schedule on the 1st day regeneration of each
month
relay:
enabled: true
tls:
server:
enabled: true
mtls: true
image:
useDigest: false
priorityClassName: system-cluster-critical
ipam:
mode: kubernetes
image:
useDigest: false
operator:
image:
useDigest: false
certgen:
image:
useDigest: false
socketLB:
hostNamespaceOnly: true
envoy:
image:
useDigest: false
k8sServiceHost: "{{ trimPrefix .ControlPlaneEndpoint.Host "https://" }}"
k8sServicePort: "{{ .ControlPlaneEndpoint.Port }}"
kubeProxyReplacement: true
tunnelProtocol: geneve
loadBalancer:
mode: dsr
dsrDispatch: geneve

For example, add the following values to the values.yaml file to enable Cilium with transparent encryption (Wireguard). As a result, Cilium encrypts network traffic and sets the encryption method as WireGuard:

encryption:
enabled: true
type: wireguard
Note: The values.yaml file includes all the required Helm values needed to configure the Cilium add-on. NKP
uses these values from the file during cluster creation to install and configure Cilium.
Warning: You cannot modify the Cilium IPAM configuration, mode, Pod CIDR, or Service CIDR of a running
cluster. For more information, see IPAM Configuration Change Limitation on page 544.

1. Create an additional file named preflight-values.yaml with these values to ensure completion of the preflight checks during an upgrade of this cluster:

agent: false
operator:
enabled: false
preflight:
enabled: true
envoy:
image:
useDigest: false
image:
useDigest: false
k8sServiceHost: "{{ trimPrefix .ControlPlaneEndpoint.Host "https://" }}"
k8sServicePort: "{{ .ControlPlaneEndpoint.Port }}"

1. Create a ConfigMap with the required Helm values, if you do not have a ConfigMap YAML file already:

kubectl create configmap <CLUSTER_NAME>-cilium-cni-helm-values-template
--from-file=values.yaml=values.yaml
--from-file=preflight-values.yaml=preflight-values.yaml -n <CLUSTER_NAMESPACE>

You can skip the previous step if you already have a ConfigMap YAML file prepared. To apply it, edit it as required to add helm values. Ensure the namespace in the ConfigMap YAML file is set to target the cluster’s namespace and apply it using this command:

kubectl apply <ConfigMap YAML Filepath>

1. Edit your cluster YAML file as follows:

You must first specify Cilium as the CNI provider.

Reference a ConfigMap that contains your custom Helm values. Ensure the ConfigMap name is same as the one created in the previous step:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: <NAME>
namespace: <CLUSTER_NAMESPACE>
spec:
topology:
variables:
- name: clusterConfig
value:
addons:
cni:
provider: Cilium
strategy: HelmAddon
values:
sourceRef:
name: <CLUSTER_NAME>-cilium-cni-helm-values-template
kind: ConfigMap

1. Apply the edited cluster YAML files:

kubectl apply -f cluster.yaml

The cluster is created with the custom Cilium configuration.

Note: If the self-managed flag was added in the dry-run, follow the steps to make the cluster a Self-
Managed Cluster on page 20

Setting up the Nutanix Kubernetes Platform User Interface Access

About this task

To set up the NKP UI access, follow these steps:

Procedure

1. Open a terminal with access to the NKP CLI.
2. Access the NKP UI in Kommander and retrieve your credentials:

nkp get dashboard --kubeconfig=${CLUSTER_NAME}.conf

Use these static credentials only to configure an external identity provider. For more information, see Identity Providers. Treat them as backup credentials, not for regular UI access. 3. Rotate the dashboard password:

nkp rotate dashboard-credentials

The CLI prompts for confirmation. Respond with y to continue or N to cancel:

This will rotate the NKP dashboard (static admin) password.
Existing sessions and stored credentials will stop working after rotation.
Are you sure you want to continue? [y/N]: y
The dashboard credentials have been rotated. Run nkp get dashboard to retrieve the
new password.

To skip this prompt in non-interactive terminals, add the –yes flag:

nkp rotate dashboard-credentials --yes

1. Retrieve the new dashboard credentials:

nkp get dashboard --kubeconfig=${CLUSTER_NAME}.conf

For more information, see Rotating the NKP Dashboard Password on page 308.

Updating Prism Central Credentials on Deployed NKP Clusters

Before you begin

Before you update your Prism Central credentials on deployed Nutanix Kubernetes Platform (NKP) clusters, ensure all requirements of Prism Central Credentials are met . For more information, see Prism Central Credential Requirements.

About this task

To update the Prism Central credentials using CLI, follow these steps:

Note: For an automated script to update Prism Central credentials on NKP clusters, see KB-18296.

Procedure

1. Open the terminal with access to the NKP CLI.
2. Scale down the following associated pods:

a. Scale down the following deployments or controllers of the pods:

kubectl scale deployment capx-controller-manager -n capx-system --replicas=0
kubectl scale deployment cluster-api-runtime-extensions-nutanix -n caren-system
--replicas=0
kubectl scale deployment nutanix-cloud-controller-manager -n kube-system --
replicas=0
kubectl scale deployment nutanix-csi-controller -n ntnx-system --replicas=0
kubectl scale deployment nutanix-csi-controller -n ntnx-system --replicas=0 --
kubeconfig=$KUBECONFIG_WORKLOAD
kubectl scale deployment nutanix-cloud-controller-manager -n kube-system --
replicas=0 --kubeconfig=$KUBECONFIG_WORKLOAD

b. Scale down the Container Storage Interface (CSI) pods on the management cluster:

kubectl -n ntnx-system patch daemonset nutanix-csi-node -p '{"spec":
{"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'

c. Scale down the CSI pods on each managed cluster:

kubectl -n ntnx-system patch daemonset nutanix-csi-node -p '{"spec":
{"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}' --
kubeconfig=$KUBECONFIG_WORKLOAD

1. Update the global-nutanix-credentials:

a. Update the secret for the following global-nutanix-credentials:

kubectl get secrets global-nutanix-credentials -n capx-system -o yaml
apiVersion: v1
kind: Secret
metadata:
labels:
clusterctl.cluster.x-k8s.io/move: ""
dkp-infrastructure-provider-type: nutanix-secret
name: global-nutanix-credentials
namespace: capx-system
type: Opaque
data:
additionalTrustBundle: ""
insecure: base 64 encoded Insecure Mode
password: base 64 encoded PC Password
prismURL: base 64 encoded PC Endpoint
username: base 64 encoded PC Username

b. Store the password in base64 format and encode the new pass:

echo -n 'NEW_PASSWORD' | base64
TkVXX1BBU1NXT1JE

c. Update the global-nutanix-credentials with a new password:

kubectl edit secrets global-nutanix-credentials -n capx-system -o yaml
apiVersion: v1
kind: Secret
metadata:
labels:
clusterctl.cluster.x-k8s.io/move: ""
dkp-infrastructure-provider-type: nutanix-secret
name: global-nutanix-credentials
namespace: capx-system
type: Opaque
data:
additionalTrustBundle: ""
insecure: base 64 encoded Insecure Mode
password: base 64 encoded PC Password
prismURL: base 64 encoded PC Endpoint
username: base 64 encoded PC Username

1. Update the individual cluster credentials (secrets). Ensure that the secrets are in the formats cluster-name-pc-credentials and cluster-name-pc- credentials-for-csi.
2. Update the Prism Central credentials for the management cluster:

a. Get the secret:

kubectl get secrets ${MANAGEMENT_CLUSTER_NAME}-pc-credentials -o yaml
apiVersion: v1
kind: Secret
metadata:
finalizers:
- nutanixcluster/infrastructure.cluster.x-k8s.io
labels:
cluster.x-k8s.io/provider: nutanix
konvoy.d2iq.io/cluster-name: ${MANAGEMENT_CLUSTER_NAME}
konvoy.d2iq.io/provider: nutanix
name: ${MANAGEMENT_CLUSTER_NAME}-pc-credentials
namespace: default
ownerReferences:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixCluster
name: ${MANAGEMENT_CLUSTER_NAME}-2smbz
uid: 8a9d1bba-02d5-4287-b3d8-284199d28a18
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: ${MANAGEMENT_CLUSTER_NAME}
uid: a74225d9-4367-4fc2-b424-53a0bd981020
type: Opaque
data:
credentials: base64 encoded PC credentials

Use the value of the credentials attribute in the secret:

echo base 64 String | base64 -d
[
{
"type": "basic_auth",
"data": {
"prismCentral": {
"username": "PC Username",
"password": "PC Password"
}
}
}
]

b. Decode the secret value and save it in a file:

c. Open the file using a text editor and modify the password:

cat update_new_password.txt
[
{
"type": "basic_auth",
"data": {
"prismCentral": {
"username": "PC Username",
"password": "PC Password"
}

}
}
]

d. Encode the updated secret value:

e. Edit the secret and update the secret with the new value:

kubectl edit secrets ${MANAGEMENT_CLUSTER_NAME}-pc-credentials -o yaml

f. Validate the changes:

apiVersion: v1
data:
credentials: base64 encoded PC Credentials
kind: Secret
metadata:
creationTimestamp: "2024-09-30T16:16:07Z"
finalizers:
- nutanixcluster/infrastructure.cluster.x-k8s.io
labels:
cluster.x-k8s.io/provider: nutanix
konvoy.d2iq.io/cluster-name: ${MANAGEMENT_CLUSTER_NAME}
konvoy.d2iq.io/provider: nutanix
name: ${MANAGEMENT_CLUSTER_NAME}-pc-credentials
namespace: default
ownerReferences:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixCluster
name: ${MANAGEMENT_CLUSTER_NAME}-2smbz
uid: 8a9d1bba-02d5-4287-b3d8-284199d28a18
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: ${MANAGEMENT_CLUSTER_NAME}
uid: a74225d9-4367-4fc2-b424-53a0bd981020
resourceVersion: "4982392"
uid: d382fa23-5e0e-46d7-a1c0-b5b3ee0bddd3
type: Opaque

1. Update the Prism Central credentials for the workload clusters:

a. Get the secret:

kubectl get secrets ${WORKLOAD_CLUSTER_NAME}-pc-credentials -n kommander-default-
workspace -o yaml
apiVersion: v1
data:
credentials: base64 encoded PC credentials
kind: Secret
metadata:
creationTimestamp: "2024-09-30T21:46:05Z"
finalizers:
- nutanixcluster/infrastructure.cluster.x-k8s.io
labels:
cluster.x-k8s.io/provider: nutanix
name: ${MANAGED_CLUSTER_NAME}-pc-credentials
namespace: kommander-default-workspace
ownerReferences:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixCluster
name: ${MANAGED_CLUSTER_NAME}-7bqc2

uid: b4832f32-970b-4990-8d26-e5b134bca625
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: ${MANAGED_CLUSTER_NAME}
uid: 435ab407-b366-4bd7-b5e3-4b7c125244e8
resourceVersion: "5006043"
uid: d132a4aa-7906-4584-b178-effdfcf3c5ce
type: Opaque

echo base 64 String | base64 -d
[{"type":"basic_auth","data":{"prismCentral":{"username":"PC
Username","password":"PC Password"}}}]

Ensure that the secrets are in the format cluster-name-pc-credentials.

b. Decode the secret and save it in a file:

echo -n '[{"type":"basic_auth","data":{"prismCentral":{"username":"PC
Username","password":"PC Password"}}}]' | base64
base 64 String

c. Edit the secret and update the secret with the new value:

kubectl edit secrets ${WORKLOAD_CLUSTER_NAME}-pc-credentials -o yaml

d. Validate the changes:

kubectl edit secrets ${WORKLOAD_CLUSTER_NAME}-pc-credentials -n
apiVersion: v1
data:
credentials: base 64 encoded PC Password
kind: Secret
metadata:
creationTimestamp: "2024-09-30T21:46:05Z"
finalizers:
- nutanixcluster/infrastructure.cluster.x-k8s.io
labels:
cluster.x-k8s.io/provider: nutanix
name: ${WORKLOAD_CLUSTER_NAME}-pc-credentials
namespace: kommander-default-workspace
ownerReferences:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixCluster
name: ${WORKLOAD_CLUSTER_NAME}-7bqc2
uid: b4832f32-970b-4990-8d26-e5b134bca625
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: ${WORKLOAD_CLUSTER_NAME}
uid: 435ab407-b366-4bd7-b5e3-4b7c125244e8
resourceVersion: "5006043"
uid: d132a4aa-7906-4584-b178-effdfcf3c5ce
type: Opaque

1. Update the CSI credentials for the management cluster:

a. Update the CSI credentials:

kubectl get secrets ${MANAGEMENT_CLUSTER_NAME}-pc-credentials-for-csi -o yaml
apiVersion: v1
data:
key: Key ID
kind: Secret
metadata:
creationTimestamp: "2024-09-30T16:16:06Z"
labels:
cluster.x-k8s.io/provider: nutanix
konvoy.d2iq.io/cluster-name: ${MANAGEMENT_CLUSTER_NAME}
konvoy.d2iq.io/provider: nutanix
name: ${MANAGEMENT_CLUSTER_NAME}-pc-credentials-for-csi
namespace: default
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: ${MANAGEMENT_CLUSTER_NAME}
uid: a74225d9-4367-4fc2-b424-53a0bd981020
resourceVersion: "5000354"
uid: e76fe412-d904-4b5f-b029-4800571898b4
type: Opaque

Ensure that you use the cluster-name-for-csi format for the management cluster name.

b. Validate the base64 value:

echo base 64 String | base64 -d
PC IP:9440:PC Username:PC Password

c. Update the password and encode in base64:

d. Edit the secret and update the base64 value:

kubectl edit secrets ${MANAGEMENT_CLUSTER_NAME}-pc-credentials-for-csi -o yaml

e. Validate the updated secret:

kubectl get secrets ${MANAGEMENT_CLUSTER_NAME}-pc-credentials-for-csi -o yaml
apiVersion: v1
data:
key: NEW_BASE64ENCODED_PASSWORD
kind: Secret
metadata:
creationTimestamp: "2024-09-30T16:16:06Z"
labels:
cluster.x-k8s.io/provider: nutanix
konvoy.d2iq.io/cluster-name: ${MANAGEMENT_CLUSTER_NAME}
konvoy.d2iq.io/provider: nutanix
name: ${MANAGEMENT_CLUSTER_NAME}-pc-credentials-for-csi
namespace: default
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: ${MANAGEMENT_CLUSTER_NAME}
uid: a74225d9-4367-4fc2-b424-53a0bd981020
resourceVersion: "5000354"
uid: e76fe412-d904-4b5f-b029-4800571898b4
type: Opaque

1. Update the konnector-agent credentials for the management cluster:

a. Update the konnector-agent credentials:

kubectl get secret ${MANAGEMENT_CLUSTER_NAME}-pc-credentials-for-konnector-agent
-o yaml

• echo -n ‘; PC IP; :9440:admin:NEW PASSWORD’; base64; base 64 String; Col5

| — | — | — | — | — | — |

apiVersion: v1
data:
password: base64encoded-password
username: base64encoded-username
kind: Secret
metadata:
creationTimestamp: "2025-12-18T04:46:58Z"
labels:
cluster.x-k8s.io/provider: nutanix
konvoy.d2iq.io/cluster-name: ${MANAGEMENT_CLUSTER_NAME}
konvoy.d2iq.io/provider: nutanix
name: ${MANAGEMENT_CLUSTER_NAME}-pc-credentials-for-konnector-agent
namespace: default
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: ${MANAGEMENT_CLUSTER_NAME}
uid: 3075d1d4-2357-4ffc-a88a-3563c87513d8
resourceVersion: "10962"
uid: 6bd05762-3007-4c39-a56a-707564f52b8d
type: Opaque

Ensure your management cluster uses this format for the cluster name: ${MANAGEMENT_CLUSTER_NAME}- pc-credentials-for-konnector-agent

b. Edit the secret with an updated base64 password:

kubectl edit secrets ${MANAGEMENT_CLUSTER_NAME}-pc-credentials-for-konnector-
agent -o yaml

c. Validate the updated secret:

kubectl get secret ${MANAGEMENT_CLUSTER_NAME}-pc-credentials-for-konnector-agent
-o yaml
apiVersion: v1
data:
password: UPDATED-BASE64-ENCODED-PASSWORD
username: base64encoded-username
kind: Secret
metadata:
creationTimestamp: "2025-12-18T04:46:58Z"
labels:
cluster.x-k8s.io/provider: nutanix
konvoy.d2iq.io/cluster-name: ${MANAGEMENT_CLUSTER_NAME}
konvoy.d2iq.io/provider: nutanix
name: ${MANAGEMENT_CLUSTER_NAME}-pc-credentials-for-konnector-agent
namespace: default
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: ${MANAGEMENT_CLUSTER_NAME}
uid: 3075d1d4-2357-4ffc-a88a-3563c87513d8
resourceVersion: "10962"
uid: 6bd05762-3007-4c39-a56a-707564f52b8d
type: Opaque

1. Update the Konnector-agent credentials for each workload cluster:

a. Get the workload cluster konnector-agent credentials:

kubectl get secret konnector-agent -n ntnx-system --kubeconfig <WORKLOAD-CLUSTER-
KUBECONFIG> -o yaml
apiVersion: v1
data:
password: base64encoded-password
username: base64encoded-username
kind: Secret
metadata:
creationTimestamp: "2025-12-18T04:33:19Z"
name: konnector-agent
namespace: ntnx-system
resourceVersion: "433"
uid: ff0d18db-7ff8-4df3-a641-1d5fc474d1aa
type: Opaque

b. Edit the secret and update the base64 value:

kubectl edit secret konnector-agent -n ntnx-system --kubeconfig <WORKLOAD-
CLUSTER-KUBECONFIG>

Though the -kubeconfig changes for each cluster, the steps remain the same. 10. Update the CSI credentials for each workload cluster in the management cluster:

a. Update the CSI credentials:

kubectl get secret ${WORKLOAD_CLUSTER_NAME}-pc-credentials-for-csi -n kommander-
default-workspace -o yaml
apiVersion: v1
data:
key: Key ID
kind: Secret
metadata:
creationTimestamp: "2024-09-30T21:46:05Z"
labels:
cluster.x-k8s.io/provider: nutanix
name: ${WORKLOAD_CLUSTER_NAME}-pc-credentials-for-csi
namespace: kommander-default-workspace
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: ${WORKLOAD_CLUSTER_NAME}
uid: 435ab407-b366-4bd7-b5e3-4b7c125244e8
resourceVersion: "5008388"
uid: bdf259ec-fa26-48a7-9be9-7712d3614802
type: Opaque

Ensure that you use the cluster-name-for-csi format for workload cluster name.

b. Validate the base64 value:

c. Update the password and encode in base64:

d. Edit the secret and update the base64 value:

kubectl edit secret ${WORKLOAD_CLUSTER_NAME}-pc-credentials-for-csi -n
kommander-default-workspace -o yaml
kubectl get secret ${WORKLOAD_CLUSTER_NAME}-pc-credentials-for-csi -n kommander-
default-workspace -o yaml
apiVersion: v1
data:
key: base64 encoded New Password
kind: Secret

• echo; base 64 String; base64 -d; PC IP; :9440:; PC Username; :; PC Password; Col9

| — | — | — | — | — | — | — | — | — | — |

• echo -n ‘; PC IP; :9440:admin:NEW PASSWORD’; base64; base 64 String; Col5

| — | — | — | — | — | — |

metadata:
creationTimestamp: "2024-09-30T21:46:05Z"
labels:
cluster.x-k8s.io/provider: nutanix
name: ${WORKLOAD_CLUSTER_NAME}-pc-credentials-for-csi
namespace: kommander-default-workspace
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: ${WORKLOAD_CLUSTER_NAME}
uid: 435ab407-b366-4bd7-b5e3-4b7c125244e8
resourceVersion: "5008388"
uid: bdf259ec-fa26-48a7-9be9-7712d3614802
type: Opaque

1. Update the cloud controller manager (CCM) and CSI credentials at the individual cluster level:

a. Update the CCM and CSI credentials:

kubectl get secrets nutanix-ccm-credentials -n kube-system --kubeconfig=
$KUBECONFIG_MANAGEMENT -o yaml
apiVersion: v1
data:
credentials: base64 encoded PC Credentials
kind: Secret
metadata:
creationTimestamp: "2024-09-30T16:12:20Z"
name: nutanix-ccm-credentials
namespace: kube-system
resourceVersion: "5017728"
uid: 3290249e-facb-45e3-a785-62ac620d5861
type: Opaque

For each cluster, –kubeconfig changes. However, the steps remain the same.

b. Validate the base64 value:

echo base 64 String | base64 -d
[
{
"type": "basic_auth",
"data": {
"prismCentral": {
"username": "PC Username",
"password": "PC Password"
}
}
}
]
echo base 64 String | base64 -d > update_ccm_password.txt
cat update_ccm_password.txt
[
{
"type": "basic_auth",
"data": {
"prismCentral": {
"username": "PC Username",
"password": "PC Password"
}
}
}
]
cat update_ccm_password.txt | base64 base 64 String

kubectl edit secrets nutanix-ccm-credentials -n kube-system --kubeconfig=
$KUBECONFIG_MANAGEMENT -o yaml

kubectl get secrets nutanix-ccm-credentials -n kube-system --kubeconfig=
$KUBECONFIG_MANAGEMENT -o yaml
apiVersion: v1
data:
credentials: base64 encoded New Password
kind: Secret
metadata:
creationTimestamp: "2024-09-30T16:12:20Z"
name: nutanix-ccm-credentials
namespace: kube-system
resourceVersion: "5017728"
uid: 3290249e-facb-45e3-a785-62ac620d5861
type: Opaque

c. Update the CSI password:

kubectl get secrets nutanix-csi-credentials -n ntnx-system -o yaml
apiVersion: v1
data:
key: Key ID
kind: Secret
metadata:
creationTimestamp: "2024-09-30T16:12:20Z"
name: nutanix-csi-credentials
namespace: ntnx-system
resourceVersion: "5023111"
uid: e78d3eef-3d86-4c26-966f-4672531b30aa
type: Opaque

echo base 64 String | base64 -d
PC IP:9440:PC Username:PC Password
echo -n 'PC IP:9440:admin:NEW_PASSWORD' | base64 base 64 String

kubectl edit secrets nutanix-csi-credentials -n ntnx-system -o yaml
apiVersion: v1
data:
key: base64 encoded New Password
kind: Secret
metadata:
creationTimestamp: "2024-09-30T16:12:20Z"
name: nutanix-csi-credentials
namespace: ntnx-system
resourceVersion: "5023111"
uid: e78d3eef-3d86-4c26-966f-4672531b30aa
type: Opaque

1. Scale up the pods on the management cluster:

kubectl -n ntnx-system patch daemonset nutanix-csi-node --type json -p='[{"op":
"remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]'
kubectl scale deployment cluster-api-runtime-extensions-nutanix -n caren-system --
replicas=1
kubectl scale deployment capx-controller-manager -n capx-system --replicas=1
kubectl scale deployment nutanix-csi-controller -n ntnx-system --replicas=1
kubectl scale deployment nutanix-cloud-controller-manager -n kube-system --
replicas=1

1. Scale up the pods on the workload cluster:

kubectl -n ntnx-system patch daemonset nutanix-csi-node --type json -p='[{"op":
"remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]' --kubeconfig=
$KUBECONFIG_WORKLOAD
kubectl scale deployment nutanix-cloud-controller-manager -n kube-system --
replicas=1 --kubeconfig=$KUBECONFIG_WORKLOAD
kubectl scale deployment nutanix-csi-controller -n ntnx-system --replicas=1 --
kubeconfig=$KUBECONFIG_WORKLOAD

Pre-provisioned Infrastructure

Configuration types for installing the Nutanix Kubernetes Platform (NKP) on a Pre-provisioned Infrastructure.

Create a Kubernetes cluster on pre-provisioned nodes in a bare metal infrastructure.

Completing this procedure results in a Kubernetes cluster that includes a Container Networking Interface (CNI) and a Local Persistence Volume Static Provisioner that is ready for workload deployment.

For more information, see:

• Container Networking Interface (CNI): <https://docs.projectcalico.org/>
• Local Persistence Volume Static Provisioner: <<https://github.com/kubernetes-> sigs/sig-storage-local-static-> provisioner

Before moving to a production environment, you might add applications for logging and monitoring, storage, security, and other functions. You can use NKP to select and deploy applications or deploy your own. For more information, see Deploying Platform Applications Using CLI on page 353.

Pre-provisioned Prerequisites and Environment Variables

Pre-provisioning is the process of setting up an environment that authorized users, devices, and servers can access. Network provisioning primarily concerns connectivity and security, which means a heavy focus on device and identity management. Pre-provisioning can bring enterprises greater efficiency and more secure operations.

A cloud-based or on-premises server must first be provisioned with the correct data, software, and configuration to function on a network.

The steps in this process typically include:

• Installing an operating system, device drivers, and partitioning and setup tools
• Installing enterprise software and applications
• Setting parameters such as IP addresses
• Performing partitioning or installation of virtualization software
• Connectivity, whether an air-gapped or non-air-gapped environment, meaning it is connected to the internet

Pre-provisioned: Prerequisites

Infrastructure and machine requirements will be required to fulfill all the prerequisites for a successful implementation in a Pre-provisioned environment. Read all the sections on this page to ensure you have met all prerequisites. Before you begin using Nutanix Kubernetes Platform, you must have the following set:

• An x86_64-based Linux or macOS machine.
• The nkp binary for Linux or macOS.
• kubectl for interacting with the running cluster.
• Pre-provisioned hosts with SSH access enabled.
• An unencrypted SSH private key, whose public key is configured on the above hosts.
• A Container engine/runtime installed is required to install NKP and bootstrap:
• Docker container engine version 27.4.0 installed for Linux or MacOS. For more information, see https:// docs.docker.com/get-docker/.

Docker runs in a virtual machine which needs configured with at least 8 GB of memory.

• Podman Version 4.0 or later for Linux. For more information, see <https://podman.io/getting-started/> installation. For host requirements, see <https://kind.sigs.k8s.io/docs/user/rootless/#host-requirements>.
• Air-Gapped Environment Setup:
• NKP Bundle: Download and extract the complete NKP Air-gapped Bundle (NKP Air-gapped Bundle for this release (for example, nkp-air-gapped-bundle_nkp- version_linux_amd64.tar.gz) to load the registry).
• Local Registry: Set up a working registry.local on a bastion host or another accessible machine to host the loaded images.
• Additional steps: Follow the specific guidance in the Pre-provisioned Air- gapped: Prerequisites and Docker Registry documentation.
• Ensure your Control Plane Nodes and Worker Nodes Resource Requirements for Nutanix Kubernetes Platform on page 721 meet the specific CPU, RAM, and Disk specifications.

When in an air-gapped environment, you must also follow the steps described in Pre-provisioned Air-gapped: Prerequisites on page 795 and Docker Registry as a prerequisite.

Warning: NKP uses localvolumeprovisioner as the default storage provider. However,
localvolumeprovisioner is not suitable for production use. Use a Kubernetes CSI compatible storage that is
suitable for production.

You can choose from any of the storage options available for Kubernetes. To disable the default that Konvoy deploys, set the default StorageClass localvolumeprovisioner as non-default. Then, set your newly created StorageClass as the default by following the commands in the Kubernetes documentation called Changing the Default Storage Class (<<https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-> class/>).

Machine Specifications

You need to have at least three Control Plane Machines.

Each control plane machine must have the following:

• 4 cores
• 16 GB memory

• Approximately 80 GB of free space for the volume used for /var/lib/kubelet and /var/lib/containerd.
• 15% free space on the root file system.
• Multiple ports are open as described in the NKP Ports and Protocols page.
• firewalld systemd service disabled. If it exists and is enabled, use the commands systemctl stop firewalld then systemctl disable firewalld, so that firewalld remains disabled after the machine restarts.

Note: Swap is disabled. The kubelet does not have generally available support for swap. Due to variable commands,
refer to your operating system documentation.

Worker Machines

You need to have at least four worker machines. The specific number of worker machines required for your environment can vary depending on the cluster workload and size of the machines.

Each worker machine must have the following:

• 8 cores
• 32 GiB memory
• Around 80 GiB of free space for the volume used for /var/lib/kubelet and /var/lib/containerd
• 15% free space on the root file system
• If you plan to use local volume provisioning to provide persistent volumes for your workloads, you must mount at least four volumes to the /mnt/disks/ mount point on each machine. Each volume must have at least 55 GB capacity.
• Ensure your disk meets the resource requirements for Rook Ceph in Block mode for ObjectStorageDaemons (<<https://rook.io/docs/rook/v1.10/CRDs/Cluster/ceph-cluster-crd/#storage-> selection-settings>) as specified in the requirements table (Rook Ceph: Configuration on page 682).
• Multiple ports are open as described in the NKP Ports and Protocols page.
• firewalld systemd service disabled. If it exists and is enabled, use the commands systemctl stop firewalld then systemctl disable firewalld, so that firewalld remains disabled after the machine restarts.

Note: Swap is disabled. The kubelet does not generally support swap. Due to variable commands, refer to your
operating system documentation.

Pre-provisioned: Defining the Set Infrastructure

About this task

The Konvoy component of Nutanix Kubernetes Platform (NKP) must know how to access your cluster hosts, so you must define the cluster hosts and infrastructure. This is done using inventory resources. For initial cluster creation, you must define a control-plane and at least one worker pool for air-gapped and non-air-gapped environments.

Complete the steps to set the necessary environment variables and specify the control plane and worker nodes:

Procedure

1. Export the following environment variables, ensuring that all control plane and worker nodes are included:

export CONTROL_PLANE_1_ADDRESS="<control-plane-address-1>"
export CONTROL_PLANE_2_ADDRESS="<control-plane-address-2>"
export CONTROL_PLANE_3_ADDRESS="<control-plane-address-3>"
export WORKER_1_ADDRESS="<worker-address-1>"
export WORKER_2_ADDRESS="<worker-address-2>"
export WORKER_3_ADDRESS="<worker-address-3>"
export WORKER_4_ADDRESS="<worker-address-4>"
export SSH_USER="<ssh-user>"
export SSH_PRIVATE_KEY_SECRET_NAME="$CLUSTER_NAME-ssh-key"

1. Use the following template to define your infrastructure. The environment variables you set in this step automatically replace the variable names when the inventory YAML file is created.

cat <<EOF > preprovisioned_inventory.yaml
---
apiVersion: infrastructure.cluster.konvoy.d2iq.io/v1alpha1
kind: PreprovisionedInventory
metadata:
name: $CLUSTER_NAME-control-plane
namespace: default
labels:
cluster.x-k8s.io/cluster-name: $CLUSTER_NAME
clusterctl.cluster.x-k8s.io/move: ""
spec:
hosts:
# Create as many of these as needed to match your infrastructure
# Note that the command line parameter --control-plane-replicas determines how
many control plane nodes will actually be used.
#
- address: $CONTROL_PLANE_1_ADDRESS
- address: $CONTROL_PLANE_2_ADDRESS
- address: $CONTROL_PLANE_3_ADDRESS
sshConfig:
port: 22
# This is the username used to connect to your infrastructure. This user must be
root or
# have the ability to use sudo without a password
user: $SSH_USER
privateKeyRef:
# This is the name of the secret you created in the previous step. It must
exist in the same
# namespace as this inventory object.
name: $SSH_PRIVATE_KEY_SECRET_NAME
namespace: default
---
apiVersion: infrastructure.cluster.konvoy.d2iq.io/v1alpha1
kind: PreprovisionedInventory
metadata:
name: $CLUSTER_NAME-md-0
namespace: default
labels:
cluster.x-k8s.io/cluster-name: $CLUSTER_NAME
clusterctl.cluster.x-k8s.io/move: ""
spec:
hosts:
- address: $WORKER_1_ADDRESS
- address: $WORKER_2_ADDRESS
- address: $WORKER_3_ADDRESS
- address: $WORKER_4_ADDRESS
sshConfig:
port: 22
user: $SSH_USER
privateKeyRef:
name: $SSH_PRIVATE_KEY_SECRET_NAME
namespace: default
EOF

1. To tell the bootstrap cluster which nodes you want to be control plane nodes and which nodes are worker nodes. Apply the file to the bootstrap cluster using the command kubectl apply.

kubectl apply -f preprovisioned_inventory.yaml

Example:

preprovisionedinventory.infrastructure.cluster.konvoy.nutanix.io/cluster_name-
control-plane created
preprovisionedinventory.infrastructure.cluster.konvoy.nutanix.io/cluster_name-md-0
created

What to do next

Pre-provisioned Cluster Creation Customization Choices

Pre-provisioned: Loading the Registry for an Air-gapped Kubernetes Cluster

About this task

The complete Nutanix Kubernetes Platform (NKP) air-gapped bundle is needed for an air-gapped environment but can also be used in a non-air-gapped environment. The bundle contains all the NKP components needed for an air- gapped environment installation and a local registry in a non-air-gapped environment.

Before you begin

Extract Air-gapped Images

Follow these steps to extract the air-gapped image bundles into your private registry.

If you are operating in an air-gapped environment, a local container registry containing all the necessary installation images, including the Kommander images is required. See below for prerequisites to download and then how to push the necessary images to this registry.

1. Download the Complete NKP Air-gapped Bundle for this release (that is, nkp- air-gapped-bundle_nkp-

version_linux_amd64.tar.gz) to load registry images as explained below. 2. Connectivity with clusters attaching to the management cluster is required:

• Both management and attached clusters must be able to connect to the local registry.
• The management cluster must be able to connect to all attached cluster’s API servers.
• The management cluster must be able to connect to any load balancers created for platform services on the management cluster.

Load the Bootstrap Image

Procedure

1. Assuming you have downloaded the air-gapped bundle nkp-air-gapped-bundle_nkp-

version_linux_amd64.tar.gz and extracted the tar file to a local directory using the command tar -xzvf nkp-air-gapped-bundle_nkp- version_linux_amd64.tar.gz.

Example directory structure after extraction

nkp-nkp-version/
### application-charts
# ### NOTICES.txt
# ### nkp-insights-charts-bundle-v2.8.0-rc.2.tar.gz
# ### nkp-kommander-charts-bundle-v2.8.0-rc.2.tar.gz
### application-repositories
# ### nkp-insights-v2.8.0-rc.2.tar.gz
# ### kommander-applications-v2.8.0-rc.2.tar.gz
### container-images
# ### NOTICES.txt (2)
# ### nkp-insights-image-bundle-v2.8.0-rc.2.tar
# ### kommander-image-bundle-v2.8.0-rc.2.tar
# ### konvoy-image-bundle-v2.8.0-rc.2.tar
### nkp
### kib
# ### LICENSE
# ### README.md
# ### ansible
# ### artifacts
# ### goss
# ### images
# ### konvoy-image
# ### overrides
### konvoy-bootstrap-image-nkp-version-rc.2.tar
### kubectl

1. The directory structure after extraction can be accessed in subsequent steps using commands to access files from different directories. For the bootstrap, change your directory to the nkp-<version> directory similar to example below depending on your current location.
2. Load the bootstrap container image on your bastion machine replacing docker with podman if needed:

Set Environment Variables

Procedure

Set an environment variable with your registry address and any other needed variables using this command.

export REGISTRY_URL="<https/http>://<registry-address>:<registry-port>"
REGISTRY_URL: the address of an existing local registry accessible in the VPC that the new cluster nodes will be
configured to use a mirror registry when pulling images.

Additional Registry Variables: More environment variables if needed:

export REGISTRY_USERNAME=<username>
export REGISTRY_PASSWORD=<password>

export REGISTRY_CA=<path to the cacert file on the bastion>

Load Images to your Private Registry - Konvoy

About this task

Before creating or upgrading a Kubernetes cluster, you need to load the required images in a local registry if operating in an air-gapped environment. This registry must be accessible from both the bastion machine and either the AWS EC2 instances or other machines that will be created for the Kubernetes cluster.

Warning: If you do not already have a local registry set up, refer to Local Registry Tools page for more information.

Procedure

Execute the following command to load the air-gapped image bundle into your private registry: nkp push bundle –bundle ./container-images/konvoy-image- bundle-nkp-version.tar –to-registry= ${REGISTRY_URL} –to-registry- username=${REGISTRY_USERNAME} –to-registry-password= ${REGISTRY_PASSWORD}

Note: It might take some time to push all the images to your image registry, depending on the network's performance
between the machine you are running the script on and the registry.

Load Images to your Private Registry - Kommander

About this task

Load Kommander images to your Private Registry.

For the air-gapped kommander image bundle, run the command below.

Procedure

Run the following command to load the image bundle:

nkp push bundle --bundle ./container-images/kommander-image-bundle-nkp-version.tar --
to-registry=${REGISTRY_URL} --to-registry-username=${REGISTRY_USERNAME} --to-registry-
password=${REGISTRY_PASSWORD}

Pre-provisioned: Replacing the Driver with the Azure Disk CSI Driver

About this task

The Nutanix Kubernetes Platform (NKP) Pre-provisioned provider installs by default the storage-local-static- provisioner CSI driver, which is not suitable for production environments. For this reason, it needs to be replaced by the Azure Disk CSI Driver.

Before you begin

• An x86_64-based Linux or macOS machine.

• Download the nkp binary for Linux or macOS. To check which version of NKP you installed for compatibility reasons, run the nkp version command.

• bundle –bundle ./container-images/konvoy-image-bundle-; nkp-version; .tar –to-registry=

| — | — | — |

• A Container engine or runtime installed is required to install NKP and bootstrap:
• Docker container engine version 27.4.0 installed for Linux or MacOS. For more information, see https:// docs.docker.com/get-docker/.
• Podman Version 4.0 or later for Linux. For more information, see <https://podman.io/getting-started/> installation. For host requirements, see <https://kind.sigs.k8s.io/docs/user/rootless/#host-requirements>.
• CLI tool Kubectl is used to interact with the running cluster. <https://kubernetes.io/docs/tasks/tools/#kubectl>
• Azure CLI. For more information, see <<https://docs.microsoft.com/en-> us/cli/azure/install-azure-cli>.
• A valid Azure account with credentials configured. For more information, see <https://learn.microsoft.com/en-> us/azure/aks/concepts-identity.
• Create a custom Azure image using NIB. For more information, see Building a Custom Image with Azure on page 61.
• For air-gapped environments only:
• Ability to download artifacts from the internet and then copy those onto your Bastion machine.
• Download the Complete NKP Air-gapped Bundle for this release - nkp-air- gapped-bundle_nkp- version_linux_amd64.tar.gz.
• An existing local registry to seed the air-gapped environment. For more information, see Registry and Registry Mirrors on page 705.

Note: On macOS, Docker runs in a virtual machine. Configure this virtual machine with at least 8GB of memory.

About this task

Set Environment Variables with Credentials - An Azure Service Principal is needed for deploying resources. To configure your Azure environment, follow below.

Procedure

1. Log in to Azure using the command az login. Example output:

[
{
"cloudName": "AzureCloud",
"homeTenantId": "a1234567-b132-1234-1a11-1234a5678b90",
"id": "b1234567-abcd-11a1-a0a0-1234a5678b90",
"isDefault": true,
"managedByTenants": [],
"name": "Mesosphere Developer Subscription",
"state": "Enabled",
"tenantId": "a1234567-b132-1234-1a11-1234a5678b90",
"user": {
"name": "[email protected]",
"type": "user"
}
}
]

1. Create an Azure Service Principal (SP) by using the command az ad sp create-for-rbac –role contributor –name “$(whoami)-konvoy” –scopes=/subscriptions/$(az account show – query id -o tsv). This command will rotate the password if an SP with the name exists. Example output:

{
"appId": "7654321a-1a23-567b-b789-0987b6543a21",
"displayName": "azure-cli-2021-03-09-23-17-06",
"password": "DUMMY_CLIENT_SECRET",
"tenant": "a1234567-b132-1234-1a11-1234a5678b90"
}

• For air-gapped environments, you need to create a resource management private link (https:// learn.microsoft.com/en-us/azure/azure-resource- manager/management/create-private-link-access- portal) with a private endpoint to ensure the Azure CSI driver will run correctly in further steps. Private links enable you to access Azure services over a private endpoint in your virtual network. For more information, see <<https://learn.microsoft.com/en-us/azure/azure-resource-> manager/management/create-private-> link-access-portal.

To set up a private link resource, use the following process.

1. Create a private resource management link using Azure CLI. For more information, see https:// learn.microsoft.com/en- us/azure/azure-resource-manager/management/create-private-link- access- commands?tabs=azure-cli#create-resource- management-private-link 2. Create a private link association for the root management group, which also references the resource ID for the resource management private link. For more information, see <https://learn.microsoft.com/> en- us/azure/azure-resource-manager/management/create-private-link-access- commands? tabs=azure-cli#create-private-link- association. 3. Add a private endpoint referencing the resource management private link using the Azure Documentation. For more information, see <<https://learn.microsoft.com/en-us/azure/private-> link/> create-private-endpoint- cli?tabs=dynamic-ip.
2. Set the required environment variables. Example:

export AZURE_SUBSCRIPTION_ID="<id>" # b1234567-abcd-11a1-a0a0-1234a5678b90
export AZURE_TENANT_ID="<tenant>" # a1234567-b132-1234-1a11-1234a5678b90
export AZURE_CLIENT_ID="<appId>" # 7654321a-1a23-567b-b789-0987b6543a21
export AZURE_CLIENT_SECRET="<password>" # DUMMY_CLIENT_SECRET
export AZURE_RESOURCE_GROUP="<resource group name>" # set to the name of the
resource group
export AZURE_LOCATION="westus" # set to the location you are using

1. Set your KUBECONFIG environment variable using the command export kubeconfig= ${CLUSTER_NAME}.conf
2. Create the Secret with the Azure credentials. The Azure CSI driver will use this.

a. Create an azure.json file.

cat <<EOF > azure.json
{
"cloud": "AzurePublicCloud",
"tenantId": "$AZURE_TENANT_ID",
"subscriptionId": "$AZURE_SUBSCRIPTION_ID",
"aadClientId": "$AZURE_CLIENT_ID",
"aadClientSecret": "$AZURE_CLIENT_SECRET",
"resourceGroup": "$AZURE_RESOURCE_GROUP",
"location": "$AZURE_LOCATION"
}
EOF

b. Create the Secret using the command kubectl create secret generic azure- cloud-provider –

namespace=kube-system –type=Opaque –from-file=cloud-config=azure.json. 6. Install the Azure Disk CSI driver using the command $ curl -skSL https:// raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/v<kubernetes- version>/deploy/install-driver.sh | bash -s v<kubernetes-version> snapshot -.

Note: For more information about the supported Kubernetes version, see Supported Kubernetes Versions section
in the NKP Release Notes.

1. Check the status to see if the driver is ready for use using the command.

kubectl -n kube-system get pod -o wide --watch -l app=csi-azuredisk-controller
kubectl -n kube-system get pod -o wide --watch -l app=csi-azuredisk-node

Kubernetes knows this is Azure disk and will create clusters on Azure. 8. Create the StorageClass for the Azure Disk CSI Driver using the command kubectl create -f https:// raw.githubusercontent.com/kubernetes- sigs/azuredisk-csi-driver/master/deploy/ example/storageclass-azuredisk- csi.yaml. 9. Change the default storage class to this new StorageClass so that every new disk will be created in the Azure environment using the command.

kubectl patch sc/localvolumeprovisioner -p '{"metadata": {"annotations":
{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch sc/managed-csi -p '{"metadata": {"annotations":
{"storageclass.kubernetes.io/is-default-class":"true"}}}'

1. Verify that the StorageClass chosen is currently the default using the command kubectl get storageclass. For more information about Azure Disk CSI for persistent storage and changing the default StorageClass, see Default Storage Providers in NKP.

Pre-provisioned Cluster Creation Customization Choices

Below are two methods to customize your cluster during creation. If none of these choices apply, proceed to the next section.

• Pre-provisioned Install in a Non-air-gapped Environment
• Pre-provisioned Install in an Air-gapped Environment

Pre-provisioned Section Topics

Many options are available when creating clusters, such as those listed in this documentation section. A brief explanation of each choice is given in the following topic summaries with a link to the more descriptive page. To use these, proceed to the cluster choice page for detailed instructions:

• Pre-provisioned Customizing CAPI Clusters : Familiarize yourself with the Cluster API before editing the cluster objects, as edits can prevent the cluster from deploying successfully.

• Pre-provisioned Registry Mirrors: In an air-gapped environment, you need a local repository to store Helm charts, Docker images, and other artifacts. In an environment with access to the Internet, you can retrieve artifacts from specialized repositories dedicated to them, such as Docker images contained in DockerHub and Helm Charts that come from a dedicated Helm Chart repository.

• raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/v<; kubernetes-

| — | — |

• version; >/deploy/install-driver.sh; bash -s v<; kubernetes-version;> snapshot –

| — | — | — | — | — |

• Pre-provisioned Create Secrets and Overrides: Create necessary secrets and overrides for pre-provisioned clusters. Most applications deployed through Kubernetes (<https://kubernetes.io/docs/concepts/configuration/> secret/) require access to databases, services, and other external resources. The easiest way to manage the login information necessary to access those resources is using secrets, which help organize and distribute sensitive information across a cluster while minimizing the risk of sensitive information exposure.
• Pre-provisioned Define Control Plane Endpoint: A control plane needs to have three, five, or seven nodes to remain available if one, two, or three nodes fail. A control plane with one node is not for production use.
• Pre-provisioned Configure MetalLB: An external load balancer (LB) is recommended to be the control plane endpoint. To distribute request load among the control plane machines, configure the load balancer to send requests to all the control plane machines. Configure the load balancer to send requests only to control plane machines responding to API requests. If you do not have one, you can use Metal LB to create MetalLB configmap for your Pre-provisioned infrastructure.
• Pre-provisioned Modify the Calico Installation: Calico is a networking and security solution that enables Kubernetes and non-Kubernetes/legacy workloads to communicate seamlessly and securely. Sometimes, changes are needed, so use the information on this Pre-provisioned Modify the Calico Installation page.
• Pre-provisioned Built-in Virtual IP: As explained in Define the Control Plane Endpoint, we recommend using an external load balancer for the control plane endpoint but provide a built-in virtual IP when one is not available.
• Pre-provisioned Use HTTP Proxy: When you require HTTP proxy configurations, you can apply them during the create operation by adding the appropriate flags to the nkp create cluster command.
• Pre-provisioned Use Alternate Pod or Service Subnets: Some subnets are reserved by Kubernetes and can prevent proper cluster deployment if you unknowingly configure NKP so that the Node subnet collides with either the Pod or Service subnet.
• Pre-provisioned Output Directory YAML: You can create individual files with different smaller manifests for ease in editing using the –output-directory flag used with –output=json|yaml. You create the directory where resources are outputted to files.

Single-Node Control Plane

Caution: Do not use a single-node control plane in a production cluster.

A control plane with one node can use its single node as the endpoint, so you will not require an external load balancer or a built-in virtual IP. At least one control plane node must always be running. Therefore, a spare machine must be available in the control plane inventory to upgrade a cluster with one control plane node. This machine is used to provision the new node before the old node is deleted.

When the API server endpoints are defined, you can create the cluster.

Note: For more information on modifying Control Plane Audit logs settings, see Configuring the Control Plane.

Pre-provisioned: Customizing CAPI Clusters

Familiarize yourself with Cluster API before editing the cluster objects because edits can prevent the cluster from deploying successfully.

The result of this command will allow such edits:

nkp create cluster preprovisioned \
--cluster-name ${CLUSTER_NAME} \
--control-plane-endpoint-host <control plane endpoint host> \
--pre-provisioned-inventory-file preprovisioned_inventory.yaml \
--ssh-private-key-file <path-to-ssh-private-key> \
--dry-run \
--output=yaml \
> ${CLUSTER_NAME}.yaml

To edit the YAML, you need to understand the CAPI components to avoid breaking the cluster.

Pre-provisioned: Registry Mirrors

In an air-gapped environment, you need a local repository to store Helm charts, Docker images, and other artifacts. In an environment with access to the Internet, you can retrieve artifacts from specialized repositories dedicated to them, such as Docker images contained in DockerHub and Helm Charts that come from a dedicated Helm Chart repository.

Kubernetes does not natively provide a registry for hosting the container images you will use to run the applications you want to deploy on Kubernetes. Instead, Kubernetes requires you to use an external solution to store and share container images. A variety of Kubernetes-compatible registry options are compatible with NKP.

How the Registry Mirror Works

The first time you request an image from your local registry mirror, it pulls the image from the public registry (such as Docker) and stores it locally before handing it back to you. On subsequent requests, the local registry mirror can serve the image from its storage.

Air-gapped vs. Non-air-gapped Environments

In a non-air-gapped environment, you can access the Internet. You retrieve artifacts from specialized repositories dedicated to them, such as Docker images contained in DockerHub and Helm Charts that come from a dedicated Helm Chart repository. You can also create your local repository to hold the downloaded container images needed or any custom images you’ve created with the Nutanix Image Builder on page 51 tool.

In an air-gapped environment, you need a local repository to store Helm charts, Docker images, and other artifacts. Private registries provide security and privacy in enterprise container image storage, whether hosted remotely or on- premises locally in an air-gapped environment. NKP in an air-gapped environment requires a local container registry of trusted images to enable production-level Kubernetes cluster management. However, a local registry is also an option in a non-air-gapped environment for speed and security.

If you want to use images from this local registry to deploy applications inside your Kubernetes cluster, you will need to set up a secret for a private registry. The secret contains your login data, which Kubernetes needs to connect to your private repository.

Pre-provisioned: Creating Secrets and Overrides

About this task

Most applications deployed through Kubernetes require external access to databases, services, and other resources. The easiest way to manage the login information necessary to access those resources is by using secrets to help organize and distribute sensitive information across a cluster while minimizing the risk of sensitive information exposure.

Nutanix Kubernetes Platform (NKP) requires SSH access to your infrastructure with superuser privileges. You must provide an unencrypted SSH private key to NKP , so secrets are a good way to achieve this. Populate the key and create the required secret on your bootstrap cluster using the following procedure.

Before you begin

Create a Unique Cluster Name

Give your cluster a unique name suitable for your environment.

Procedure

Set the environment variable to be used throughout this procedure using the command export CLUSTER_NAME=<cluster_name>

Note: A cluster name can include only the following characters: a-z, 0-9, ., and -. The cluster creation fails if the
name has uppercase letters. For more instructions on naming, see Object Names and IDs.

When specifying the cluster-name, use the same cluster-name defined in your inventory objects to ensure consistency.

(Optional) If you want to create a unique cluster name using the command.

export CLUSTER_NAME=cluster_name-$(LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | fold -w 5
| head -n1)
echo $CLUSTER_NAME

Note: This creates a unique name every time you run it, so use it carefully.

cluster_name-pf4a3

Create a Secret

About this task

Procedure

1. Export the key using the command export SSH_PRIVATE_KEY_FILE=“<path-to- ssh-private-key>”
2. Export the secret using the command export SSH_PRIVATE_KEY_SECRET_NAME=$CLUSTER_NAME-ssh-key.
3. Create the secret using the command.

kubectl create secret generic ${SSH_PRIVATE_KEY_SECRET_NAME} --from-file=ssh-
privatekey=${SSH_PRIVATE_KEY_FILE}
kubectl label secret ${SSH_PRIVATE_KEY_SECRET_NAME} clusterctl.cluster.x-k8s.io/move=

Example output:

secret/cluster_name-ssh-key created
secret/cluster_name-ssh-key labeled

Create Overrides

About this task

Create a Secret (2)

Procedure

1. Example CentOS7 and Docker - If you want to provide an override with Docker credentials and a different source for EPEL on a CentOS7 machine, create a file like this.

cat > overrides.yaml << EOF
image_registries_with_auth:
- host: "registry-1.docker.io"
username: "my-user"
password: "my-password"
auth: ""
identityToken: ""
epel_centos_7_rpm: https://my-rpm-repostory.org/epel/epel-release-latest-7.noarch.rpm
EOF

You can then create the related secret by using the command.

kubectl create secret generic $CLUSTER_NAME-user-overrides --from-
file=overrides.yaml=overrides.yaml
kubectl label secret $CLUSTER_NAME-user-overrides clusterctl.cluster.x-k8s.io/move=

1. When using Oracle 7 OS, you might wish to deploy the RHCK kernel instead of the default UEK kernel. To do so, add the following text to your overrides.yaml.

cat > overrides.yaml << EOF
---
oracle_kernel: RHCK
EOF
kubectl create secret generic $CLUSTER_NAME-user-overrides --from-
file=overrides.yaml=overrides.yaml
kubectl label secret $CLUSTER_NAME-user-overrides clusterctl.cluster.x-k8s.io/move=

Pre-provisioned: Creating FIPS Secrets and Overrides

About this task

You must provide an unencrypted SSH private key to NKP , so secrets are a good way to achieve this. Populate the key and create the required secret on your bootstrap cluster using the following procedure.

Before you begin

Create a Unique Cluster Name

Give your cluster a unique name suitable for your environment.

Procedure

Set the environment variable to be used throughout this procedure using the command export CLUSTER_NAME=<cluster_name>

Note: A cluster name can include only the following characters: a-z, 0-9, ., and -. The cluster creation fails if the
name has uppercase letters. For more instructions on naming, see Object Names and IDs.

When specifying the cluster-name, use the same cluster-name defined in your inventory objects to ensure consistency.

(Optional) If you want to create a unique cluster name, use the command.

export CLUSTER_NAME=cluster_name-$(LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | fold -w 5
| head -n1)
echo $CLUSTER_NAME

Note: This creates a unique name every time you run it, so use it carefully.

cluster_name-pf4a3

Create a Secret (3)

About this task

Procedure

1. Export the key using the command export SSH_PRIVATE_KEY_FILE=“<path-to- ssh-private-key>” .
2. Export the secret using the command

export SSH_PRIVATE_KEY_SECRET_NAME=$CLUSTER_NAME-ssh-key

1. Create the secret using the command.

kubectl create secret generic ${SSH_PRIVATE_KEY_SECRET_NAME} --from-file=ssh-
privatekey=${SSH_PRIVATE_KEY_FILE}
kubectl label secret ${SSH_PRIVATE_KEY_SECRET_NAME} clusterctl.cluster.x-k8s.io/move=

Example output:

secret/cluster_name-ssh-key created
secret/cluster_name-ssh-key labeled

Create Overrides (2)

About this task

Procedure

1. Create a secret that includes the customization Overrides for FIPS compliance.

cat > overrides.yaml << EOF
---
k8s_image_registry: docker.io/mesosphere
fips:
enabled: true
build_name_extra: -fips
kubernetes_build_metadata: fips.0
default_image_repo: hub.docker.io/mesosphere
kubernetes_rpm_repository_url: "https://packages.nutanix.com/konvoy/stable/linux/
repos/el/kubernetes-v{{ kubernetes_version }}-fips/x86_64"
docker_rpm_repository_url: "\
https://containerd-fips.s3.us-east-2.amazonaws.com\
/{{ ansible_distribution_major_version|int }}\
/x86_64"
EOF

1. If your pre-provisioned machines need customization with alternate package libraries, Docker image or other container registry image repositories, or other Custom Override Files, add more lines to the same Overrides file.

a. Example One - If you want to provide an override with Docker credentials and a different source for EPEL on a CentOS7 machine, create a file like this.

cat > overrides.yaml << EOF
---
# fips configuration
k8s_image_registry: docker.io/mesosphere
fips:
enabled: true
build_name_extra: -fips
kubernetes_build_metadata: fips.0
default_image_repo: hub.docker.io/mesosphere
kubernetes_rpm_repository_url: "https://packages.nutanix.com/konvoy/stable/linux/
repos/el/kubernetes-v{{ kubernetes_version }}-fips/x86_64"
docker_rpm_repository_url: "\
https://containerd-fips.s3.us-east-2.amazonaws.com\
/{{ ansible_distribution_major_version|int }}\
/x86_64"
EOF

b. Example Two - When using Oracle 7 OS, you may wish to deploy the RHCK kernel instead of the default UEK kernel. To do so, add the following text to your overrides.yaml.

cat > overrides.yaml << EOF
---
# fips configuration (2)
k8s_image_registry: docker.io/mesosphere
fips:
enabled: true
build_name_extra: -fips
kubernetes_build_metadata: fips.0
default_image_repo: hub.docker.io/mesosphere
kubernetes_rpm_repository_url: "https://packages.nutanix.com/konvoy/stable/linux/
repos/el/kubernetes-v{{ kubernetes_version }}-fips/x86_64"
docker_rpm_repository_url: "\
https://containerd-fips.s3.us-east-2.amazonaws.com\
/{{ ansible_distribution_major_version|int }}\
/x86_64"
# custom configuration
oracle_kernel: RHCK
EOF

1. Create the related secret by using the command.

kubectl create secret generic $CLUSTER_NAME-user-overrides --from-
file=overrides.yaml=overrides.yaml
kubectl label secret $CLUSTER_NAME-user-overrides clusterctl.cluster.x-k8s.io/move=

Pre-provisioned: Defining Control Plane Endpoint

Define the control plane endpoint for your cluster and the connection mechanism. A control plane must have three, five, or seven nodes to remain available if one or more nodes fail. A control plane with one node is not for production use.

In addition, the control plane should have an endpoint that remains available if some nodes fail.

-------- cp1.example.com:6443
|
lb.example.com:6443 ---------- cp2.example.com:6443
|
-------- cp3.example.com:6443

In this example, the control plane endpoint host is lb.example.com, and the control plane endpoint port is 6443. The control plane nodes are cp1.example.com, cp2.example.com, and cp3.example.com. The port of each API server is 6443.

The control plane endpoint port is also used as the API server port on each control plane machine. The default port is 6443. Before creating the cluster, ensure the port is available on each control plane machine.

Select your Connection Mechanism

A virtual IP is the client’s address to which to connect to the service. A load balancer is a device that distributes the client connections to the backend servers. Before you create a new Nutanix Kubernetes Platform (NKP) cluster, choose an external load balancer (LB) or virtual IP.

• External load balancer

It is recommended that an external load balancer be the control plane endpoint. To distribute request load among the control plane machines, configure the load balancer to send requests to all the control plane machines. Configure the load balancer to send requests only to control plane machines responding to API requests.

• Built-in virtual IP

You can use the built-in virtual IP if an external load balancer is unavailable. The virtual IP is not a load balancer; it does not distribute request load among the control plane machines. However, if the machine receiving requests does not respond, the virtual IP automatically moves to another machine.

Single-Node Control Plane

Caution: Do not use a single-node control plane in a production cluster.

A control plane with one node can use its single node as the endpoint, so you will not require an external load balancer or a built-in virtual IP. At least one control plane node must always be running. Therefore, a spare machine must be available in the control plane inventory to upgrade a cluster with one control plane node. This machine is used to provision the new node before the old node is deleted. When the API server endpoints are defined, you can create the cluster using the link in the Next Step below.

Note: Modify Control Plane Audit log settings using the information on the page Configure the Control Plane.

Pre-provisioned: Configuring MetalLB

Create MetalLB configuration for your Pre-provisioned infrastructure.

Nutanix recommends that an external load balancer (LB) be the control plane endpoint. To distribute request load among the control plane machines, configure the load balancer to send requests to all the control plane machines. Configure the load balancer to send requests only to control plane machines responding to API requests. If you do not have one, you can use Metal LB to create MetalLB custom resources for your Pre-provisioned infrastructure.

Choose one of the two protocols you want to use to announce service IPs. If your environment is not currently equipped with a load balancer, you can use MetalLB, a load balancer implementation for Kubernetes. Otherwise, your load balancer will work, and you can continue the installation process with Pre-provisioned: Installing Kommander on page 80. To use MetalLB, create MetalLB custom resources for your Pre-provisioned infrastructure. MetalLB uses one of two protocols to expose Kubernetes services.

Select one of the following procedures to create your MetalLB manifest for further editing:

• Layer 2, with Address Resolution Protocol (ARP)
• Border Gateway Protocol (BGP)

Layer 2 Configuration

Layer 2 mode is the simplest to configure: in many cases, you do not need any protocol-specific configuration, only IP addresses. It does not require the IPs to be bound to the network interfaces of your worker nodes. It responds to ARP requests on your local network directly and gives clients the machine’s MAC address.

Warning:

• MetalLB IP address ranges or CIDRs must be within the node’s primary network subnet. For more information, see Cluster Pod and Services Subnets on page

• MetalLB IP address ranges, CIDRs, and node subnets must not conflict with the Kubernetes cluster pod and service subnets.

For example, the following configuration gives MetalLB control over IPs from 192.168.1.240 to 192.168.1.250 and configures Layer 2 mode:

The following values are generic; enter your specific values into the fields where applicable.

cat << EOF > metallb-conf.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: default
namespace: metallb-system
spec:
addresses:
- 192.168.1.240-192.168.1.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: default
namespace: metallb-system
spec:
ipAddressPools:
- default
EOF

After this completes, run the following kubectl command.

kubectl apply -f metallb-conf.yaml

BGP Configuration

For a basic configuration featuring one BGP router and one IP address range, you need four pieces of information:

• The router IP address that MetalLB needs to connect to.
• The router’s autonomous systems (AS) number.
• The AS number MetalLB.
• An IP address range is a Classless Inter-Domain Routing (CIDR) prefix.

As an example, if you want to give MetalLB the range 192.168.10.0/24 and AS number 64500 and connect it to a router at 10.0.0.1 with AS number 64501, your configuration will look like this:

Note: The following values are generic, enter your specific values into the fields where applicable.

cat << EOF > metallb-conf.yaml
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
name: default
namespace: metallb-system
spec:
myASN: 64500
peerASN: 64501
peerAddress: 10.0.0.1
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: default
namespace: metallb-system
spec:
addresses:
- 192.168.10.0/24
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: example
namespace: metallb-system
spec:
ipAddressPools:
- default
EOF

After this completes, run the following kubectl command.

kubectl apply -f metallb-conf.yaml

Pre-provisioned: Modifying the Calico Installation

About this task

Set the Interface

By default, Calico automatically detects the IP address to use for each node using the first-found method. This is not always appropriate for your particular nodes. In that case, you must modify Calico’s configuration to use a different method. An alternative is to use the interface method by providing the interface ID.

Note: Azure does not set the interface. Proceed to Change the Encapsulation Type section below.

In this example, all cluster nodes use ens192 as the interface name.

Procedure

1. Get the pods running on your cluster with this command.

kubectl get pods -A --kubeconfig ${CLUSTER_NAME}.conf

Output:

NAMESPACE NAME
READY STATUS RESTARTS AGE
calico-system calico-kube-controllers-57fbd7bd59-vpn8b
1/1 Running 0 16m
calico-system calico-node-5tbvl
1/1 Running 0 16m
calico-system calico-node-nbdwd
1/1 Running 0 4m40s
calico-system calico-node-twl6b
0/1 PodInitializing 0 9s
calico-system calico-node-wktkh
1/1 Running 0 5m35s
calico-system calico-typha-54f46b998d-52pt2
1/1 Running 0 16m
calico-system calico-typha-54f46b998d-9tzb8
1/1 Running 0 4m31s
default cuda-vectoradd
0/1 Pending 0 0s
kube-system coredns-78fcd69978-frwx4
1/1 Running 0 16m
kube-system coredns-78fcd69978-kkf44
1/1 Running 0 16m
kube-system etcd-ip-10-0-121-16.us-west-2.compute.internal
0/1 Running 0 8s
kube-system etcd-ip-10-0-46-17.us-west-2.compute.internal
1/1 Running 1 16m
kube-system etcd-ip-10-0-88-238.us-west-2.compute.internal
1/1 Running 1 5m35s
kube-system kube-apiserver-ip-10-0-121-16.us-west-2.compute.internal
0/1 Running 6 7s
kube-system kube-apiserver-ip-10-0-46-17.us-west-2.compute.internal
1/1 Running 1 16m
kube-system kube-apiserver-ip-10-0-88-238.us-west-2.compute.internal
1/1 Running 1 5m34s
kube-system kube-controller-manager-ip-10-0-121-16.us-
west-2.compute.internal 0/1 Running 0 7s
kube-system kube-controller-manager-ip-10-0-46-17.us-
west-2.compute.internal 1/1 Running 1 (5m25s ago) 15m
kube-system kube-controller-manager-ip-10-0-88-238.us-
west-2.compute.internal 1/1 Running 0 5m34s
kube-system kube-proxy-gclmt
1/1 Running 0 16m
kube-system kube-proxy-gptd4
1/1 Running 0 9s
kube-system kube-proxy-mwkgl
1/1 Running 0 4m40s
kube-system kube-proxy-zcqxd
1/1 Running 0 5m35s
kube-system kube-scheduler-ip-10-0-121-16.us-west-2.compute.internal
0/1 Running 1 7s
kube-system kube-scheduler-ip-10-0-46-17.us-west-2.compute.internal
1/1 Running 3 (5m25s ago) 16m
kube-system kube-scheduler-ip-10-0-88-238.us-west-2.compute.internal
1/1 Running 1 5m34s
kube-system local-volume-provisioner-2mv7z
1/1 Running 0 4m10s
kube-system local-volume-provisioner-vdcrg
1/1 Running 0 4m53s
kube-system local-volume-provisioner-wsjrt
1/1 Running 0 16m
node-feature-discovery node-feature-discovery-master-84c67dcbb6-m78vr
1/1 Running 0 16m
node-feature-discovery node-feature-discovery-worker-vpvpl
1/1 Running 0 4m10s
tigera-operator tigera-operator-d499f5c8f-79dc4
1/1 Running 1 (5m24s ago) 16m
Note: If a calico-node pod is not ready on your cluster, you must edit the default Installation resource. To
edit the Installation resource, run the command:
kubectl edit installation default --kubeconfig ${CLUSTER_NAME}.conf