419 min read

Cluster Operations Management

Cluster Operations Management (2)

Operations

Applications on page 327
Workspaces on page 369
Projects on page 415
Cluster Management on page 458
Backup and Restore on page 555
Logging on page 590
Security on page 618
Networking on page 632
GPU Management on page 651
Monitoring and Alerts on page 663
Storage for Applications on page 680

You can manage your cluster and deployed applications using platform applications.

In most cases, a production cluster requires additional advanced configuration tailored for your environment, ongoing maintenance, authentication and authorization, and other common activities. For example, it is important to monitor cluster activity and collect metrics to ensure application performance and response time, evaluate network traffic patterns, manage user access to services, and verify workload distribution and efficiency.

In addition to the configurations, you can also control the appearance of your NKP UI by adding banners and footers. There are different options available depending on the NKP level that you license and install.

Access Control on page 285
Identity Providers on page 297
Kubectl API Access Using an Identity Provider on page 305
Infrastructure Providers on page 309
Header, Footer, and Logo Implementation on page 325

Access Control

You can centrally manage access across clusters and define role-based authorization within the NKP UI to control resource access on the management cluster for a set or all of the target clusters. These resources are similar to Kubernetes RBAC but with crucial differences, and they make it possible to define the roles and role bindings once and federate them to clusters within a given scope.

Kommander Roles: control access to resources on the management clusters.
Cluster Roles: control access to resources on all target clusters.

Table 19: Managing Access Across Scopes

Global : Manages access to the entire environment.

Create ClusterRoles on the management cluster.

Federates ClusterRoles on all target clusters across all workspaces.

Workspace: Manages access to
clusters in a specific workspace,
for example, in the scope of multi-
tenancy. See Multi-Tenancy in
NKP on page 412.

Create namespaced Roles on the management cluster in the workspace namespace.

Federates ClusterRoles on all target clusters in the workspace.

Project: Manages access for
clusters in a specific project, for
example, in the scope of multi-
tenancy. See Multi-Tenancy in
NKP on page 412.

Create namespaced Roles on the management cluster in the project namespace.

Federates namespaced Roles on all target clusters in the project in the project namespace.

Create the role bindings for each level and type create RoleBindings or ClusterRoleBindings on the clusters that apply to each category.

This approach gives you maximum flexibility over who has access to what resources, conveniently mapped to your existing identity providers’ claims.

Limitation for Kommander Roles

Access Control (2)

The property for the subjects.name varies depending on the context for which you have established an Identity Provider.

If you have set up an identity provider for All Workspaces:
For groups: configure the subjects.name field to oidc:<IdP_user_group>. For example, oidc:engineering.
For users: configure the subjects.name field to <user_email>. For example, <[email protected]>

Environment Context	Kommander Roles	Cluster Roles

If you have set up an identity provider for a Specific Workspace:
For groups: configure the subjects.name field to oidc:<workspace_ID>:<IdP_user_group>. For example, oidc:tenant-z:engineering.
For users: configure the subjects.name field to <workspace_ID>:<user_email>. For example, tenant-z:[email protected].

Note: Run kubectl get workspaces to obtain a list of all existing workspaces. The workspace_ID
is listed under the NAME column.
cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: eng-kommander-dashboard
labels:
"workspaces.kommander.mesosphere.io/rbac": ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dkp-kommander-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: oidc:engineering
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: eng-nkp-routes
labels:
"workspaces.kommander.mesosphere.io/rbac": ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dkp-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: oidc:engineering
EOF

Types of Access Control Objects

Manage Kubernetes role-based access control with three different object categories: Groups, Roles, and Policies.

Groups

Access control groups are configured in the Groups tab of the Identity Providers page.

You can map group and user claims made by your configured identity providers to Kommander groups by selecting administration or identity providers in the left sidebar in the global workspace level, and then select the Groups tab.

The syntax for the Identity Provider groups you add to a Group varies depending on the context for which you have established an Identity Provider.

If you have set up an identity provider globally, for All Workspaces:
For groups: Add an Identity Provider Group in the oidc:<github_org>:<github_team> format. For example, oidc:org:team-a.
For users: Add an Identity Provider User in the <user_email>. For example, <[email protected]>
If you have set up an identity provider for a Specific Workspace:
For groups: Add an Identity Provider Group in the oidc:<workspace_ID>:<github_org>:<github_team>. For example, oidc:tenant-z:org:team-a.
For users: Add an Identity Provider User in the <workspace_ID>:<user_email>. For example, tenant-z:[email protected].

Note: Run kubectl get workspaces to obtain a list of all existing workspaces. The workspace_ID
is listed under the NAME column.

Roles

ClusterRoles are named collections of rules defining which verbs can be applied to what resources.

Kommander Roles apply specifically to resources in the management cluster.
Cluster Roles apply to target clusters within their scope at these levels:
Global level - this is all target clusters in all workspaces.
Workspace level - all target clusters in the workspaces.
Project level - this i all target clusters that are added to the project.

Propagating Workspace Roles to Projects

About this task

Each workspace has roles defined using KommanderWorkspaceRole resources. Automatic propagation is controlled using the annotation “workspace.kommander.mesosphere.io/sync-to-project”: “true” on a KommanderWorkspaceRole resource. You can manage this only by using the CLI.

Procedure

Run the command kubectl get kommanderworkspaceroles -n <WORKSPACE_NAMESPACE>.

NAME DISPLAY NAME AGE
kommander-workspace-admin Kommander Workspace Admin Role 2m18s
kommander-workspace-edit Kommander Workspace Edit Role 2m18s
kommander-workspace-view Kommander Workspace View Role 2m18s

To prevent propagation of the kommander-workspace-view role, remove this annotation from the

KommanderWorkspaceRole resource.

kubectl annotate kommanderworkspacerole -n <WORKSPACE_NAMESPACE> kommander-workspace-
view workspace.kommander.mesosphere.io/sync-to-project=false --overwrite

To enable propagation of the role, add this annotation to the relevant KommanderWorkspaceRole resource.

kubectl annotate kommanderworkspacerole -n <WORKSPACE_NAMESPACE> kommander-workspace-
view workspace.kommander.mesosphere.io/sync-to-project=true --overwrite

Limitation for Workspace

During the inheritance of the Project Role, when granting users access to a workspace, you must manually grant access to the projects within that workspace. Each project is created with a set of admin, edit, or view roles, and you can choose to add RoleBinding to each group or user of the workspace for one of these project roles. Usually, these are prefixed with one of the roles kommander-project-(admin/edit/view).

This is an example of RoleBinding that grants the Kommander Project Admin role access for the project namespace to the engineering group:

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: workspace-admin-project1-admin
namespace: <my-project-namespace-xxxxx>
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: <kommander-project-admin-xxxxx>
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: oidc:engineering
EOF

Role Bindings

Access to Kubernetes and Kommander Resources

You can grant access to Kommander and Kubernetes resources using RBAC.

Initially, users and groups from an external identity provider have no access to Kubernetes resources. Privileges must be granted explicitly by interacting with the RBAC API. This section provides some basic examples for general usage. For more information on the RBAC API, see the Using RBAC Authorization section in the Kubernetes documentation at <https://kubernetes.io/docs/reference/access-authn-authz/rbac/>.

Role Bindings (2)

For example, if you want to make <[email protected]> a cluster administrator, bind their username to the cluster- admin default role as follows:

cat << EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: mary-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: [email protected]
EOF

If you have configured an Identity Provider for a specific workspace, configure the subjects.name field to <workspace_ID>:<user_email>. For example, tenant-z:[email protected].

User Namespace Restriction

A common example is granting users access to specific namespaces by creating a RoleBinding (RoleBindings are namespaced scoped). For example, to make the user <[email protected]> a reader of the baz namespace, bind the user to the view role:

cat << EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: bob-view
namespace: baz
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: view
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: [email protected]
EOF

If you have configured an Identity Provider for a specific workspace, configure the subjects.name field to <workspace_ID>:<user_email>. For example, tenant-z:[email protected].

The user can now only perform non-destructive operations targeting resources in the #baz# namespace.

Groups

If your external identity provider supports group claims, you can also bind groups to roles. To make the engineering LDAP group administrators of the production namespace bind the group to the admin role:

cat << EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: engineering-admin
namespace: production
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: oidc:engineering
EOF

One important distinction from adding users is that all external groups are prefixed with oidc:, so a group name is oidc:devops. This prevents collision with locally defined groups.

The property for the subjects.name varies depending on the context for which you have established an Identity Provider.

If you have set up an identity provider for All Workspaces:
For groups: configure the subjects.name field to oidc:<IdP_user_group>. For example, oidc:engineering.
For users: configure the subjects.name field to <user_email>. For example, <[email protected]>
If you have set up an identity provider for a Specific Workspace:
For groups: configure the subjects.name field to oidc:<workspace_ID>:<IdP_user_group>. For example, oidc:tenant-z:engineering.
For users: configure the subjects.name field to <workspace_ID>:<user_email>. For example, tenant-z:[email protected].

Note: Run kubectl get workspaces to obtain a list of all existing workspaces. The workspace_ID
is listed under the NAME column.

NKP UI Authorization

The NKP UI and other HTTP applications protected by Kommander forward authentication, are also authorized by the Kubernetes RBAC API. In addition to the Kubernetes API resources, it is possible to define rules which map to HTTP URIs and HTTP verbs. Kubernetes RBAC refer to these as nonResourceURLs, Kommander forward authentication uses these rules to grant or deny access to HTTP endpoints.

Default Roles

Roles are created to grant access to the dashboard and select applications that expose an HTTP server through the ingress controller. The cluster-admin role is a system role that grants permission to all actions (verbs) on any resource, including non-resource URLs. The default dashboard user is bound to this role.

Note:

With the NKP Starter license, you can only select the following cluster roles: cluster-admin or cluster view
Granting user administrator privileges on /dkp/* grants admin privileges to all sub-resources, even if the bindings exist for sub-resources with fewer privileges.

Table 20: Table

cluster-admin * read, write, delete

kommander dkp-view /dkp/* read

kommander dkp-edit /dkp/* read, write

Dashboard	Role	Path	access

kommander dkp-admin /dkp/* read, write, delete

kommander-dashboard dkp-kommander-view /dkp/kommander/ dashboard/*

read

kommander-dashboard dkp-kommander-edit /dkp/kommander/ dashboard/*

read, write

kommander-dashboard dkp-kommander-admin /dkp/kommander/ dashboard/*

read, write, delete

alertmanager dkp-kube-prometheus- stack-alertmanager-view

/dkp/alertmanager/* read

alertmanager dkp-kube-prometheus- stack-alertmanager-edit

/dkp/alertmanager/* read, write

alertmanager dkp-kube-prometheus- stack-alertmanager- admin

/dkp/alertmanager/* read, write, delete

centralized-grafana dkp-centralized-grafana- grafana-view

/dkp/kommander/ monitoring/grafana/*

read

centralized-grafana dkp-centralized-grafana- grafana-edit

/dkp/kommander/ monitoring/grafana/*

read, write

centralized-grafana dkp-centralized-grafana- grafana-admin

/dkp/kommander/ monitoring/grafana/*

read, write, delete

centralized-opencost dkp-centralized- opencost-view

/dkp/kommander/ opencost/*

read

centralized-opencost dkp-centralized- opencost-edit

/dkp/kommander/ opencost/*

read, write

centralized-opencost dkp-centralized- opencost-admin

/dkp/kommander/ opencost/*

read, write, delete

grafana dkp-kube-prometheus- stack-grafana-view

/dkp/grafana/* read

grafana dkp-kube-prometheus- stack-grafana-edit

/dkp/grafana/* read, write

grafana dkp-kube-prometheus- stack-grafana-admin

/dkp/grafana/* read, write, delete

grafana-logging dkp-grafana-logging-view /dkp/logging/grafana/* read

grafana-logging dkp-grafana-logging-edit /dkp/logging/grafana/* read, write

grafana-logging dkp-grafana-logging- admin

/dkp/logging/grafana/* read, write, delete

karma dkp-karma-view /dkp/kommander/ monitoring/karma/*

read

karma dkp-karma-edit /dkp/kommander/ monitoring/karma/*

read, write

karma dkp-karma-admin /dkp/kommander/ monitoring/karma/*

read, write, delete

Dashboard	Role	Path	access

kubernetes-dashboard dkp-kubernetes- dashboard-view

/dkp/kubernetes/* read

kubernetes-dashboard dkp-kubernetes- dashboard-edit

/dkp/kubernetes/* read, write

kubernetes-dashboard dkp-kubernetes- dashboard-admin

/dkp/kubernetes/* read, write, delete

prometheus dkp-kube-prometheus- stack-prometheus-view

/dkp/prometheus/* read

prometheus dkp-kube-prometheus- stack-prometheus-edit

/dkp/prometheus/* read, write

prometheus dkp-kube-prometheus- stack-prometheus-admin

/dkp/prometheus/* read, write, edit

traefik dkp-traefik-view /dkp/traefik/* read

traefik dkp-traefik-edit /dkp/traefik/* read, edit

traefik dkp-traefik-admin /dkp/traefik/* read, edit, delete

thanos dkp-thanos-query-view /dkp/kommander/ monitoring/query/*

read

thanos dkp-thanos-query-edit /dkp/kommander/ monitoring/query/*

read, write

thanos dkp-thanos-query-admin /dkp/kommander/ monitoring/query/*

read, write, delete

Examples of Default Roles

This topic provides a few examples of binding subjects to the default roles defined for the NKP UI endpoints.

User

To grant the user <[email protected]> administrative access to all Kommander resources, bind the user to the dkp- admin role:

cat << EOF | kubectl apply -f -
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dkp-admin-mary
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dkp-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: [email protected]
EOF

Dashboard	Role	Path	access

If you inspect the role, you can see what access is now granted:

kubectl describe clusterroles dkp-admin
Name: dkp-admin
Labels: app.kubernetes.io/instance=kommander
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/version=v2.13.1
helm.toolkit.fluxcd.io/name=kommander
helm.toolkit.fluxcd.io/namespace=kommander
rbac.authorization.k8s.io/aggregate-to-admin=true
Annotations: meta.helm.sh/release-name: kommander
meta.helm.sh/release-namespace: kommander
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
[/dkp/*] [] [delete]
[/dkp] [] [delete]
[/dkp/*] [] [get]
[/dkp] [] [get]
[/dkp/*] [] [head]
[/dkp] [] [head]
[/dkp/*] [] [post]
[/dkp] [] [post]
[/dkp/*] [] [put]
[/dkp] [] [put]

The user can now use the HTTP verbs HEAD, GET, DELETE, POST, and PUT when accessing any URL at or under /dkp. The downstream application follows REST conventions. This effectively allows privileges to be read, edited, and deleted.

Note: To enable users to access the NKP UI, ensure they have the appropriate dkp-kommander role and the
Kommander roles granted in the NKP UI.

Group

To grant view access to the /dkp/* endpoints and edit access to the grafana logging endpoint to group logging- ops, create the following ClusterRoleBindings:

cat << EOF | kubectl apply -f -
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: nkp-view-logging-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dkp-view
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: oidc:logging-ops
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dkp-logging-edit-logging-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dkp-logging-edit
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: oidc:logging-ops
EOF

Note: External groups must be prefixed by oidc:

Members of logging-ops to view all the resources under /nkp and edit all the resources under /nkp/logging/ grafana.

Creating Custom Roles

If one of the predefined roles from NKP does not include all the permissions you need, you can create a custom role.

About this task

Perform the following tasks to assign actions and permissions to roles:

Procedure

In the Administration section of the sidebar menu, select Access Control.
Select the Cluster Roles tab, and then select + Create Role .
Enter a descriptive name for the role and ensure that Cluster Role is selected as the type.
For example, to configure a read-only role, select Add Rule.

a. In the Resources input, select All Resource Types.

b. Select the get, list, and watch options.

c. Click Save. You can assign your newly created role to the developer’s group.

Kubernetes Dashboard

cat << EOF | kubectl apply -f -
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: developer-dashboard-access
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dkp-kubernetes-dashboard-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: oidc:developers
EOF

However, access to the underlying Kubernetes resources exposed by the dashboard are protected by the cluster RBAC policy.

Onboarding a User to an NKP Cluster

Before you begin

You must have administrator rights. Also, ensure that:

You have an LDAP Connector.
You are a cluster administrator.
You have a valid NKP license (Ultimate, Starter, or Pro)
You have a running cluster.

For information about adding users using other types of connectors, see:

<https://dexidp.io/docs/connectors/oidc/>
<https://dexidp.io/docs/connectors/saml/>
<https://dexidp.io/docs/connectors/github/>

To onboard a user:

Procedure

Create an LDAP Connector definition and name the file ldap.yaml.

apiVersion: v1
kind: Secret
metadata:
name: ldap-password
namespace: kommander
type: Opaque
stringData:
password: superSecret
---
apiVersion: dex.mesosphere.io/v1alpha1
kind: Connector
metadata:
name: ldap
namespace: kommander
spec:
enabled: true
type: ldap
displayName: LDAP Test Connector
ldap:
host: ldapdce.testdomain
insecureNoSSL: true
bindDN: cn=ldapconnector,cn=testgroup,ou=testorg,dc=testdomain
bindSecretRef:
name: ldap-password
userSearch:
baseDN: dc=testdomain
filter: "(objectClass=inetOrgPerson)"
username: uid
idAttr: uid
emailAttr: uid
groupSearch:
baseDN: ou=testorg,dc=testdomain
filter: "(objectClass=posixGroup)"
userMatchers:
- userAttr: uid
groupAttr: memberUid
nameAttr: cn

Add the connector and use the command kubectl apply -f ldap.yaml.

The following output is displayed.

secret/ldap-password created
connector.dex.mesosphere.io/ldap created

Add the appropriate role bindings and name the file new_user.yaml.

The property for the subjects.name varies depending on the context for which you have established an Identity Provider.

If you have set up an identity provider for All Workspaces:
For groups: configure the subjects.name field to oidc:<IdP_user_group>. For example, oidc:engineering.
For users: configure the subjects.name field to <user_email>. For example, <[email protected]>
If you have set up an identity provider for a Specific Workspace:
For groups: configure the subjects.name field to oidc:<workspace_ID>:<IdP_user_group>. For example, oidc:tenant-z:engineering.
For users: configure the subjects.name field to <workspace_ID>:<user_email>. For example, tenant-z:[email protected].

Note: Run kubectl get workspaces to obtain a list of all existing workspaces. The
workspace_ID is listed under the NAME column.

See the following examples for both Single User and Group Bindings.

» For Single Users:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: newUser

» For Group Binding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cluster-admin
namespace: ml
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: oidc:kommanderAdmins

Add the role binding(s) use the command kubectl apply -f new_user.yaml.

Note:

ClusterRoleBindings permissions are applicable at the global level.
RoleBindings permissions are applicable at the namespace level.

For additional information about the other roles in NKP and their permissions, see Access to Kubernetes and Kommander Resources on page 288.

Identity Providers

Configuring a dedicated identity provider per workspace can be useful if you want to retain access to your workspaces separately. In this case, users of a specific workspace have a dedicated login 2-factor authentication page with the identity provider options configured for their workspace. This setup is particularly helpful if you have multiple tenants. For more information, see Multi-Tenancy in NKP on page 412.

Advantages of Using an External Identity Provider

Using an external identity provider is beneficial for:

Centralized management of multiple users and multiple clusters.
Centralized management of password rotation, expiration, and so on.
Support of 2-factor-authentication methods for increased security.
Separate storage of user credentials.

Access Limitations

The GitHub provider allows you to specify any organizations and teams that are eligible for access.
The LDAP provider allows you to configure search filters for either users or groups.
The OIDC provider cannot limit users based on identity.
The SAML provider allows users to log in using a single sign-on (SSO).

Configuring an Identity Provider Through the UI

Before you begin

Limit Access

The GitHub provider allows you to specify any of the organizations and teams are eligible for access.
The LDAP provider allows you to configure search filters for either users or groups.
The OIDC provider cannot limit users based on identity.
The SAML provider allows users to log in using a single sign-on (SSO) profile.

To configure an identity provider:

Procedure

Log into the Kommander UI. See Pre-provisioned: Logging In To the UI on page 83.
From the dropdown list, select the Global workspace.
Select Administration > Identity Providers.
Select the Identity Providers tab.
Select Add Identity Provider.
Select an identity provider.
Select the target workspace for the identity provider and complete the fields with the relevant details.

Note: You can configure an identity provider globally for your entire organization using theAll Workspaces
option or per workspace, enabling multi-tenancy.

Click Save.

Disabling an Identity Provider

Procedure

Select the three-dot button on the Identity Providers table.
Select Disable from the dropdown menu. The identity provider is disabled. You can view the disabled identity providers in the Identity Providers tab.

GitHub Identity Provider Configuration

You can configure GitHub as an identity provider and grant access to NKP.

NKP allows authorizing access to your clusters and the UI with GitHub credentials but it must be configured in the dashboard. To ensure every developer in your GitHub organization has access to your Kubernetes clusters using their GitHub credentials, add that option for login by adding an identity provider with the information from your GitHub profile in the OAuth application settings

The first login requires you to authorize the GitHub account. As an administrator of the cluster, select the Authorize github-username button on the page that follows the login. After setting up the GitHub authorization, the future login screens will have the Log in with github-auth button as an option.

Adding an Identity Provider Using GitHub

To authorize all developers to access your clusters using their GitHub credentials, set up GitHub as an identity provider login option.

Procedure

Start by creating a new OAuth Application in your GitHub organization by completing the registration form.

To view the form, see <https://github.com/settings/applications/new>. 2. In the Application name field, enter a name for your application. 3. In the Homepage URL field, enter your cluster URL. 4. In the Authorization callback URL field, use your cluster URL followed by /dex/callback by adding this to the end of your URL. 5. Click Register application. After you complete the application, the Settings page. appears 6. You need the Client ID and Client Secret from this page for the NKP UI.

If you do not have a Client Secret for the application, to generate a new client secret, select Generate a new client secret. 7. Log in to your NKP UI from the top menu bar, and select the Global workspace. 8. Select Identity Providers in the Administration section of the sidebar menu. 9. Select the Identity Providers tab and then click Add Identity Provider . 10. Select GitHub as the identity provider type, and select the target workspace. 11. Copy the Client ID and Client Secret values from GitHub into this form. 12. To configure dex to load all the groups configured in the user’s GitHub identity, select the Load All Groups check box. This allows you to configure group-specific access to NKP and Kubernetes resources.

Note: Do not select the Enable Device Flow checkbox before selecting <Register the Application> . 13. Click Save.

Mapping the Identity Provider Groups to the Kubernetes Groups

You can map the identity provider groups to the Kubernetes groups.

Procedure

In the NKP UI, select the Groups tab from the Identity Provider screen, and then click Create Group.
In the Enter Name field, enter a descriptive name.

The syntax for the Identity Provider groups you add to a Group varies depending on the context for which you have established an Identity Provider.

If you have set up an identity provider globally, for All Workspaces:
For groups: Add an Identity Provider Group in the oidc:<github_org>:<github_team> format. For example, oidc:org:team-a.
For users: Add an Identity Provider User in the <user_email>. For example, <[email protected]>
If you have set up an identity provider for a Specific Workspace:
For groups: Add an Identity Provider Group in the oidc:<workspace_ID>:<github_org>:<github_team>. For example, oidc:tenant- z:org:team-a.
For users: Add an Identity Provider User in the <workspace_ID>:<user_email>. For example, tenant-z:[email protected].

Note: Run kubectl get workspaces to obtain a list of all existing workspaces. The
workspace_ID is listed under the NAME column.

Add the groups or teams from your GitHub provider under Identity Provider Groups.

For more information on finding the teams to which you are assigned in GitHub, see the Changing team visibility section at <<https://docs.github.com/en/organizations/organizing-members-into-> teams/changing-team-> visibility. 4. Click Save.

Assigning a Role to the Developers Group

After defining a group, bind one or more roles to this group. This topic describes how to bind the group to the View Only role.

Procedure

In the NKP UI, from the top menu bar, select Global or the target workspace.
Select the Cluster Role Bindings tab and then select Add roles.
Select View Only role from the Roles dropdown list and select Save.

For more information on granting users access to Kommander paths on your cluster, see Access to Kubernetes and Kommander Resources on page 288. 4. At a minimum, add a read only path for access to all the Kommander Dashboard views:

Table 21: Kommander Dashboard Views

kommander-dashboard nkp-kommander-view /nkp/kommander/ dashboard/*

read

When you check your attached clusters and login as a user from your matched groups, every resource, is listed. Do delete or edit them.

Future Log In

The first login will require you to authorize the GitHub account, so the administrator of the cluster will select the <Authorize github-username> button on the page that follows the login. After setting up GitHub authorization, future login screens will have that as an option: <Log in with github-auth> button.

External LDAP Directory Configuration

You can connect your cluster to an external LDAP directory. Configure your NKP cluster for logging in with the credentials stored in an external LDAP directory service.

Each LDAP directory is set up in unique ways. You may add the LDAP authentication mechanism using the CLI or UI.

Adding an LDAP Connector

Each LDAP directory is set up in unique ways. So, these steps are important. Add the LDAP authentication mechanism using the CLI or UI.

About this task

This topic describes the configuration of an NKP cluster to connect to the Online LDAP Test Server in Forum Systems Web site at <<https://www.forumsys.com/tutorials/integration-how-to/ldap/online-> ldap-test-server/>. For demonstration purpose, the configuration shown uses insecureNoSSL: true. In production, you should protect LDAP communication with a properly configured transport layer security (TLS). When using TLS, as an admin, you can add insecureSkipVerify: true to spec.ldap to skip server certificate verification, if needed.

Procedure

Choose whether to establish an external LDAP globally or for a specific workspace.

» Global LDAP - identity provider serves all workspaces: Create and apply the following objects:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: ldap-password
namespace: kommander
type: Opaque
stringData:
password: password
---
apiVersion: dex.mesosphere.io/v1alpha1
kind: Connector
metadata:

Dashboard	Role	Path	Access

name: ldap
namespace: kommander
spec:
enabled: true
type: ldap
displayName: LDAP Test
ldap:
host: ldap.forumsys.com:389
insecureNoSSL: true
bindDN: cn=read-only-admin,dc=example,dc=com
bindSecretRef:
name: ldap-password
userSearch:
baseDN: dc=example,dc=com
filter: "(objectClass=inetOrgPerson)"
username: uid
idAttr: uid
emailAttr: mail
groupSearch:
baseDN: dc=example,dc=com
filter: "(objectClass=groupOfUniqueNames)"
userMatchers:
- userAttr: DN
groupAttr: uniqueMember
nameAttr: ou
EOF
Note: The value for the LDAP connector spec:displayName (here LDAP Test) appears on the Login button
for this identity provider in the NKP UI. Enter a name for the users.

» Workspace LDAP - identity provider serves a specific workspace: Create and apply the following objects:

Note: Establish LDAP for a specific workspace in the scope of multiple tenants.

1. Obtain the workspace name for which you are establishing an LDAP authentication server.

kubectl get workspaces

Note down the value under the WORKSPACE NAMESPACE column. 2. Set the WORKSPACE_NAMESPACE environment variable to that namespace.

export WORKSPACE_NAMESPACE=<your-namespace>

Create and apply the following objects on that workspace.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: ldap-password
namespace: ${WORKSPACE_NAMESPACE}
type: Opaque
stringData:
password: password
---
apiVersion: dex.mesosphere.io/v1alpha1
kind: Connector
metadata:
name: ldap
namespace: ${WORKSPACE_NAMESPACE}
spec:
enabled: true
type: ldap
displayName: LDAP Test
ldap:
host: ldap.forumsys.com:389
insecureNoSSL: true
bindDN: cn=read-only-admin,dc=example,dc=com
bindSecretRef:
name: ldap-password
userSearch:
baseDN: dc=example,dc=com
filter: "(objectClass=inetOrgPerson)"
username: uid
idAttr: uid
emailAttr: mail
groupSearch:
baseDN: dc=example,dc=com
filter: "(objectClass=groupOfUniqueNames)"
userMatchers:
- userAttr: DN
groupAttr: uniqueMember
nameAttr: ou
EOF
Note: The value for the LDAP connector spec:displayName (here LDAP Test) appears
on the Login button for this identity provider in the NKP UI. Choose a name for the users.

Testing the LDAP Connector

You can test the LDAP connector.

Procedure

To retrieve a list of connectors using the kubectl get connector.dex.mesosphere.io -A command.
Run the kubectl get Connector.dex.mesosphere.io -n kommander <LDAP- CONNECTOR-NAME> -o

yaml command to verify that the LDAP connector is created successfully.

Logging In for Global LDAP

Global LDAP identity provider serves all workspaces.

Procedure

Visit <https://><YOUR-CLUSTER-HOST>/token and initiate a login flow.
On the login page, click Log in with <ldap-name>.
Enter the LDAP credentials and log in.

Note: In the UI, after the LDAP authentication is enabled, additional access rights must be configured using the
Add Identity Provider page in the UI.

Logging In for Workspace LDAP

Workspace LDAP identity provider serves a specific workspace.

https://	`<YOUR-CLUSTER-HOST>`	/token

Procedure

Complete the steps in Generating a Dedicated Login URL for Each Tenant on page 414.
On the login page, click Log in with <ldap-name>.
Enter the LDAP credentials and log in.

Note: In the UI, after the LDAP authentication is enabled, additional access rights must be configured using the
Add Identity Provider page in the UI.

LDAP Troubleshooting

If the Dex LDAP connector configuration is incorrect, you must debug the problem, and iterate on it to identify and resolve the issue. The Dex log output contains error messages that must be reviewed, as indicated in the following examples:

Reading Errors During Dex Startup

About this task

If the Dex configuration fragment provided results in an invalid Dex config log file, Dex does not properly start up. Then, read the error details by reviewing the Dex logs.

Procedure

Use the following command to retrieve the Dex logs.

kubectl logs -f dex-66675fcb7c-snxb8 -n kommander

You may see an error similar to the following example:

error parse config file /etc/dex/cfg/config.yaml: error unmarshaling JSON: parse
connector config: illegal base64 data at input byte 0

Another reason for Dex not starting up correctly is that <https://><YOUR- CLUSTER-HOST>/token displays a 5xx HTTP error response after timing out.

Most problems with the Dex LDAP connector configuration become apparent only after a login attempt. A login that fails from misconfiguration results in an error displaying only Internal Server Error and Login error. You can find the root cause by reading the Dex log, as shown in the following example.

kubectl logs -f dex-5d55b6b94b-9pm2d -n kommander

You can look for output similar to this example.

[...]
time="2019-07-29T13:03:57Z" level=error msg="Failed to login user: failed to connect:
LDAP Result Code 200 \"Network Error\": dial tcp: lookup freeipa.example.com on
10.255.0.10:53: no such host"

Here, the directory’s DNS name was misconfigured, which should be easy to address.

A more difficult problem occurs when a login through Dex through LDAP fails because Dex cannot find the specified user unambiguously in the directory. That is the result of an invalid LDAP user search configuration. Here’s an example error message from the Dex log.

time="2019-07-29T14:21:27Z" level=info msg="performing ldap search
cn=users,cn=compat,dc=demo1,dc=freeipa,dc=org sub (&(objectClass=posixAccount)
(uid=employee))"
time="2019-07-29T14:21:27Z" level=error msg="Failed to login user: ldap: filter
returned multiple (2) results: \"(&(objectClass=posixAccount)(uid=employee))\""

Solving problems like this requires you to review the directory structures carefully. Directory structures can be very different between different LDAP setups. You must carefully assemble a user search configuration matching the directory structure.

Notably, with some directories, it can be hard to distinguish between the cases such as properly configured and user not found where login fails in an expected way and displays not properly configured, and therefore user not found where login fails in an unexpected way.

For comparison, here are some sample log lines issued by Dex for a successful login:

time="2019-07-29T15:35:51Z" level=info msg="performing ldap search
cn=accounts,dc=demo1,dc=freeipa,dc=org sub (&(objectClass=posixAccount)
(uid=employee))"
time="2019-07-29T15:35:52Z" level=info msg="username \"employee\" mapped to entry
uid=employee,cn=users,cn=accounts,dc=demo1,dc=freeipa,dc=org"
time="2019-07-29T15:35:52Z" level=info msg="login successful: connector \"ldap\",
username=\"\", email=\"[email protected]\", groups=[]"

Kubectl API Access Using an Identity Provider

To allow other users and user groups to access your environment, Nutanix recommends setting up an external identity provider. Users added through an identity provider do not have static credentials, but have to generate a token to gain access to your environment’s kubectl API. This token ensures that certificates are rotated continuously for security reasons.

There are two options for the generation of this token:

Table 22: Token Generation

Generating a token User must log in with credentials and manually generate a kubeconfig file with a fresh token every 24 hours.

Enabling the Konvoy Async Plugin

User configures the Konvoy Async Plugin so the authentication is routed through Dex’s oidc and the token is generated automatically. By enabling the plugin, the user is routed to an additional login procedure for authentication, but they no longer have to generate a token manually in the UI.

The instructions for either generating a token manually or enabling the Konvoy Async Plugin differ slightly depending on whether you configured the identity provide globally for all the workspaces, or individually for a single workspace.

Method	How Often Does the User Have to Generate a Token?

Configuring Token Authentication for Global Identity Providers

In this scenario, the Identity Provider serves all workspaces.

About this task

Note: You must manually generate a new token every 24 hours:

About this task

Procedure

Log in to the NKP UI with your credentials.
Select your username.
Select Generate Token.
Login again.
If there are several clusters, select the target cluster.
Follow the instructions on the displayed page.

Enabling the Konvoy Async Plugin for Global Identity Providers

Enable the Konvoy Async Plugin to automatically update the token.

Before you begin

You or a global admin must configure an identity provider to see this option.

Procedure

Open the login URL.
To authenticate, select Konvoy credentials plugin instructions.
Follow the instructions on the displayed (Konvoy) Credentials plugin instructionspage.

If you use Method 1 in the instructions documented in the (Konvoy) Credentials plugin instructions, then download a kubeconfig file that includes the contexts for all clusters.

Alternatively, to switch between clusters, you can use Method 2 to create a kubeconfig file per cluster and use the –kubeconfig= flag or export KUBECONFIG= commands.

Warning: If you choose Method 2, the Set profile name field is not optional if you have multiple clusters in
your environment. Ensure you change the name of the profile for each cluster for which you want to generate a
kubeconfig file. Otherwise, all clusters will use the same token, which makes cluster authentication vulnerable
and can let users access clusters for which they do not have authorization.

Configuring Token Authentication for Workspace Identity Providers

In this scenario, the identity provider serves a specific workspace or tenant.

About this task

Note: You must manually generate a new token every 24 hours:

Procedure

Open the login link you obtained from the global administrator, which they generated for your workspace or tenant.
Select Generate Kubectl Configuration.
If there are several clusters in the workspace, select the cluster for which you want to generate a token.
Log in with your credentials.
Follow the instructions on the page displayed.

Enabling the Konvoy Async Plugin for Workspace Identity Providers

Enable the Konvoy Async Plugin to automatically update the token.

Before you begin

You or a global admin must configure a workspace-scoped identity provider to see this option.

Procedure

Open the login link you obtained from the global administrator, which they generated for your workspace or tenant.
Select Credentials plugin instructions.
Follow the instructions on the (Konvoy) Credentials plugin instructions page.

If you use Method 1 in the instructions documented in the (Konvoy) Credentials plugin instructions, then download a kubeconfig file that includes the contexts for all clusters.

Alternatively, to switch between clusters, you can use Method 2 to create a kubeconfig file per cluster and use the –kubeconfig= flag or export KUBECONFIG= commands.

Warning: If you choose Method 2, the Set profile name field is not optional if you have multiple clusters in
your environment. Ensure you change the name of the profile for each cluster for which you want to generate a
kubeconfig file. Otherwise, all clusters will use the same token, which makes cluster authentication vulnerable
and can let users access clusters for which they do not have authorization.

Frequently Asked Questions

What happens if I have several clusters in my environment?

Each cluster has a unique API server address, and each cluster requires a unique token.

There are several things you can do to have access to several clusters.

If you decide to manually create a kubeconfig file with a fresh token every 24 hours, you must do this for each cluster in your environment.

If you decide to enable the Konvoy Async Plugin to automatically refresh the token, you can use the Method 1 steps documented in the (Konvoy) Credentials plugin instructions page to download a kubeconfig that includes the contexts for all clusters.

Alternatively, you can use the Method 2 steps documented in the (Konvoy) Credentials plugin instructions page to create a kubeconfig file per cluster, and use the –kubeconfig= flag export KUBECONFIG= or commands to switch between clusters.

Warning: If you choose Method 2, the Set profile name field is not optional if you have multiple clusters in
your environment. Ensure you change the name of the profile for each cluster for which you want to generate a
kubeconfig file. Otherwise, all clusters will use the same token, which makes cluster authentication vulnerable and
can let users access clusters for which they do not have authorization.

How does NKP ensure that users only see the login options for their specific workspace?

NKP supports multi-tenancy by enabling administrators to configure dedicated Identity Providers per workspace/ tenant. Given that a user employs a dedicated workspace login URL to access the NKP UI, the user only sees the login options (and IdPs) available in said workspace.

What is a kubeconfig file?

A file that is used to configure access to clusters is called a kubeconfig file.

The kubectl and NKP command-line tools use kubeconfig files to find the information it needs to choose a cluster and communicate with the API server of a cluster.

In NKP, a kubeconfig file is created automatically when you create a cluster, but it requires a valid token to obtain access to it.

For more information, see Provide Context for Commands with a kubeconfig File.

Rotating the NKP Dashboard Password

About this task

The NKP dashboard uses a static local administrator account as a backup credential. Rotating the password invalidates existing sessions and any stored copies of the previous password. Retrieve the new password with the nkp get dashboard command after the rotation completes.

Procedure

Open a terminal with access to the NKP CLI.
Rotate the dashboard password:

nkp rotate dashboard-credentials

The CLI prompts for confirmation. Respond with y to continue or N to cancel:

This will rotate the NKP dashboard (static admin) password.
Existing sessions and stored credentials will stop working after rotation.
Are you sure you want to continue? [y/N]: y
The dashboard credentials have been rotated. Run nkp get dashboard to retrieve the
new password.

If you decline the prompt, the command aborts without changing the password:

Aborted. The dashboard password was not rotated.

To skip this prompt in non-interactive terminals, add the –yes flag:

nkp rotate dashboard-credentials --yes

Retrieve the new dashboard credentials:

nkp get dashboard --kubeconfig=${CLUSTER_NAME}.conf

The user name, new password, and dashboard URL appear in the command output. Store the new password securely.

What to do next

The static local credentials are backup credentials. For regular user access, configure an external identity provider. For more information, see Identity Providers.

Infrastructure Providers

NKP Pro supports multi-cluster environment, with a single management cluster and one or more attached or managed clusters of the same infrastructure provider.

Configuring a Nutanix Infrastructure Provider in the UI

For AWS, see Configuring an AWS Infrastructure Provider with a User Role on page 310
For Azure, Configuring an Azure Infrastructure Provider in the UI on page 323
For vSphere, see Configuring a vSphere Infrastructure Provider in the UI on page 324

Infrastructure provider credentials are configured in each workspace. The name you assign must be unique across all the other namespaces in your cluster.

Infrastructure Providers (2)

About this task

Before you provision Nutanix clusters using the NKP user interface, you must first create a Nutanix infrastructure provider to contain your Prism Central credentials.

To add a single or multiple Nutanix infrastructures and contain your Prism Central credentials, follow these steps:

Procedure

Log in to the NKP dashboard.
In the top-left corner, from the dropdown list, select the target workspace.

By default, NKP displays the Global workspace. 4. Click Add Infrastructure Provider.

The Add Infrastructure Provider page appears. 5. In the Workspace dropdown menu, select the workspace for your infrastructure provider.

NKP automatically creates the new infrastructure provider in that workspace. 6. From the Select Infrastructure Provider, select Nutanix and add the following information:

a. In the Name field, enter a name for your infrastructure provider.

b. In the Prism Central Endpoint field, enter the endpoint URL of Prism Central.

The Prism Central endpoint must be in the format https://<Prism Central IP Address or FQDN>:9440

c. In the Username field, enter a valid Prism Central username.

d. In the Password field, enter a valid Prism Central user password.

e. To upload the trust bundle, follow these steps:

Click Upload File. 2. Browse to the location of the Prism Central endpoint certificate and click Open.

f. Click Save.

Configuring an AWS Infrastructure Provider with a User Role

Before you begin

Before creating an AWS Infrastructure Provider in the NKP user interface, ensure that you manually create a user role. For more information, see Create a Role Manually on page 311

About this task

Create your infrastructure provider to add resources to your AWS account. For more flexible credential configuration and third-party access, NKP offers a role-based authentication method with an optional external ID. For more information, see IAM roles for Amazon EC2.

Note:

Nutanix recommends using the role-based method as it is more secure.
If your management cluster runs on AWS, you can use the role authentication method.

To create a single or multiple AWS infrastructure provider with a user role, follow these steps:

Procedure

Log in to the NKP dashboard.
In the top-left corner, from the dropdown list, select the target workspace.

By default, NKP displays the Global workspace. 3. From the left navigation pane, click Administration > Infrastructure Providers. 4. Click Add Infrastructure Provider.

The Add Infrastructure Provider page appears. 5. In the Workspace dropdown menu, select the workspace for your infrastructure provider.

NKP automatically creates the new infrastructure provider in that workspace. 6. From the Select Infrastructure Provider, select Amazon Web Services (AWS) and add the following information:

a. In the Name field, enter a name for your infrastructure provider.

Select a name that matches the AWS user.

b. To configure the AWS infrastructure provider with the user role, in the Authentication Method dropdown list, select Role.

Ensure to select Role as the authentication method only when your management cluster runs on AWS.

c. In the Role ARN field, enter the unique identifier of the IAM role.

d. (Optional) To share the role with a third-party account or service, in the External ID field, add a secret string or unique identifier of that account or service.

External IDs secure your environment from accidentally used roles. For more information see Access to AWS accounts owned by third parties.

e. Click Save.

Create a Role Manually

About this task

The role must grant permissions to create the following resources in the AWS account:

EC2 Instances
VPC
Subnets
Elastic Load Balancer (ELB)
Internet Gateway
NAT Gateway
Elastic Block Storage (EBS) Volumes
Security Groups
Route Tables
IAM Roles

Procedure

The user you delegate from your role must have a minimum set of permissions. The following snippet is the minimal IAM policy required.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AllocateAddress",
"ec2:AssociateRouteTable",
"ec2:AttachInternetGateway",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateInternetGateway",
"ec2:CreateNatGateway",
"ec2:CreateRoute",
"ec2:CreateRouteTable",
"ec2:CreateSecurityGroup",
"ec2:CreateSubnet",
"ec2:CreateTags",
"ec2:CreateVpc",
"ec2:ModifyVpcAttribute",
"ec2:DeleteInternetGateway",
"ec2:DeleteNatGateway",
"ec2:DeleteRouteTable",
"ec2:DeleteSecurityGroup",
"ec2:DeleteSubnet",
"ec2:DeleteTags",
"ec2:DeleteVpc",
"ec2:DescribeAccountAttributes",
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInstances",
"ec2:DescribeInternetGateways",
"ec2:DescribeImages",
"ec2:DescribeNatGateways",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeNetworkInterfaceAttribute",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcs",
"ec2:DescribeVpcAttribute",
"ec2:DescribeVolumes",
"ec2:DetachInternetGateway",
"ec2:DisassociateRouteTable",
"ec2:DisassociateAddress",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyNetworkInterfaceAttribute",
"ec2:ModifySubnetAttribute",
"ec2:ReleaseAddress",
"ec2:RevokeSecurityGroupIngress",
"ec2:RunInstances",
"ec2:TerminateInstances",
"tag:GetResources",
"elasticloadbalancing:AddTags",
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:ConfigureHealthCheck",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:RemoveTags",
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeInstanceRefreshes",
"ec2:CreateLaunchTemplate",
"ec2:CreateLaunchTemplateVersion",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeLaunchTemplateVersions",
"ec2:DeleteLaunchTemplate",
"ec2:DeleteLaunchTemplateVersions",
"ec2:DescribeKeyPairs"
],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": [
"autoscaling:CreateAutoScalingGroup",
"autoscaling:UpdateAutoScalingGroup",
"autoscaling:CreateOrUpdateTags",
"autoscaling:StartInstanceRefresh",
"autoscaling:DeleteAutoScalingGroup",
"autoscaling:DeleteTags"
],
"Resource": [
"arn:*:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*"
]
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:*:iam::*:role/aws-service-role/autoscaling.amazonaws.com/
AWSServiceRoleForAutoScaling"
],
"Condition": {
"StringLike": { "iam:AWSServiceName": "autoscaling.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:*:iam::*:role/aws-service-role/elasticloadbalancing.amazonaws.com/
AWSServiceRoleForElasticLoadBalancing"
],
"Condition": {
"StringLike": {
"iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
}
}
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:*:iam::*:role/aws-service-role/spot.amazonaws.com/
AWSServiceRoleForEC2Spot"
],
"Condition": {
"StringLike": { "iam:AWSServiceName": "spot.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["iam:PassRole"],
"Resource": ["arn:*:iam::*:role/*.cluster-api-provider-aws.sigs.k8s.io"]
},
{
"Effect": "Allow",
"Action": [
"secretsmanager:CreateSecret",
"secretsmanager:DeleteSecret",
"secretsmanager:TagResource"
],
"Resource": ["arn:*:secretsmanager:*:*:secret:aws.cluster.x-k8s.io/*"]
},
{
"Effect": "Allow",
"Action": ["ssm:GetParameter"],
"Resource": ["arn:*:ssm:*:*:parameter/aws/service/eks/optimized-ami/*"]
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:*:iam::*:role/aws-service-role/eks.amazonaws.com/
AWSServiceRoleForAmazonEKS"
],
"Condition": {
"StringLike": { "iam:AWSServiceName": "eks.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:*:iam::*:role/aws-service-role/eks-nodegroup.amazonaws.com/
AWSServiceRoleForAmazonEKSNodegroup"
],
"Condition": {
"StringLike": { "iam:AWSServiceName": "eks-nodegroup.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:aws:iam::*:role/aws-service-role/eks-fargate-pods.amazonaws.com/
AWSServiceRoleForAmazonEKSForFargate"
],
"Condition": {
"StringLike": { "iam:AWSServiceName": "eks-fargate.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["iam:GetRole", "iam:ListAttachedRolePolicies"],
"Resource": ["arn:*:iam::*:role/*"]
},
{
"Effect": "Allow",
"Action": ["iam:GetPolicy"],
"Resource": ["arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"]
},
{
"Effect": "Allow",
"Action": [
"eks:DescribeCluster",
"eks:ListClusters",
"eks:CreateCluster",
"eks:TagResource",
"eks:UpdateClusterVersion",
"eks:DeleteCluster",
"eks:UpdateClusterConfig",
"eks:UntagResource",
"eks:UpdateNodegroupVersion",
"eks:DescribeNodegroup",
"eks:DeleteNodegroup",
"eks:UpdateNodegroupConfig",
"eks:CreateNodegroup",
"eks:AssociateEncryptionConfig"
],
"Resource": ["arn:*:eks:*:*:cluster/*", "arn:*:eks:*:*:nodegroup/*/*/*"]
},
{
"Effect": "Allow",
"Action": [
"eks:ListAddons",
"eks:CreateAddon",
"eks:DescribeAddonVersions",
"eks:DescribeAddon",
"eks:DeleteAddon",
"eks:UpdateAddon",
"eks:TagResource",
"eks:DescribeFargateProfile",
"eks:CreateFargateProfile",
"eks:DeleteFargateProfile"
],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": ["iam:PassRole"],
"Resource": ["*"],
"Condition": {
"StringEquals": { "iam:PassedToService": "eks.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["kms:CreateGrant", "kms:DescribeKey"],
"Resource": ["*"],
"Condition": {
"ForAnyValue:StringLike": {
"kms:ResourceAliases": "alias/cluster-api-provider-aws-*"
}
}
}
]
}

Ensure to add a correct trust relationship to the created role.

This provides everyone within the same account to assign AssumeRole with the created role. 2. Replace YOURACCOUNTRESTRICTION with the AWS Account ID that you want AssumeRole from.

Note: Never add a */ wildcard. This opens your account to the public.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com",
"AWS": "arn:aws:iam::YOURACCOUNTRESTRICTION:root"
},
"Action": "sts:AssumeRole"
}
]
}

To use the role created, attach the following policy to the role which is already attached to your managed or attached cluster. Replace YOURACCOUNTRESTRICTION with the AWS Account ID where the role AssumeRole is saved. Also, replace THEROLEYOUCREATED with the AWS Role name.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AssumeRoleKommander",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::YOURACCOUNTRESTRICTION:role/THEROLEYOUCREATED"
}
]
}

Configuring an AWS Infrastructure Provider with Static Credentials

About this task

When configuring AWS infrastructure provider with static credentials, you need an access key ID and a secret access key for the IAM user with a set of minimum capabilities. For more information, see

To create a single or multiple AWS infrastructure provider with static credentials, follow these steps:

Procedure

Log in to the NKP dashboard.
In the top-left corner, from the dropdown list, select the target workspace.

By default, NKP displays the Global workspace. 3. From the left navigation pane, click Administration > Infrastructure Providers. 4. Click Add Infrastructure Provider.

The Add Infrastructure Provider page appears. 5. In the Workspace dropdown menu, select the workspace for your infrastructure provider.

NKP automatically creates the new infrastructure provider in that workspace. 6. From the Select Infrastructure Provider, select Amazon Web Services (AWS) and add the following information:

a. In the Name field, enter a name for your infrastructure provider.

Select a name that matches the AWS user.

b. To configure the AWS infrastructure provider with the static credentials, in the Authentication Method dropdown list, select Static.

c. In the Access Key field, enter the access key ID of the IAM user.

d. In the Secret Key field, enter the secret access key of the IAM user.

e. Click Save.

Creating a User Using CLI

You can create a user using CLI.

Before you begin

Ensure that you install the AWS CLI utility. For more information, see Install or update to the latest version of the AWS CLI.

About this task

To create a IAM user using AWS CLI, follow these steps:

Procedure

Create a user:

aws iam create-user --user-name Kommander

Create an IAM policy:

aws iam create-policy --policy-name kommander-policy --policy-document
'{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":
["ec2:AllocateAddress","ec2:AssociateRouteTable","ec2:AttachInternetGateway","ec2:AuthorizeSecur
["*"]},{"Effect":"Allow","Action":
["autoscaling:CreateAutoScalingGroup","autoscaling:UpdateAutoScalingGroup","autoscaling:CreateOr
["arn:*:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*"]},
{"Effect":"Allow","Action":["iam:CreateServiceLinkedRole"],"Resource":
["arn:*:iam::*:role/aws-service-role/autoscaling.amazonaws.com/
AWSServiceRoleForAutoScaling"],"Condition":{"StringLike":
{"iam:AWSServiceName":"autoscaling.amazonaws.com"}}},
{"Effect":"Allow","Action":["iam:CreateServiceLinkedRole"],"Resource":
["arn:*:iam::*:role/aws-service-role/elasticloadbalancing.amazonaws.com/
AWSServiceRoleForElasticLoadBalancing"],"Condition":{"StringLike":
{"iam:AWSServiceName":"elasticloadbalancing.amazonaws.com"}}},
{"Effect":"Allow","Action":["iam:CreateServiceLinkedRole"],"Resource":
["arn:*:iam::*:role/aws-service-role/spot.amazonaws.com/
AWSServiceRoleForEC2Spot"],"Condition":{"StringLike":
{"iam:AWSServiceName":"spot.amazonaws.com"}}},{"Effect":"Allow","Action":
["iam:PassRole"],"Resource":["arn:*:iam::*:role/*.cluster-
api-provider-aws.sigs.k8s.io"]},{"Effect":"Allow","Action":
["secretsmanager:CreateSecret","secretsmanager:DeleteSecret","secretsmanager:TagResource"],"Reso
["arn:*:secretsmanager:*:*:secret:aws.cluster.x-k8s.io/*"]},
{"Effect":"Allow","Action":["ssm:GetParameter"],"Resource":["arn:*:ssm:*:*:parameter/
aws/service/eks/optimized-ami/*"]},{"Effect":"Allow","Action":
["iam:CreateServiceLinkedRole"],"Resource":["arn:*:iam::*:role/aws-service-
role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS"],"Condition":{"StringLike":
{"iam:AWSServiceName":"eks.amazonaws.com"}}},{"Effect":"Allow","Action":
["iam:CreateServiceLinkedRole"],"Resource":["arn:*:iam::*:role/aws-service-role/
eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup"],"Condition":
{"StringLike":{"iam:AWSServiceName":"eks-nodegroup.amazonaws.com"}}},
{"Effect":"Allow","Action":["iam:CreateServiceLinkedRole"],"Resource":
["arn:aws:iam::*:role/aws-service-role/eks-fargate-pods.amazonaws.com/
AWSServiceRoleForAmazonEKSForFargate"],"Condition":{"StringLike":
{"iam:AWSServiceName":"eks-fargate.amazonaws.com"}}},{"Effect":"Allow","Action":
["iam:GetRole","iam:ListAttachedRolePolicies"],"Resource":["arn:*:iam::*:role/
*"]},{"Effect":"Allow","Action":["iam:GetPolicy"],"Resource":
["arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"]},{"Effect":"Allow","Action":
["eks:DescribeCluster","eks:ListClusters","eks:CreateCluster","eks:TagResource","eks:UpdateClust
["arn:*:eks:*:*:cluster/*","arn:*:eks:*:*:nodegroup/*/*/*"]},
{"Effect":"Allow","Action":
["eks:ListAddons","eks:CreateAddon","eks:DescribeAddonVersions","eks:DescribeAddon","eks:DeleteA
["*"]},{"Effect":"Allow","Action":["iam:PassRole"],"Resource":
["*"],"Condition":{"StringEquals":{"iam:PassedToService":"eks.amazonaws.com"}}},
{"Effect":"Allow","Action":["kms:CreateGrant","kms:DescribeKey"],"Resource":
["*"],"Condition":{"ForAnyValue:StringLike":{"kms:ResourceAliases":"alias/cluster-
api-provider-aws-*"}}}]}'

Create an IAM attach-user policy:

aws iam attach-user-policy --user-name Kommander --policy-arn $(aws iam list-policies
--query 'Policies[?PolicyName==`kommander-policy`].Arn' | grep -o '".*"' | tr -d
'"')

Create an IAM access key:

aws iam create-access-key --user-name Kommander

Using an Existing User to Configure an AWS Infrastructure

You can use an existing AWS user with the credentials configured.

Before you begin

Ensure that you are authorized to create the following resources in the AWS account:

EC2 Instances
VPC
Subnets
Elastic Load Balancer (ELB)
Internet Gateway
NAT Gateway
Elastic Block Storage (EBS) Volumes
Security Groups
Route Tables
IAM Roles

For more information, see Configuration and credential file settings in the AWS CLI

Procedure

The following is the minimal IAM policy required.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AllocateAddress",
"ec2:AssociateRouteTable",
"ec2:AttachInternetGateway",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateInternetGateway",
"ec2:CreateNatGateway",
"ec2:CreateRoute",
"ec2:CreateRouteTable",
"ec2:CreateSecurityGroup",
"ec2:CreateSubnet",
"ec2:CreateTags",
"ec2:CreateVpc",
"ec2:ModifyVpcAttribute",
"ec2:DeleteInternetGateway",
"ec2:DeleteNatGateway",
"ec2:DeleteRouteTable",
"ec2:DeleteSecurityGroup",
"ec2:DeleteSubnet",
"ec2:DeleteTags",
"ec2:DeleteVpc",
"ec2:DescribeAccountAttributes",
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInstances",
"ec2:DescribeInternetGateways",
"ec2:DescribeImages",
"ec2:DescribeNatGateways",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeNetworkInterfaceAttribute",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcs",
"ec2:DescribeVpcAttribute",
"ec2:DescribeVolumes",
"ec2:DetachInternetGateway",
"ec2:DisassociateRouteTable",
"ec2:DisassociateAddress",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyNetworkInterfaceAttribute",
"ec2:ModifySubnetAttribute",
"ec2:ReleaseAddress",
"ec2:RevokeSecurityGroupIngress",
"ec2:RunInstances",
"ec2:TerminateInstances",
"tag:GetResources",
"elasticloadbalancing:AddTags",
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:ConfigureHealthCheck",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:RemoveTags",
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeInstanceRefreshes",
"ec2:CreateLaunchTemplate",
"ec2:CreateLaunchTemplateVersion",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeLaunchTemplateVersions",
"ec2:DeleteLaunchTemplate",
"ec2:DeleteLaunchTemplateVersions",
"ec2:DescribeKeyPairs"
],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": [
"autoscaling:CreateAutoScalingGroup",
"autoscaling:UpdateAutoScalingGroup",
"autoscaling:CreateOrUpdateTags",
"autoscaling:StartInstanceRefresh",
"autoscaling:DeleteAutoScalingGroup",
"autoscaling:DeleteTags"
],
"Resource": [
"arn:*:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*"
]
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:*:iam::*:role/aws-service-role/autoscaling.amazonaws.com/
AWSServiceRoleForAutoScaling"
],
"Condition": {
"StringLike": { "iam:AWSServiceName": "autoscaling.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:*:iam::*:role/aws-service-role/elasticloadbalancing.amazonaws.com/
AWSServiceRoleForElasticLoadBalancing"
],
"Condition": {
"StringLike": {
"iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
}
}
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:*:iam::*:role/aws-service-role/spot.amazonaws.com/
AWSServiceRoleForEC2Spot"
],
"Condition": {
"StringLike": { "iam:AWSServiceName": "spot.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["iam:PassRole"],
"Resource": ["arn:*:iam::*:role/*.cluster-api-provider-aws.sigs.k8s.io"]
},
{
"Effect": "Allow",
"Action": [
"secretsmanager:CreateSecret",
"secretsmanager:DeleteSecret",
"secretsmanager:TagResource"
],
"Resource": ["arn:*:secretsmanager:*:*:secret:aws.cluster.x-k8s.io/*"]
},
{
"Effect": "Allow",
"Action": ["ssm:GetParameter"],
"Resource": ["arn:*:ssm:*:*:parameter/aws/service/eks/optimized-ami/*"]
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:*:iam::*:role/aws-service-role/eks.amazonaws.com/
AWSServiceRoleForAmazonEKS"
],
"Condition": {
"StringLike": { "iam:AWSServiceName": "eks.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:*:iam::*:role/aws-service-role/eks-nodegroup.amazonaws.com/
AWSServiceRoleForAmazonEKSNodegroup"
],
"Condition": {
"StringLike": { "iam:AWSServiceName": "eks-nodegroup.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": [
"arn:aws:iam::*:role/aws-service-role/eks-fargate-pods.amazonaws.com/
AWSServiceRoleForAmazonEKSForFargate"
],
"Condition": {
"StringLike": { "iam:AWSServiceName": "eks-fargate.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["iam:GetRole", "iam:ListAttachedRolePolicies"],
"Resource": ["arn:*:iam::*:role/*"]
},
{
"Effect": "Allow",
"Action": ["iam:GetPolicy"],
"Resource": ["arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"]
},
{
"Effect": "Allow",
"Action": [
"eks:DescribeCluster",
"eks:ListClusters",
"eks:CreateCluster",
"eks:TagResource",
"eks:UpdateClusterVersion",
"eks:DeleteCluster",
"eks:UpdateClusterConfig",
"eks:UntagResource",
"eks:UpdateNodegroupVersion",
"eks:DescribeNodegroup",
"eks:DeleteNodegroup",
"eks:UpdateNodegroupConfig",
"eks:CreateNodegroup",
"eks:AssociateEncryptionConfig"
],
"Resource": ["arn:*:eks:*:*:cluster/*", "arn:*:eks:*:*:nodegroup/*/*/*"]
},
{
"Effect": "Allow",
"Action": [
"eks:ListAddons",
"eks:CreateAddon",
"eks:DescribeAddonVersions",
"eks:DescribeAddon",
"eks:DeleteAddon",
"eks:UpdateAddon",
"eks:TagResource",
"eks:DescribeFargateProfile",
"eks:CreateFargateProfile",
"eks:DeleteFargateProfile"
],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": ["iam:PassRole"],
"Resource": ["*"],
"Condition": {
"StringEquals": { "iam:PassedToService": "eks.amazonaws.com" }
}
},
{
"Effect": "Allow",
"Action": ["kms:CreateGrant", "kms:DescribeKey"],
"Resource": ["*"],
"Condition": {
"ForAnyValue:StringLike": {
"kms:ResourceAliases": "alias/cluster-api-provider-aws-*"
}
}
}
]
}

Configuring an Azure Infrastructure Provider in the UI

About this task

Before you provision Azure clusters using the NKP user interface, you must first create an Azure infrastructure provider to contain your Azure credentials.

To add a single or multiple AWS infrastructures and contain your Azure credentials, follow these steps:

Procedure

az login

Create an Azure service principal.

az ad sp create-for-rbac --role contributor --name "$(whoami)-konvoy" --scopes=/
subscriptions/$(az account show --query id -o tsv)

Log in to the NKP dashboard.
In the top-left corner, from the dropdown list, select the target workspace.

By default, NKP displays the Global workspace. 5. From the left navigation pane, click Administration > Infrastructure Providers. 6. Click Add Infrastructure Provider.

The Add Infrastructure Provider page appears. 7. In the Workspace dropdown menu, select the workspace for your infrastructure provider.

NKP automatically creates the new infrastructure provider in that workspace. 8. From the Select Infrastructure Provider, select Microsoft Azure and add the following information:

a. In the Subscription ID field, copy and paste the ID from the output of the login command.

b. In the Tenant ID field, copy and paste the tenant value used for creating Azure service principal.

c. In the Client ID field, copy and paste the appId value used for creating Azure service principal.

d. In the Client Secret field, copy and paste the password value used for creating Azure service principal.

e. Click Save.

Configuring a vSphere Infrastructure Provider in the UI

About this task

To add a single or multiple vSphere infrastructures, follow these steps:

Procedure

Log in to the NKP dashboard.
In the top-left corner, from the dropdown list, select the target workspace.

By default, NKP displays the Global workspace. 3. From the left navigation pane, click Administration > Infrastructure Providers. 4. Click Add Infrastructure Provider.

The Add Infrastructure Provider page appears. 5. In the Workspace dropdown menu, select the workspace for your infrastructure provider.

NKP automatically creates the new infrastructure provider in that workspace. 6. From the Select Infrastructure Provider, select vSphere and add the following information:

a. In the Name field, enter a name your infrastructure provider.

b. In the Username field, enter a valid vSphere vCenter username.

c. In the Password field, enter a valid vSphere vCenter user password.

d. In the Host URL field, enter the vSphere vCenter server URL.

Ensure that you add only the domain for the URL, such as vcenter.ca1.your-org- platform.domain.cloud. Do not specify the protocol http:// to avoid errors during cluster creation.

e. (Optional) In the TLS Certificate Thumbprint field, enter a valid SHA-1 thumbprint of the vSphere server TLS certificate.

The TLS certificate thumbprint creates a secure connection to VMware vCenter. If you do not have a thumbprint, NKP might mark your connection as insecure. You only need the TLS certificate thumbprint when you have a self-signed vCenter instance.

To generate the SHA-1 thumbprint for the TLS certificate of vSpheres server, the terminal with access to the NKP CLI and run the following command:

openssl s_client -connect <host url>:443 2>/dev/null | openssl x509 -noout -
fingerprint -sha1

f. Click Save.

Viewing and Modifying Infrastructure Providers

About this task

To view and modify an infrastructure provider, follow these steps:

Procedure

Log in to the NKP dashboard.
In the top-left corner, from the dropdown list, select the target workspace.

By default, NKP displays the Global workspace. 3. From the left navigation pane, click Administration > Infrastructure Providers. The list of available infrastructure providers appears. 4. To modify an infrastructure provider information, click the triple dot vertical icon on that infrastructure provider. 5. Click Edit.

Note: You cannot modify the default or any Nutanix infrastructure provider information.

The Add Infrastructure Provider page appears. 6. After updating the required information specific to your infrastructure provider, click Save.

Deleting an infrastructure provider

Before you begin

Ensure that no clusters are currently associated with the infrastructure provider that you intend to delete.

About this task

To delete an infrastructure provider, follow these steps:

Procedure

Log in to the NKP dashboard.
In the top-left corner, from the dropdown list, select the target workspace.

By default, NKP displays the Global workspace. 3. From the left navigation pane, click Administration > Infrastructure Providers. The list of available infrastructure providers appears. 4. To delete an infrastructure provider, click the triple dot vertical icon next to that infrastructure provider. 5. Click Delete.

Note: You cannot delete the default Nutanix infrastructure provider.

Header, Footer, and Logo Implementation

NKP displays your header and footer banner in a default typeface and size, which cannot be changed.

About this task

The Color selection control uses the style of your browser for its color picker tool. This control allows you to select a color for your header banner:

Procedure

Enter the color’s Hex code.
Select a general color range, and then select a specific shade or tint. The color input uses the style of your browser for its color selection tool.
Select the eyedropper, move it to a sample of the color you want and select once to select that color’s location.

About this task

The Color selection control uses the style of your browser for its color picker tool. This control allows you to select a color for your footer banner:

Procedure

Enter the color’s Hex code.
Select a general color range from the slider bar, and then select a specific shade or tint with your mouse cursor.
Select the eyedropper, move it to a sample of the color you want and select once to select that color’s location.

Adding Your Organization’s Logo Using the Drag and Drop Option

Before you begin

Your logo graphic must meet the following criteria:

Use a suggested file format: PNG, SVG, or JPEG.
The file size cannot exceed 200 KB.

Error messages affecting the file to upload appear below the image in red, inside the shaded logo area.

Note: To provide security against certain kinds of malicious activity, your browser has a same-origin policy for
accessing resources. When you upload a file, the browser creates a unique identifier for the file. This prevents you from
selecting a file more than once.

Procedure

Locate the required file in the MacOS Finder or Windows File Explorer.
Drag and drop an image of the appropriate file type into the shaded area to see a preview of the image and display the file name.

You can select X on the upper-right or Remove on the lower-right to clear the image, if needed. 3. Click Save.

Warning: You cannot select a file for drag-and-drop if it does not have a valid image format.

Adding Your Organization’s Logo Using the Upload Option

Procedure

Select Browse Files.
To clear the image, select X or click the Remove link, if needed.
Click Save.

Applications

AppDeployment Resources

Customizing Your Application on page 328
Printing and Reviewing the Current State of an AppDeployment Resource on page 329

Applications (2)

For example, this is the default AppDeployment for the Kube Prometheus Stack platform application:

apiVersion: apps.kommander.nutanix.io/v1alpha3
kind: AppDeployment
metadata:
name: kube-prometheus-stack
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: kube-prometheus-stack-82.13.6
kind: ClusterApp

Customizing Your Application

About this task

For workspace applications, you can also enable and customize them on a per- cluster basis. For instructions on how to enable and customize an application per cluster in a given workspace, see Cluster-scoped Application for Existing AppDeployments on page 373.

Before you begin

Set the WORKSPACE_NAMESPACE environment variable to the name of the workspace’s namespace where the cluster is attached:

export WORKSPACE_NAMESPACE=<your_workspace_namespace>

You can now copy the following commands without replacing the placeholder with your workspace namespace every time you run a command.

Here’s an example of how to customize the AppDeployment of Kube Prometheus Stack:

Procedure

Provide the name of a ConfigMap with the custom configuration in the AppDeployment.

cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: kube-prometheus-stack
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: kube-prometheus-stack-82.13.6
kind: ClusterApp
configOverrides:
name: kube-prometheus-stack-overrides-attached
EOF

Create the ConfigMap with the name provided in the previous step, which provides the custom configuration on top of the default configuration.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
namespace: ${WORKSPACE_NAMESPACE}
name: kube-prometheus-stack-overrides-attached
data:
values.yaml: |
prometheus:
prometheusSpec:
storageSpec:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 150Gi
EOF

Printing and Reviewing the Current State of an AppDeployment Resource

About this task

You can review all the AppDeployments in a workspace or a specific AppDeployments of an application in a workspace.

Procedure

You can run the following commands to review AppDeployments.

» Review all AppDeployments in a workspace: To review the state of the AppDeployment resource for a specific workspace, run the get command with the name of your workspace. Here’s as example:

nkp get appdeployments -w kommander-workspace

The output displays a list of all your applications:

NAME APP CLUSTERS
[...]
kube-oidc-proxy kube-oidc-proxy-0.3.8 management
kube-prometheus-stack kube-prometheus-stack-82.13.6 management
[...]

» Review a Specific AppDeployment of an application in a workspace: To review the state of a specific AppDeployment of an application, run the get command with the name of the application and your workspace. Here’s an example:

nkp get appdeployment kube-prometheus-stack -w kommander-workspace

The output is as follows:

NAME APP CLUSTERS
kube-prometheus-stack kube-prometheus-stack-82.13.6 management

Note: For more information on how to create, or get an AppDeployment, see the CLI documentation.

Deployment Scope

In a single-cluster environment with a Starter license, AppDeployments do not allow for any customization.

In a single-cluster environment with a Pro license, AppDeployments enable customizing any platform application.

In a multi-cluster environment with an Ultimate license, you can customize AppDeployments to enable resource requirements, project applications, and cluster-scoped applications for existing AppDeployments.

For more information on how to create, or get an AppDeployment, see the CLI documentation.

Logging Stack Application Sizing Recommendations

For information on how you customize your AppDeployments, see AppDeployment Resources on page 327.

Note: When configuring storage for logging-operator-logging-overrides, ensure that you create a
ConfigMap in your workspace namespace for every cluster in that workspace.

Keep in mind that you can configure logging-operator-logging-overrides only through the CLI.

Table 23: Table

Logging Operator Logging Override Config

values.yaml: |-
clusterOutputs:
- name: loki
spec:
loki:
# change
${WORKSPACE_NAMESPACE} to
the actual value of your
workspace namespace
url: http://grafana-
loki-loki-distributed-gateway.
${WORKSPACE_NAMESPACE}.svc.cluster.local:8

extract_kubernetes_labels:
true

configure_kubernetes_labels:
true
buffer:
disabled: true
retry_forever:
false
retry_max_times: 5
flush_mode:
interval
flush_interval:
10s

flush_thread_count: 8
extra_labels:
log_source:
kubernetes_container
fluentbit:
inputTail:
Mem_Buf_Limit: 512MB
fluentd:
bufferStorageVolume:
emptyDir:
medium: Memory
disablePvc: true
scaling:
replicas: 10
resources:
requests:
memory: 1000Mi
cpu: 1000m
limits:
memory: 2000Mi
cpu: 1000m

50 1.4 MB/s

Loki ingester: replicas: 10 distributor: replicas: 2

values.yaml: |-
clusterOutputs:
- name: loki
spec:
loki:
# change (2)
${WORKSPACE NAMESPACE} to

No. of Worker Nodes; Log Generating Load; Application; Suggested Configuration

| — | — | — | — |

100 8.5 MB/s Logging Operator Logging Override Config; values.yaml:; - clusterOutputs: - name: loki

| — | — | — |

Rook Ceph Cluster Sizing Recommendations

For information on how you customize your AppDeployments, see AppDeployment Resources on page 327.

Note: To add more storage to rook-ceph-cluster, copy and paste storageClassDeviceSets
list from the rook-ceph-cluster-your-ceph-version-d2iq-defaults ConfigMap
into your workspace where rook-ceph-cluster is present and then modify count and
volumeClaimTemplates.spec.resource.requests.storage .

rook-ceph-cluster-	your-ceph-version	-d2iq-defaults

Table 24: Table

100 dkp: grafana-loki: additionalConfig:

No. of Worker Nodes	Application	Suggested Configuration

50 Rook Ceph Cluster; cephClusterSpec: labels: monitoring: prometheus.kommander.d2iq.io/select: “true” storage: storageClassDeviceSets: - name: rook-ceph-osd-set1 count: 4 portable: true encrypted: false placement: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone # The nodes in the same rack have the same topology.kubernetes.io/zone label. whenUnsatisfiable: ScheduleAnyway labelSelector: matchExpressions: - key: app operator: In values: - rook-ceph-osd - rook-ceph-osd- prepare - maxSkew: 1 topologyKey: kubernetes.io/ hostname whenUnsatisfiable: ScheduleAnyway labelSelector: matchExpressions: - key: app operator: In values: - rook-ceph-osd - rook- ceph-osd- prepare volumeClaimTemplates: # If there are some faster devices and some slower devices, it is more efficient to use # separate metadata, wal, and data devices. # Refer <https://rook.io/docs/> rook/v1.10/CRDs/Cluster/pvc-cluster/ #dedicated-metadata-and-wal-device- for- osd-on-pvc - metadata: name: data spec: resources: requests: storage: 120Gi volumeMode: Block accessModes: - ReadWriteOnce

| — | — |

Application Management Using the UI

To ensure that the applications are deployed successfully, review them carefully before customizing. There may be dependencies between the applications, which are listed in Platform Applications Dependencies For All Clusters on page 354.

Note: To use the NKP CLI to deploy or uninstall applications, see Deploying Platform Applications Using CLI
on page 353.

Ultimate: Enabling an Application Using the UI

This topic describes how to enable your platform applications from the UI.

Procedure

Log into the NKP UI dashboard.
In the workspace header drop-down list, select your target workspace.
From the sidebar, browse through the available applications from your configured repositories, and select Applications.
Select the three-dot button of the desired application card > Enable.
If available, select a version from the dropdown list. The dropdown list is only visible if there are more than one versions to choose from.
Select the clusters where you want to deploy the application.
For customizations only: to override the default configuration values, follow these steps:

Note: If there are customization Overrides at the workspace and cluster level, they are combined for
implementation. Cluster-level Overrides take precedence over Workspace Overrides.

a. To customize an application for all clusters in a workspace, copy your customized values into the text editor under Workspace Application Configuration or upload your YAML file that contains the values.

someField: someValue

b. To add a customization per cluster, copy the customized values into the text editor of each cluster under Cluster Application Configuration Override or upload your YAML file that contains the values.

someField: someValue

Verify that the details are correct and select Enable.

Warning: There may be dependencies between the applications, which are listed in Platform Applications
Dependencies For All Clusters on page 354. Review them carefully before customizing to ensure that the
applications are deployed successfully.

Ultimate: Customizing an Application Using the UI

You can enable an application and customize it using the UI.

About this task

Note: If you want to enable an application for the first time and customize it, see Ultimate: Enabling an
Application Using the UI on page 334.

To customize the applications that are deployed to a workspace’s cluster using the UI:

Procedure

From the top menu bar, select your target workspace.
From the sidebar, browse through the available applications from your configured repositories, and select Applications.
In the Application card you want to customize, select the three dot menu and Edit.
To override the default configuration values, follow these steps:

Note: If there are customization Overrides at the workspace and cluster levels, they are combined for
implementation. Cluster-level Overrides take precedence over Workspace Overrides.

someField: someValue

someField: someValue

Verify that the details are correct and select Save.

Ultimate: Customizing an Application For a Specific Cluster

You can also customize an application for a specific cluster from the Clusters view:

Procedure

From the sidebar menu, select Clusters.
Select the target cluster.
Select the Applications tab.
Navigate to the target Applications card.
Select the triple dot vertical icon > Edit.

Ultimate: Verifying an Application using the UI

The application has now been enabled.

About this task

To verify that the application is deployed correctly:

Procedure

From the top menu bar, select your target workspace.
Select the cluster you want to verify.

a. Select Management Cluster if your target cluster is the Management Cluster Workspace.

b. Otherwise, select Clusters, and choose your target cluster. 3. Select the Applications tab and navigate to the application you want to verify. 4. If the application was deployed successfully, the status Deployed appears in the application card. Otherwise, hover over the failed status to obtain more information on why the application failed to deploy.

Note: It can take several minutes for the application to deploy completely. If the Deployed or Failed status is not
displayed, the deployment process is not finished.

Ultimate: Disabling an Application Using the UI

You can disable an application using the UI.

About this task

Follow these steps to disable an application with the UI:

Procedure

From the top menu bar, select your target workspace.
From the sidebar, browse through the available applications from your configured repositories, and select Applications.
Select the three-dot button of the desired application card and select Disable.
Follow the instructions on the confirmation pop-up message and select Disable Application.

Pro: Application Management Using the UI

You can enable an application and customize it using the UI.

You can deploy and uninstall an application using the UI.

Note: To use the CLI to deploy or uninstall applications, see Deploying Platform Applications Using CLI on
page 353.

Pro: Enabling an Application Using the UI

About this task

To enable your platform applications from the UI in Kommander:

Procedure

Log into the NKP UI dashboard.
In the workspace header drop-down list, select your target workspace.
From the sidebar to browse through the available applications from your configured repositories, select Applications.
Select the three-dot button of the desired application card > Enable.
If available, select a version from the dropdown list. This dropdown list is only visible if there is more than one version to choose from.
For customizations only: to override the default configuration values, follow these steps:

someField: someValue

Confirm the details are correct and then select Enable.

Warning: There may be dependencies between the applications, which are listed in Platform Applications
Dependencies For All Clusters on page 354. Review them carefully before customizing to ensure that the
applications are deployed successfully.

Pro: Customizing an Application Using the UI

About this task

Note: If you want to enable an application for the first time and customize it, see Ultimate: Enabling an
Application Using the UI on page 334.

To customize the applications that are deployed to your Management Cluster cluster using the UI:

Procedure

Log into the NKP UI dashboard.
In the workspace header drop-down list, select your target workspace.
From the sidebar, browse through the available applications from your configured repositories and select Applications
In the Application card you want to customize, select the three dot menu and Edit.
To override the default configuration values, follow these steps:

Note: If there are customization Overrides at the workspace and cluster level, they are combined for
implementation. Cluster-level Overrides take precedence over Workspace Overrides.

someField: someValue

Verify that the details are correct and select Save.

Pro: Verifying an Application using the UI

The application has now been enabled.

About this task

To ensure that the application is deployed correctly:

Procedure

From the sidebar, select Management Cluster.
Select the Applications tab and navigate to the application you want to verify.
If the application was deployed successfully, the status Deployed appears in the application card. Otherwise, hover over the failed status to obtain more information on why the application failed to deploy.

Note: It can take several minutes for the application to deploy completely. If the Deployed or Failed status is not
displayed, the deployment process is not finished.

Pro: Disabling an Application Using the UI

About this task

To disable an application with the UI:

Procedure

From the sidebar, browse through the available applications from your configured repositories and select Applications
Select the three-dot button of the desired application card and select Disable.
Follow the instructions on the confirmation pop-up message, and select Disable Application.

Deploying Harbor on a Management Cluster

Before you begin

Ensure you complete the following:

Deploy CloudNativePG.
Enable the CloudNativePG application on a NKP cluster.
Get access to any of the following S3-compatible object storage options:
Nutanix Objects: Nutanix Objects are well-integrated with Nutanix and managed by the COSI Driver for Nutanix. Nutanix Objects include a built-in lifecycle management for storage and manages the entire bucket lifecycle.
S3 storage from other providers such as AWS, Azure, GCP, or any enterprise- grade S3 storage. The lifecycle management of external buckets are managed separately.
Integrated Rook Ceph: You can set up integrated Rook Ceph as an S3 bucket within your infrastructure. You can install the Rook Ceph on the same NKP cluster. However, Rook Ceph shares storage with other internal applications which may cause space limitations.

It is recommended you choose an S3-compatible object storage based on the your requirements:

For production environments, choose either enterprise-grade S3 storage or Nutanix Object Storage, because these provide better scalability and lifecycle management.
To avoid external dependency, choose integrated Rook Ceph instead of Nutanix Objects.
Install and configure the Kubernetes CLI kubectl to interact with your cluster.
Appropriate user credentials with permissions to create and manage resources.
Ensure that storage requirements are according to Storage Requirements for Harbor on page 341.

To deploy Harbor on a management cluster, follow these steps:

Procedure

From the top menu bar, select Management Cluster Workspace.
Search for the required application:

» From the sidebar, select Applications, then browse through the available applications,and navigate to the General section.

» In the Filter by Name field, enter CloudNativePG, Harbor, or COSI Driver For Nutanix.

CloudNativePG, Harbor, or COSI Driver For Nutanix are displayed. 3. Enable CloudNativePG.

a. In the search field, enter CloudNativePG, select the triple dot vertical icon on the CloudNativePG application card.

b. Click Enable.

c. (Optional) Enter application overrides as per the environment. For example, you can enter replicaCount: 2. 4. In the search field, enter Harbor, select the triple dot vertical icon on the Harbor application card, and click Enable. 5. In the Enable Workspace Platform Application page, under S3 Configuration > S3 Objects Store drop-down list, select one of the following options:

Nutanix Objects: You do not have to provide additional information after selecting Nutanix Objects.

Note: Ensure that you pre-install and configure the COSI Driver for Nutanix application. For more
information, see Configuring and Enabling COSI Driver for Nutanix in an NKP Cluster on
page 344.

Rook Ceph: You do not have to provide additional information after selecting Rook Ceph.

Note: Ensure that you have enable the Rook Ceph Cluster application.

Manual Bucket Configuration: Provide the necessary URL and credentials for accessing the S3- compatible storage as follows:
Endpoint: The endpoint URL for the S3-compatible object storage.
Access Key: The key used to access the S3-compatible storage.
Secret Key: The secret associated with the access key.
Region: The geographical region where the bucket is hosted.
Bucket: The name of the specific bucket where you store the container images.

Click Enable.
In the Platform Applications page, click the Configuration tab and verify the Harbor application details.
Retrieve the Harbor registry URL:

echo "https://$(kubectl get kommandercluster -A -l 'kommander.d2iq.io/host=true' -o
jsonpath='{.items[0].status.ingress.address}'):5000"

Retrieve default password for Harbor admin:

kubectl get secrets -n ncr-system harbor-admin-password -o
jsonpath='{.data.HARBOR_ADMIN_PASSWORD}' | base64 -d

Create a custom user for Harbor: Harbor has its own user management system, and the UI login credentials are validated by Harbor’s authentication services. You can create separate accounts such as machine accounts for pulling images and human accounts for UI access. For more information on Harbor user management, see Managing Users.

a. Log in to Harbor UI with default credentials:

'admin/<retrieved-password>'

b. Select Administration > Users > NEW USER.

c. Enter information in all the fields and save.

d. (Optional) To assign an administration role to a newly created user, select the custom user and click SET AS ADMIN. An end users can use this user credentials for administration tasks instead of the default user generated by Harbor.

What to do next

On completion of the steps above, you can perform the following steps to deploy harbor:

Preparing a Local Registry Mirror on page 736
Pushing Images to the Registry on page 737

Storage Requirements for Harbor

Ensure that storage requirements are according to the following recommendations.

It is recommended you have 100 GB storage space for most use cases.
To use Harbor as an Open Container Initiative (OCI) registry for charts and other artifacts, ensure that you have sufficient storage based on the size of the artifacts and charts.
To use Harbor as a pull-through cache for temporarily storing images, ensure that you have sufficient storage based on the number of images you need to cache.
To use integrated Rook Ceph, increase the available storage. This is because, integrated Rook Ceph provides 120 GB of space by default which is shared with other services such as logging and backups. When you expand the integrated Rook Ceph storage, note that integrated Rook Ceph uses erasure coding. Therefore, you can use only 75% of the space. For example, if you add 100 GB, you can use only 75 GB to store data. For more information on how to configure auto expansion of Ceph OSDs, see Rook Ceph Cluster Sizing Recommendations and Auto Expansion of OSDs.

For example, to increase the integrated Rook Ceph storage from the default 4 x 40G to 8 x 40G, you can configure the following overrides:

cephClusterSpec:
labels:
monitoring:
prometheus.kommander.d2iq.io/select: "true"
storage:
storageClassDeviceSets:
- name: rook-ceph-osd-set1
count: 8 # Refers to the number of PVs
portable: true
encrypted: false
placement:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone # The nodes in the same rack
have the same topology.kubernetes.io/zone label.
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-osd
- rook-ceph-osd-prepare
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-osd
- rook-ceph-osd-prepare
volumeClaimTemplates:
- metadata:
name: data
spec:
resources:
requests:
storage: 40Gi # Size of each PV
volumeMode: Block
accessModes:
- ReadWriteOnce

Authentication types for launching workloads

Configure Certificate authority (CA) certificates to ensure the following:

Secure TLS connections and cluster-wide credentials for management and workload clusters.
Enabling secure and streamlined access to Harbor.
Authenticated and encrypted communication between NKP clusters and the registry.
Protection of sensitive image data.
Simplify deployments.

Configure the CA certificate only if the NKP cluster where you install Harbor is installed is using an ingress TLS certificate that was signed by a CA that is not included in the OS trust bundles (private CA).

The authentication is different for Nutanix infrastructure and non-Nutanix infrastructure.

Authentication for Nutanix Providers

You can configure NKP clusters using the Nutanix provider with multiple image registries, including authentication and the CA certificate setup. You can configure through the CLI or UI. However, there are limited options through the UI only when launching a new NKP cluster. NKP clusters with Nutanix provider can be configured with a registry mirror.

Configuring Through NKP UI

Configure the Harbor registry of the management cluster as a private registry when launching a managed cluster. You can configure this through the NKP UI

Image Registry Mirror: This is a general registry for pulling images from public registries. The NKP platform application images will be pulled from this registry. If you don’t have a registry mirror, provide docker.io credentials here to avoid DockerHub registry pull limits.
Private Registry: Enter information about the Harbor registry :

URL: Specify the location where the Harbor registry is running.

Username and Password: Specify the username and password. The username and password are required for cluster-wide authentication of the newly launched cluster.

CA Certificate: Provide a custom CA certificate if Harbor is running on an NKP cluster with ingress TLS certificate that was signed by a private CA.

You cannot change this configuration using the UI. For reconfiguring private image registries, reconfigure through CLI.

Configuring Through CLI

Use CLI to configure additional image registries on NKP Nutanix clusters after the initial cluster launch. You can adjust configuration on management and managed clusters. It is recommended to create a dedicated Harbor user or a Harbor robot user and provide a dedicated set of credentials for each NKP cluster.

Create a secret with credentials and CA certificate that will be used by containerd.

REGISTRY_USERNAME="username"
REGISTRY_PASSWORD="password"
kubectl create secret generic harbor-registry-credentials \
--from-literal username=$REGISTRY_USERNAME \
--from-literal password=$REGISTRY_PASSWORD \
--from-file=ca.crt=<(kubectl -n kommander get nkpcluster -l 'kommander.d2iq.io/
host=true' -o jsonpath='{.items[0].kommanderCluster.status.ingress.caBundle}' | base64
-d)

Include username and password only if cluster-wide authentication is required.
Include ca.crt if Harbor is installed on cluster with a TLS certificate that is signed by a private CA
You can create a secret with ca.crt without providing a username and password.
If you include the username and password in the secret, the registry is authenticated at the cluster level.
If only the ca.crt value will be present in the secret then the containerd will be able establish verified TLS connections to the private registry but won’t be able to pull images from Harbor projects that require authentication. A Kubernetes imagePullSecret can be then used to provide credentials per workload.
Get the address of Harbor installed on management cluster:

echo "https://$(kubectl -n kommander get nkpcluster -l 'kommander.d2iq.io/host=true'
-o jsonpath='{.items[0].kommanderCluster.status.ingress.address}'):5000"

Modify the NKP cluster resource and add new image registry configuration to the .spec.capiCluster.topology.variables.imageRegistries section and replace the HARBOR_ADDRESS with the address from the previous step.

apiVersion: clusters.nkp.nutanix.com/v1alpha1
kind: NKPCluster
metadata:
name: <NAME>
namespace: kommander
spec:
capiCluster:
topology:
variables:
- name: clusterConfig
value:
imageRegistries:
- url: <HARBOR_ADDRESS>
credentials:
secretRef:
name: harbor-registry-credentials

Changing the image registry configuration will result in a rolling upgrade of all nodes in a cluster.

Configuring through Harbor

Harbor supports multiple authentication modes, including database, LDAP/AD, and OpenID Connect (OIDC), which should be selected during initial deployment. Once user accounts are created in database mode, switching to another authentication method is not allowed. For more information, see Configuring Authentication.

You can install Harbor within the management cluster using catalog applications. After you install, you must ensure you have the correct authorizations for image access. You can use Harbor to store and scan images from both management and managed clusters. Therefore, you must manage authorizations as per your requirements.

Limitations for non-Nutanix Infrastructure Providers

For non-Nutanix infrastructure providers such as AWS, vSphere, or GCP, you can provide Harbor configuration only as a single cluster-wide registry mirror. Therefore, each image pulled to the cluster will be first attempted with the registry mirror, even for container images that are hosted in public registries like docker.io which can increase the load on Harbor.

The limitations of non-Nutanix infrastructure providers are as follows:

You can configure and apply Harbor settings only during the initial creation of the cluster.
You can only use Harbor as a registry mirror which you can use as a local cache for images pulled from a remote registry.
After you launch the cluster, you cannot change the Harbor configuration. You cannot modify the registry URL, credentials, or any other related settings.

Configuring and Enabling COSI Driver for Nutanix in an NKP Cluster

This topic provides the instructions to configure the COSI Driver for Nutanix on an NKP cluster to enable seamless integration with Nutanix Object Storage.

About this task

To configure the COSI Driver for Nutanix on an NKP cluster, follow these steps:

Procedure

Configure Nutanix Object Storage in Prism Central, if not configured already.

a. Enable Objects in Nutanix Marketplace.

For more information, see Enabling Objects.

b. Create an Object Store from Prism Central.

For more information, see Creating or Deploying an Object Store on Prism Central. 2. Configure the COSI Driver for Nutanix in the NKP UI.

a. Log in to the NKP Management Cluster UI.

b. Select Management Cluster Workspace from the dropdown menu.

c. On the left navigation page, select Applications as the storage provider.

d. Search for COSI Driver for Nutanix.

e. Select the triple dot vertical icon on the application card and click Enable.

f. In the Enable Workspace Platform Application dialog box, enter the following details:

Prism Central Endpoint: Enter the PC endpoint URL as <https://><PRISM_CENTRAL_IP>:9440
Prism Central Username: Enter the PC administrator username.
Prism Central Password: Enter the PC administrator password.
Enable TLS Validation: You can provide the Prism Central CA (Certificate Authority) certificate. This is a recommended but optional step. This certificate is used to validate the TLS connection and ensure secure communication between the NKP Cluster and Prism Central.

To provide the certificate, select the Enable TLS Validation checkbox. Then, you can either select the Upload File button to import the CA certificate, or paste the certificate into the text field.

Nutanix Object Storage Endpoint: Enter the Nutanix Object Storage Endpoint URL: http:// <nutanix-object-endpoint-ip/fqdn>:<port-number> or https://<nutanix-object- endpoint-ip/fqdn>:<port-number>
The Nutanix Object Storage Endpoint URL is referred as Public Network static IPs in Nutanix Objects.
If you do not enter a port number , then port 80 is configured as the default port.
Enable TLS Validation: You can provide the Nutanix Object Storage CA (Certificate Authority) certificate. This is a recommended but optional step. This certificate is used to validate the TLS connection and ensure secure communication between the NKP Cluster and Nutanix Object Storage .

To provide the certificate, select the Enable TLS Validation checkbox. Then, you can either select the Upload File button to import the CA certificate, or paste the certificate into the text field.

Nutanix Object Access Key
Nutanix Object Secret Key

Note: TLS requires a secure connection. Please ensure the URL starts with https:// when you enable TLS.

g. Click Enable. The COSI Driver for Nutanix application is enabled.

Using Harbor on an NKP Cluster

Before you begin

Ensure you complete the following:

Install NKP Management or Workload Cluster running on Nutanix infrastructure.
Access to the NKP UI with administrative privileges.
Install and configure the Kubernetes CLI kubectl to interact with your cluster.
Access to Kubernetes cluster with the necessary permissions to create secrets and modify cluster configurations.
Deploy and enable Harbor.
Create Harbor registry credentials.
Install Docker on your local machine.

For more information, see Managing Users.

A valid container image to push to Harbor.
Network connectivity between the cluster nodes and Harbor.

To use Harbor on an NKP management cluster on Nutanix infrastructure, follow these steps:

Procedure

Use admin credentials to create a robot account for every cluster.

For more information on how to create a robot account , see Create Project Robot Accounts. 2. Create a secret in Kubernetes for Harbor authentication using the robot account credentials you created in the previous step. 3. Modify the managed cluster resource:

kubectl edit nkpcluster -n kommander <WORKLOAD_CLUSTER_NAME>

Update the imageRegistries:

imageRegistries:
- url: https://<HARBOR_ADDRESS>:5000
credentials:
secretRef:
name: harbor-registry-credentials

Save and verify the changes. 5. Export the Harbor registry address:

export HARBOR_ADDRESS="$(kubectl -n kommander get kommandercluster -l
'kommander.d2iq.io/host=true' -o jsonpath='{.items[0].status.ingress.address}')"

This authenticates Docker with Harbor. 6. Modify deployment configuration to use Harbor:

spec:
containers:
- name: my-app
image: <HARBOR_ADDRESS>:5000/library/my-app:latest

a. Deploy the application:

kubectl apply -f my-app-deployment.yaml

b. Verify the deployment:

kubectl get pods -n <namespace>

(Optional) Configure authentication for NKP clusters that need to access images for launching workloads:

a. Log on to Harbor, navigate to the dashboard, and create a new username and password.

b. In the management cluster, create a secret (“harbor-registry-credentials” ) to store your Harbor credentials (“username”/“password”) This Harbor registry credentials allow the Management cluster to pull images from Harbor on the Management Cluster to deploy an app.

c. After the deployment, fetch the Ingress TLS certificate (TLS certificate) in case it is generated.

kubectl get kommandercluster -n kommander -l 'kommander.d2iq.io/host=true' -o
jsonpath='{.items[0].status.ingress.caBundle}' | base64 -d > ca.crt

Store the ca.crt for future configuration.

d. To enable secure access to Harbor, update the registry configuration by creating a secret.

REGISTRY_USERNAME="username"
REGISTRY_PASSWORD="password"
kubectl create secret generic harbor-registry-credentials \
--from-literal username=$REGISTRY_USERNAME \
--from-literal password=$REGISTRY_PASSWORD \
--from-file=ca.crt=<(kubectl -n kommander get kommandercluster -o
jsonpath='{.status.ingress.caBundle}' | base64 -d)

Note: Include the ca.crt only if the Harbor installation uses a private CA signed certificate.

Using Harbor as a Mirror Registry or a Private Registry for Clusters

This topic provides instructions for configuring Harbor running on the NKP management cluster as a mirror registry for NKP clusters. Using Harbor as a mirror registry ensures that clusters pull images from a local registry, reducing latency, external dependencies, and security risks. This topic described the process to configure Harbor as a mirror registry or a private registry through the NKP UI.

About this task

Note: You can deploy Harbor as mirror registry for clusters across all infrastructure providers.

Before you begin

NKP Management or Workload Cluster installed and running on your infrastructure.
Access to the NKP UI with administrative privileges.
The Kubernetes CLI kubectl is installed and configured to interact with your cluster.
Access to Kubernetes cluster with the necessary permissions to create secrets and modify cluster configurations.
Deploy and enable Harbor.
Create Harbor registry credentials.
Install Docker on your local machine.

To use Harbor as a mirror registry or a private registry on an NKP cluster, follow these steps:

Procedure

Deploy Harbor as described in Deploying Harbor on a Management Cluster on page 338.
Log into the Harbor UI using the following credentials.

Username: admin
Password: Retrieve the password.

kubectl get secrets -n ncr-system harbor-admin-password -o
jsonpath='{.data.HARBOR_ADMIN_PASSWORD}' | base64 -d

Go to Users > New User and complete the following fields: Use robot accounts for machine or cluster authentication. For more information on how to create a robot account , see Create Project Robot Accounts.

Username: <custom_user>
Email
First and last name
Password: <secure_password>
Confirm Password

a. Click OK.

b. To provide administrator privileges for users, select the check box of the corresponding user and click SET AS ADMIN. 4. Create a secret in Kubernetes for Harbor authentication using the robot account credentials you created in the previous step. 5. To configure the NKP management cluster to use Harbor, retrieve the Harbor address.

echo "https://$(kubectl -n kommander get kommandercluster -l 'kommander.d2iq.io/
host=true' -o jsonpath='{.items[0].status.ingress.address}'):5000"

a. Modify the cluster resource through the NKP UI.

Select Clusters > Create Cluster > Image Registries

b. Enter details. of Harbor.

c. In the CA Certificate section, click Upload File to upload the ca.crt certificate. 6. To deploy a managed cluster with Harbor mirror registry.

a. Navigate to Clusters > Create New Cluster.

b. Under Private Registry, select Harbor Mirror Registry.

c. Upload the ca.crt file.

d. Verify the managed cluster configuration:

kubectl get nkpclusters -A
kubectl get nkpcluster -n <WORKSPACE_NAMESPACE> <NKPCLUSTER_NAME> -o yaml

To deploy and validate the image usage from the Harbor mirror registry, push and pull an image to the Harbor mirror registry.

docker pull alpine
docker tag alpine $HARBOR_ADDRESS:5000/library/alpine
docker push $HARBOR_ADDRESS:5000/library/alpine

a. Deploy a pod from the Harbor mirror registry:

apiVersion: v1
kind: Pod
metadata:
name: image-registry-test
spec:
containers:
- name: alpine
image: $HARBOR_ADDRESS:5000/library/alpine
command: ["/bin/sh", "-c", "sleep 100000"]
kubectl apply -f image-registry-pod-test.yaml

To validate the registry configuration on a managed cluster, check the secret on the managed cluster.

kubectl get secret -n <WORKSPACE_NAMESPACE> <NKPCLUSTER_NAME>-image-registry-
credentials -o yaml

Pushing a Container Image to Harbor

You can push a container image to Harbor.

About this task

To push a container image to Harbor, follow these steps:

Procedure

Configure TLS trust.

For more information, see Configuring Docker with a Harbor TLS Certificate on page 350. 2. Authenticate the local docker daemon. Ensure that you have the HARBOR_ADDRESS environment variable before running the commands in this procedure. You can configure this when Harbor is running on a management cluster.

export HARBOR_ADDRESS="$(kubectl -n kommander get kommandercluster nkp-cluster -o
jsonpath='{.status.ingress.address}')"

Push a container image to Harbor:

a. Pull a sample image from Docker Hub:

docker pull alpine:latest

b. Tag the image to match the address of Harbor:

docker tag alpine:latest $HARBOR_ADDRESS:5000/library/alpine:latest

c. Push the image to Harbor:

docker push $HARBOR_ADDRESS:5000/library/alpine:latest

d. Verify if the image is available in the Harbor UI.

Configuring Docker with a Harbor TLS Certificate

If Harbor is running in an NKP cluster that uses an ingress TLS certificate that is from a private Certificate Authority or a self-signed certificate, then configure the Docker daemon to trust the ingress certificate.

About this task

Perform this procedure on the user machine where the docker CLI is used to push and pull from the Harbor registry.

To configure the Docker daemon to trust the ingress certificate, follow these steps:

Procedure

Verify if the cluster is using a custom certificate:

kubectl get nkpcluster -n kommander -l 'kommander.d2iq.io/host=true' -o
jsonpath='{.items[0].kommanderCluster.status.ingress.caBundle}'

Switch to a privileged user as sudo -s and configure the Docker daemon to trust the ingress certificate.

sudo mkdir -p /etc/docker/certs.d/$HARBOR_ADDRESS:5000/
kubectl -n kommander get nkpcluster -l 'kommander.d2iq.io/host=true' -o
jsonpath='{.items[0].kommanderCluster.status.ingress.caBundle}' | base64 -d | sudo
tee /etc/docker/certs.d/$HARBOR_ADDRESS:5000/ca.crt > /dev/null

(Optional) If you use MacOS, save these certificates to the ~/.docker/certs.d/ directory. For more information, see

Directory structures for certificates 4. Restart Docker Desktop after making any certificate configuration changes.

Platform Applications

For more information on the applications supported for management and workload clusters, see Supported Platform Applications on page 350.

This topic contains all the applications that are currently for use with Nutanix Kubernetes Platform (NKP).

Table 25: Supported Applications per NKP Licenses

AI Navigator App

X X

Col7

Applications; Management Cluster; Workload Cluster; Management Cluster; Workload Cluster; Management Cluster; Workload Cluster

AI Navigator Cluster Info Agent

X X

AI Navigator RAG

X X

Centralized Grafana

Cert Manager X X X X X X

Cilium Hubble Relay Traefik

X X X X

Cloudnative PG

X X X X

COSI Driver Nutanix

X X X X

Dex X X X

Dex K8s Authenticator

X X X

External DNS X X X X

External Secrets

X X X

Fluent Bit X X X X

Flux X X X X X X

Gatekeeper X X X X X X

Gateway API CRDs

X X X X X X

Git Operator X X X

Grafana Logging

X X X X

Grafana Loki X X X X

Harbor X X

Istio Helm X X X X

Jaeger X X X X

Karma X

Kiali X X X X

Knative X X X X

Col7

Applications; Management Cluster; Workload Cluster; Management Cluster; Workload Cluster; Management Cluster; Workload Cluster

Kube OIDC Proxy

X X X X X X

Kube Prometheus Stack

X X X X

OpenCost X X

Kubernetes Dashboard

X X X X

Kubetunnel X X X X

Logging Operator

X X X X

NKP Insights X X

NKP Insights Management

NKP Pulse Management

X X X

NKP Pulse Workspace

X X X

NVIDIA GPU Operator

X X X X

Project Grafana Logging

X X

Project Grafana Loki

X X

Project Logging

X X

Prometheus Adapter

X X X X

Reloader X X X X X X

Rook Ceph X X X X

Rook Ceph Cluster

X X X X

Thanos X

Traefik X X X X X X

Traefik ForwardAuth

X X X

Col7

Applications; Management Cluster; Workload Cluster; Management Cluster; Workload Cluster; Management Cluster; Workload Cluster

Traefik Forward Auth Mgmt

X X X

Velero X X X X

vGPU Token Operator

X X X X

Deploying Platform Applications Using CLI

Before you begin

Before you begin, you must have:

A running cluster with Kommander installed.
An existing Kubernetes cluster attached to Kommander (see Attaching an Existing Kubernetes Cluster on page 474).
Determine the name of the workspace where you wish to perform the deployments. You can use the nkp get workspaces command to view the list of workspace names and their corresponding namespaces.
Set the WORKSPACE_NAMESPACE environment variable to the name of the workspace’s namespace where the cluster is attached:

export WORKSPACE_NAMESPACE=<workspace_namespace>

Set the WORKSPACE_NAME environment variable to the name of the workspace where the cluster is attached:

export WORKSPACE_NAME=<workspace_name>
Warning: From the CLI, you can enable applications to deploy in the workspace. Verify that the application has
successfully deployed through the CLI.

To create the AppDeployment, enable a supported application to deploy to your existing attached or managed cluster with an AppDeployment resource (see AppDeployment Resources on page 327).

Procedure

Obtain the APP ID and Version of the application. For more information, see NKP Release Notes on the Nutanix Support Portal. You must add them in the <APP-ID>-<Version> format, for example, kiali-2.24.0.

Col7

Applications; Management Cluster; Workload Cluster; Management Cluster; Workload Cluster; Management Cluster; Workload Cluster

Run the following command and define the –app flag to specify which platform application and version will be enabled.

nkp create appdeployment kiali --app kiali-2.24.0 --workspace ${WORKSPACE_NAME}

Note:

The –app flag must match the APP NAME from the list of available platform applications.
Observe that the nkp create command must be run with the WORKSPACE_NAME instead of the WORKSPACE_NAMESPACE flag.

This instructs Kommander to create and deploy the AppDeployment to the KommanderClusters in the specified WORKSPACE_NAME.

Verifying the Deployed Platform Applications

Procedure

Connect to the attached cluster and watch the HelmReleases to verify the deployment. In this example, we are checking whether kiali is deployed correctly.

kubectl get helmreleases kiali -n ${WORKSPACE_NAMESPACE} -w

HelmRelease must be marked as Ready.

NAMESPACE NAME READY STATUS AGE
workspace-test-vjsfq kiali True Release reconciliation succeeded 7m3s

Some supported applications have dependencies on other applications. For more information, see Platform Applications Dependencies For All Clusters on page 354.

Platform Applications Dependencies For All Clusters

As you deploy or troubleshoot platform applications, understand how they interact and rely on other applications as dependencies. If a required dependency of a platform application fails to deploy, any dependent application also fails to deploy. For more information on the applications supported per license for management and workload clusters, see Supported Platform Applications on page 350.

To view the applications that are enabled or disabled in each category, verify their status:

kubectl get apps,clusterapps,appdeployments -A

After enabling the applications, deploy the HelmReleases.

To check whether the HelmReleases are deployed, connect to the attached cluster and monitor their status. In this example, check if kiali is deployed successfully:

kubectl get helmreleases kiali -n ${WORKSPACE_NAMESPACE} -w

You must see the HelmRelease marked as Ready:

NAMESPACE NAME READY STATUS AGE
workspace-test-vjsfq kiali True Release reconciliation succeeded 7m3s

Foundational Applications

Provides the foundation for all platform application capabilities and deployments on management and managed clusters. You must enable the foundational applications for any platform applications to work properly.

The foundational applications comprise of the following platform applications:

flux: A tool that keeps Kubernetes clusters in sync with sources of configuration such as Git repositories, and automates updates to configuration when there is new code to deploy. For more information, see Flux.
reloader: A Kubernetes controller that automatically reloads values for ConfigMaps and Secrets in Pods. For more information, see Reloader.
traefik: Provides an HTTP reverse proxy and load balancer. For more information, see Traefik.

Table 26: Foundational Applications

Flux

[Applicable for both management and managed clusters]

kommander-flux Yes N/A N/A

Reloader

[Applicable for both management and managed clusters]

reloader No N/A N/A

Traefik

[Applicable for both management and managed clusters]

traefik No N/A

cert-manager
reloader

Logging

Collects logs over time from Kubernetes and applications deployed on management and managed clusters. Logging enables you to visualize and query the aggregated logs.

fluent-bit: An open source and multi-platform log processor tool that is designed to be a generic Swiss knife for logs processing and distribution. For more information, see Fluent Bit documentation.
grafana-logging: A logging dashboard to view the logs aggregated to Grafana Loki. For more information, see Grafana.
grafana-loki: A horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. For more information, see Grafana Loki.
logging-operator: An operator automates the deployment and configuration of a Kubernetes logging pipelines. For more information, see Logging Operator.
project-grafana-logging: A logging dashboard to view the project logs aggregated to Grafana Loki. For more information, see Grafana.
project-grafana-loki: A horizontally-scalable, highly-available, multi- tenant log aggregation system inspired by Prometheus. For more information, see Grafana Loki.
project-logging: A Nutanix application that defines resources for the logging operator, which are used for directing project logs to its Loki application. For more information, see Logging operator.
Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

rook-ceph-cluster: The Ceph cluster is managed by Rook Ceph operator and it provides storage services in cloud native environments. For more information, see Ceph Cluster Helm Chart.
rook-ceph: Rook orchestrates the Ceph storage solution, with a specialized Kubernetes operator to automate management. For more information, see Ceph Operator Helm Chart.

Table 27: Logging

Fluent Bit

[Applicable for both management and managed clusters]

fluent-bit No N/A N/A

Grafana Logging

[Applicable for both management and managed clusters]

grafana-logging No N/A grafana-loki

Grafana Loki

[Applicable for both management and managed clusters]

grafana-loki No N/A rook-ceph-cluster

Logging Operator

[Applicable for both management and managed clusters]

logging-operator No N/A N/A

Project Grafana Logging

[Applicable for both management and managed clusters]

project-grafana- logging

No N/A project-grafana-loki

Project Grafana Loki

[Applicable for both management and managed clusters]

project-grafana- loki

No N/A

project-logging
grafana-loki
logging-operator
Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Project Logging

[Applicable for both management and managed clusters]

project-logging No N/A logging-operator

Rook Ceph Cluster

[Applicable for both management and managed clusters]

rook-ceph-cluster No N/A rook-ceph

Note: You can
override the
configuration
to remove the
dependency, as
needed.

Rook Ceph

[Applicable for both management and managed clusters]

rook-ceph No N/A N/A

Note: By default, NKP deploys only the monitoring stack and not the logging stack.

Monitoring

Enables you to collect performance and cost metrics for Kubernetes and applications deployed on clusters. Monitoring visualizes metrics and evaluates rule-based expressions to trigger alerts when specific conditions are met.

centralized-grafana: A dashboard that collects centralized metrics and provides visualization and alerting capabilities. For more information, see Grafana.
opencost: Provides real-time cost visibility and insights for Kubernetes workloads. OpenCost helps you to continuously reduce your cloud costs. For more information, see OpenCost.
karma: An alert dashboard for the Prometheus alertmanager. For more information, see Karma.
kube-prometheus-stack: A stack of applications that collect metrics to provide visualization and configuring alerts. For more information, see Kube Prometheus Stack.

Note: Kube Prometheus Stack includes Prometheus, Prometheus Alertmanager, and Grafana. For more
information, see Prometheus, Alertmanager, and Grafana.

kubernetes-dashboard: A general purpose, web-based UI that enables you to manage applications deployed in the cluster, troubleshoot them, and manage the Kubernetes clusters. For more information, see Kubernetes Dashboard.
prometheus-adapter: Provides cluster metrics from Prometheus. For more information, see Prometheus Adapter.
thanos: An open-source, highly available Prometheus-compatible solution that offers unlimited data retention through object storage while remaining fully compatible with existing Prometheus tools. For more information, see Thanos.
Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Table 28: Monitoring

Centralized Grafana

[Only applicable for management clusters]

centralized-grafana Yes N/A N/A

OpenCost

[Applicable for both management and managed clusters]

opencost No N/A kube-prometheus- stack

Karma

[Only applicable for management clusters]

karma Yes karma-traefik N/A

Full Prometheus Monitoring Stack

[Applicable for both management and managed clusters]

kube-prometheus- stack

No N/A N/A

Kubernetes Dashboard

[Applicable for both management and managed clusters]

kubernetes- dashboard

No N/A traefik

Prometheus Adapter

[Applicable for both management and managed clusters]

prometheus- adapter

No kube-prometheus- stack

N/A

Thanos

[Only applicable for management clusters]

thanos Yes N/A N/A

Security

Enables you to manage the security constraints and capabilities for the clusters.

cert-manager: Automates the management and issuance of TLS certificate from various issuing sources. For more information, see cert-manager.
external-secrets: External secrets operator is a Kubernetes operator that integrates external secret management systems such as AWS secrets manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Secrets Manager, Pulumi ESC, and so on. The external secrets operator reads information from external APIs and automatically injects the values into a Kubernetes secret. For more information, see External Secrets Operator.
Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

gatekeeper: A customizable admission webhook for Kubernetes to enforce policies executed by open policy agent (OPA). For more information, see Gatekeeper.

Table 29: Security

Cert-Manager

[Applicable for both management and managed clusters]

cert-manager No N/A N/A

External Secrets Operator

[Applicable for both management and managed clusters]

external-secrets No N/A N/A

Gatekeeper

[Applicable for both management and managed clusters]

gatekeeper No N/A N/A

Single Sign On (SSO)

A group of platform applications that enable single sign on (SSO) integration on attached clusters. SSO provides a centralized system to securely connect the attached clusters to the centralized authority hosted on the managed cluster.

dex-k8s-authenticator: A helper web application that integrates with one or more Dex identity services and provides a unified web UI for managing one or more clusters. For more information, see Dex K8 Authenticator.
dex: An identity service that leverages openID connect (OIDC) to provide authentication for other applications. For more information, see Dex.
kube-oidc-proxy: A proxy that consistently authenticates the managed Kubernetes clusters on multi-cloud using openID connect (OIDC). For more information, see Kube-OIDC-Proxy.
traefik-forward-auth and traefik-forward-auth-mgmt: Installs a forward authentication service from Nutanix that provides Google OAuth based authentication for Traefik. For more information, see Traefik ForwardAuth.

Table 30: SSO

Dex K8s Authenticator

[Only applicable for management clusters]

dex-k8s- authenticator

Yes

dex
kommander
traefik

N/A

Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Dex

[Only applicable for management clusters]

dex Yes N/A N/A

Kube OIDC Proxy

[Applicable for both management and managed clusters]

kube-oidc-proxy Yes N/A

cert-manager
traefik

Traefik ForwardAuth

[Only applicable for managed clusters]

traefik-forward-auth Yes traefik N/A

Traefik ForwardAuth Mgmt

[Only applicable for management clusters]

traefik-forward- auth-mgmt

Yes

dex
kommander
traefik

N/A

Backup

A platform application that assists you securely back up and restore your environment.

velero: An open source tool to safely back up and restore Kubernetes cluster resources, perform disaster recovery, and migrate resources and persistent volumes across Kubernetes cluster. For more information, see Velero.

Table 31: Backup

Velero

[Applicable for both management and managed clusters]

velero No N/A

rook-ceph-cluster
Velero is an optional dependency. You can override the configuration to remove the dependency, as needed.

Review the resource requirements of workspace platform application and ensure that the attached clusters have sufficient resources. For more information, see Workspace Platform Application Defaults and Resource Requirements on page 726

Platform applications are bundled and tested as a single unit. You must deploy or upgrade the entire bundle in one operation per workspace. As a result, all clusters within a workspace run the same set and version of the deployed platform applications.

Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Service Mesh

Enables service mesh deployment on clusters to manage microservices in cloud- native applications. Service mesh offers several benefits, including:

Observability into communication
Secure connections
Automated retries and back-off for failed requests
istio-helm: Addresses the challenges faced with distributed or microservices architecture. For more information, see Istio.
jaeger: A distributed tracing system used for monitoring and troubleshooting microservices-based distributed systems. For more information, see Jaeger.
kiali: A management console for an Istio-based service mesh that provides dashboards, observability tools, and enables you to operate your service with robust configuration and validation. For more information, see Kiali.

Table 32: Service Mesh

Istio Helm

[Applicable for both management and managed clusters]

istio-helm No kube-prometheus- stack

N/A

Jaeger

[Applicable for both management and managed clusters]

jaeger No istio N/A

Kiali

[Applicable for both management and managed clusters]

kiali No istio jaeger

(optional for monitoring purposes)

Nutanix Kubernetes Platform (NKP) AI Navigator

Coupled with the AI Navigator, it analyses the data of your cluster to include live information on queries made through the AI Navigator chatbot.

ai-navigator-info-api: A chatbot with advanced artificial intelligence used for seamless Kubernetes cluster management and NKP environment optimization. The AI Navigator application is enabled by default and included in the AI Navigator.
ai-navigator-cluster-info-agent: Includes the cluster information agent and API service. The AI Navigator Cluster Info Agent collects information about the NKP cluster and sends it to the AI Navigator service.
ai-navigator-rag: A component that provides NKP AI Navigator with documentation and interfaces it with an external large language models (LLM).

For more information, see AI Navigator on page 1100.

Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Table 33: NKP AI Navigator

AI Navigator App

[Only applicable for management clusters]

ai-navigator-app No N/A N/A

AI Navigator Cluster Info Agent

[Only applicable for management clusters]

ai-navigator-cluster- info-agent

No cloudnative-pg N/A

AI Navigator RAG

[Only applicable for management clusters]

ai-navigator-rag No cloudnative-pg N/A

NKP Insights

Analyses the health of your health, detects current and future anomalies, and creates alerts that offer root cause analysis and recommended solutions.

nkp-insights: Also known as NKP Insights engine that collects events and metrics from your cluster and applies rule-based heuristics to detect potential anomalies of across different levels of severity.
nkp-insights-management: Also known as NKP Insights Management that collects anomaly alerts from one or more clusters and displays root cause analysis and recommended solutions in the UI.

For more information, see Nutanix Kubernetes Platform Insights Guide on page 1111.

Table 34: NKP Insights

NKP Insights

[Applicable for both management and managed clusters]

nkp-insights No

kube-prometheus- stack
cloudnative-pg

N/A

NKP Insights Management

[Only applicable for management clusters]

nkp-insights- management

Yes kubefed N/A

General

cloudnative-pg: A comprehensive open source platform designed to seamlessly manage PostgreSQL databases within Kubernetes environments. For more information, see CloudNativePG.
cosi-driver-nutanix: An interface that seamlessly manage provisioning of object storage resources on Nutanix platforms within Kubernetes environments adhering to COSI standard. For more information, see COSI Driver Nutanix.
Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

harbor: A cloud native registry used for storing, signing, and scanning container images. For more information, see Harbor.
knative: A Kubernetes-based platform to build, deploy, and manage modern serverless workloads. For more information, see Knative Technical Overview.
nvidia-gpu-operator: The NVIDIA GPU Operator manages the NVIDIA GPU resources in Kubernetes cluster and automates tasks related to bootstrapping the GPU nodes. For more information, see NVIDIA GPU Operator.
vgpu-token-operator: A Kubernetes native way to handle VGPU tokens and license validation. For more information, see vGPU Token Operator.

Table 35: General

CloudNativePG

[Applicable for both management and managed clusters]

cloudnative-pg No N/A N/A

COSI Driver Nutanix

[Applicable for both management and managed clusters]

cosi-driver-nutanix No N/A N/A

Harbor

[Only applicable for management clusters]

harbor No cloudnative-pg N/A

Knative

[Applicable for both management and managed clusters]

knative No istio N/A

NVIDIA GPU Operator

[Applicable for both management and managed clusters]

nvidia-gpu-operator No N/A N/A

vGPU Token Operator

[Applicable for both management and managed clusters]

vgpu-token- operator

No N/A N/A

Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Tools

external-dns: ExternalDNS synchronizes the exposed Kubernetes services and ingresses with the DNS providers. For more information, see External DNS.

Table 36: Tools

External DNS

[Applicable for both management and managed clusters]

external-dns No N/A N/A

Networking

Interna apps: - Need category, definition, and reference links for the following:

cilium-hubble-relay-traefik: Exposes the Cilium Hubble Relay gRPC service through Traefik with TLS passthrough enabled, since Hubble Relay handles mTLS termination.
gateway-api-crds: Gateway API is an interface that defines a set of Kubernetes resources and models service networking in Kubernetes. For more information, see Gateway API.
kubetunnel: Enables access to the API server of a remote cluster without ingress by establishing a secure tunnel. For more information, see Cluster Attachment with Networking Restrictions on page 492

Table 37: Networking

Cilium Hubble Relay Traefik

[Applicable for both management and managed clusters]

cilium-hubble-relay- traefik

Yes traefik N/A

Gateway API CRDs

[Applicable for both management and managed clusters]

gateway-api-crds Yes N/A N/A

Kubetunnel

[Applicable for both management and managed clusters]

kubetunnel Yes N/A N/A

Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

NKP Pulse

nkp-pulse-management: An application or component within NKP that automatically sends telemetry data to improve diagnostics and product performance. NKP Pulse management simplifies Kubernetes lifecycle management by provisioning, scaling, and upgrades across hybrid and multi-cloud environments.
nkp-pulse-workspace: An application or component within NKP that handles pulse telemetry and potentially integrates with the pulse workspace.

For more information, see Pulse Telemetry in NKP on page 693.

Table 38: NKP Pulse

NKP Pulse Management

[Only applicable for management clusters]

nkp-pulse- management

Yes N/A N/A

NKP Pulse Workspace

[Only applicable for managed clusters]

nkp-pulse- workspace

Yes N/A N/A

External Secrets Operator

From NKP 2.17, the management cluster includes ESO pre-installed by default.

The two key roles of ESO in NKP are:

Platform Operations: NKP uses ESO internally to manage critical platform secrets, including OCI registry credentials and certificates, ensuring secure propagation from management to workload clusters.
User Workloads: ESO can be enabled on workload clusters as a platform application to manage secrets for your own applications. This allows you to centralize sensitive data management outside of Kubernetes while ensuring your workloads have the credentials they need to run securely.

Note:

Disabling configurations of ESO on the management cluster is not permitted, and the option is not available in the NKP UI.
The createPushSecrets and processPushSecrets keys in the charts are true by default, and must not be changed to false.

Enable ESO on a Workload Cluster Using UI

Enable ESO on a Workload Cluster using NKP UI.

Platform Application; APP ID; Internal; Required Dependency; Optional Dependency

| — | — | — | — | — |

Procedure

Log in to the NKP UI.
Navigate to the Management Cluster Workspace dashboard.
Select the Applications tab from the left navigation bar.
Locate the External Secrets Operator or ESO card in the application catalog.
Click Enable.
Select the workload cluster to enable ESO on.
(Optional) Configure the Overrides

If you need to customize the installation (for example, change the target namespace or resource limits), edit the configuration YAML in the provided editor.
By default, ESO is deployed to the external-secrets namespace.

Click Enable to begin the deployment.

Migrate to The ESO Application

For existing users with External Secrets Operator (ESO) already deployed using helm-charts on workload clusters (a brownfield scenario), you must complete the following procedure to migrate to the ESO Platform Application to manage your secrets. This will ensure existing secrets and credentials are not lost, and there are no Custom Resource Definition (CRD) conflicts and lifecycle management issues. On completion of the migration procedure, you can use the NKP UI for operating ESO or making any updates.

Procedure

Backup the resources and secrets referred to in these custom resources.

kubectl get externalsecrets -n <namespace> -o yaml > backup/externalsecrets.yaml
kubectl get pushsecrets -n <namespace> -o yaml > backup/pushsecrets.yaml
kubectl get secretstores -n <namespace> -o yaml > backup/secretstores.yaml
kubectl get clustersecretstores --all-namespaces -o yaml > backup/
clustersecretstores.yaml
kubectl get clusterPushSecrets --all-namespaces -o yaml > backup/
clusterpushsecrets.yaml
kubectl get clusterexternalsecrets --all-namespaces -o yaml > backup/
clusterexternalsecrets.yaml
helm get values manual-eso -n custom-eso-ns > backup/values.yaml

Uninstall the helm installed ESO.

helm uninstall external-secrets
-n external-secrets \

Enable ESO from the Kommander UI. During this process, you must use the values backed up in the

values.yaml file that was created in Step 1 to retain the older configurations. 4. To retain the values and apply the overrides that were configured earlier, use the .yaml file that was created in Step 1. For example:

k apply -f externalsecrets_default.yaml
externalsecret.external-secrets.io/my-first-es created
k apply -f pushsecrets.yaml
pushsecrets.external-secrets.io/push-secrets created
k apply -f secretstores.yaml
secretstores.external-secrets.io/my-secretstores created
k apply -f clustersecretstores.yaml
clustersecretstore.external-secrets.io/kubernetes-backend created

k apply -f clusterPushSecrets.yaml
clusterPushSecrets.external-secrets.io/cluster-push-secrets created
k apply -f clusterexternalsecrets.yaml
clusterexternalsecrets.external-secrets.io/cluster-es created

What to do next

To verify the successful migration, use the command k get externalsecrets -n external-secrets. The ‘ExternalSecret’ resource status displays Synced if the procedure is executed successfully.

In this example, ClusterSecretStore is executed successfully and the status SecretSynced confirms it.

k get externalsecrets -n external-secrets NAME STORETYPE STORE REFRESH INTERVAL STATUS READY my-first-es ClusterSecretStore kubernetes-backend 1m0s SecretSynced True

Setting PriorityClasses in NKP Applications

About this task

By default, the PriorityClasses of Platform Applications are set by NKP.

For more information about the default PriorityClasses for NKP applications, see the following pages:

Workspace Platform Application Defaults and Resource Requirements on page 726
Nutanix Kubernetes Platform Management Cluster Application Requirements on page 725
Project Platform Application Configuration Requirements on page 421

This topic provides instructions on how to override the default PriorityClass of any application in NKP to a different one.

NKP Priority Classes: The PriorityClasses that are available in NKP are as follows:

Before you begin

Table 39: PriorityClasses

NKP High nkp-high-priority 100001000 This is the PriorityClass that is used for high priority NKP workloads.

Class	Value Name	Value	Description

NKP Critical nkp-critical-priority 100002000 This is the highest PriorityClass that is used for critical priority NKP workloads.

Set the WORKSPACE_NAMESPACE environment variable to the name of the workspace’s namespace where the cluster is attached: xport WORKSPACE_NAMESPACE=<your_workspace_namespace>.

export WORKSPACE_NAMESPACE=<your_workspace_namespace>

You are now able to copy the following commands without having to replace the placeholder with your workspace namespace every time you run a command.

Follow these steps.

Note: Keep in mind that the overrides for each application appears differently and is dependent on how the
application's helm chart values are configured.

For more information about the helm chart values used in the NKP, see NKP Release Notes on the Nutanix Support Portal.

Generally speaking, performing a search for the priorityClassName field allows you to find out how you can set the PriorityClass for a component.

In the example below which uses the helm chart values in Grafana Loki, the referenced priorityClassName field is nested under the ingester component. The PriorityClass can be set for several other components, including distributor, ruler, and on a global level.

Procedure

Create a ConfigMap with custom PriorityClass configuration values for Grafana Loki. The following example sets the PriorityClass of ingester component to the NKP critical priority class.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
namespace: ${WORKSPACE_NAMESPACE}
name: grafana-loki-v3-overrides
data:
values.yaml: |
ingester:
priorityClassName: nkp-critical-priority
EOF

Edit the grafana-loki-v3 AppDeployment to set the value of spec.configOverrides.name to grafana-

loki-v3-overrides. After your editing is complete, the AppDeployment resembles this example.

apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: grafana-loki-v3
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: grafana-loki-v3-3.6.7
kind: ClusterApp
configOverrides:

Class	Value Name	Value	Description

name: grafana-loki-v3overrides

It will take a few minutes to reconcile but you can check the ingester pod’s PriorityClass after reconciling.

kubectl get pods -n ${WORKSPACE_NAMESPACE} -o custom-
columns=NAME:.metadata.name,PRIORITY:.spec.priorityClassName,PRIORITY:.spec.priority
|grep ingester

The results appears as follows::

NAME PRIORITY
PRIORITY
grafana-loki-loki-distributed-ingester-0 nkp-critical-
priority 100002000

For more information about PriorityClasses, see <https://kubernetes.io/docs/concepts/scheduling-eviction/> pod-priority- preemption/.

Workspaces

Deploying Platform Applications Using CLI on page 353
Platform Applications on page 350
Platform Applications Dependencies For All Clusters on page 354
Workspace Platform Application Defaults and Resource Requirements on page 726

Global or Workspace UI

The UI is designed to be accessible for different roles at different levels:

Workspace: DevOps administrators manage multiple clusters within a workspace.
Projects: DevOps administrators or developers manage configuration and services across multiple clusters.

Default Workspace

To get started immediately, you can use the default workspace deployed in NKP. However, take into account that you cannot move clusters from one workspace to another after creating/attaching them.

About this task

To create a workspace:

Procedure

From the workspace selection dropdown list in the top menu bar, select Create Workspace.
Type a name and description.
Click Save. The workspace is now accessible from the workspace selection dropdown list.

Adding or Editing Workspace Annotations and Labels

About this task

To perform an action in workspace:

Procedure

From the top menu bar, select your target workspace.
Select the Actions from the dropdown list and click Edit.
Enter in new Key and Value labels for your workspace, or edit existing Key and Value labels.

Note: Labels that are added to a workspace are also applied to all the kommanderclusters in the workspace.

Deleting a Workspace

About this task

To delete a workspace:

Note: Workspaces can only be deleted if all the clusters in the workspace have been deleted or detached.

Procedure

From the top menu bar, select Global.
From the sidebar menu, select Workspaces.
Select the three-dot button to the right of the workspace you want to delete, and then click Delete.
Confirm the workspace deletion in the Delete Workspace dialog box. The following procedures are supported for workspaces:

Deploying Platform Applications Using CLI on page 353
Platform Applications on page 350
Platform Applications Dependencies For All Clusters on page 354
Workspace Platform Application Defaults and Resource Requirements on page 726

Workspace Applications

Platform Applications on page 350 are applications integrated into NKP.
Catalog Applications: are either pre-packaged applications from the Nutanix Application Catalog or custom applications that you maintain for your teams or organization.

Cluster-scoped Application Configuration from the NKP UI

This functionality allows you to use NKP in a multi-cluster scenario without restricting the management of multiple clusters from a single workspace.

Note: NKP Pro users are only be able to configure and deploy applications to a single cluster within a workspace.
Selecting an application to deploy to a cluster skips cluster selection and takes you directly to the workspace
configuration overrides page.

Enabling a Cluster-scoped Application Using the NKP UI

Before you begin

Ensure that you’ve provisioned or attached clusters in one of the following environments:

Amazon Web Services (AWS): Creating the NKP Management Cluster on AWS on page 818
Amazon Elastic Kubernetes Service (EKS):EKS: Creating an EKS Cluster from the UI on page 838
Microsoft Azure:Creating a Managed Azure Cluster Through the NKP UI on page 468

For more information, see the current list of Catalog and Platform Applications:

Workspace Catalog Applications on page 379
Platform Applications on page 350

Navigate to the workspace containing the clusters you want to deploy to by selecting the appropriate workspace name from the dropdown list at the top of the NKP dashboard.

Procedure

Log into the NKP UI dashboard.
In the workspace header drop-down list, select your target workspace.
From the left navigation pane, find the application you want to deploy to the cluster, and select Applications.
Select the triple dot vertical icon in the desired application’s tile and select Enable.

Note: You can also access the Application Enablement by selecting the triple dot vertical icon > View >
Details. Then, select Enable from the application's details page.

The Application Enablement page appears. 5. Select the cluster(s) that you want to deploy the application to.

The available clusters are sorted by Name, Type, Provider and any Labels that you added. 6. In the lower-right of the Application Enablement page, deploy the application to the clusters by selecting Enable. You are automatically redirected to either the Applications or View Details page.

To view the application enabled in your chosen cluster, navigate to the Clusters page on the left navigation bar. The application appears in the Applications pane of the appropriate cluster.

Note: Once you enable an application at the workspace level, NKP automatically enables that app on any other
cluster you create or attach.

Configuring a Cluster-scoped Application Using the NKP UI

About this task

For scenarios where applications require different configurations on a per- cluster basis, navigate to the Applications page and select Edit from the triple dot vertical icon of the appropriate application to return to the application enablement page.

Procedure

Log into the NKP UI dashboard.
In the workspace header drop-down list, select your target workspace.
From the left navigation pane, find the application you want to deploy to the cluster, and select Applications.

The available clusters can be sorted by Name, Type, Provider and any Labels you’ve added. 4. The Applications page contains two separate types of code editors, where you can enter your specified overrides and configurations.

» Workspace Application Configuration: A workspace-level code editor that applies all configurations and overrides to the entirety of the workspace and its clusters for this application.

» Cluster Application Configuration Override: A cluster-scoped code editor that applies configurations and overrides to the cluster specified. These customizations will merge with the workspace application configuration. If there is no cluster-scoped configuration, the workspace configuration applies. 5. If you already have a configuration to apply in a text or .yaml file, you can upload the file by selecting Upload File. If you want to download the displayed set of configurations, select Download File. 6. Finish configuring the cluster-scoped applications by selecting Save. You are automatically redirected to either the Applications or View Details page. To view the custom configurations of the application in the cluster, select the Configurations tab on the details page of the application.

Note: Editing is disabled in the code boxes displayed in the application's details page. To edit the configuration,
click Edit in the top right of the page and repeat the steps in this section.

Removing a Cluster-scoped Application

About this task

Navigate to the cluster you’ve deployed your applications to by selecting Clusters from the left navigation bar.

Procedure

Click on the Applications tab.
Select the triple dot vertical icon in the application tile that you want and select Disable. A prompt appears to confirm your decision to disable the application.
Follow the instructions in the prompt and select Disable
Refresh the page to confirm that the application has been removed from the cluster. This process only removes the application from the specific cluster you have navigated to. To remove this application from other clusters, navigate to the Clusters page and repeat the process.

Cluster-scoped Application for Existing AppDeployments

When you enable an application for a workspace, you deploy this application to all clusters within that workspace. You can also choose to enable or customize an application on certain clusters within a workspace. This functionality allows you to use NKP in a multi-cluster scenario without restricting the management of multiple clusters from a single workspace.

Your NKP cluster comes bundled with a set of default application configurations. If you want to override the default configuration of your applications, you can define workspace configOverrides on top of the default workspace configuration. And if you want to further customize your workspace by enabling applications on a per-cluster basis or by defining per-cluster customizations, you can create and apply clusterConfigOverrides.

The cluster-scoped enablement and customization of applications is an Ultimate-only feature, which allows the configuration of all workspace Platform Applications on page 350, Workspace Catalog Applications on page 379, and Custom Applications on page 402 through the CLI in your managed and attached clusters regardless of your environment setup (air-gapped or non-air- gapped). This capability is not provided for project applications.

Cluster-scope Application Prerequisites

Prerequisites for enabling an application per cluster for the first time or after it is enabled at the workspace level.

Any application you wish to enable or customize at a cluster level, first needs to be enabled at the workspace-level through an AppDeployment. See Deploying Platform Applications Using CLI on page 353 and Workspace Catalog Applications on page 379.
For custom configurations, you must created a ConfigMap. For all the required spec fields for each customization you want to add to an application in a cluster, see AppDeployment Resources on page 327.

You can apply a ConfigMap to several clusters, or create a ConfigMap for each cluster, but the ConfigMap object must exist in the Management cluster.

Determine the name of the workspace where you wish to perform the deployments. You can use the nkp get workspaces command to see the list of workspace names and their corresponding namespaces.
Set the WORKSPACE_NAMESPACE environment variable to the name of the workspace’s namespace where the cluster is attached.

export WORKSPACE_NAMESPACE=<workspace_namespace>

Set the WORKSPACE_NAME environment variable to the name of the workspace where the cluster is attached.

export WORKSPACE_NAME=<workspace_name>

Enabling an Application Per Cluster For the First Time

Before you begin

See Cluster-scope Application Prerequisites on page 373.

About this task

When you enable an application on a workspace, it is deployed to all clusters in the workspace by default. If you want to deploy it only to a subset of clusters when enabling it on a workspace for the first time, you can follow the steps in this topic.

To enable an application per cluster for the first time:

Procedure

Create an AppDeployment for your application, selecting a subset of clusters within the workspace to enable it on. You can use the nkp get clusters –workspace ${WORKSPACE_NAME} command to see the list of clusters in the workspace. The following snippet is an example. Replace the application name, version, workspace name and cluster names according to your requirements. For compatible components and application versions, see NKP Release Notes on the Nutanix Support Portal.

nkp create appdeployment kube-prometheus-stack --app kube-prometheus-stack-82.13.6 --
workspace ${WORKSPACE_NAME} --clusters attached-cluster1,attached-cluster2

(Optional) Check the current status of the AppDeployment to see the names of the clusters where the application is currently enabled.

Enabling an Application Per Cluster After it is Enabled at the Workspace

Level

You can enable or disable an application per cluster after it has been enabled at the workspace level.

Before you begin

See Cluster-scope Application Prerequisites on page 373.

About this task

You can enable or disable applications at any time. After you have enabled the application at the workspace level, the spec.clusterSelector field populates.

Note: For clusters that are newly attached into the workspace, all applications enabled for the workspace are
automatically enabled on and deployed to the new clusters.

If you want to see on what clusters your application is currently deployed, see the print and review the current state of your AppDeployment. For more information, see AppDeployment Resources on page 327.

Procedure

Edit the AppDeployment YAML by adding or removing the names of the clusters where you want to enable your application in the clusterSelector section: The following snippet is an example. Replace the application name, version, workspace name and cluster names according to your requirements.

cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: kube-prometheus-stack
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: kube-prometheus-stack-46.8.0
kind: ClusterApp
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster1
- attached-cluster3-new
EOF

Customizing an Application Per Cluster

You can customize the application for each cluster occurrence of said application. If you want to customize the application for a cluster that is not yet attached, refer to the instructions below, so the application is deployed with the custom configuration during attachment.

About this task

To enable per-cluster customizations:

Procedure

Reference the name of the ConfigMap to be applied per cluster in the spec.clusterConfigOverrides fields. In this example, you have three different customizations specified in three different ConfigMaps for three different clusters in one workspace.

cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: kube-prometheus-stack
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: kube-prometheus-stack-82.13.6
kind: ClusterApp
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster1
- attached-cluster2
- attached-cluster3-new
clusterConfigOverrides:
- configMapName: kps-cluster1-overrides
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster1
- configMapName: kps-cluster2-overrides
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster2
- configMapName: kps-cluster3-overrides
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster3-new
EOF

If you have not done so yet, create the ConfigMaps referenced in each clusterConfigOverrides entry.

Note:

The changes are applied only if the YAML file has a valid syntax.
Set up only one cluster override ConfigMap per cluster. If there are several ConfigMaps configured for a cluster, only one will be applied.
Cluster override ConfigMaps must be created on the Management cluster.

Customizing an Application Per Cluster at Attachment

You can customize the application configuration for a cluster prior to its attachment, so that the application is deployed with this custom configuration on attachment. This is preferable, if you do not want to redeploy the application with an updated configuration after it has been initially installed, which may cause downtime.

About this task

To enable per-cluster customizations, follow these steps before attaching the cluster

Procedure

Set the CLUSTER_NAME environment variable to the cluster name that you will give your to-be-attached cluster.

export CLUSTER_NAME=<your_attached_cluster_name>

Reference the name of the ConfigMap you want to apply to this cluster in the

spec.clusterConfigOverrides fields. You do not need to update the spec.clusterSelector field. In this example, you have the kps- cluster1-overrides customization specified for attached-cluster-1 and a different customization (in kps-your-attached-cluster-overrides ConfigMap) for your to-be- attached cluster.

cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: kube-prometheus-stack
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: kube-prometheus-stack-82.13.6
kind: ClusterApp
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:

kps-your-attached-cluster-overrides	ConfigMap

- attached-cluster1
clusterConfigOverrides:
- configMapName: kps-cluster1-overrides
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster1
- configMapName: kps-your-attached-cluster-overrides
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- ${CLUSTER_NAME}
EOF

If you have not done so yet, create the ConfigMap referenced for your to- be-attached cluster.

Note:

The changes are applied only if the YAML file has a valid syntax.
Cluster override ConfigMaps must be created on the Management cluster.

Disabling the Custom Configuration of an Application Per Cluster

Enabled customizations are defined in a ConfigMap which, in turn, is referenced in the spec.clusterConfigOverrides object of your AppDeployment.

Procedure

Review your current configuration to establish what you want to remove.

kubectl get appdeployment -n ${WORKSPACE_NAMESPACE} kube-prometheus-stack -o yaml

The result appears as follows.

apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: kube-prometheus-stack
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: kube-prometheus-stack-82.13.6
kind: ClusterApp
configOverrides:
name: kube-prometheus-stack-overrides-attached
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster1
- attached-cluster2
clusterConfigOverrides:
- configMapName: kps-cluster1-overrides
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster1
- configMapName: kps-cluster2-overrides
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster2

Here you can see that kube-prometheus-stack has been enabled for the attached- cluster1 and attached-cluster2. There is also a custom configuration for each of the clusters: kps-cluster1- overrides and kps-cluster2-overrides. 2. To delete the customization, delete the configMapName entry of the cluster. This is located under

clusterConfigOverrides.

cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: kube-prometheus-stack
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
kind: ClusterApp
name: kube-prometheus-stack-82.13.6
configOverrides:
name: kube-prometheus-stack-ws-overrides
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster1
clusterConfigOverrides:
- configMapName: kps-cluster1-overrides
clusterSelector:
matchExpressions:
- key: kommander.d2iq.io/cluster-name
operator: In
values:
- attached-cluster1
EOF

Note: Compare steps one and two for a reference of how an entry should be deleted. 3. Before deleting a ConfigMap that contains your customization, ensure you will NOT require it at a later time. It is not possible to restore a deleted ConfigMap. If you choose to delete it, run.

kubectl delete configmap <name_configmap> -n ${WORKSPACE_NAMESPACE}
Note: It is not possible to delete a ConfigMap that is being actively used and referenced in the
configOverride of any AppDeployment.

Verify the Configuration of your Application

Procedure

Workspace Catalog Applications

If you want to know how the AppDeployment resource is currently configured, refer to the print and review the state of your AppDeployments.

Nutanix Enterprise AI in Workspace

Nutanix Enterprise AI (NAI) is a comprehensive inference endpoint management product designed to streamline and optimize your AI model orchestration experience. NAI allows you to select, deploy, and manage large language models (LLMs) on a Kubernetes cluster. You can deploy NAI on any Kubernetes distribution, including Nutanix Kubernetes Platform(NKP) and public cloud Kubernetes platforms such as Amazon Elastic Kubernetes Service (EKS).

Nutanix Enterprise AI (NAI) includes the following features:

Built-in support for Nutanix products: NAI supports seamless integration with existing Nutanix products, thus providing a robust and compliant solution.
Model access integration: NAI supports selecting and deploying text-based generative AI LLMs from Hugging Face.
API endpoint integration: NAI supports creating an API endpoint and sharing it with developers to incorporate into their AI applications. You can also manage and validate these endpoints to ensure they are configured correctly.
API-based access control: NAI supports accessing an endpoint by an application using API keys. API key management allows you to provide and revoke API access to ensure proper access security controls.
User access and role-based access control: NAI supports role-based access control (RBAC), which you can configure to provide customized access permissions to users based on their assigned roles. The user dashboard displays information about all the defined roles.
Enterprise user interface: NAI has a simple and dynamic user interface that streamlines your deployment processes using one-click deployment.
Metrics dashboard: NAI has a dynamic dashboard where you can monitor the health of your deployment with real-time monitoring tools that identify bottlenecks, track performance, and troubleshoot issues.

Nutanix Enterprise AI on NKP Prerequisites

Before configuring Nutanix Enterprise AI on NKP, verify that you have completed the following:

Create or use your existing NKP workload cluster in a workspace.

For more information on creating the management cluster, see Creating a Managed Nutanix Cluster Through the NKP UI on page 458.

Locate the DockerHub credentials that is provided as part of your Nutanix Enterprise AI purchase.

For more information, see Generating Nutanix Docker Hub Access Tokens.

Ensure to apply your NKP Pro or Ultimate License to your cluster.

For more information, see Kubernetes Cluster Setup.

Deploying Nutanix Enterprise AI 2.7.0 on NKP

This topic describes how to deploy Nutanix Enterprise AI (NAI) 2.7.0 on an NKP cluster.

Before you begin

Ensure your cluster meets all the Nutanix Enterprise AI Requirements.
Enable the following applications on your managed cluster:
Cert Manager
Prometheus Monitoring
NVIDIA GPU Operator
NAI - Envoy Gateway (Supported by NAI) (v1.7.0)
Kserve (v0.15.0)
OpenTelemetry Operator (v0.102.0)

About this task

The NKP catalog includes NAI as an application. This allows you to deploy NAI on your clusters with a single click.

Note:

Starting with NKP 2.16, you can access Nutanix AI directly from the NKP UI. The nkp create catalog CLI command is no longer required and is now deprecated.
NAI deployment is supported on Management and Workload clusters.

To enable the NAI catalog applications from the NKP UI, perform the following steps.

Procedure

Set the KUBECONFIG environment variable: The KUBECONFIG environment variable must be set to the configuration file of the cluster where NAI is being installed.

export KUBECONFIG=<cluster.conf>

Create the Nutanix file storage class. Use the following manifest to replace nfs-path and nfs-server:

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nai-nfs-storage
parameters:
nfsPath: <nfs-path>
nfsServer: <nfs-server>
storageType: NutanixFiles
provisioner: csi.nutanix.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF

Create the docker registry secrets.

Create the nai-system namespace and the docker-registry secret in both nai- system and envoy-gateway-system namespaces:

export REGISTRY_SECRET_NAME=nai-regcred
export DOCKER_SERVER=https://index.docker.io/v1/
export DOCKER_USERNAME=<docker-username>
export DOCKER_PASSWORD=<docker-password>
export DOCKER_EMAIL=<docker-email>
kubectl create namespace nai-system --dry-run=client -o yaml | kubectl apply -f -
kubectl -n nai-system create secret docker-registry ${REGISTRY_SECRET_NAME} \
--docker-server=${DOCKER_SERVER} \
--docker-username=${DOCKER_USERNAME} \
--docker-password=${DOCKER_PASSWORD} \
--docker-email=${DOCKER_EMAIL} \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n envoy-gateway-system create secret docker-registry
${REGISTRY_SECRET_NAME} \
--docker-server=${DOCKER_SERVER} \
--docker-username=${DOCKER_USERNAME} \
--docker-password=${DOCKER_PASSWORD} \
--docker-email=${DOCKER_EMAIL} \
--dry-run=client -o yaml | kubectl apply -f -

Note:

The envoy-gateway-system namespace is already present on the cluster. 4. Log in to the NKP dashboard. 5. Enable NAI from the UI.

a. Go to the Applications tab.

b. Select Nutanix Enterprise AI.

c. Click Enable. 6. From the Version dropdown, select 2.7.0. 7. Configure the application override settings.

In the Application Configuration Override window:

a. Select the target cluster(s).

b. Paste the following configuration:

global:
imagePullSecrets:
- name: nai-regcred

For information on generating the Docker Hub Access Tokens, see Generating Nutanix Docker Hub Access Tokens.
For information on the configuration override fields, see Nutanix Enterprise AI Configuration Parameters.

Access the NAI dashboard.

You can use the cluster’s application dashboard NAI card to access the Nutanix Enterprise AI dashboard.

a. Go to the cluster where NAI is deployed.

b. In Enabled Applications, search for Nutanix Enterprise AI.

c. Click the Dashboard link.

This will open NAI using the service external IP. 9. Create a DNS record. A DNS record is required to map the service IP to the FQDN. 10. (Optional) Update the Dashboard link:

export NAI_UI_ENDPOINT=https://<FQDN>/
kubectl patch cm nai-ui -n <workspace-namespace> -p '{"data":
{"dashboardLink":"'${NAI_UI_ENDPOINT}'"}}'
Note: This NAI deployment by default provisions a self-signed certificate to access NAI. For more information,
refer to Setting TLS Encryption for NAI.

What to do next

After you enable an application for a workspace, you can deploy this application on all the clusters within that workspace. For more information on managing cluster-scoped applications in the NKP UI, see Cluster- scoped Application Configuration from the NKP UI on page 371.

Upgrading Nutanix Enterprise AI to 2.7.0 on NKP

This topic describes how to upgrade Nutanix Enterprise AI (NAI) 2.7.0 on an NKP cluster.

Before you begin

You must disable Envoy Gateway (v1.6.3) and enable NAI - Envoy Gateway (Supported by NAI) (v1.7.0) on your managed cluster as NAI 2.7.0 requires the AI gateway powered Envoy Gateway.
Enable the following applications on your managed cluster:
Cert Manager
Prometheus Monitoring
NVIDIA GPU Operator
NAI - Envoy Gateway (Supported by NAI) (v1.7.0)
Kserve (v0.15.0)
OpenTelemetry Operator (v0.102.0)

About this task

The NKP catalog includes NAI as an application. This allows you to deploy NAI on your clusters with a single click.

Note:

Upgrades to Nutanix Enterprise AI version 2.7.0 are supported only from NAI version 2.6.0.
For a fresh installation of Nutanix Enterprise AI 2.7.0, see Deploying Nutanix Enterprise AI 2.7.0 on NKP on page 380.
Starting with NKP 2.16, you can access Nutanix AI directly from the NKP UI. The nkp create catalog CLI command is no longer required and is now deprecated.
NAI deployment is supported on Management and Workload clusters.

To upgrade the NAI catalog applications from the NKP UI, perform the following steps:

Procedure

Set the KUBECONFIG environment variable: The KUBECONFIG environment variable must be set to the configuration file of the cluster where NAI is being upgraded:

export KUBECONFIG=<cluster.conf>

Update the docker registry secret in the nai-system and envoy-gateway- system namespaces with your current credentials:

export REGISTRY_SECRET_NAME=nai-regcred
export DOCKER_SERVER=https://index.docker.io/v1/
export DOCKER_USERNAME=<docker-username>
export DOCKER_PASSWORD=<docker-password>
export DOCKER_EMAIL=<docker-email>
kubectl -n nai-system create secret docker-registry ${REGISTRY_SECRET_NAME} \
--docker-server=${DOCKER_SERVER} \
--docker-username=${DOCKER_USERNAME} \
--docker-password=${DOCKER_PASSWORD} \
--docker-email=${DOCKER_EMAIL} \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n envoy-gateway-system create secret docker-registry
${REGISTRY_SECRET_NAME} \
--docker-server=${DOCKER_SERVER} \
--docker-username=${DOCKER_USERNAME} \
--docker-password=${DOCKER_PASSWORD} \
--docker-email=${DOCKER_EMAIL} \
--dry-run=client -o yaml | kubectl apply -f -

Log in to the NKP dashboard.
Edit the application:

a. Go to the Applications tab.

b. Select Nutanix Enterprise AI.

c. Click Edit. 5. From the Version dropdown, select 2.7.0. 6. Update the application configuration override:

In the Application Configuration Override window:

a. Select the target cluster(s) where NAI 2.6.0 is already deployed.

b. Replace the override content:

global:
imagePullSecrets:
- name: nai-regcred

Access the NAI dashboard: Use the cluster’s Application Dashboard NAI card to access the Nutanix Enterprise AI dashboard:

a. Go to the cluster where NAI is deployed.

b. In Enabled Applications, search for Nutanix Enterprise AI.

c. Click Dashboard link. This will open NAI on service External IP. 8. Verify the DNS record:

Ensure a DNS record exists that maps the service IP to the FQDN. If the service IP has changed as part of the upgrade, update the record accordingly. 9. Verify cluster health post-upgrade:

Confirm that all NAI pods in the nai-system namespace are Running and that the new NAI - Envoy Gateway is healthy in the envoy-gateway-system namespace. 10. (Optional) Update the dashboard link:

export NAI_UI_ENDPOINT=https://<FQDN>/
kubectl patch cm nai-ui -n <workspace-namespace> -p '{"data":
{"dashboardLink":"'${NAI_UI_ENDPOINT}'"}}'
Note: This NAI deployment by default provisions a self-signed certificate to access NAI. For more information,
refer to Setting TLS Encryption for NAI.

What to do next

Deploying Nutanix Enterprise AI 2.6.0 on NKP

This topic describes how to deploy Nutanix Enterprise AI (NAI) 2.6.0 on an NKP cluster.

Before you begin

Ensure your cluster meets all the Nutanix Enterprise AI Requirements.
Enable the following applications on your managed cluster:
Cert Manager
Prometheus Monitoring
NVIDIA GPU Operator
Envoy Gateway (v1.6.3)
Kserve (v0.15.0)
OpenTelemetry Operator (v0.102.0)

About this task

The NKP catalog includes NAI as an application. This allows you to deploy NAI on your clusters with a single click.

Note:

Starting with NKP 2.16, you can access Nutanix AI directly from the NKP UI. The nkp create catalog CLI command is no longer required and is now deprecated.
NAI deployment is supported on Management and Workload clusters.

To enable the NAI catalog applications from the NKP UI, perform the following steps.

Procedure

Set the KUBECONFIG environment variable: The KUBECONFIG environment variable must be set to the configuration file of the cluster where NAI is being installed.

export KUBECONFIG=<cluster.conf>

Create the Nutanix file storage class. Use the following manifest to replace nfs-path and nfs-server:

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nai-nfs-storage
parameters:
nfsPath: <nfs-path>
nfsServer: <nfs-server>
storageType: NutanixFiles
provisioner: csi.nutanix.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF

Create the docker registry secrets.

Create the nai-system namespace and the docker-registry secret in both nai- system and envoy-gateway-system namespaces:

export REGISTRY_SECRET_NAME=nai-regcred
export DOCKER_SERVER=https://index.docker.io/v1/
export DOCKER_USERNAME=<docker-username>
export DOCKER_PASSWORD=<docker-password>
export DOCKER_EMAIL=<docker-email>
kubectl create namespace nai-system --dry-run=client -o yaml | kubectl apply -f -
kubectl -n nai-system create secret docker-registry ${REGISTRY_SECRET_NAME} \
--docker-server=${DOCKER_SERVER} \
--docker-username=${DOCKER_USERNAME} \
--docker-password=${DOCKER_PASSWORD} \
--docker-email=${DOCKER_EMAIL} \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n envoy-gateway-system create secret docker-registry
${REGISTRY_SECRET_NAME} \
--docker-server=${DOCKER_SERVER} \
--docker-username=${DOCKER_USERNAME} \
--docker-password=${DOCKER_PASSWORD} \
--docker-email=${DOCKER_EMAIL} \
--dry-run=client -o yaml | kubectl apply -f -

Note:

The envoy-gateway-system namespace is already present on the cluster. 4. Log in to the NKP dashboard. 5. Enable NAI from the UI.

a. Go to the Applications tab.

b. Select Nutanix Enterprise AI.

c. Click Enable. 6. From the Version dropdown, select 2.6.0. 7. Configure the application override settings.

In the Application Configuration Override window:

a. Select the target cluster(s).

b. Paste the following configuration:

global:
imagePullSecrets:
- name: nai-regcred

For information on generating the Docker Hub Access Tokens, see Generating Nutanix Docker Hub Access Tokens.
For information on the configuration override fields, see Nutanix Enterprise AI Configuration Parameters.

Access the NAI dashboard.

You can use the cluster’s application dashboard NAI card to access the Nutanix Enterprise AI dashboard.

a. Go to the cluster where NAI is deployed.

b. In Enabled Applications, search for Nutanix Enterprise AI.

c. Click the Dashboard link.

This will open NAI using the service external IP. 9. Create a DNS record. A DNS record is required to map the service IP to the FQDN. 10. (Optional) Update the Dashboard link:

export NAI_UI_ENDPOINT=https://<FQDN>/
kubectl patch cm nai-ui -n <workspace-namespace> -p '{"data":
{"dashboardLink":"'${NAI_UI_ENDPOINT}'"}}'

What to do next

Upgrading Nutanix Enterprise AI to 2.6.0 on NKP

This topic describes how to upgrade Nutanix Enterprise AI (NAI) 2.6.0 on an NKP cluster.

Before you begin

Enable the following applications on your managed cluster:
Cert Manager
Prometheus Monitoring
NVIDIA GPU Operator
Envoy Gateway (v1.6.3)
Kserve (v0.15.0)
OpenTelemetry Operator (v0.102.0)

About this task

The NKP catalog includes NAI as an application. This allows you to deploy NAI on your clusters with a single click.

Note:

Upgrades to Nutanix Enterprise AI version 2.6.0 are supported only from NAI version 2.5.0.
For a fresh installation of Nutanix Enterprise AI 2.6.0, see Deploying Nutanix Enterprise AI 2.6.0 on NKP on page 384.
Starting with NKP 2.16, you can access Nutanix AI directly from the NKP UI. The nkp create catalog CLI command is no longer required and is now deprecated.
NAI deployment is supported on Management and Workload clusters.

To upgrade the NAI catalog applications from the NKP UI, perform the following steps:

Procedure

Set the KUBECONFIG environment variable. The KUBECONFIG environment variable must be set to the configuration file of the cluster where NAI is being upgraded:

export KUBECONFIG=<cluster.conf>

Update the docker registry secret in the nai-system and envoy-gateway- system namespaces with your current credentials:

export REGISTRY_SECRET_NAME=nai-regcred
export DOCKER_SERVER=https://index.docker.io/v1/
export DOCKER_USERNAME=<docker-username>
export DOCKER_PASSWORD=<docker-password>
export DOCKER_EMAIL=<docker-email>
kubectl -n nai-system create secret docker-registry ${REGISTRY_SECRET_NAME} \
--docker-server=${DOCKER_SERVER} \
--docker-username=${DOCKER_USERNAME} \
--docker-password=${DOCKER_PASSWORD} \
--docker-email=${DOCKER_EMAIL} \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n envoy-gateway-system create secret docker-registry ${REGISTRY_SECRET_NAME}
\
--docker-server=${DOCKER_SERVER} \
--docker-username=${DOCKER_USERNAME} \
--docker-password=${DOCKER_PASSWORD} \
--docker-email=${DOCKER_EMAIL} \
--dry-run=client -o yaml | kubectl apply -f -

Log in to the NKP dashboard.
Edit the application.

a. Go to the Applications tab.

b. Select Nutanix Enterprise AI.

c. Click Edit. 5. From the Version dropdown, select 2.6.0. 6. Update the application configuration override.

In the Application Configuration Override window:

a. Select the target cluster(s) where NAI 2.5.0 is already deployed.

b. Replace the override content:

global:
imagePullSecrets:
- name: nai-regcred

Access the NAI dashboard. Use the cluster’s Application Dashboard NAI card to access the Nutanix Enterprise AI dashboard:

a. Go to the cluster where NAI is deployed.

b. In Enabled Applications, search for Nutanix Enterprise AI.

c. Click Dashboard link. This will open NAI on service External IP.

What to do next

Nutanix Data Services for Kubernetes in Workspace

NDK provides the following features and services for applications running on Kubernetes:

Business Continuity and Disaster Recovery (BCDR)
Data Management
Data Protection and Restoration
Application Delivery Acceleration
Application Self-Service Management (meeting specific requirements)
Infrastructure Automation and Monitoring (for optimal operations)
Standardization of Governance and Compliance Policies

For more information, see the Nutanix Data Services for Kubernetes user guide.

Nutanix Data Services for Kubernetes on NKP Prerequisites

Note the requirements for deploying Nutanix Data Services for Kubernetes (NDK) on Nutanix Kubernetes Platform (NKP).

Ensure to apply your NKP Pro, Ultimate, or Full Stack (NKPFS) License to your cluster.

For more information, see Licenses on page 24.

Ensure the user account that interacts with Prism Central has admin privileges.
Ensure that you assign a virtual IP address to the Prism Element clusters for iSCSI Data Services.
Ensure that you configure the Prism Central virtual IP address on the scale- out Prism Central deployments.
Ensure that the iscsi-initiator-utils is installed on the worker nodes:

sh-5.1# rpm -q iscsi-initiator-utils
iscsi-initiator-utils-6.2.1.9-1.gita65a472.el9.x86_64

For NDK version 2.0, ensure that the CSI version 3.3.8 or later is installed in the NKP cluster.

For more information, see Install the CSI Drivers.

Ensure that you maintain sufficient IP addresses in the MetalLB address pool for Kubernetes service type:LoadBalancer.

For information about service type:LoadBalancer, see Service type: LoadBalancer in NDK Custom Resources.

Ensure to locate the DockerHub credentials provided as part of your NDK purchase.

You must log in to the Nutanix Support Portal to generate the access token for image pull secret. Your organization can have up to two access tokens. For more information, see Generating Nutanix Docker Hub Access Tokens.

When enabling NDK in the NKP UI, securely reference the credentials or secret in values.yaml to authenticate the storage registry:

imageCredentials:
credentials:
registry: https://index.docker.io/v1/ # Leave as-is for Docker Hub, or enter
your private registry URL
username: <your-username>
password: <your-password> # For Docker Hub, use a generated access token
email: <your-email>

If you already configured the credential secret, provide the imagePullSecretName for enabling NDK in the NKP UI:

imageCredentials:
# Name of the secret used to pull images.
imagePullSecretName: <name of the secret with image pull credentials>

The secret must exist in the same namespace where you install NDK. For information on generating docker hub access tokens, see Generating Nutanix Docker Hub Access Tokens.

To create a custom secret in the ntnx-system namespace, follow these steps:

Enter the DockerHub credentials in config.json file:

cat > config.json << EOF
{
"auths":
{
"https://index.docker.io/v1/":
{
"auth": "$(echo -n 'username:password' | base64)",
"email": "[email protected]"
}
}
}
EOF

Create a generic secret:

$ kubectl create secret generic <name of image pull secret> -n ntnx-system --from-
file=.dockerconfigjson=./config.json --type=kubernetes.io/dockerconfigjson

Verify that the secret created exists in ntnx-system namespace:

$ kubectl get secret <name of image pull secret> -n ntnx-system
Note: If you use the pre-configured secret approach imagePullSecretName, you must manually create this
secret on every workload cluster where you deployed NDK. The secret is not automatically referenced across
clusters, so each cluster requires its own copy of the secret in the appropriate namespace.

Deploying Nutanix Data Services for Kubernetes on Air-Gapped NKP

Environments

You can deploy Nutanix Data Services for Kubernetes (NDK) on a Nutanix Nutanix Kubernetes Platform(NKP) cluster in an air-gapped environment.

About this task

The NKP catalog includes NDK as an application, enabling you to deploy NDK on your Nutanix clusters with a single click.

Note:

Starting with NKP 2.17, you can install Nutanix Data Services for Kubernetes directly from the NKP UI.

For more information on the NDK functionalities and workflows, see Nutanix Data Services for Kubernetes.

NDK deployment is supported on management and workload clusters.
NDK is a cluster-scoped application and therefore each cluster runs its own NDK instance.

To enable NDK catalog application in the Nutanix air-gapped environment:

Procedure

Clone the repository:

git clone https://github.com/nutanix-cloud-native/nkp-nutanix-product-catalog.git

Go to the cloned repository:

cd nkp-nutanix-product-catalog/

$ docker login --username nutanixndk
Password:

For air-gapped environments, the bundle includes Helm charts, container images, and NDK images pulled from the Nutanix DockerHub, which requires corresponding credentials. 4. Create a bundle for the catalog application:

$ nkp create catalog-bundle –airgapped –apps=ndk=“	ndk-version	“

Push the bundle to the registry mirror that you configured:

$ nkp push bundle --bundle ndk-ndk-version-airgapped.tar
--to-registry-mirror-url=${REGISTRY_URL}
--to-registry-mirror-username=${REGISTRY_USERNAME} \
--to-registry-mirror-password=${REGISTRY_PASSWORD}

Create a catalog application. Copy the nkp create catalog-application command from the output of nkp create catalog- bundle:

$ nkp create catalog-application --url oci://<registry-url>/nkp-nutanix-product-
catalog/ndk --tag ndk-version --workspace kommander-workspace

Adding the catalog in kommander-workspace automatically propagates it to all workspaces. 7. Log in to the NKP UI dashboard. 8. From the workspace header dropdown list, select your target workspace. 9. Click View Details in the cluster widget (management, managed, or attached clusters). The General Cluster Configuration page is displayed. 10. In the Applications tab, select Nutanix Data Services for Kubernetes and click Enable. 11. In the General page, follow these steps:

a. In the Selected Cluster section, choose at least one target cluster.

b. In the Workspace Application Configuration Override code editor, enter the configuration override:

(Optional) To enable TLS for the webhook and NDK server, provide the following configuration override:

Note: By default, the TLS is in a disabled state.

For self-signed certificates by cert-manager:

tls:
server:
enable: true
clusterName: <name of NKP cluster on which this configuration is
applied>

For custom issuer (a pre-configured issuer in ntnx-system namespace):

tls:
server:
enable: true
mode: "ISSUER"
issuer:
issuerName: <issuer-name>
clusterName: <name of NKP clusters>

For custom secret (a pre-configured secret in ntnx-system namespace):

tls:
server:
enable: true
mode: "SECRET"
secretName: <name of pre-existing secret in ntnx-system namespace>
clusterName: <name of NKP cluster>

For more information about creating issuer and secret, see Installing NDK with Server Side TLS Using CA-signed Certificates in the Nutanix Data Services for Kubernetes guide.

(Optional) To enable MTLS between the NDK intercom service on the source cluster and the replication Kubernetes cluster, provide the following configuration override:

tls:
server:
enableMTLS: true

Note: By default, the MTLS is in a disabled state.

The same trusted CA must sign the certificates used by the primary and replication clusters.

(Optional) To enable communication between the NDK intercom service on the source cluster and the replication Kubernetes cluster using your own load balancer:

intercomService:
type: LoadBalancer
loadBalancerClass:
loadBalancerIP:

Provide both loadBalancerClass and loadBalancerIP.
If you do not provide both loadBalancerClass and loadBalancerIP, MetalLB assigns a free IP address from its pool.
IP assignment fails with an error, in the following cases:
No free IP address is available in the MetalLB address pool.
The loadBalancerIP you specify is not present in the MetalLB address pool.

Click Enable.

What to do next

After you enable an application for a workspace, you can deploy this application on all the clusters within that workspace. To configure the various policies and schedules in NDK, see Configure NDK.

For more information on managing cluster-scoped applications in the NKP UI, see Cluster-scoped Application Configuration from the NKP UI on page 371.

Deploying Nutanix Data Services for Kubernetes on Non Air-Gapped NKP

Environments

You can deploy Nutanix Data Services for Kubernetes (NDK) on a Nutanix Nutanix Kubernetes Platform(NKP) cluster in a non-air-gapped environment.

About this task

The NKP catalog includes NDK as an application, enabling you to deploy NDK on your clusters with a single click.

Note:

Starting with NKP 2.17, you can install Nutanix Data Services for Kubernetes directly from the NKP UI.

For more information on the NDK functionalities and workflows, see Nutanix Data Services for Kubernetes.

NDK deployment is supported on Management and Workload clusters.
NDK is a cluster-scoped application and therefore each cluster runs its own NDK instance.

To enable NDK catalog application in the Nutanix non-air-gapped environment:

Procedure

Log into the NKP UI dashboard.
From the workspace header drop-down list, select your target workspace.
Click View Details in the cluster widget (management, managed, or attached clusters). The General Cluster Configuration page is displayed.
In the Applications tab, select Nutanix Data Services for Kubernetes and click Enable.
In the General page, follow these steps:

a. In the Selected Cluster section, choose at least one target cluster.

b. In the Workspace Application Configuration Override code editor, enter the configuration override to customize the application deployment:

To pull the image from the container registry, update the <username>, <password> and <email> fields with the Docker Hub credentials provided as part of the NDK purchase.

imageCredentials:
credentials:
registry: https://index.docker.io/v1/ # Leave as-is for Docker Hub, or
enter your private registry URL
username: <your-username>
password: <your-password> # For Docker Hub, use a generated access token
email: <your-email>

If you already configured the credential secret, provide the imagePullSecretName:

imageCredentials:
# Name of the secret that will be created to pull images.
imagePullSecretName: <name of the secret with image pull credentials>

The secret must exist in the same namespace where you install NDK. For information on generating docker hub access tokens, see Generating Nutanix Docker Hub Access Tokens.

Note: By default, the TLS is in a disabled state.

(Optional) To enable TLS for the webhook and NDK server, provide the following configuration override:

For self-signed certificates by cert-manager:

tls:
server:
enable: true
clusterName: <name of NKP cluster on which this configuration is applied>

For custom issuer (a pre-configured issuer in ntnx-system namespace):

tls:
server:
enable: true
mode: "ISSUER"
issuer:
issuerName: <issuer-name>
clusterName: <name of NKP cluster>

For custom secret (a pre-configured secret in ntnx-system namespace):

tls:
server:
enable: true
mode: "SECRET"
secretName: <name of pre-existing secret in ntnx-system namespace>
clusterName: <name of NKP cluster>

For more information about creating issuer and secret, see Installing NDK with Server Side TLS Using CA-signed Certificates in the Nutanix Data Services for Kubernetes guide.

Note: By default, the MTLS is in a disabled state.

(Optional) To enable MTLS between the NDK intercom service on the source cluster and the replication Kubernetes cluster, provide the following configuration override:

tls:
server:
enableMTLS: true

The same trusted CA must sign the certificates used by the primary and replication clusters.

(Optional) To enable communication between the NDK intercom service on the source cluster and the replication Kubernetes cluster using your own load balancer:

intercomService:
type: LoadBalancer
loadBalancerClass:
loadBalancerIP:

To make the configuration valid, provide both loadBalancerClass and loadBalancerIP.
If you do not provide both loadBalancerClass and loadBalancerIP, MetalLB assigns a free IP address from its pool.
IP assignment fails with an error, when:
No free IP address is available in the MetalLB address pool.
The loadBalancerIP you specify is not present in the MetalLB address pool.

Click Enable.

What to do next

After you enable an application for a workspace, you can deploy this application on all the clusters within that workspace. To configure the various policies and schedules in NDK, see Configure NDK.

For more information on managing cluster-scoped applications in the NKP UI, see Cluster-scoped Application Configuration from the NKP UI on page 371.

Deployment of Catalog Applications in Workspaces

To deploy an application to selected clusters within a workspace, see Cluster- scoped Application for Existing AppDeployments on page 373.

Enabling the Catalog Application Using the UI

Before you begin

Before you begin, you must have:

A running cluster with Kommander installed. The cluster must be on a supported Kubernetes version for this release of NKP and also compatible with the catalog application version you want to install.
Attach an Existing Kubernetes Cluster section of the documentation completed. For more information, see Attaching an Existing Kubernetes Cluster on page 474.
Set the WORKSPACE_NAMESPACE environment variable to the name of the workspace’s namespace the attached cluster exists in.

export WORKSPACE_NAMESPACE=<workspace_namespace>

A Git Repository. After creating a GitRepository, use either the NKP UI or the CLI to enable your catalog applications. For more information, see Creating a Catalog Collection or an Application on page 404.

Warning: From within a workspace, you can enable applications to deploy. Verify that an application has
successfully deployed through the CLI.

About this task

Follow these steps to enable your catalog applications from the NKP UI:

Procedure

Ultimate only: From the top menu bar, select your target workspace.
From the sidebar menu to browse the available applications from your configured repositories and select Applications.
Select the three dot button on the required application tile and select Enable.
If available, select a version from the dropdown list. This dropdown list will only be visible if there is more than one version.
(Optional) If you want to override the default configuration values, copy your customized values into the text editor under Workspace Configuration or upload your YAML file that contains the values.

someField: someValue

Confirm the details are correct, and then click Enable. For all applications, you must provide a display name and an ID which is automatically generated based on what you enter for the display name, unless or until you edit the ID directly. The ID must be compliant with Kubernetes DNS subdomain name validation rules in the Kubernetes documentation. Alternately, you can use the CLI to enable your catalog applications.

Enabling the Catalog Application Using the CLI

See Workspace Catalog Applications for the list of available applications that you can deploy on the attached cluster.

Before you begin

Procedure

Enable a supported application to deploy to your attached Kubernetes cluster with an AppDeployment resource.

For more information, see Attaching an Existing Kubernetes Cluster on page \474. 2. Within the AppDeployment, define the appRef to specify which App to enable.

cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: operator
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: operator-0.25.1
kind: App
EOF

Note:

The appRef.name must match the app name from the list of available catalog applications.
Create the resource in the workspace you just created, which instructs Kommander to deploy the AppDeployment to the KommanderClusters in the same workspace.

Enabling the Catalog Application With a Custom Configuration Using the CLI

About this task

To enable the catalog application:

Procedure

Provide the name of a ConfigMap in the AppDeployment, which provides custom configuration on top of the default configuration.

cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: operator
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: operator-0.25.1
kind: App
configOverrides:
name: operator-overrides
EOF

Create the ConfigMap with the name provided in the step above, with the custom configuration.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
namespace: ${WORKSPACE_NAMESPACE}
name: operator-overrides
data:
values.yaml: |
operator:
verboseLogging: true
EOF

Kommander waits for the ConfigMap to be present before deploying the AppDeployment to the managed or attached clusters.

Verify the Catalog Applications

The applications are now enabled.

Procedure

Connect to the attached cluster and check the HelmReleases to verify the deployment.

kubectl get helmreleases -n ${WORKSPACE_NAMESPACE}

The result appears as follows.

NAMESPACE NAME READY STATUS
AGE
workspace-test-vjsfq operator True Release reconciliation succeeded 7m3s

Workspace Catalog Application Upgrade

Before upgrading, keep in mind the distinction between Platform applications and Catalog applications. Platform applications are deployed and upgraded as a set for each cluster or workspace. Catalog applications are deployed separately, so that you can deploy and upgrade them individually for each workspace or project.

Upgrading the Catalog Applications Using the UI

Before you begin

Complete the upgrade prerequisites tasks. For more information, see Plan the Upgrade on page 1042.

About this task

To upgrade an application from the NKP UI:

Procedure

From the top menu bar, select your target workspace.
From the sidebar menu, select Applications.
Select the three dot button on the required application tile, and then select Edit.
Select the Version from the dropdown list and select a new version. This dropdown list is only available if there is a newer version to upgrade to.
Click Save.

Upgrading the Catalog Applications Using the CLI

Before you begin

Note: The commands use the workspace name and not namespace.

You can retrieve the workspace name by running the following command.

nkp get workspaces

To view a list of the deployed apps to your workspace, run the following command.

nkp get appdeployments --workspace=<workspace-name>

Complete the upgrade prerequisites tasks. For more information, see Plan the Upgrade on page 1042.

About this task

To upgrade an application from the CLI:

Procedure

To see what app(s) and app versions are available to upgrade, run the following command.

Note: You can reference the app version by going into the app name (e.g. <APP ID>-<APP VERSION>.

kubectl get apps -n ${WORKSPACE_NAMESPACE}

You can also use this command to display the apps and app versions, for example.

kubectl get apps -n ${WORKSPACE_NAMESPACE} -o jsonpath='{range .items[*]}
{@.spec.appId}{"----"}{@.spec.version}{"\n"}{end}'

The output displays the different application and application versions. 2. Run the following command to upgrade an application from the NKP CLI.

nkp upgrade catalogapp <appdeployment-name> --workspace=<my-workspace-name> --to-
version=<version.number>
Note: Platform applications cannot be upgraded on a one-off basis, and must be upgraded in a single process for
each workspace. If you attempt to upgrade a platform application with these commands, you receive an error and
the application is not upgraded.

Partner Catalog in NKP

You can contribute to partner catalog only if you are a Nutanix Elevate Technology Alliance Partner with Kubernetes-based applications that meet NKP quality and security standards.

NKP partner catalog helps NKP customers discover their product or application across hybrid-cloud, multi-cloud, and multi-cluster environments. By submitting applications to the catalog, you can:

Reach the enterprise customer base of NKP without requiring separate installation channels. You can deploy and update your software through a familiar NKP workflow.
Tag releases against specific NKP versions so that customers see only the compatible versions.

After onboarding the application, you are granted access to a dedicated Git repository.

Prepare and test your application. Package the application into a Helm chart, test installation, and upgrade flows locally before submission.
Fork and contribute. Fork the catalog repository, place the application under the appropriate directory (for NKP specific apps or cross-cluster apps), and create a pull request (PR).
Provide metadata. Each submission must include a metadata file, specifying description, version, supported NKP releases, support contacts, categories, keywords and a logo. NKP requires capabilities, container image, creation date, support contact, and repository fields, and NKP will require analogous metadata.
Run automated tests. When a PR is opened, the CI system of NKP automatically validates if the application is installed and can be upgraded. Applications that fail installation or upgrade tests are flagged for partner correction.
Review and approval. The NKP engineering team reviews submissions for security, licensing, and quality. Once approved and merged, the application appears in the partner catalog Git repository and is visible when enabled in the catalog.

Note:

Partners can update or remove applications by submitting additional PRs.
Each application version must be tagged with the NKP releases it supports. Only the versions compatible with the your NKP version is displayed.

In NKP UI, Applications Catalog displays Preferred Partner Apps category containing all approved partner applications. You can browse descriptions, see which NKP versions are supported, and deploy the apps with a few clicks. The catalog ensures that only applications qualified for your NKP version are displayed, reducing the risk of incompatible deployments. The support contacts are provided by the partner for assistance.

Creating a New Application

This page details the creation of a new application in Partner Catalog.

About this task

To create a new application and update an existing application, follow these steps:

Procedure

Clone the repository.

git clone [email protected]:nutanix-cloud-native/nkp-partner-catalog.git

Note:

You must fork and work off the repository.
You must download the NKP binary to create a new application in partner catalog.

Install devbox.

Devbox is a CLI tool used for creating isolated shells for development. For more information, see devbox. 3. Run devbox shell to enter the devbox environment in the cloned repository directory. An environment with all the necessary dependencies is set up. Subsequently, you can run commands like nkp, just, and go directly. 4. Generate a placeholder application version directory by running the following command:

nkp generate catalog-repository --apps=<app-name>=<app-version>

A placeholder directory layout for your new application is generated in applications/ directory with your application name and version.

Overview of the Application Structure

This page details about the partner catalog application repository structure.

Each application can have multiple versions with its own directory.
Each version must include metadata.yaml used by NKP UI for rendering metadata description.
After merging, the OCI artifacts are built and pushed to the ghcr.io/nutanix-cloud-native/nkp- partner-catalog/<app-name> automatically.

Sample directory structure:

...
### applications (3)
# ### <app-name>
# ### <app-version1>
# ### helmrelease
# # ### cm.yaml // ConfigMap specifying default
configuration overrides for HelmRelease
# # ### helmrelease.yaml // OCIRepository and HelmRelease
# # ### kustomization.yaml // Kubernetes Kustomization file
# ### helmrelease.yaml // FluxCD Kustomization file for the
HelmRelease
# ### kustomization.yaml // Kubernetes Kustomization file
# ### metadata.yaml // Application details
# ### <app-version2>
# ### helmrelease (2)
# # ### cm.yaml
# # ### helmrelease.yaml
# # ### kustomization.yaml
# ### helmrelease.yaml
# ### kustomization.yaml
# ### metadata.yaml
...
Note: For more information about Git Repository Structure and Application Metadata, see
Custom Applications on page 402.

Structuring an Application in Partner Catalog

This page details the structuring of application in the partner catalog.

About this task

To structure an application in the partner catalog.

Procedure

Update open container initiative OCIRepository URL to point the Helm chart of your application.
Add optional overrides for application HelmRelease into cm.yaml as necessary. The default layout (generated in the previous step) addresses most use cases that deploy a simple application with no dependencies. If there are dependencies in deploying various components of the application, see Flux Kustomization. Flux Kustomization provides the necessary details to deploy an application with multiple

top level Flux Kustomization files that depends on each other and the ability to define health checks for each kustomization. For more information, see Dependencies and Health checks.

Note: Following are the substitution variables in the application:

${releaseName}: Dynamically substituted with the AppDeployment value provided during installation. For more information, see Applications on page

${releaseNamespace}: Dynamically substituted with the namespace where the app is deployed.

Validate your directory by running the following command.

nkp validate catalog-repository --repo-dir=/path/to/nkp-partner-catalog

(Optional) Customize the validation behavior by adding a bloodhound configuration file. Bloodhound is the linter configuration used for catalog application validation during CI/CD processes. You can configure validation settings by creating a .bloodhound.yaml or .bloodhound.yml file at the repository root or application level. For more information, see Bloodhound Configuration.

Custom Applications

Git Repository Structure

Git repositories must be structured in a specific manner for defined applications to be processed by Kommander.

You must structure your Git repository based on the following guidelines, for your applications to be processed properly by Kommander so that they can be deployed.

Git Repository Directory Structure

Run the following command to generate a new Git repository for hosting your catalog applications:

mkdir -p /new/directory/for/catalog
cd /new/directory/for/catalog
nkp generate catalog-repository

Run the following command, to generate a directory for your application version. You can run this command multiple times (once for each version of a given application).

nkp generate catalog-repository --apps <app-name>=<app-version>

Above commands will generate following basic directory structure:

...
### applications (4)
# ### <app-name> (2)
# ### <app-version1> (2)
# ### helmrelease (3)
# # ### cm.yaml // ConfigMap specifying default (2)
configuration overrides for HelmRelease
# # ### helmrelease.yaml // OCIRepository and HelmRelease (2)
# # ### kustomization.yaml // Kubernetes Kustomization file (2)
# ### helmrelease.yaml // FluxCD Kustomization file for the (2)
HelmRelease
# ### kustomization.yaml // Kubernetes Kustomization file (2)
# ### metadata.yaml // Application details (2)
# ### <app-version2> (2)
# ### helmrelease (4)
# # ### cm.yaml (2)
# # ### helmrelease.yaml (2)
# # ### kustomization.yaml (2)
# ### helmrelease.yaml (2)
# ### kustomization.yaml (2)
# ### metadata.yaml (2)
...

Remember the following guidelines:

Define applications in the applications/ directory.
You can define multiple versions of an application, under different directories nested under the applications/ <app name>/ directory.
Populate the HelmRelease and OCIRepository details in the versioned directory applications/<app name>/ <app version>/helmrelease in the helmrelease.yaml file. For more information, see HelmRelease, OCIRepository in the Flux documentation, and The Kustomization File in the SIG CLI documentation.
Define the default values ConfigMap for HelmReleases in the applications/<app name>/<app version>/helmrelease/cm.yaml file accompanied by a kustomization.yaml Kubernetes Kustomization file pointing to the ConfigMap file.
Define the metadata.yaml of each application under the applications/<app name>/<app version> directory. For more information, see Application Metadata on page 405.
You can generate a new application version directory with the required layout and files by running:

nkp generate catalog-repository --apps=<app-name>=<app-version>

For an example of how to structure catalog Git repositories, see Overview of the Application Structure on page 401.

OCI Repositories

You must include the OCIRepository that is referenced in each HelmRelease’s Chart spec.

Each applications/<app name>/<app version>/helmrelease/helmrelease.yaml file must include YAML definition of the OCIRepository along with the HelmRelease definition.

For more information, see OCI Repository in the Flux documentation.

Substitution Variables

Some substitution variables are provided. For more information, see Kustomization in the Flux documentation.

${releaseName}: For each App deployment, this variable is set to the AppDeployment name. Use this variable to prefix the names of any resources that are defined in the application directory in the Git repository so that multiple instances of the same application can be deployed. If you create resources without using the releaseName prefix (or suffix) in the name field, there can be conflicts if the same named resource is created in that same namespace.
${releaseNamespace}: The namespace of the workspace.
${workspaceNamespace}: The namespace of the workspace that the Workspace belongs to.

Creating a Catalog Collection or an Application

Use CLI to create a Flux OCIRepository resource in your workspace or project that acts as a source for your catalog applications.

About this task

Create an OCIRepository in the workspace or project namespace.

Procedure

To build an OCI artifact from your catalog git repository run the following command. This generates a .tar file locally.

nkp create catalog-bundle --collection-tag <tag>

Above command will bundle up all versions of all applications in the current directory.

Note:

To include only selected applications use –apps flag.
To include the container images use –airgapped flag. This is required for air-gapped environments.

Push the OCI artifact to a registry that is accessible to the cluster. The registry used must be OCI Compliant.

nkp push bundle --bundle <path-to-file-generated-above.tar> --to-registry <your-
registry-url>

You can either create a catalog collection (group of applications) or just one application in your NKP Cluster. Note that creating a catalog in the Kommander workspace will automatically propagate it to all other workspaces/projects/. If you prefer to limit the scope of catalog to a single workspace or project, use the –workspace or –project flags accordingly. Run either one of the following commands:

Create a Catalog Collection

nkp create catalog-collection --url oci://<registry-url>/<repo-name>/collection --
workspace <workspace-or-project-namespace> --tag <tag>

Create a Catalog Application

nkp create catalog-collection --url oci://<registry-url>/<repo-name>/<app-name> --
workspace <workspace-or-project-namespace> --tag <app-version>

Optionally, ensure the status of the OCIRepository signals a ready state.

kubectl get ocirepository -n <workspace-or-project-namespace>${WORKSPACE_NAMESPACE}
<name-printed-in-above-create-command>

The repository commit also displays the ready state.

NAME URL READY STATUS

AGE
example-repo https://github.com/example-org/example-repo True stored
artifact for digest
'2.16@sha256:c7cbee436dc81ac09c7b985696ba798c4d55f6964634c0f1e89e282e632aaaf3'
1m

Troubleshooting an OCIRepository

Logs used for troubleshooting issues with creating OCIRepository sources.

Procedure

Review the following logs:

kubectl -n kommander-flux logs -l app=source-controller
[...]
kubectl -n kommander-flux logs -l app=kustomize-controller
[...]
kubectl -n kommander-flux logs -l app=helm-controller
[...]

For more information, see:

Flux: <https://fluxcd.io/>
Flux documentation: <https://fluxcd.io/docs>

Application Metadata

You can define how custom applications display in the NKP UI by defining a metadata.yaml file for each version of an application in the git repository. You must define this file at applications/<app name>/<app version>/metadata.yaml for it to process correctly.

You can define the following fields:

displayName App ID Display name of the application for the UI.

allowMultipleInstances true Whether multiple instances of the application can be installed.

category [general] One or more categories for this application. Categories are used to group applications in the UI.

description “” Short description, must be a sentence or two, displayed in the UI on the application card.

dependencies List of applications that must be installed in order for the application to function properly. The UI will not block the installation of the application if the dependencies are not installed

requiredDependencies List of applications that are required to be installed in order for the application to be enabled in the UI.

k8sVersionSupport A string indicating the compatible Kubernetes version or range. For example, from 1.29 to 1.32).

icon Base64 encoded icon SVG file used for application logos in the UI.

nkpVersionSupport A string indicating the compatible NKP version or range.

Field	Default	Description

overview Markdown overview used on the application detail page in the UI.

supportLink A link to the support page

scope [project] List of scopes, can be set only to project or workspace currently.

upgradesFrom A string indicating the version or range of versions that the application can be upgraded from.

licensing [Pro, Ultimate] Cluster must have one of these licenses applied in order for the application to be installable.

certifications List of certifications that the application has.

type custom Type of application

The following JSON Schema must be adhered to in the metadata.yaml file for it to be processed accurately. To validate metadata.yaml against the schema, run the following command:

nkp validate catalog-repository --repo-dir=/path/to/catalog-repository
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "catalog.nkp.nutanix.com/v1/application-metadata",
"properties": {
"schema": {
"type": "string",
"description": "Identifies the schema used."
},
"displayName": {
"type": "string",
"description": "Display name of the application for the UI.\nFalls back to App ID
if not given."
},
"allowMultipleInstances": {
"type": "boolean",
"description": "Whether multiple instances of the application can be installed.
Defaults to true.",
"default": true
},
"category": {
"items": {
"type": "string"
},
"type": "array",
"description": "1 or more categories for this application. Categories are used to
group applications in the UI.\nDefaults to [general].",
"default": [
"general"
]
},
"description": {
"type": "string",
"description": "Short description, should be a sentence or two, displayed in the
UI on the application card."
},
"dependencies": {
"items": {
"type": "string"
},

Field	Default	Description

"type": "array",
"description": "List of applications that should be installed in order for the
application to function properly.\nThe UI will not block the installation of the
application if the dependencies are not installed."
},
"requiredDependencies": {
"items": {
"type": "string"
},
"type": "array",
"description": "List of applications that are required to be installed in order
for the application to be enabled in the UI."
},
"k8sVersionSupport": {
"type": "string",
"pattern": "^(0|[1-9]\\d*)(\\.(0|[1-9]\\d*)){0,2}$|^((>=?|<=?)\\s*(v)?(0|[1-9]\
\d*)(\\.(0|[1-9]\\d*)){0,2})(\\s*(>=?|<=?)\\s*(v)?(0|[1-9]\\d*))?(\\.(0|[1-9]\\d*))
{0,2}?$",
"description": "A string indicating the compatible Kubernetes version or range."
},
"icon": {
"type": "string",
"description": "Base64 encoded icon SVG file used for application logos in the
UI."
},
"nkpVersionSupport": {
"type": "string",
"pattern": "^(0|[1-9]\\d*)(\\.(0|[1-9]\\d*)){0,2}$|^((>=?|<=?)\\s*(v)?(0|[1-9]\
\d*)(\\.(0|[1-9]\\d*)){0,2})(\\s*(>=?|<=?)\\s*(v)?(0|[1-9]\\d*))?(\\.(0|[1-9]\\d*))
{0,2}?$",
"description": "A string indicating the compatible NKP version or range."
},
"overview": {
"type": "string",
"description": "Markdown overview used on the application detail page in the UI."
},
"supportLink": {
"type": "string",
"pattern": "^https?:\\/\\/([a-zA-Z0-9\\.-]+)\\.([a-zA-Z]{2,6})(:[0-9]{1,5})?(\\/
[^\\s]*)?$",
"description": "A link to the support page."
},
"scope": {
"items": {
"type": "string",
"enum": [
"project",
"workspace"
]
},
"type": "array",
"description": "List of scopes, can be set only to project and/or workspace.
Defaults to project.",
"default": [
"project"
]
},
"upgradesFrom": {
"type": "string",
"pattern": "^(0|[1-9]\\d*)(\\.(0|[1-9]\\d*)){0,2}$|^((>=?|<=?)\\s*(v)?(0|[1-9]\
\d*)(\\.(0|[1-9]\\d*)){0,2})(\\s*(>=?|<=?)\\s*(v)?(0|[1-9]\\d*))?(\\.(0|[1-9]\\d*))
{0,2}?$",
"description": "A string indicating the version or range of versions that the
application can be upgraded from."
},
"licensing": {
"items": {
"type": "string",
"enum": [
"Starter",
"Pro",
"Ultimate"
]
},
"type": "array",
"description": "Cluster must have one of these licenses applied in order for the
application to be installable.\nDefaults to [Pro, Ultimate].",
"default": [
"Pro",
"Ultimate"
]
},
"certifications": {
"items": {
"type": "string",
"enum": [
"airgapped",
"qualified",
"nutanix-supported"
]
},
"type": "array",
"description": "List of certifications that the application has."
},
"type": {
"type": "string",
"enum": [
"internal",
"nkp-core-platform",
"nkp-catalog",
"preferred-partner",
"custom"
],
"description": "Type of application.",
"default": "custom"
}
},
"additionalProperties": false,
"type": "object",
"required": [
"schema"
],
"description": "Metadata for an application."
}

None of these fields are required for the application to display in the UI.

Here is an example metadata.yaml file.

schema: "catalog.nkp.nutanix.com/v1/application-metadata"
displayName: Prometheus Monitoring Stack
description: Stack of applications that collect metrics and provides visualization and
alerting capabilities. Includes Prometheus, Prometheus Alertmanager and Grafana.
category:
- monitoring
overview: >
# Overview
A stack of applications that collects metrics and provides visualization and alerting
capabilities. Includes Prometheus, Prometheus Alertmanager and Grafana.
## Dashboards
By deploying the Prometheus Monitoring Stack, the following platform applications and
their respective dashboards are deployed. After deployment to clusters in a workspace,
the dashboards are available to access from a respective cluster's detail page.
### Prometheus
A software application for event monitoring and alerting. It records real-time
metrics in a time series database built using a HTTP pull model, with flexible and
real-time alerting.
- [Prometheus Documentation - Overview](https://prometheus.io/docs/introduction/
overview/)
### Prometheus Alertmanager
A Prometheus component that enables you to configure and manage alerts sent by the
Prometheus server and to route them to notification, paging, and automation systems.
- [Prometheus Alertmanager Documentation - Overview](https://prometheus.io/docs/
alerting/latest/alertmanager/)
### Grafana
A monitoring dashboard from Grafana that can be used to visualize metrics collected
by Prometheus.
- [Grafana Documentation](https://grafana.com/docs/)
icon:
PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAzMDAgMzAwIiBzdHlsZT0iZW5hYm

Custom Application from the Workspace Catalog

After creating a catalog collection or an application, you can either use the NKP user interface or the CLI to enable your custom applications. To deploy an application to selected clusters within a workspace, see Cluster-scoped Application Configuration from the NKP UI on page 371.

From within a workspace, you can enable applications to deploy. Verify that the application has successfully deployed through the CLI.

Enabling the Custom Application Using the UI

About this task

Procedure

From the top menu bar, select your target workspace.
From the sidebar menu to browse the available applications from your configured repositories, select Applications.
Select the three dot button on the required application tile and click Enable.
If available, select a version from the dropdown list. This dropdown list will only be visible if there is more than one version.
(Optional) If you want to override the default configuration values, copy your customized values into the text editor under Workspace Configuration or upload your YAML file that contains the values.

someField: someValue

Confirm the details are correct, and then click Enable.

For all applications, you must provide a display name and an ID which is automatically generated based on what you enter for the display name, unless or until you edit the ID directly. The ID must be compliant with Kubernetes DNS subdomain name validation rules. For more information, see DNS Subdomain Names section in the Kubernetes documentation.

Alternately, you can use the CLI to enable your catalog applications.

Enabling the Custom Application Using the CLI

Before you begin

Determine the name of the workspace where you wish to perform the deployments. You can use the nkp get workspaces command to see the list of workspace names and their corresponding namespaces.
Set the WORKSPACE_NAMESPACE environment variable to the name of the workspace’s namespace where the cluster is attached:

export WORKSPACE_NAMESPACE=<workspace_namespace>

Procedure

Get the list of available applications to enable using the following command.

kubectl get apps -n ${WORKSPACE_NAMESPACE}

Deploy one of the supported applications from the list with an AppDeployment resource.
Within the AppDeployment, define the appRef to specify which App to enable.

cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: my-custom-app
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: custom-app-0.0.1
kind: App
EOF

Note:

The appRef.name must match the app name from the list of available catalog applications.
Create the resource in the workspace you just created, which instructs Kommander to deploy the AppDeployment to the KommanderClusters in the same workspace.

Enabling the Custom Application With Custom Configuration Using the CLI

About this task

Procedure

Provide the name of a ConfigMap in the AppDeployment, which provides custom configuration on top of the default configuration.

cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
name: my-custom-app
namespace: ${WORKSPACE_NAMESPACE}
spec:
appRef:
name: custom-app-0.0.1
kind: App
configOverrides:
name: my-custom-app-overrides
EOF

Create the ConfigMap with the name provided in the step above, with the custom configuration.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
namespace: ${WORKSPACE_NAMESPACE}
name: my-custom-app-overrides
data:
values.yaml: |
someField: someValue
EOF

Kommander waits for the ConfigMap to be present before deploying the AppDeployment to the managed or attached clusters.

Verify the Custom Applications

After completing the previous steps, your applications are enabled.

Procedure

Connect to the attached cluster and check the HelmReleases to verify the deployment.

kubectl get helmreleases -n ${WORKSPACE_NAMESPACE}

The output is as follows.

NAMESPACE NAME READY STATUS AGE
workspace-test-vjsfq my-custom-app True Release reconciliation succeeded 7m3s

Configuring Workspace Role Bindings

Before you begin

Before you can create a Workspace Role Binding, ensure you have created a workspace Group. A Group can contain one or several Identity Provider users, groups or both.

The syntax for the Identity Provider groups you add to a Group varies depending on the context for which you have established an Identity Provider.

If you have set up an identity provider globally, for All Workspaces:
For groups: Add an Identity Provider Group in the oidc:<IdP_user_group> format. For example, oidc:engineering.
For users: Add an Identity Provider User in the <user_email>. For example, <[email protected]>.
If you have set up an identity provider for a Specific Workspace:
For groups: Add an Identity Provider Group in the oidc:<workspace_name>:<IdP_user_group> format. For example, oidc:tenant-z:engineering.
For users: Add an Identity Provider User in the <workspace_ID>:<user_email> format. For example, tenant-z:[email protected].

Note: Run kubectl get workspaces to obtain a list of all existing workspaces. The workspace_ID is listed
under the NAME column.

About this task

You can assign a role to this Kommander Group:

Procedure

From the top menu bar, select your target workspace.
Select Access Control in the Administration section of the sidebar menu.
Select the Cluster Role Bindings tab, and then select Add Roles next to the group you want.
Select the Role, or Roles, you want from the dropdown list and click Save. It will take a few minutes for the resource to be created.

Multi-Tenancy in NKP

Here are some important concepts:

Multi-tenant environments have at least two participating parties: the Ultimate license administrator (for example, an MSP), and one or several tenants.

Managed Service Providers or MSPs are partner organizations that use NKP to facilitate cloud infrastructure services to their customers or tenants.
Tenants can be customers of Managed Service Provider partners. They outsource their cloud management requirements to MSPs, so they can focus on the development of their products.

Tenants can also be divisions within an organization that require a strict isolation from other divisions, for example, through differentiated access control.

In NKP, a workspace is assigned to a tenant.

Access Control in Multi-Tenant Environments

To isolate each tenant’s information and environment, multi-tenancy allows you to configure an identity provider per workspace or tenant. In this setup, NKP keeps all workspaces and tenants separate and isolated from each other.

You, as a global administrator, manage tenant access at the Workspace level. A tenant can further adapt user access at the Project level.

Figure 9: Multi-tenant Cluster

Here are some important concepts:

Workspaces: In a multi-tenant system, workspaces and tenants are synonymous. You can set up an identity provider to control all workspaces, including the Management cluster’s kommander workspace. You can then set up additional identity providers for each workspace/tenant, and generate a dedicated Login URL so each tenant has its own user access.

Projects: After you set up an identity provider per workspace or tenant, the tenant can choose to further narrow down access with an additional layer. A tenant can choose to organize clusters into projects and assign differentiated access to user groups with Project Role Bindings.

For more information, see Project Role Bindings on page 442.

By assigning clusters to one or several projects, you can enable more complex user access.

Multi-Tenancy Enablement

To enable multi-tenancy, you must:

If you want to use a single IdP to access all of your tenant’s environments, configure an Identity Provider globally.
Configure an Identity Provider per workspace. This way, each tenant has a dedicated IdP to access their workspace.
Create NKP Identity Provider groups with the correct prefixes to map your existing IdP groups.
Create a dedicated login URL for each tenant. You can provide a workspace login link to each tenant for access to the NKP UI and for the generation of kubectl API access tokens.

To enforce security, every tenant should be in a different AWS account, so they are truly independent of each other.

About this task

By making this URL available to your tenant, you provide them with a dedicated login page, where users can enter their SSO credentials to access their workspace in the NKP UI and to where users can create a token to access a cluster’s kubectl API. Other tenants and their SSO configurations are not visible.

Before you begin

Complete the steps in Multi-Tenancy in NKP on page 412.
Ensure you have administrator permissions and access to all workspaces.

Procedure

Set an environment variable to point at the workspace for which you want to generate a URL: Replace <name_target_workspace> with the workspace name. If you do not know the exact name of the workspace, run kubectl get workspace to get a list of all workspace names.

export WORKSPACE_NAME=<name_target_workspace>

Generate an NKP UI login URL for that workspace.

echo https://$(kubectl get nkpcluster -n kommander -l 'kommander.d2iq.io/host=true'
-o jsonpath='{.items[0].kommanderCluster.status.ingress.address }')/token/landing/
${WORKSPACE_NAME}

The output is as follows.

https://example.com/token/landing/<WORKSPACE_NAME>

Share the output login URL with your tenant, so users can start accessing their workspace from the NKP UI.

Note: The login page displays:

Identity providers set globally.
Identity providers set for that specific workspace.

The login page does not display any resources or workspaces for which the tenant has no permissions.

Projects

Multi-cluster Configuration Management

Projects support the management of configMaps, continuous deployments, secrets, services, quotas, and role-based access control and multi-tenant logging by leveraging federated resources. When a Project is created, NKP creates a federated namespace that is propagated to the Kubernetes clusters associated with this Project.

Federation in this context means that a common configuration is pushed out from a central location (NKP) to all Kubernetes clusters, or a pre-defined subset group, under NKP management. This pre-defined subset group of Kubernetes clusters is called a Project.

Projects enable teams to deploy their configurations and services to clusters in a consistent way. Projects enable central IT or a business unit to share their Kubernetes clusters among several teams. Using Projects, NKP leverages Kubernetes Cluster Federation (KubeFed) to coordinate the configuration of multiple Kubernetes clusters.

Kommander allows a user to use labels to select, manually or dynamically, the Kubernetes clusters associated with a Project.

Project Namespaces

A Project Namespace is a Kommander-specific concept. Project Namespaces isolate configurations across clusters and are created on all clusters matching the project labels. When creating a new project, you can customize the name of the Kubernetes namespace that is created.

Creating a Project Using the UI

About this task

When you create a Project, you must specify a Project Name, a Namespace Name (optional) and a way to allow Kommander to determine which Kubernetes clusters will be part of this project.

As mentioned previously, a Project Namespace corresponds to a Kubernetes Federated Namespace. By default, the name of the namespace is auto-generated based on the project name (first 57 characters) plus 5 unique alphanumeric characters. You can specify a namespace name, but you must ensure it does not conflict with any existing namespace on the target Kubernetes clusters, that will be a part of the Project.

To determine which Kubernetes clusters will be part of this project, you can either select manually existing clusters or define labels that Kommander will use to dynamically add clusters. The latter is recommended because it will allow you to deploy additional Kubernetes clusters later and to have them automatically associated with Projects based on their labels.

To create a Project, you can either use the NKP UI or create a Project object on the Kubernetes cluster where Kommander is running (using kubectl or the Kubernetes API). The latter allows you to configure Kommander resources in a declarative way. It is available for all kinds of Kommander resources.

To create a project using the NKP UI:

Procedure

Enter the following fields:

Project Name
ID / Namespace
Description

In the Clusters field, add one or more active clusters.
Click Create.

Creating a Project Using the CLI

About this task

The following sample is a YAML Kubernetes object for creating a Kommander Project. This example does not work verbatim because it depends on a workspace name that has been previously created and does not exist by default in your cluster.

Procedure

Use this as an example format and fill in the workspace name and namespace name appropriately along with the proper labels.

apiVersion: workspaces.kommander.mesosphere.io/v1alpha1
kind: Project
metadata:
name: My-Project-Name
namespace: my-project-k8s-namespace-name
spec:
workspaceRef:
name: myworkspacename
namespaceName: myworkspacename-di3tx
placement:
clusterSelector:
matchLabels:
cluster: prod

The following procedures are supported for projects:

Project Applications on page 417
Project Deployments on page 432
Project Role Bindings on page 442
Project Roles on page 445
Project ConfigMaps on page 448
Project Secrets on page 449
Project Quotas and Limit Ranges on page 451
Project Network Policies on page 453

Project Applications

Application types are:

Workspace Catalog Applications on page 379 that are either pre-packaged applications from the Nutanix Application Catalog or custom applications that you maintain for your teams or organization.
NKP Applications on page 327 are applications that are provided by Nutanix and added to the Catalog.
Custom Applications on page 402 are applications integrated into Kommander.
Platform Applications on page 350

When deploying and upgrading applications, platform applications come as a bundle; they are tested as a single unit and you must deploy or upgrade them in a single process, for each workspace. This means all clusters in a workspace have the same set and versions of platform applications deployed. Whereas catalog applications are individual, so you can deploy and upgrade them individually, for each project.

Project Platform Applications

The following table describes the list of applications that can be deployed to attached clusters within a project.

Review the Project Platform Application Configuration Requirements on page 421 to ensure that the attached clusters in the project have sufficient resources.

From within a project, you can enable applications to deploy. Verify that an application has successfully deployed through the CLI.

Platform Applications

Table 41: Platform Applications

project-grafana-logging-11.3.3 project-grafana-logging False

project-grafana-loki-v3-3.6.7 project-grafana-loki False

project-logging-1.1.0 project-logging False

Enabling the Platform Application Using the UI

About this task

Follow these steps to:

Procedure

Log into the NKP UI dashboard.
In the workspace header drop-down list, select your target workspace.
From the left navigation menu, select Projects.

Name	APP ID	Deployed by default

From the list of project, select your project.
To browse the available applications, select the Applications tab.
Select the triple dot vertical from the bottom-right corner of the desired application tile, and then select Enable.
If you want to override the default configuration values, copy your customized values into the text editor under Configure Application or upload your YAML file that contains the values.

someField: someValue

Confirm the details are correct, and then click Enable.

To use the CLI to enable or disable applications, see Deploying Platform Applications Using CLI on page 353

Warning: There may be dependencies between the applications, which are listed in Project Platform
Application Dependencies on page 420. Review them carefully prior to customizing to ensure that the
applications are deployed successfully.

Platform Applications Upgrade Using the CLI

Platform Applications within a Project are automatically upgraded when the Workspace that a Project belongs to is upgraded.

For more information on how to upgrade these applications, see Upgrading Platform Applications on Managed and Attached Clusters on page 1050.

Deploying Project Platform Applications Using the CLI

Deploy applications to attached clusters in a project using the CLI.

About this task

This topic describes how to use the CLI to deploy an application to attached clusters within a project.

For a list of all applications and those that are enabled by default, see Project Platform Applications on page 417.

Before you begin

Ensure that you have:

A running cluster with Kommander installed.
An existing Kubernetes cluster attached to Kommander.
Set the WORKSPACE_NAME environment variable to the name of the workspace where the cluster is attached.

export WORKSPACE_NAME=<workspace_name>

Set the WORKSPACE_NAMESPACE environment variable to the namespace of the above workspace.

export WORKSPACE_NAMESPACE=$(kubectl get namespace --
selector='workspaces.kommander.mesosphere.io/workspace-name=${WORKSPACE_NAME}' -o
jsonpath='{.items[0].metadata.name}')

Set the PROJECT_NAME environment variable to the name of the project in which the cluster is included:

export PROJECT_NAME=<project_name>

Set the PROJECT_NAMESPACE environment variable to the name of the above project’s namespace:

export PROJECT_NAMESPACE=$(kubectl get project ${PROJECT_NAMESPACE} -n
${WORKSPACE_NAMESPACE} -o jsonpath='{.status.namespaceRef.name}')

Procedure

Deploy one of the supported applications to your existing attached cluster with an AppDeployment resource. Provide the appRef and application version to specify which App is deployed.

nkp create appdeployment project-grafana-logging --app project-grafana-logging-11.3.3
--workspace ${WORKSPACE_NAME} --project ${PROJECT_NAME}

Create the resource in the project you just created, which instructs Kommander to deploy the AppDeployment to the KommanderClusters in the same project.

Note:

The appRef.name must match the app name from the list of available catalog applications.
Observe that the nkp create command must be run with both the –workspace and –project flags for project platform applications.

Deploying the Project Platform Application With Custom Configuration

Using the CLI

About this task

To perform custom configuration using the CLI:

Procedure

Create the AppDeployment and provide the name of a ConfigMap, which provides custom configuration on top of the default configuration.

nkp create appdeployment project-grafana-logging --app project-grafana-logging-11.3.3
--config-overrides project-grafana-logging-overrides --workspace ${WORKSPACE_NAME}
--project ${PROJECT_NAMESPACE}

Create the ConfigMap with the name provided in the step above, with the custom configuration.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
namespace: ${PROJECT_NAMESPACE}
name: project-grafana-logging-overrides
data:
values.yaml: |
datasources:
datasources.yaml:
apiVersion: 1
datasources:
- name: Loki
type: loki
url: "http://project-grafana-loki-loki-distributed-gateway"
access: proxy
isDefault: false
EOF

Kommander waits for the ConfigMap to be present before deploying the AppDeployment to the managed or attached clusters.

Verify the Project Platform Applications

After completing the previous steps, your applications are enabled.

Procedure

Export the project_namespace with this command.

export PROJECT_NAMESPACE=<project_namespace>

Connect to the attached cluster and check the HelmReleases to verify the deployment.

kubectl get helmreleases -n ${PROJECT_NAMESPACE}
NAMESPACE NAME READY STATUS
AGE
project-test-vjsfq project-grafana-logging True Release reconciliation
succeeded 7m3s
Note: Some of the supported applications have dependencies on other applications. See Project Platform
Application Dependencies on page 420 for that table.

Project Platform Application Dependencies

Dependencies between project platform applications.

There are many dependencies between the applications that are deployed to a project’s attached clusters. It is important to note these dependencies when customizing the platform applications to ensure that your services are properly deployed to the clusters. For more information on how to customize platform applications, see Project Platform Applications on page 417.

Application Dependencies

When deploying or troubleshooting applications, it helps to understand how applications interact and may require other applications as dependencies.

If an application’s dependency does not successfully deploy, the application requiring that dependency does not successfully deploy.

The following sections detail information about the platform applications.

Logging

Collects logs over time from Kubernetes pods deployed in the project namespace. Also provides the ability to visualize and query the aggregated logs.

project-logging: Defines resources for the Logging Operator which uses them to direct the project’s logs to its respective Grafana Loki application. For more information, see <https://grafana.com/oss/grafana/>.
project-grafana-loki: A horizontally-scalable, highly-available, multi- tenant log aggregation system inspired by Prometheus. For more information, see <https://grafana.com/oss/loki/>.
project-grafana-logging: Logging dashboard used to view logs aggregated to Grafana Loki. For more information, see <https://grafana.com/oss/grafana/>.

Warning: The project logging applications depend on the Enabling Logging Applications Using the UI on
page 600 being deployed.

Table 42: Project Platform Application Dependencies

project-logging logging-operator (workspace)

project-grafana-loki project-logging, grafana-loki (workspace), logging- operator (workspace)

project-grafana-logging project-grafana-loki

Project Platform Application Configuration Requirements

Project Platform Application Descriptions and Resource Requirements

Platform applications require more resources than solely deploying or attaching clusters into a project. Your cluster must have sufficient resources when deploying or attaching to ensure that the applications are installed successfully.

The following table describes all the platform applications that are available to the clusters in a project, minimum resource and persistent storage requirements, and whether they are enabled by default.

project-grafana- logging

cpu: 200m

memory: 100Mi

No NKP Critical (100002000)

project-grafana-loki # of PVs: 3

PV sizes: 10Gi x 3 (total: 30Gi)

No NKP Critical (100002000)

project-logging No NKP Critical (100002000)

Project Catalog Applications

Before upgrading your catalog applications, verify the current and supported versions of the application. Also, keep in mind the distinction between Platform applications and Catalog applications. Platform applications are deployed and upgraded as a set for each cluster or workspace. Catalog applications are deployed separately, so that you can deploy and upgrade them individually for each project.

Application	Required Dependencies

Name; Minimum Resources Suggested; Minimum Persistent Storage Required; Deployed by Default; Default PriorityClass

| — | — | — | — | — |

About this task

Catalog applications must be upgraded to the latest version BEFORE upgrading the Konvoy component for Managed clusters or Kubernetes version for attached clusters.

To upgrade an application from the NKP UI:

Procedure

From the top menu bar, select your target workspace.
From the side menu bar, select Projects.
Select your target project.
Select Applications from the project menu bar.
Select the three dot button from the bottom-right corner of the desired application tile, and then click Edit.
Select the Version dropdown list, and select a new version. This dropdown list will only be available if there is a newer version to upgrade to.
Click Save.

Upgrading Project Catalog Applications Using the CLI

About this task

To upgrade project catalog applications:

Procedure

To see what app(s) and app versions are available to upgrade, run the following command:

Note: The APP ID column displays the available apps and the versions available to upgrade.

kubectl get apps -n ${PROJECT_NAMESPACE}

Run the following command to upgrade an application from the NKP CLI.

nkp upgrade catalogapp <appdeployment-name> --workspace=my-workspace --project=my-
project --to-version=<version.number>
Note: Platform applications cannot be upgraded on a one-off basis, and must be upgraded in a single process for
each workspace. If you attempt to upgrade a platform application with these commands, you receive an error and
the application is not upgraded.

Project-level NKP Applications

NKP applications are catalog applications provided by Nutanix for use in your environment.

Some NKP workspace catalog applications will provision CustomResourceDefinitions, which allow you to deploy Custom Resources to a Project. See your NKP workspace catalog application’s documentation for instructions.

Usage of Custom Resources with Workspace Catalog Applications

Some workspace catalog applications provision one or more CustomResourceDefinition (CRD) objects as part of their deployment process.

The CRDs allow you to deploy and manage the corresponding custom resources within your environment. For more information on how to configure and use the custom resources effectively, see the documentation specific to your workspace catalog application.

Custom Project Applications

Custom applications are third-party applications you have added to the Kommander Catalog.

Custom applications are any third-party applications that are not provided in the NKP Application Catalog. Custom applications can leverage applications from the NKP Catalog or be fully-customized. There is no expectation of support by Nutanix for a Custom application. Custom applications can be deployed on Konvoy clusters or on any Nutanix supported 3rd party Kubernetes distribution.

Git Repository Structure (2)

Creating a Catalog Collection or an Application on page 404
Application Metadata on page 405
Enabling a Custom Application From the Project Catalog Using the UI on page 430 and Enabling a Custom Application From the Project Catalog Using the CLI on page 431

Git repositories must be structured in a specific manner for defined applications to be processed by Kommander.

You must structure your Git repository based on the following guidelines, for your applications to be processed properly by Kommander so that they can be deployed.

Git Repository Directory Structure

Run the following command to generate a new Git repository for hosting your catalog applications:

mkdir -p /new/directory/for/catalog
cd /new/directory/for/catalog
nkp generate catalog-repository

Run the following command, to generate a directory for your application version. You can run this command multiple times (once for each version of a given application).

nkp generate catalog-repository --apps <app-name>=<app-version>

Above commands will generate following basic directory structure:

...
### applications (5)
# ### <app-name> (3)
# ### <app-version1> (3)
# ### helmrelease (5)
# # ### cm.yaml // ConfigMap specifying default (3)
configuration overrides for HelmRelease
# # ### helmrelease.yaml // OCIRepository and HelmRelease (3)
# # ### kustomization.yaml // Kubernetes Kustomization file (3)
# ### helmrelease.yaml // FluxCD Kustomization file for the (3)
HelmRelease
# ### kustomization.yaml // Kubernetes Kustomization file (3)
# ### metadata.yaml // Application details (3)
# ### <app-version2> (3)
# ### helmrelease (6)
# # ### cm.yaml (3)
# # ### helmrelease.yaml (3)
# # ### kustomization.yaml (3)
# ### helmrelease.yaml (3)
# ### kustomization.yaml (3)
# ### metadata.yaml (3)
...

Remember the following guidelines:

Define applications in the applications/ directory.
You can define multiple versions of an application, under different directories nested under the applications/ <app name>/ directory.
Populate the HelmRelease and OCIRepository details in the versioned directory applications/<app name>/ <app version>/helmrelease in the helmrelease.yaml file. For more information, see HelmRelease, OCIRepository in the Flux documentation, and The Kustomization File in the SIG CLI documentation.
Define the default values ConfigMap for HelmReleases in the applications/<app name>/<app version>/helmrelease/cm.yaml file accompanied by a kustomization.yaml Kubernetes Kustomization file pointing to the ConfigMap file.
Define the metadata.yaml of each application under the applications/<app name>/<app version> directory. For more information, see Application Metadata on page 405.
You can generate a new application version directory with the required layout and files by running:

nkp generate catalog-repository --apps=<app-name>=<app-version>

For an example of how to structure catalog Git repositories, see Overview of the Application Structure on page 401.

OCI Repositories

You must include the OCIRepository that is referenced in each HelmRelease’s Chart spec.

Each applications/<app name>/<app version>/helmrelease/helmrelease.yaml file must include YAML definition of the OCIRepository along with the HelmRelease definition.

For more information, see OCI Repository in the Flux documentation.

Substitution Variables

Some substitution variables are provided. For more information, see Kustomization in the Flux documentation.

${releaseName}: For each App deployment, this variable is set to the AppDeployment name. Use this variable to prefix the names of any resources that are defined in the application directory in the Git repository so that multiple instances of the same application can be deployed. If you create resources without using the releaseName prefix (or suffix) in the name field, there can be conflicts if the same named resource is created in that same namespace.
${releaseNamespace}: The namespace of the workspace.
${workspaceNamespace}: The namespace of the workspace that the Workspace belongs to.

Creating a Catalog Collection or an Application (2)

Use CLI to create a Flux OCIRepository resource in your workspace or project that acts as a source for your catalog applications.

About this task

Create an OCIRepository in the workspace or project namespace.

Procedure

To build an OCI artifact from your catalog git repository run the following command. This generates a .tar file locally.

nkp create catalog-bundle --collection-tag <tag>

Above command will bundle up all versions of all applications in the current directory.

Note:

To include only selected applications use –apps flag.
To include the container images use –airgapped flag. This is required for air-gapped environments.

Push the OCI artifact to a registry that is accessible to the cluster. The registry used must be OCI Compliant.

nkp push bundle --bundle <path-to-file-generated-above.tar> --to-registry <your-
registry-url>

You can either create a catalog collection (group of applications) or just one application in your NKP Cluster. Note that creating a catalog in the Kommander workspace will automatically propagate it to all other workspaces/projects/. If you prefer to limit the scope of catalog to a single workspace or project, use the –workspace or –project flags accordingly. Run either one of the following commands:

Create a Catalog Collection

nkp create catalog-collection --url oci://<registry-url>/<repo-name>/collection --
workspace <workspace-or-project-namespace> --tag <tag>

Create a Catalog Application

nkp create catalog-collection --url oci://<registry-url>/<repo-name>/<app-name> --
workspace <workspace-or-project-namespace> --tag <app-version>

Optionally, ensure the status of the OCIRepository signals a ready state.

kubectl get ocirepository -n <workspace-or-project-namespace>${WORKSPACE_NAMESPACE}
<name-printed-in-above-create-command>

The repository commit also displays the ready state.

NAME URL READY STATUS

AGE
example-repo https://github.com/example-org/example-repo True stored
artifact for digest
'2.16@sha256:c7cbee436dc81ac09c7b985696ba798c4d55f6964634c0f1e89e282e632aaaf3'
1m

Troubleshooting an OCIRepository (2)

Procedure

Review the following logs:

kubectl -n kommander-flux logs -l app=source-controller
[...]
kubectl -n kommander-flux logs -l app=kustomize-controller
[...]
kubectl -n kommander-flux logs -l app=helm-controller
[...]

For more information, see:

Flux: <https://fluxcd.io/>
Flux documentation: <https://fluxcd.io/docs>

Application Metadata (2)