Unified Deployment

In this guide, we’ll show you how to update your cluster configuration to support a unified installation and deployment of HPE Machine Learning Data Management + Machine Learning Development Environment.

Before You Start #

This guide assumes that you have already completed all of the following:

1. Deployed HPE Machine Learning Data Management using one of the cloud deployment guides (AWS, GCP, or Azure).
2. Added an active Enterprise License Key.
3. Set up TLS (SSL, HTTPS) for your HPE Machine Learning Data Management cluster.
4. Set up an OIDC connector for your HPE Machine Learning Data Management cluster.

Self-Signed Certificates #

If you are using a self-signed certificate for your implementation, you must update the Determined Helm values.yaml file at .Values.externalCaCertSecretName to include the name of the secret containing the root certificate.

How to Configure a Unified Setup #

1. Create Necessary Secrets #

You will need to create two secrets for Determined:

1. A Determined Enterprise docker image credentials secret (e.g., det-image)

   kubectl create secret docker-registry det-image \
   --docker-server=https://index.docker.io/v1/ \
   --docker-username=<username> \
   --docker-password=<password> \
   --docker-email=<email> \
   --output=json > det-image-secret.json

   Copy

2. A Determined Enterprise admin credentials secret (e.g., det-creds)

   kubectl create secret generic det-creds \
   --from-literal=determined-username=admin \
   --from-literal=determined-password="" \
   --output=json > det-creds-secret.json

   Copy

2. Update the HPE Machine Learning Data Management Helm Chart #

1. Open your values.yaml file or generate a local copy using the following command:

   helm get values pachyderm > values.yaml

   Copy
2. Update the determined.enabled field in values.yaml file to true.
3. Update the determined.oidc section of your values.yaml file to include the OIDC provider, client ID, and URLs:

   determined:
     oidc:
       enabled: true
       provider: "" # your oidc.upstreamIDPs.config.id; e.g., Auth0 or Okta
       idpRecipientUrl: "" # https://<proxy.host.value.com>:8080 
       idpSsoUrl: "" # https://<proxy.host.value.com>/dex
       clientId: "determined"
       clientSecretKey: ""
       clientSecretName: ""
       authenticationClaim: ""
       scimAuthenticationAttribute: ""
       autoProvisionUsers: false
       groupsAttributeName: ""
       displayNameAttributeName: ""

   Copy
4. Provide a determined.tlsSecret if applicable.
5. Update the pachd section of your values.yaml file to include the full endpoint address and the name of the Determined admin credentials secret:

   pachd:
     determined:
       apiEndpoint: # https://determined-master-service-internal-<HELM RELEASE NAME>:8082
       credentialsSecretName: det-creds 
     activateEnterprise: true

   Copy

Adding Users to Pipelines #

You can add a determined section to your pipeline specification file and make use of a user via the $DET_USER and $DET_PASS environment variables. This can be used by the user code that run determined work to talk back to HPE Machine Learning Data Management and can be used with the Pachyderm SDK.

{
    "pipeline": {
      "name": "<PIPELINE NAME>"
    },
    "description": "<PIPELINE DESCRIPTION>",
    "input": {
      "pfs": {
        "name": "data",
        "repo": "input",
        "branch": "master",
        "glob": "/",
        "emptyFiles": true
      }
    },
    "transform": {
      "cmd": ["/bin/sh"],
      "stdin": ["pip install determined && echo $DET_PASS | det user login $DET_USER && det model list -w WORKSPACE-NAME  > /pfs/out/WORKSPACE-NAME.txt"],
      "image": "python:3.8"
    },
    "determined": {
      "workspaces": ["WORKSPACE-NAME"]
    }
  }

Note

These tokens life cycles are scoped to the jobs and are revoked after the job ends.