Release Notes Highlights for MLDM

• Feature: Users can now manage metadata (as key:value pairs) in Console for projects and repositories from the User Metadata tab of the details side panel.
• Enhancement: Projects, pipelines, branches, and commits now include the following dervied metadata by default: created_at, created_by, updated_at, and updated_by.
• Feature: Pre-built Jsonnet templates are now available in Console when creating a pipeline:
  • Snowflake Integration: Creates a cron pipeline that can execute a query against a Snowflake database and return the results in a single output file.
  • Hugging Face Downloader: Creates a cron pipeline to download datasets or models from huggingface on demand.
• Enhancement: Several enhancements have been made to improve the integration between HPE Machine Learning Data Management and Machine Learning Development Environment.
• Feature: The Pachyderm SDK now has an extras package cdr that you can install (pip install pachyderm_sdk[cdr]) to make use of Common Data Refs (CDRs) in your user code. CDRs improve performance and speed by downloading version-controlled data directly from HPE Machine Learning Data Management’s underlying Object Storage bucket and caching that data locally on your machine, allowing datasets to be assembled entirely locally and incrementally updated.
• Security: The HPE Machine Learning Data Management repository is now available at Iron Bank, a hardened container image repository owned and maintained by the U.S. Department of Defense (DoD) that supports the end-to-end lifecycle for modern software development. If you plan to download and install from Iron Bank, please reach out to ai-support@hpe.com or your Customer Success Engineer for assistance.
• Notice: The gocdk_enabled attribute has been removed from the Helm Chart Values as it is now the default object storage driver.

• Feature: You can now create pipelines in Console using jsonnet-compatible pipeline templates.
• Enhancement: Console now displays more information in the side panel for repos and pipelines, including the pipeline’s active workers and parallelism spec.
• Enhancement: Several web-responsive improvements have been made to Console: the side-panel now supports responsive sizing, and the DAG view side panel tab headers become scrollable on small screen sizes.
• Fix: Action buttons in the sidebar now become dropdown ellipsis menus when resizing the sidebar to a narrower size.
• Fix: When trying to download a single file, the file browser will download the file instead of opening the file in a new tab
• Fix: Correctly configured custom TLS certificates in Console.
• Fix: Improved our consistent hashing algorithm’s use of locking.

• Enhancement: Editing Project and Cluster Metadata is now available in Console.
• Enhancement: pachctl is now available in Determined notebooks.
• Fix: Resolve an unexpected file loading error when opening multiple files in the Jupyter extension explore view.
• Fix: Pachyderm Clients are now correctly loaded into Determined notebooks.

• Enhancement: Editing Commit and Repo Metadata is now available in Console.

• Feature: For enterprise customers using MLDM (Pachyderm) & MLDE (Determined) in a combined cluster environment, the MLDE Notebooks now include support for the Jupyter Pachyderm extension by default.
• Feature: You can now set up and maintain metadata on your HPE Machine Learning Data Management artifacts. This includes clusters, projects, repos, branches, and commits.
• Enhancement: The blog storage configuration attribute GOCDK_ENABLED is now set to true by default in your Helm chart values; in 2.11.0 the option to disable it will be removed.
• Enhancement: The Console UI has undergone several improvements, including:
  • Improved file browsing experience
  • Improved DAG visualizations
    • Interactive DAG edge highlighting
    • Distinguishing colors and/or patterns based on pipeline types
    • Enhancing the ability to understand more about connections by their edges (like joins)
    • Visual indications of parallelism for pipelines when spec calls for it and when running
  • Pipeline and repo table paging

• Enhancement: Colors in the Console’s file browser have been improved for accessibility.
• Enhancement: Console’s callback page now properly redirects to the original page after login instead of redirecting to the home page (/).
• Enhancement: The visualization of the DAG in Console have been improved for better sizing and scaling with your browser window.
• Fix Resolved a bug where pipeline and job statuses were not correctly displaying.

• Enhancement: Updated to Go 1.22.2 and latest x/net.
• Enhancement: Upgraded to Envoy 1.27.4

• Enhancement: Updated to Go 1.22.2 and latest x/net.
• Enhancement: pinned Pachyderm-SDK versino to >=2.8.0<2.9.0 for the Jupyter Mount Extension.
• Enhancement: Upgraded to Envoy 1.27.4

• Enhancement: Updated to Go 1.22.2 and latest x/net.
• Enhancement: Upgraded to Envoy 1.27.4
• Fix: Resolved issue in Console where proxy port caused issues with downloading files.
• Feature: Added new args to debug dumps. You can review the full debug template by running pachctl debug template in the CLI.

Full Default Debug template #

$ pachctl debug template
defaults:
    clusterDefaults: true
inputRepos: true
starlarkScripts:
    - builtin: starlark/basic.star
      timeout: 60s
    - builtin: starlark/list_rcs.star
      timeout: 60s
system:
    database: true
    describes:
        - name: console                                                                                                                                                                                                                                                     - name: etcd
        - name: pachd
        - name: pachyderm-kube-event-tail
        - name: pachyderm-proxy
        - name: pg-bouncer
    helm: true
    logs:
        - name: console
        - name: etcd
        - name: pachd
        - name: pachyderm-kube-event-tail
        - name: pachyderm-proxy
        - name: pg-bouncer
    lokiLogs:
        - lokiArgs:
            maxLogs: "30000"
          name: console
        - lokiArgs:
            maxLogs: "30000"
          name: etcd
        - lokiArgs:
            maxLogs: "30000"
          name: pachd
        - lokiArgs:
            maxLogs: "30000"
          name: pachyderm-kube-event-tail
        - lokiArgs:
            maxLogs: "30000"
          name: pachyderm-proxy
        - lokiArgs:
            maxLogs: "30000"
          name: pg-bouncer
    profiles:
        - name: pachd
          profileArgs:
            profiles:
                - name: heap
                - name: goroutine
    version: true

• Enhancement: Refactored the Console’s Explore View.
• Enhancement: You can now adjust the Refecth Interval rate for Console in the Helm Chart.
• New Feature: You can now set roles on the Cluster level from within Console.