Proxy-Related Inference Failures

Symptom #

On clusters that use a proxy, inferencing requests for the nv-embedqa-e5-v5 or nv-embedqa-mistral-7b-v2 models may fail with the following error:

{
"object": "error",
"message": "Triton service is currently unavailable and may be currently initializing. Please check the health endpoints and try again in a few moments. If problem persists then please check the Triton logs for more details.",
"detail": {},
"type": "service_unavailable"
}

Additionally, the deployment pod logs will show the following error:

2024-10-01T18:08:46Z ERROR: root - Got error from Triton: failed to connect to all addresses; last error: UNKNOWN: ipv4:[INTERNAL_IP]:443: HTTP proxy returned response code 502
2024-10-01T18:08:46Z ERROR: root - Triton service unavailable
2024-10-01T18:08:46Z INFO: uvicorn.access - 127.0.0.6:0 - "POST /v1/embeddings HTTP/1.1" 503
127.0.0.6:0 - "POST /v1/embeddings HTTP/1.1" 503

Cause #

The deployment pod is unable to connect to the Triton service due to proxy-related issues.

Resolution #

Add the IP address 0.0.0.0 to your no_proxy environment variable.
Update your Helm values file (values.yaml) or use the --set flag during installation to include the proxy configuration. For more details on setting global environment variables, refer to the Global Environment Variables section in the Helm Chart Values documentation.
```
global:
  env:
    - name: no_proxy
      value: "localhost,127.0.0.1,0.0.0.0"
```

If you’ve already installed MLIS, upgrade your Helm release with the new configuration:

helm upgrade <release_name> <chart_name> --values values.yaml

Or, if you’re using the --set flag:

helm upgrade <release_name> <chart_name> --set global.env[0].name=no_proxy --set global.env[0].value=0.0.0.0

After applying the changes, redeploy the affected models.

Verification #

After applying the fix, attempt to run inference requests again. If successful, you should no longer encounter the error.

For HPE AI Essentials Users

To edit the values.yaml file:

You can easily access the configuration by visiting the configuration button on the HPE MLIS product card.
Edit the NO_PROXY & no_proxy values to include 0.0.0.0..
Then click Configure to apply the changes.