{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "453524a2-0d79-4512-8d97-0240ec1ea08e",
   "metadata": {},
   "source": [
    "<img src=\"./images/DLI_Header.png\" width=400/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c57b540f-18e8-42c3-a688-c30103c5f462",
   "metadata": {},
   "source": [
    "# Fundamentals of Accelerated Data Science # "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61aa05f5-febf-487c-bd59-d489efd98709",
   "metadata": {},
   "source": [
    "## 07 - Triton ##\n",
    "\n",
    "**Table of Contents**\n",
    "<br>\n",
    "This notebook covers the process of deploying a model with Triton Inference Server and walks through the various tools available within Triton. This notebook covers the below sections: \n",
    "1. [Background](#s1)\n",
    "2. [Preparing the Model](#s1)\n",
    "    * [Load the Model](#s1)\n",
    "    * [Creating the Folder Structure](#s1)\n",
    "    * [Creating the Configuration File](#s1)\n",
    "3. [Loading the Model in Triton](#s1)\n",
    "    * [Starting Triton](#s1)\n",
    "    * [Check Status of Triton Server](#s1)\n",
    "4. [Testing Inference](#s1)\n",
    "    * [Triton Client](#s1)\n",
    "    * [Verifying Results for Local Model and Triton](#s1)\n",
    "5. [Analyzing Performance](#s1)\n",
    "    * [Customizing Perf Analyzer](#s1)\n",
    "    * [Exercise #1 - Testing Perf Analyzer](#s1)\n",
    "6. [Model Analyzer](#s1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4d08487-8161-4d8c-b402-cdeab3fcbe2a",
   "metadata": {},
   "source": [
    "# Background\n",
    "NVIDIA offers a framework for deploying ML models called Triton. Triton will automatically handle all inference requests that comes to the server. Triton supports multiple backends including PyTorch, TensorFlow, Forest Inference Library (FIL), etc. In this notebook, we will focus on the FIL backend using the xgboost model that was trained previously. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa0a7d94-c0a9-4b86-92bb-fafffa1ed021",
   "metadata": {},
   "source": [
    "## Preparing the Model\n",
    "### Load the Model\n",
    "Let's start with loading the previous XGBoost model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b6b2149c-3f83-46cd-a1eb-70db7067cf5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "model = xgb.Booster({'nthread': 4})  # init model\n",
    "model.load_model('xgboost_model.json')  # load model data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7896106-8410-40c3-8b6d-e0bdb94f69c3",
   "metadata": {},
   "source": [
    "### Creating the Folder Structure\n",
    "In the previous notebook, we just saved the XGBoost model to the working directory. Triton expects the model to be in a particular structure: Model name should be the top level directory, and the version number should be next. This allows for multiple models and versions of those models to be hosted simulatenously. Let's create that folder structure and save the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "28c42b97-2691-46d1-9b3a-7d19685d1d7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Create the model repository directory. The name of this directory is arbitrary.\n",
    "REPO_PATH = os.path.abspath('models')\n",
    "os.makedirs(REPO_PATH, exist_ok=True)\n",
    "\n",
    "# The name of the model directory determines the name of the model as reported by Triton\n",
    "model_dir = os.path.join(REPO_PATH, \"virus_prediction\")\n",
    "\n",
    "# We can store multiple versions of the model in the same directory. In our case, we have just one version, so we will add a single directory, named '1'.\n",
    "version_dir = os.path.join(model_dir, '1')\n",
    "os.makedirs(version_dir, exist_ok=True)\n",
    "\n",
    "# The default filename for XGBoost models saved in json format is 'xgboost.json'.\n",
    "# It is recommended that you use this filename to avoid having to specify a name in the configuration file.\n",
    "model_file = os.path.join(version_dir, 'xgboost.json')\n",
    "model.save_model(model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "900e5837-4807-4fd4-af21-0973a4942f1b",
   "metadata": {},
   "source": [
    "### Creating the Configuration File\n",
    "Triton also requires a configuration file that provides details about the model and deployment. For this notebook, we will assume default parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "61d898fb-a8d2-4d1c-a13f-2c4be6c18969",
   "metadata": {},
   "outputs": [],
   "source": [
    "config_text = f\"\"\"backend: \"fil\"\n",
    "max_batch_size: 32768\n",
    "input [                                 \n",
    " {{  \n",
    "    name: \"input__0\"\n",
    "    data_type: TYPE_FP32\n",
    "    dims: [ 4 ]                    \n",
    "  }} \n",
    "]\n",
    "output [\n",
    " {{\n",
    "    name: \"output__0\"\n",
    "    data_type: TYPE_FP32\n",
    "    dims: [ 1 ]\n",
    "  }}\n",
    "]\n",
    "instance_group [{{ kind: KIND_GPU }}]\n",
    "parameters [\n",
    "  {{\n",
    "    key: \"model_type\"\n",
    "    value: {{ string_value: \"xgboost_json\" }}\n",
    "  }},\n",
    "  {{\n",
    "    key: \"output_class\"\n",
    "    value: {{ string_value: \"false\" }}\n",
    "  }},\n",
    "  {{\n",
    "    key: \"storage_type\"\n",
    "    value: {{ string_value: \"AUTO\" }}\n",
    "  }}\n",
    "]\n",
    "\n",
    "dynamic_batching {{\n",
    "  max_queue_delay_microseconds: 100\n",
    "}}\"\"\"\n",
    "config_path = os.path.join(model_dir, 'config.pbtxt')\n",
    "with open(config_path, 'w') as file_:\n",
    "    file_.write(config_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f262d4d9-819a-4d28-bb26-7eb737c3387a",
   "metadata": {},
   "source": [
    "Now the model is ready to be loaded in Triton! The model repository should look like this.\n",
    "\n",
    "```\n",
    "model/\n",
    "`-- virus_prediction\n",
    "    |-- 1\n",
    "    |   `-- xgboost.json\n",
    "    `-- config.pbtxt\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5876d7a-3416-4ac9-827a-53796a1f7f93",
   "metadata": {},
   "source": [
    "## Loading the Model in Triton\n",
    "Next, we will need to start the Triton server. For this course, the server has already been started, but we will briefly go the steps required."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "709cf499-9faa-4746-9252-16f41fa548e6",
   "metadata": {},
   "source": [
    "### Starting Triton\n",
    "Triton is available as buildable source code or a pre-built docker image. For simplicity, we recommend that most users start with the docker images. This is how you would start the docker container in the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1283515c-c812-41c5-a100-e4dc5788781e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!docker run --gpus=1 --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 -v /full/path/to/docs/examples/model_repository:/models nvcr.io/nvidia/tritonserver:<xx.yy>-py3 tritonserver --model-repository=/models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ded6e36-0a1d-488e-8b10-f482ced8ea07",
   "metadata": {},
   "source": [
    "Wow! That's a lot of inputs. Let's break it down a little bit more.\n",
    "\n",
    "- **gpus=1** : Passes through the first GPU to the Triton Inference Server\n",
    "- **rm** : Removes the container after completing execution\n",
    "- **p 8000:8000** : Forwards the GTPCInferenceService Port\n",
    "- **p 8001:8001** : Forwards the HTTPService Port\n",
    "- **p 8002:8002** :  Forwards the Metrics Port\n",
    "- **v /full/path/to/docs/examples/model_repository:/models** : Mounts the path to the models folder on the host machine to the Triton Inference Server container\n",
    "- **nvcr.io/nvidia/tritonserver:<xx.yy>-py3** : Name of the Triton Inference Server image. The version number will change depending on the latest release\n",
    "- **tritonserver --model-repository=/models**: The command to run in the container. In this case, we start the Triton Inference Server and point to the models folder\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42967a82-fdfd-413e-8006-eb78a3ca805b",
   "metadata": {},
   "source": [
    "As we mentioned before, the server has already been started for this lab. Let's check our connection to the server! We are using \"triton\" as the hostname because the default docker network will resolve \"triton\" to the ip address of the Triton Inference Server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ac240cdc-f0dc-45ad-9838-58066a85696a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*   Trying 172.18.0.4:8000...\n",
      "* Connected to triton (172.18.0.4) port 8000 (#0)\n",
      "> GET /v2/health/ready HTTP/1.1\n",
      "> Host: triton:8000\n",
      "> User-Agent: curl/7.81.0\n",
      "> Accept: */*\n",
      "> \n",
      "* Mark bundle as not supporting multiuse\n",
      "< HTTP/1.1 200 OK\n",
      "< Content-Length: 0\n",
      "< Content-Type: text/plain\n",
      "< \n",
      "* Connection #0 to host triton left intact\n"
     ]
    }
   ],
   "source": [
    "!curl -v triton:8000/v2/health/ready"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d92b9b35-51c6-4eba-a9c0-9a92d19e0a0b",
   "metadata": {},
   "source": [
    "Now let's see if the model has been loaded!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d80b37c5-1242-489f-8463-387e657264a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{\"name\":\"virus_prediction\"}]"
     ]
    }
   ],
   "source": [
    "!curl -X POST http://triton:8000/v2/repository/index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06902d34-7763-4693-b919-f70ee4d60dfa",
   "metadata": {},
   "source": [
    "If all went well, we should be able to see the model \"virus prediction\" show up as ready."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2429b1bf-9372-42a5-86ea-be07cabf30ed",
   "metadata": {},
   "source": [
    "## Testing Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c2616f6-2de8-4e84-96d1-ad92b5f4b732",
   "metadata": {},
   "source": [
    "### Triton Client\n",
    "To test the deployment we will use the Triton Client library. Let's see how we can create an instance of the client. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ec652ce4-a2d6-4b3a-8e2f-faac51da5487",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import tritonclient.grpc as triton_grpc\n",
    "from tritonclient import utils as triton_utils\n",
    "HOST = \"triton\"\n",
    "PORT = 8001\n",
    "TIMEOUT = 60"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "22af3af5-1d15-406f-91cb-2208c938d7ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = triton_grpc.InferenceServerClient(url=f'{HOST}:{PORT}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84c3a594-e377-47c9-b3e7-0da321138bf5",
   "metadata": {},
   "source": [
    "Now let's make sure the Triton server is ready by sending a sample inference request. First let's load the training data. We will only load 32768 rows, as that's the max batch size we specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b57396e8-bc06-43dd-bf4c-2e34b24f5962",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         infected\n",
      "4506149       0.0\n",
      "1456192       0.0\n",
      "2065740       0.0\n",
      "1072379       0.0\n",
      "424532        0.0\n",
      "...           ...\n",
      "1204134       0.0\n",
      "3330908       0.0\n",
      "2066864       0.0\n",
      "1643995       0.0\n",
      "4661849       0.0\n",
      "\n",
      "[32768 rows x 1 columns]\n"
     ]
    }
   ],
   "source": [
    "import cudf \n",
    "import numpy as np\n",
    "df = cudf.read_csv('./data/clean_uk_pop_full.csv', usecols=['age', 'sex', 'northing', 'easting', 'infected'], nrows=5000000)\n",
    "df = df.sample(32768)\n",
    "input_data = df.drop('infected', axis=1)\n",
    "target = df[['infected']]\n",
    "print(target)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d24f3792-a541-4b9f-acdc-e72c64897804",
   "metadata": {},
   "source": [
    "Now we convert it to a numpy array and force the type to be float32 (the same type we specified in the config file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1ab43811-f273-48cf-b473-ea3d4786cab4",
   "metadata": {},
   "outputs": [],
   "source": [
    "converted_df = input_data.to_numpy(dtype='float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed49f912-5e35-4fe8-b6fa-b91f025cc07e",
   "metadata": {},
   "source": [
    "Since we limited our batch size to 32768, let's splice the array and attempt the inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "77e90afc-9ea6-45ad-83bb-2ac7f154b3f8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.61 ms, sys: 1.34 ms, total: 3.96 ms\n",
      "Wall time: 6.44 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "batched_data = converted_df[:32768]\n",
    "# Prepare the input tensor\n",
    "input_tensor = triton_grpc.InferInput(\"input__0\", batched_data.shape, 'FP32')\n",
    "input_tensor.set_data_from_numpy(batched_data)\n",
    "\n",
    "# Prepare the output\n",
    "output = triton_grpc.InferRequestedOutput(\"output__0\")\n",
    "\n",
    "# Send inference request\n",
    "response = client.infer(\"virus_prediction\", [input_tensor], outputs=[output])\n",
    "\n",
    "# Get the output data\n",
    "output_data = response.as_numpy(\"output__0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "923c2923-acc1-48cb-bc41-3adafbf7eca2",
   "metadata": {},
   "source": [
    "Let's make sure the results we got were the same as with the local model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f809c2db-11f9-48de-9913-3e032e14a553",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import auc\n",
    "\n",
    "xgb_data = xgb.DMatrix(input_data)\n",
    "y_test = model.predict(xgb_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "52ada8b8-0f47-435b-90df-57c5056bfc26",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38932 characters]",
      "text/plain": [
       "<Figure size 500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check that we got the same accuracy as previously\n",
    "#target = target.to_numpy()\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "false_pos_rate, true_pos_rate, thresholds = roc_curve(target.to_numpy(), y_test)\n",
    "auc_result = auc(false_pos_rate, true_pos_rate)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(5, 5))\n",
    "ax.plot(false_pos_rate, true_pos_rate, lw=3,\n",
    "        label='AUC = {:.2f}'.format(auc_result))\n",
    "ax.plot([0, 1], [0, 1], 'k--', lw=2)\n",
    "ax.set(\n",
    "    xlim=(0, 1),\n",
    "    ylim=(0, 1),\n",
    "    title=\"ROC Curve\",\n",
    "    xlabel=\"False Positive Rate\",\n",
    "    ylabel=\"True Positive Rate\",\n",
    ")\n",
    "ax.legend(loc='lower right');\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "30562baa-f7c9-40d8-ab5d-40b3c642b666",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38948 characters]",
      "text/plain": [
       "<Figure size 500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check that we got the same accuracy as previously\n",
    "#target = target.to_numpy()\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "false_pos_rate, true_pos_rate, thresholds = roc_curve(target.to_numpy(), output_data)\n",
    "auc_result = auc(false_pos_rate, true_pos_rate)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(5, 5))\n",
    "ax.plot(false_pos_rate, true_pos_rate, lw=3,\n",
    "        label='AUC = {:.2f}'.format(auc_result))\n",
    "ax.plot([0, 1], [0, 1], 'k--', lw=2)\n",
    "ax.set(\n",
    "    xlim=(0, 1),\n",
    "    ylim=(0, 1),\n",
    "    title=\"ROC Curve\",\n",
    "    xlabel=\"False Positive Rate\",\n",
    "    ylabel=\"True Positive Rate\",\n",
    ")\n",
    "ax.legend(loc='lower right');\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b00984cb-6e04-4951-a864-a67cc4a22b6f",
   "metadata": {},
   "source": [
    "As we can see, our AUC score is the same between both inference options!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a459c429-b902-4056-ad2e-05d53f82537c",
   "metadata": {},
   "source": [
    "## Analyzing Performance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e724bd09-057b-408a-896a-b48a7a4362d1",
   "metadata": {},
   "source": [
    "Earlier, we tested a *relatively* small inference request. What if we want to see the max throughput of the model? Luckily, Triton offers a performance analysis tool that generates synthetic data to collect latency and throughput numbers. Let's try it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "15129aef-3d3f-4c65-b2ff-1d0b2599ffd7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Measurement Settings ***\n",
      "  Batch size: 1\n",
      "  Service Kind: TRITON\n",
      "  Using \"time_windows\" mode for stabilization\n",
      "  Stabilizing using average latency and throughput\n",
      "  Measurement window: 5000 msec\n",
      "  Using synchronous calls for inference\n",
      "\n",
      "Request concurrency: 1\n",
      "  Client: \n",
      "    Request count: 39308\n",
      "    Throughput: 2173.85 infer/sec\n",
      "    Avg latency: 456 usec (standard deviation 107 usec)\n",
      "    p50 latency: 428 usec\n",
      "    p90 latency: 608 usec\n",
      "    p95 latency: 664 usec\n",
      "    p99 latency: 689 usec\n",
      "    Avg HTTP time: 452 usec (send/recv 41 usec + response wait 411 usec)\n",
      "  Server: \n",
      "    Inference count: 39309\n",
      "    Execution count: 39309\n",
      "    Successful request count: 39309\n",
      "    Avg request latency: 319 usec (overhead 2 usec + queue 219 usec + compute input 27 usec + compute infer 48 usec + compute output 22 usec)\n",
      "\n",
      "Inferences/Second vs. Client Average Batch Latency\n",
      "Concurrency: 1, throughput: 2173.85 infer/sec, latency 456 usec\n"
     ]
    }
   ],
   "source": [
    "!perf_analyzer -m virus_prediction -u \"triton:8000\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12fe9a4c-e5db-4474-bc9f-76e0f3485506",
   "metadata": {},
   "source": [
    "That's a lot of information to take in. Let's break it down.\n",
    "\n",
    "**Measurement window**: Timeframe that measurements are taken in \n",
    "\n",
    "**Batch Size**: Number of inputs in each request\n",
    "\n",
    "\n",
    "**Concurrency**: Number of simulatenous connections\n",
    "**Latency**: Time taken to recieve results\n",
    "**p50/90/95/99**: Different percentiles for latency\n",
    "\n",
    "Based on these results, we can see that our throughput is roughly ~2300 inferences per second with a single concurrent connection, and the average latency for each requst is 434 usec."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27e8c5c0-7f3e-41e3-881d-38f9d0aa5d3e",
   "metadata": {},
   "source": [
    "# Customizing Perf Analyzer\n",
    "The Performance Analyzer tool for Triton has many knobs that can be turned to analyze results. Let's enable GPU metric collection, and increase the batch size and concurrency range values!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6353c5bb-ebd5-492e-ad62-90e8929849d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Measurement Settings ***\n",
      "  Batch size: 8\n",
      "  Service Kind: TRITON\n",
      "  Using \"time_windows\" mode for stabilization\n",
      "  Stabilizing using average latency and throughput\n",
      "  Measurement window: 5000 msec\n",
      "  Latency limit: 0 msec\n",
      "  Concurrency limit: 8 concurrent requests\n",
      "  Using synchronous calls for inference\n",
      "\n",
      "Request concurrency: 2\n",
      "  Client: \n",
      "    Request count: 82618\n",
      "    Throughput: 36421.1 infer/sec\n",
      "    Avg latency: 436 usec (standard deviation 94 usec)\n",
      "    p50 latency: 429 usec\n",
      "    p90 latency: 582 usec\n",
      "    p95 latency: 628 usec\n",
      "    p99 latency: 707 usec\n",
      "    Avg HTTP time: 432 usec (send/recv 37 usec + response wait 395 usec)\n",
      "  Server: \n",
      "    Inference count: 660960\n",
      "    Execution count: 41392\n",
      "    Successful request count: 82620\n",
      "    Avg request latency: 294 usec (overhead 2 usec + queue 197 usec + compute input 25 usec + compute infer 44 usec + compute output 25 usec)\n",
      "\n",
      "  Server Prometheus Metrics: \n",
      "    Avg GPU Utilization:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 0.842105%\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 0.842105%\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 0.842105%\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 0.842105%\n",
      "    Avg GPU Power Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 25.5052 watts\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 25.6625 watts\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 25.5276 watts\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 25.1561 watts\n",
      "    Max GPU Memory Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 832569344 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 230686720 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 230686720 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 230686720 bytes\n",
      "    Total GPU Memory:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 16106127360 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 16106127360 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 16106127360 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 16106127360 bytes\n",
      "Request concurrency: 4\n",
      "  Client: \n",
      "    Request count: 170691\n",
      "    Throughput: 74290.5 infer/sec\n",
      "    Avg latency: 427 usec (standard deviation 100 usec)\n",
      "    p50 latency: 433 usec\n",
      "    p90 latency: 572 usec\n",
      "    p95 latency: 648 usec\n",
      "    p99 latency: 716 usec\n",
      "    Avg HTTP time: 424 usec (send/recv 34 usec + response wait 390 usec)\n",
      "  Server: \n",
      "    Inference count: 1365536\n",
      "    Execution count: 48705\n",
      "    Successful request count: 170691\n",
      "    Avg request latency: 272 usec (overhead 2 usec + queue 167 usec + compute input 24 usec + compute infer 45 usec + compute output 33 usec)\n",
      "\n",
      "  Server Prometheus Metrics: \n",
      "    Avg GPU Utilization:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 1.10526%\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 1.10526%\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 1.10526%\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 1.10526%\n",
      "    Avg GPU Power Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 25.517 watts\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 25.6462 watts\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 25.522 watts\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 25.1323 watts\n",
      "    Max GPU Memory Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 832569344 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 230686720 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 230686720 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 230686720 bytes\n",
      "    Total GPU Memory:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 16106127360 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 16106127360 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 16106127360 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 16106127360 bytes\n",
      "Request concurrency: 6\n",
      "  Client: \n",
      "    Request count: 329197\n",
      "    Throughput: 137543 infer/sec\n",
      "    Avg latency: 346 usec (standard deviation 82 usec)\n",
      "    p50 latency: 339 usec\n",
      "    p90 latency: 438 usec\n",
      "    p95 latency: 458 usec\n",
      "    p99 latency: 621 usec\n",
      "    Avg HTTP time: 343 usec (send/recv 31 usec + response wait 312 usec)\n",
      "  Server: \n",
      "    Inference count: 2633624\n",
      "    Execution count: 86121\n",
      "    Successful request count: 329203\n",
      "    Avg request latency: 212 usec (overhead 1 usec + queue 129 usec + compute input 16 usec + compute infer 36 usec + compute output 29 usec)\n",
      "\n",
      "  Server Prometheus Metrics: \n",
      "    Avg GPU Utilization:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 2.05%\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 2.05%\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 2.05%\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 2.05%\n",
      "    Avg GPU Power Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 25.5291 watts\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 25.6947 watts\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 25.5563 watts\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 25.1661 watts\n",
      "    Max GPU Memory Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 832569344 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 230686720 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 230686720 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 230686720 bytes\n",
      "    Total GPU Memory:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 16106127360 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 16106127360 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 16106127360 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 16106127360 bytes\n",
      "Request concurrency: 8\n",
      "  Client: \n",
      "    Request count: 508448\n",
      "    Throughput: 184400 infer/sec\n",
      "    Avg latency: 344 usec (standard deviation 82 usec)\n",
      "    p50 latency: 348 usec\n",
      "    p90 latency: 386 usec\n",
      "    p95 latency: 400 usec\n",
      "    p99 latency: 445 usec\n",
      "    Avg HTTP time: 341 usec (send/recv 31 usec + response wait 310 usec)\n",
      "  Server: \n",
      "    Inference count: 4067600\n",
      "    Execution count: 122220\n",
      "    Successful request count: 508450\n",
      "    Avg request latency: 192 usec (overhead 1 usec + queue 116 usec + compute input 14 usec + compute infer 35 usec + compute output 26 usec)\n",
      "\n",
      "  Server Prometheus Metrics: \n",
      "    Avg GPU Utilization:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 2.83333%\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 2.83333%\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 2.83333%\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 2.83333%\n",
      "    Avg GPU Power Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 25.5151 watts\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 25.7038 watts\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 25.6153 watts\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 25.1973 watts\n",
      "    Max GPU Memory Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 832569344 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 230686720 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 230686720 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 230686720 bytes\n",
      "    Total GPU Memory:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 16106127360 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 16106127360 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 16106127360 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 16106127360 bytes\n",
      "Inferences/Second vs. Client Average Batch Latency\n",
      "Concurrency: 2, throughput: 36421.1 infer/sec, latency 436 usec\n",
      "Concurrency: 4, throughput: 74290.5 infer/sec, latency 427 usec\n",
      "Concurrency: 6, throughput: 137543 infer/sec, latency 346 usec\n",
      "Concurrency: 8, throughput: 184400 infer/sec, latency 344 usec\n"
     ]
    }
   ],
   "source": [
    "!perf_analyzer --collect-metrics -m virus_prediction -u \"triton:8000\" -b 8 --concurrency-range 2:8:2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d0340a9-da5a-400f-9127-b3f4fe1facad",
   "metadata": {},
   "source": [
    "The results show that our model configuration gives a throughput of about ~156171 inferences per second. Notice how there are significant throughput gains as we increase the number of concurrent connections. With low concurrency values, Triton is idle during the time when the response is returned to the client and the next request is received at the server. Throughput increases when we increase concurrency values because Triton overlaps the processing of one request with the communication of the other. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea13ffda-b03d-435e-a01e-5f8ceaa076de",
   "metadata": {},
   "source": [
    "# Exercise \n",
    "Please take this time to experiment with the perf analyzer tool. A full list of parameters can be seen with the --help argument. Some parameters that we recommend trying out are listed below: \n",
    "* -b <value> : batch size\n",
    "* --concurrency-range <start:end:step> : range of concurrency values to test\n",
    "* --collect-metrics : enable collection of GPU metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8bb02a7-92c1-4373-afb6-c4a507ddf20d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Measurement Settings ***\n",
      "  Batch size: 8\n",
      "  Service Kind: TRITON\n",
      "  Using \"time_windows\" mode for stabilization\n",
      "  Stabilizing using average latency and throughput\n",
      "  Measurement window: 5000 msec\n",
      "  Latency limit: 0 msec\n",
      "  Concurrency limit: 16 concurrent requests\n",
      "  Using synchronous calls for inference\n",
      "\n",
      "Request concurrency: 1\n",
      "  Client: \n",
      "    Request count: 40512\n",
      "    Throughput: 17945.4 infer/sec\n",
      "    Avg latency: 442 usec (standard deviation 103 usec)\n",
      "    p50 latency: 425 usec\n",
      "    p90 latency: 596 usec\n",
      "    p95 latency: 648 usec\n",
      "    p99 latency: 705 usec\n",
      "    Avg HTTP time: 438 usec (send/recv 40 usec + response wait 398 usec)\n",
      "  Server: \n",
      "    Inference count: 324112\n",
      "    Execution count: 40514\n",
      "    Successful request count: 40514\n",
      "    Avg request latency: 309 usec (overhead 0 usec + queue 215 usec + compute input 26 usec + compute infer 46 usec + compute output 20 usec)\n",
      "\n",
      "  Server Prometheus Metrics: \n",
      "    Avg GPU Utilization:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 0.947368%\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 0.842105%\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 0.842105%\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 0.947368%\n",
      "    Avg GPU Power Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 25.4854 watts\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 25.6408 watts\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 25.5326 watts\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 25.1241 watts\n",
      "    Max GPU Memory Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 832569344 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 230686720 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 230686720 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 230686720 bytes\n",
      "    Total GPU Memory:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 16106127360 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 16106127360 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 16106127360 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 16106127360 bytes\n",
      "Request concurrency: 3\n",
      "  Client: \n",
      "    Request count: 123389\n",
      "    Throughput: 54124.1 infer/sec\n",
      "    Avg latency: 440 usec (standard deviation 103 usec)\n",
      "    p50 latency: 429 usec\n",
      "    p90 latency: 606 usec\n",
      "    p95 latency: 650 usec\n",
      "    p99 latency: 731 usec\n",
      "    Avg HTTP time: 436 usec (send/recv 38 usec + response wait 398 usec)\n",
      "  Server: \n",
      "    Inference count: 987144\n",
      "    Execution count: 41396\n",
      "    Successful request count: 123393\n",
      "    Avg request latency: 286 usec (overhead 1 usec + queue 178 usec + compute input 27 usec + compute infer 47 usec + compute output 32 usec)\n",
      "\n",
      "  Server Prometheus Metrics: \n",
      "    Avg GPU Utilization:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 1%\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 1%\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 1%\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 1%\n",
      "    Avg GPU Power Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 25.5275 watts\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 25.6826 watts\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 25.5457 watts\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 25.1294 watts\n",
      "    Max GPU Memory Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 832569344 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 230686720 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 230686720 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 230686720 bytes\n",
      "    Total GPU Memory:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 16106127360 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 16106127360 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 16106127360 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 16106127360 bytes\n",
      "Request concurrency: 5\n",
      "  Client: \n",
      "    Request count: 248663\n",
      "    Throughput: 106417 infer/sec\n",
      "    Avg latency: 373 usec (standard deviation 88 usec)\n",
      "    p50 latency: 356 usec\n",
      "    p90 latency: 454 usec\n",
      "    p95 latency: 520 usec\n",
      "    p99 latency: 698 usec\n",
      "    Avg HTTP time: 369 usec (send/recv 32 usec + response wait 337 usec)\n",
      "  Server: \n",
      "    Inference count: 1989328\n",
      "    Execution count: 77638\n",
      "    Successful request count: 248666\n",
      "    Avg request latency: 221 usec (overhead 1 usec + queue 136 usec + compute input 18 usec + compute infer 38 usec + compute output 27 usec)\n",
      "\n",
      "  Server Prometheus Metrics: \n",
      "    Avg GPU Utilization:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 1.84211%\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 1.73684%\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 1.84211%\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 1.84211%\n",
      "    Avg GPU Power Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 25.5205 watts\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 25.7027 watts\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 25.5786 watts\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 25.1956 watts\n",
      "    Max GPU Memory Usage:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 832569344 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 230686720 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 230686720 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 230686720 bytes\n",
      "    Total GPU Memory:\n",
      "      GPU-3b4a3ef9-2229-9b49-0526-7070d90a8f2d : 16106127360 bytes\n",
      "      GPU-99aab0d2-4243-f5eb-c3c7-8fe633a47ee2 : 16106127360 bytes\n",
      "      GPU-cc28cec7-3383-f02d-7890-3b72a424a18d : 16106127360 bytes\n",
      "      GPU-f3f9894c-c591-94f2-4d37-8292251c337e : 16106127360 bytes\n",
      "Request concurrency: 7\n"
     ]
    }
   ],
   "source": [
    "!perf_analyzer -u \"triton:8000\" -m virus_prediction -b 8 --concurrency-range 1:16:2 --collect-metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6ccd1e7-989b-49b5-8ece-0583e9890c66",
   "metadata": {},
   "source": [
    "## Model Analyzer\n",
    "Although it is out of scope for this course, we would like to introduce the Model Analyzer tool that is available as part of Triton. This tool searches through different parameter configurations to find optimal parameters that maximize inference throughput. With some minor processing, results can be viewed in a PDF format as well. An example of the output is shown below. More information about the Model Analyzer can be found [here](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_analyzer.html). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0c0c90f-d8e6-4ce5-a9c9-197a46e1d8a2",
   "metadata": {},
   "source": [
    "![image](images/model_analyzer.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91558f00-41d2-493d-8517-9468efcae64e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import IPython\n",
    "app = IPython.Application.instance()\n",
    "app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7e473f9-9ccf-43c6-aa86-eda25150a5ca",
   "metadata": {},
   "source": [
    "**Well Done!** Let's move to the [next notebook](3-08_k-means_dask.ipynb). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832c59bf-517c-4703-9161-1f9997214e19",
   "metadata": {},
   "source": [
    "<img src=\"./images/DLI_Header.png\" width=400/>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
