Deploying a MedNIST Classifier App with MONAI Deploy App SDK (Prebuilt Model)#
This tutorial demos the process of packaging up a trained model using MONAI Deploy App SDK into an deployable inference application which can be run as a local program, as well as an MONAI Application Package (MAP) for containerized workflow execution.
Clone the github project (the latest version of the main branch only)#
!rm -rf source \
&& git clone --branch main --depth 1 https://github.com/Project-MONAI/monai-deploy-app-sdk.git source \
&& rm -rf source/.git
!ls source/examples/apps/mednist_classifier_monaideploy/
Install monai-deploy-app-sdk package#
!pip install monai-deploy-app-sdk
Install necessary packages for the app#
!pip install monai Pillow # for MONAI transforms and Pillow
!python -c "import pydicom" || pip install -q "pydicom>=1.4.2"
!python -c "import highdicom" || pip install -q "highdicom>=0.18.2" # for the use of DICOM Writer operators
Download/Extract mednist_classifier_data.zip from Google Drive#
Note: Data files are now access controlled. Please first request permission to access the shared folder on Google Drive, then download the zip file, mednist_classifier_data.zip found in the medmist_classifier_app folder, to the same folder as this notebook example.
import os
data_dir = os.path.join(os.path.curdir, "mednist_classifier_data.zip")
if not os.path.exists(data_dir):
# Download mednist_classifier_data.zip
!pip install gdown
!gdown "https://drive.google.com/uc?id=1IoEJZFFixcNtPPKeKZfD_xSJSFQCbawl" # Redundant if already manually downloaded the file to avoid permission issue.
# Unzip the downloaded mednist_classifier_data.zip from the web browser or using gdown, to the notebook/turotials folder, and set up folders
input_folder = "input"
output_folder = "output"
models_folder = "models"
!rm -rf {input_folder}
!unzip -o "mednist_classifier_data.zip"
# Need to copy the model file to its own clean subfolder for packaging, to workaround an issue in the Packager
models_folder = "models"
!rm -rf {models_folder} && mkdir -p {models_folder}/model && cp classifier.zip {models_folder}/model && ls {models_folder}/model
Set up environment variables#
The application uses well-known environment variables for the input/output data path, working dir, as well as AI model file path if applicable. Defaults are used if these environment variable are absent.
Set the environment variables corresponding to the extracted data path.
%env HOLOSCAN_INPUT_PATH {input_folder}
%env HOLOSCAN_OUTPUT_PATH {output_folder}
%env HOLOSCAN_MODEL_PATH {models_folder}
Package app (creating MAP container image)#
Now we can use the CLI package command to build the MONAI Application Package (MAP) container image based on a supported base image
Use -l DEBUG option to see progress.
Note
This assumes that NVIDIA Container Toolkit or nvidia docker is installed on the local machine.
tag_prefix = "mednist_app"
!monai-deploy package "source/examples/apps/mednist_classifier_monaideploy/mednist_classifier_monaideploy.py" -m {models_folder} -c "source/examples/apps/mednist_classifier_monaideploy/app.yaml" -t {tag_prefix}:1.0 --platform x86_64 -l DEBUG
We can see that the MAP Docker image is created
!docker image ls | grep {tag_prefix}
We can choose to display and inspect the MAP manifests by running the container with the show command.
Furthermore, we can also extract the manifests and other contents in the MAP by using the extract command while mapping specific folder to the host’s (we know that our MAP is compliant and supports these commands).
Note
The host folder for storing the extracted content must first be created by the user, and if it has been created by Docker on running the container, the folder needs to be deleted and re-created.
!echo "Display manifests and extract MAP contents to the host folder, ./export"
!docker run --rm {tag_prefix}-x64-workstation-dgpu-linux-amd64:1.0 show
!rm -rf `pwd`/export && mkdir -p `pwd`/export
!docker run --rm -v `pwd`/export/:/var/run/holoscan/export/ {tag_prefix}-x64-workstation-dgpu-linux-amd64:1.0 extract
!ls `pwd`/export
Executing packaged app locally#
The packaged app can be run locally through MONAI Application Runner.
# Clear the output folder and run the MAP. The input is expected to be a folder.
!rm -rf {ouput_folder}
!monai-deploy run -i $HOLOSCAN_INPUT_PATH -o $HOLOSCAN_OUTPUT_PATH mednist_app-x64-workstation-dgpu-linux-amd64:1.0
!cat {output_folder}/output.json
Implementing and Packaging Application with MONAI Deploy App SDK#
In the following sections we will discuss the details of buildng the application that was packaged and run above.
Based on the Torchscript model(classifier.zip), we will implement an application that process an input Jpeg image and write the prediction(classification) result as JSON file(output.json).
In our inference application, we will define two operators:
LoadPILOperator- Load a JPEG image from the input path and pass the loaded image object to the next operator.Input: a file path (
Path)Output: an image object in memory (
Image)
MedNISTClassifierOperator- Pre-transform the given image by using MONAI’sComposeclass, feed to the Torchscript model (classifier.zip), and write the prediction into JSON file(output.json)Pre-transforms consist of three transforms –
EnsureChannelFirst,ScaleIntensity, andEnsureType.Input: an image object in memory (
Image)Output: a folder path that the prediction result(
output.json) would be written (Path)
The workflow of the application would look like this.
Setup imports#
Let’s import necessary classes/decorators and define MEDNIST_CLASSES.
import logging
import os
from pathlib import Path
from typing import Optional
import torch
from monai.deploy.conditions import CountCondition
from monai.deploy.core import AppContext, Application, ConditionType, Fragment, Image, Operator, OperatorSpec
from monai.deploy.operators.dicom_text_sr_writer_operator import DICOMTextSRWriterOperator, EquipmentInfo, ModelInfo
from monai.transforms import EnsureChannelFirst, Compose, EnsureType, ScaleIntensity
MEDNIST_CLASSES = ["AbdomenCT", "BreastMRI", "CXR", "ChestCT", "Hand", "HeadCT"]
Creating Operator classes#
LoadPILOperator#
class LoadPILOperator(Operator):
"""Load image from the given input (DataPath) and set numpy array to the output (Image)."""
DEFAULT_INPUT_FOLDER = Path.cwd() / "input"
DEFAULT_OUTPUT_NAME = "image"
# For now, need to have the input folder as an instance attribute, set on init.
# If dynamically changing the input folder, per compute, then use a (optional) input port to convey the
# value of the input folder, which is then emitted by a upstream operator.
def __init__(
self,
fragment: Fragment,
*args,
input_folder: Path = DEFAULT_INPUT_FOLDER,
output_name: str = DEFAULT_OUTPUT_NAME,
**kwargs,
):
"""Creates an loader object with the input folder and the output port name overrides as needed.
Args:
fragment (Fragment): An instance of the Application class which is derived from Fragment.
input_folder (Path): Folder from which to load input file(s).
Defaults to `input` in the current working directory.
output_name (str): Name of the output port, which is an image object. Defaults to `image`.
"""
self._logger = logging.getLogger("{}.{}".format(__name__, type(self).__name__))
self.input_path = input_folder
self.index = 0
self.output_name_image = (
output_name.strip() if output_name and len(output_name.strip()) > 0 else LoadPILOperator.DEFAULT_OUTPUT_NAME
)
super().__init__(fragment, *args, **kwargs)
def setup(self, spec: OperatorSpec):
"""Set up the named input and output port(s)"""
spec.output(self.output_name_image)
def compute(self, op_input, op_output, context):
import numpy as np
from PIL import Image as PILImage
# Input path is stored in the object attribute, but could change to use a named port if need be.
input_path = self.input_path
if input_path.is_dir():
input_path = next(self.input_path.glob("*.*")) # take the first file
image = PILImage.open(input_path)
image = image.convert("L") # convert to greyscale image
image_arr = np.asarray(image)
output_image = Image(image_arr) # create Image domain object with a numpy array
op_output.emit(output_image, self.output_name_image) # cannot omit the name even if single output.
MedNISTClassifierOperator#
class MedNISTClassifierOperator(Operator):
"""Classifies the given image and returns the class name.
Named inputs:
image: Image object for which to generate the classification.
output_folder: Optional, the path to save the results JSON file, overridingthe the one set on __init__
Named output:
result_text: The classification results in text.
"""
DEFAULT_OUTPUT_FOLDER = Path.cwd() / "classification_results"
# For testing the app directly, the model should be at the following path.
MODEL_LOCAL_PATH = Path(os.environ.get("HOLOSCAN_MODEL_PATH", Path.cwd() / "model/model.ts"))
def __init__(
self,
fragment: Fragment,
*args,
app_context: AppContext,
model_name: Optional[str] = "",
model_path: Path = MODEL_LOCAL_PATH,
output_folder: Path = DEFAULT_OUTPUT_FOLDER,
**kwargs,
):
"""Creates an instance with the reference back to the containing application/fragment.
fragment (Fragment): An instance of the Application class which is derived from Fragment.
model_name (str, optional): Name of the model. Default to "" for single model app.
model_path (Path): Path to the model file. Defaults to model/models.ts of current working dir.
output_folder (Path, optional): output folder for saving the classification results JSON file.
"""
# the names used for the model inference input and output
self._input_dataset_key = "image"
self._pred_dataset_key = "pred"
# The names used for the operator input and output
self.input_name_image = "image"
self.output_name_result = "result_text"
# The name of the optional input port for passing data to override the output folder path.
self.input_name_output_folder = "output_folder"
# The output folder set on the object can be overridden at each compute by data in the optional named input
self.output_folder = output_folder
# Need the name when there are multiple models loaded
self._model_name = model_name.strip() if isinstance(model_name, str) else ""
# Need the path to load the models when they are not loaded in the execution context
self.model_path = model_path
self.app_context = app_context
self.model = self._get_model(self.app_context, self.model_path, self._model_name)
# This needs to be at the end of the constructor.
super().__init__(fragment, *args, **kwargs)
def _get_model(self, app_context: AppContext, model_path: Path, model_name: str):
"""Load the model with the given name from context or model path
Args:
app_context (AppContext): The application context object holding the model(s)
model_path (Path): The path to the model file, as a backup to load model directly
model_name (str): The name of the model, when multiples are loaded in the context
"""
if app_context.models:
# `app_context.models.get(model_name)` returns a model instance if exists.
# If model_name is not specified and only one model exists, it returns that model.
model = app_context.models.get(model_name)
else:
model = torch.jit.load(
MedNISTClassifierOperator.MODEL_LOCAL_PATH,
map_location=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)
return model
def setup(self, spec: OperatorSpec):
"""Set up the operator named input and named output, both are in-memory objects."""
spec.input(self.input_name_image)
spec.input(self.input_name_output_folder).condition(ConditionType.NONE) # Optional for overriding.
spec.output(self.output_name_result).condition(ConditionType.NONE) # Not forcing a downstream receiver.
@property
def transform(self):
return Compose([EnsureChannelFirst(channel_dim="no_channel"), ScaleIntensity(), EnsureType()])
def compute(self, op_input, op_output, context):
import json
import torch
img = op_input.receive(self.input_name_image).asnumpy() # (64, 64), uint8. Input validation can be added.
image_tensor = self.transform(img) # (1, 64, 64), torch.float64
image_tensor = image_tensor[None].float() # (1, 1, 64, 64), torch.float32
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
image_tensor = image_tensor.to(device)
with torch.no_grad():
outputs = self.model(image_tensor)
_, output_classes = outputs.max(dim=1)
result = MEDNIST_CLASSES[output_classes[0]] # get the class name
print(result)
op_output.emit(result, self.output_name_result)
# Get output folder, with value in optional input port overriding the obj attribute
output_folder_on_compute = op_input.receive(self.input_name_output_folder) or self.output_folder
Path.mkdir(output_folder_on_compute, parents=True, exist_ok=True) # Let exception bubble up if raised.
output_path = output_folder_on_compute / "output.json"
with open(output_path, "w") as fp:
json.dump(result, fp)
Creating Application class#
Our application class would look like below.
It defines App class inheriting Application class.
LoadPILOperator is connected to MedNISTClassifierOperator by using self.add_flow() in compose() method of App.
class App(Application):
"""Application class for the MedNIST classifier."""
def compose(self):
app_context = Application.init_app_context({}) # Do not pass argv in Jupyter Notebook
app_input_path = Path(app_context.input_path)
app_output_path = Path(app_context.output_path)
model_path = Path(app_context.model_path)
load_pil_op = LoadPILOperator(self, CountCondition(self, 1), input_folder=app_input_path, name="pil_loader_op")
classifier_op = MedNISTClassifierOperator(
self, app_context=app_context, output_folder=app_output_path, model_path=model_path, name="classifier_op"
)
my_model_info = ModelInfo("MONAI WG Trainer", "MEDNIST Classifier", "0.1", "xyz")
my_equipment = EquipmentInfo(manufacturer="MOANI Deploy App SDK", manufacturer_model="DICOM SR Writer")
my_special_tags = {"SeriesDescription": "Not for clinical use. The result is for research use only."}
dicom_sr_operator = DICOMTextSRWriterOperator(
self,
copy_tags=False,
model_info=my_model_info,
equipment_info=my_equipment,
custom_tags=my_special_tags,
output_folder=app_output_path,
)
self.add_flow(load_pil_op, classifier_op, {("image", "image")})
self.add_flow(classifier_op, dicom_sr_operator, {("result_text", "text")})
Executing app locally#
We can execute the app in the Jupyter notebook. Before doing so, we also need to clean the output folder which was created by running the packaged containerizd app in the previous cell.
!rm -rf $HOLOSCAN_OUTPUT_PATH
app = App().run()
!cat $HOLOSCAN_OUTPUT_PATH/output.json
Once the application is verified inside Jupyter notebook, we can write the whole application as a file(mednist_classifier_monaideploy.py) by concatenating code above, then add the following lines:
if __name__ == "__main__":
App().run()
The above lines are needed to execute the application code by using python interpreter.
# Create an application folder
!mkdir -p mednist_app && rm -rf mednist_app/*
%%writefile mednist_app/mednist_classifier_monaideploy.py
# Copyright 2021-2023 MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
import os
from pathlib import Path
from typing import Optional
import torch
from monai.deploy.conditions import CountCondition
from monai.deploy.core import AppContext, Application, ConditionType, Fragment, Image, Operator, OperatorSpec
from monai.deploy.operators.dicom_text_sr_writer_operator import DICOMTextSRWriterOperator, EquipmentInfo, ModelInfo
from monai.transforms import EnsureChannelFirst, Compose, EnsureType, ScaleIntensity
MEDNIST_CLASSES = ["AbdomenCT", "BreastMRI", "CXR", "ChestCT", "Hand", "HeadCT"]
# @md.env(pip_packages=["pillow"])
class LoadPILOperator(Operator):
"""Load image from the given input (DataPath) and set numpy array to the output (Image)."""
DEFAULT_INPUT_FOLDER = Path.cwd() / "input"
DEFAULT_OUTPUT_NAME = "image"
# For now, need to have the input folder as an instance attribute, set on init.
# If dynamically changing the input folder, per compute, then use a (optional) input port to convey the
# value of the input folder, which is then emitted by a upstream operator.
def __init__(
self,
fragment: Fragment,
*args,
input_folder: Path = DEFAULT_INPUT_FOLDER,
output_name: str = DEFAULT_OUTPUT_NAME,
**kwargs,
):
"""Creates an loader object with the input folder and the output port name overrides as needed.
Args:
fragment (Fragment): An instance of the Application class which is derived from Fragment.
input_folder (Path): Folder from which to load input file(s).
Defaults to `input` in the current working directory.
output_name (str): Name of the output port, which is an image object. Defaults to `image`.
"""
self._logger = logging.getLogger("{}.{}".format(__name__, type(self).__name__))
self.input_path = input_folder
self.index = 0
self.output_name_image = (
output_name.strip() if output_name and len(output_name.strip()) > 0 else LoadPILOperator.DEFAULT_OUTPUT_NAME
)
super().__init__(fragment, *args, **kwargs)
def setup(self, spec: OperatorSpec):
"""Set up the named input and output port(s)"""
spec.output(self.output_name_image)
def compute(self, op_input, op_output, context):
import numpy as np
from PIL import Image as PILImage
# Input path is stored in the object attribute, but could change to use a named port if need be.
input_path = self.input_path
if input_path.is_dir():
input_path = next(self.input_path.glob("*.*")) # take the first file
image = PILImage.open(input_path)
image = image.convert("L") # convert to greyscale image
image_arr = np.asarray(image)
output_image = Image(image_arr) # create Image domain object with a numpy array
op_output.emit(output_image, self.output_name_image) # cannot omit the name even if single output.
# @md.env(pip_packages=["monai"])
class MedNISTClassifierOperator(Operator):
"""Classifies the given image and returns the class name.
Named inputs:
image: Image object for which to generate the classification.
output_folder: Optional, the path to save the results JSON file, overridingthe the one set on __init__
Named output:
result_text: The classification results in text.
"""
DEFAULT_OUTPUT_FOLDER = Path.cwd() / "classification_results"
# For testing the app directly, the model should be at the following path.
MODEL_LOCAL_PATH = Path(os.environ.get("HOLOSCAN_MODEL_PATH", Path.cwd() / "model/model.ts"))
def __init__(
self,
fragment: Fragment,
*args,
app_context: AppContext,
model_name: Optional[str] = "",
model_path: Path = MODEL_LOCAL_PATH,
output_folder: Path = DEFAULT_OUTPUT_FOLDER,
**kwargs,
):
"""Creates an instance with the reference back to the containing application/fragment.
fragment (Fragment): An instance of the Application class which is derived from Fragment.
model_name (str, optional): Name of the model. Default to "" for single model app.
model_path (Path): Path to the model file. Defaults to model/models.ts of current working dir.
output_folder (Path, optional): output folder for saving the classification results JSON file.
"""
# the names used for the model inference input and output
self._input_dataset_key = "image"
self._pred_dataset_key = "pred"
# The names used for the operator input and output
self.input_name_image = "image"
self.output_name_result = "result_text"
# The name of the optional input port for passing data to override the output folder path.
self.input_name_output_folder = "output_folder"
# The output folder set on the object can be overridden at each compute by data in the optional named input
self.output_folder = output_folder
# Need the name when there are multiple models loaded
self._model_name = model_name.strip() if isinstance(model_name, str) else ""
# Need the path to load the models when they are not loaded in the execution context
self.model_path = model_path
self.app_context = app_context
self.model = self._get_model(self.app_context, self.model_path, self._model_name)
# This needs to be at the end of the constructor.
super().__init__(fragment, *args, **kwargs)
def _get_model(self, app_context: AppContext, model_path: Path, model_name: str):
"""Load the model with the given name from context or model path
Args:
app_context (AppContext): The application context object holding the model(s)
model_path (Path): The path to the model file, as a backup to load model directly
model_name (str): The name of the model, when multiples are loaded in the context
"""
if app_context.models:
# `app_context.models.get(model_name)` returns a model instance if exists.
# If model_name is not specified and only one model exists, it returns that model.
model = app_context.models.get(model_name)
else:
model = torch.jit.load(
MedNISTClassifierOperator.MODEL_LOCAL_PATH,
map_location=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)
return model
def setup(self, spec: OperatorSpec):
"""Set up the operator named input and named output, both are in-memory objects."""
spec.input(self.input_name_image)
spec.input(self.input_name_output_folder).condition(ConditionType.NONE) # Optional for overriding.
spec.output(self.output_name_result).condition(ConditionType.NONE) # Not forcing a downstream receiver.
@property
def transform(self):
return Compose([EnsureChannelFirst(channel_dim="no_channel"), ScaleIntensity(), EnsureType()])
def compute(self, op_input, op_output, context):
import json
import torch
img = op_input.receive(self.input_name_image).asnumpy() # (64, 64), uint8. Input validation can be added.
image_tensor = self.transform(img) # (1, 64, 64), torch.float64
image_tensor = image_tensor[None].float() # (1, 1, 64, 64), torch.float32
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
image_tensor = image_tensor.to(device)
with torch.no_grad():
outputs = self.model(image_tensor)
_, output_classes = outputs.max(dim=1)
result = MEDNIST_CLASSES[output_classes[0]] # get the class name
print(result)
op_output.emit(result, self.output_name_result)
# Get output folder, with value in optional input port overriding the obj attribute
output_folder_on_compute = op_input.receive(self.input_name_output_folder) or self.output_folder
Path.mkdir(output_folder_on_compute, parents=True, exist_ok=True) # Let exception bubble up if raised.
output_path = output_folder_on_compute / "output.json"
with open(output_path, "w") as fp:
json.dump(result, fp)
# @md.resource(cpu=1, gpu=1, memory="1Gi")
class App(Application):
"""Application class for the MedNIST classifier."""
def compose(self):
# Use Commandline options over environment variables to init context.
app_context = Application.init_app_context(self.argv)
app_input_path = Path(app_context.input_path)
app_output_path = Path(app_context.output_path)
model_path = Path(app_context.model_path)
load_pil_op = LoadPILOperator(self, CountCondition(self, 1), input_folder=app_input_path, name="pil_loader_op")
classifier_op = MedNISTClassifierOperator(
self, app_context=app_context, output_folder=app_output_path, model_path=model_path, name="classifier_op"
)
my_model_info = ModelInfo("MONAI WG Trainer", "MEDNIST Classifier", "0.1", "xyz")
my_equipment = EquipmentInfo(manufacturer="MOANI Deploy App SDK", manufacturer_model="DICOM SR Writer")
my_special_tags = {"SeriesDescription": "Not for clinical use. The result is for research use only."}
dicom_sr_operator = DICOMTextSRWriterOperator(
self,
copy_tags=False,
model_info=my_model_info,
equipment_info=my_equipment,
custom_tags=my_special_tags,
output_folder=app_output_path,
)
self.add_flow(load_pil_op, classifier_op, {("image", "image")})
self.add_flow(classifier_op, dicom_sr_operator, {("result_text", "text")})
if __name__ == "__main__":
App().run()
This time, let’s execute the app on the command line.
Note
Since the environment variables have been set and contain the correct paths, it is not necessary to provide the command line options on running the application, though the following demonstrates the use of the options.
!python "mednist_app/mednist_classifier_monaideploy.py" -i {input_folder} -o {output_folder} -m {models_folder} -l DEBUG
!cat {output_folder}/output.json
Additional file required for packaging the app (creating MAP Docker image)#
In this version of the App SDK, we need to write out the configuration yaml file as well as the package requirements file, in the application folder.
%%writefile mednist_app/app.yaml
%YAML 1.2
---
application:
title: MONAI Deploy App Package - MedNIST Classifier App
version: 1.0
inputFormats: ["file"]
outputFormats: ["file"]
resources:
cpu: 1
gpu: 1
memory: 1Gi
gpuMemory: 1Gi
%%writefile mednist_app/requirements.txt
monai>=1.2.0
Pillow>=8.4.0
pydicom>=2.3.0
highdicom>=0.18.2
SimpleITK>=2.0.0
setuptools>=59.5.0 # for pkg_resources
By now, we have built the application and prepared all necessary files for create the MONAI Application Package (MAP).