Creating a Multi-AI Deploy App with Multiple Models

This tutorial shows how to create an inference application with multiple models, focusing on model files organization, accessing and inferring with named model network in the application, and finally building an app package.

Typically multiple models will work in tandem, e.g. a lung segmentation model’s output, along with the original image, are then used by a lung nodule detection and classification model. There is, however, a lack of such models in the MONAI Model Zoo as of now. So, for illustration purpose, two independent models will be used in this example, Spleen Segmentation and Pancreas Segmentation, both are trained with DICOM images of CT modality, and both are packaged in the MONAI Bundle format. A single input of a CT Abdomen DICOM Series can be used for both models within the application.

Important Steps

• Place the model TorchScripts in a defined folder structure, see below for details

• Pass the model name to the inference operator instance in the app

• Connect the input to and output from the inference operators, as required by the app

Required Model File Organization

• The model files in TorchScript, be it MONAI Bundle compliant or not, must each be placed in an uniquely named folder. The name of this folder becomes the name of the loaded model network in the application, and is used by the application to retrieve the network via the execution context.

• The folders containing the individual model file must then be placed under a parent folder. The name of this folder is chosen by the application developer.

• The path of the aforementioned parent folder is used to set the well-known environment variable for the model path, HOLOSCAN_MODEL_PATH, when the application is directly run as a program.

• When the application is packaged as an MONAI Application Package (MAP), the parent folder is used as the model path, and the Packager copies all of the sub folders to the well-known models folder in the MAP.

Example Model File Organization

In this example, the models are organized as shown below.

multi_models
├── pancreas_ct_dints
│   └── model.ts
└── spleen_ct
    └── model.ts

Please note,

• The multi_models is the parent folder, whose path is used to set the well-known environment variable for model path. When using App SDK CLI Packager to build the application package, this is the used as the path for models.

• The sub-folder names become model network names, pancreas_ct_dints and spleen_model, respectively.

In the following sections, we will demonstrate how to create and package the application using these two models.

Note

The two models are both MONAI bundles, published in MONAI Model Zoo

• spleen_ct_segmentation, v0.3.2

• pancreas_ct_dints_segmentation, v0.3.0

The DICOM CT series used as test input is downloaded from TCIA, CT Abdomen Collection ID CPTAC-PDA Subject ID C3N-00198.

Both the DICOM files and the models have been packaged and shared on Google Drive.

Creating Operators and connecting them in Application class

We will implement an application that consists of seven Operators:

• DICOMDataLoaderOperator:

  • Input(dicom_files): a folder path (Path)

  • Output(dicom_study_list): a list of DICOM studies in memory (List[DICOMStudy])

• DICOMSeriesSelectorOperator:

  • Input(dicom_study_list): a list of DICOM studies in memory (List[DICOMStudy])

  • Input(selection_rules): a selection rule (Dict)

  • Output(study_selected_series_list): a DICOM series object in memory (StudySelectedSeries)

• DICOMSeriesToVolumeOperator:

  • Input(study_selected_series_list): a DICOM series object in memory (StudySelectedSeries)

  • Output(image): an image object in memory (Image)

• MonaiBundleInferenceOperator x 2:

  • Input(image): an image object in memory (Image)

  • Output(pred): an image object in memory (Image)

• DICOMSegmentationWriterOperator x2:

  • Input(seg_image): a segmentation image object in memory (Image)

  • Input(study_selected_series_list): a DICOM series object in memory (StudySelectedSeries)

  • Output(dicom_seg_instance): a file path (Path)

Note

The DICOMSegmentationWriterOperator needs both the segmentation image as well as the original DICOM series for reusing the patient demographics and other DICOM Study level attributes, as well as referring to the original SOP instance UID.

The workflow of the application is illustrated below.

        %%{init: {"theme": "base", "themeVariables": { "fontSize": "16px"}} }%%

classDiagram
    direction TB
    DICOMDataLoaderOperator --|> DICOMSeriesSelectorOperator : dicom_study_list...dicom_study_list
    DICOMSeriesSelectorOperator --|> DICOMSeriesToVolumeOperator : study_selected_series_list...study_selected_series_list

    DICOMSeriesToVolumeOperator --|> Spleen_BundleInferenceOperator : image...image
    DICOMSeriesSelectorOperator --|> Spleen_DICOMSegmentationWriterOperator : study_selected_series_list...study_selected_series_list
    Spleen_BundleInferenceOperator --|> Spleen_DICOMSegmentationWriterOperator : pred...seg_image

    DICOMSeriesToVolumeOperator --|> Pancreas_BundleInferenceOperator : image...image
    DICOMSeriesSelectorOperator --|> Pancreas_DICOMSegmentationWriterOperator : study_selected_series_list...study_selected_series_list
    Pancreas_BundleInferenceOperator --|> Pancreas_DICOMSegmentationWriterOperator : pred...seg_image

    class DICOMDataLoaderOperator {
        <in>dicom_files : DISK
        dicom_study_list(out) IN_MEMORY
    }
    class DICOMSeriesSelectorOperator {
        <in>dicom_study_list : IN_MEMORY
        <in>selection_rules : IN_MEMORY
        study_selected_series_list(out) IN_MEMORY
    }
    class DICOMSeriesToVolumeOperator {
        <in>study_selected_series_list : IN_MEMORY
        image(out) IN_MEMORY
    }
    class Spleen_BundleInferenceOperator {
        <in>image : IN_MEMORY
        pred(out) IN_MEMORY
    }
    class Pancreas_BundleInferenceOperator {
        <in>image : IN_MEMORY
        pred(out) IN_MEMORY
    }
    class Spleen_DICOMSegmentationWriterOperator {
        <in>seg_image : IN_MEMORY
        <in>study_selected_series_list : IN_MEMORY
        dicom_seg_instance(out) DISK
    }
    class Pancreas_DICOMSegmentationWriterOperator {
        <in>seg_image : IN_MEMORY
        <in>study_selected_series_list : IN_MEMORY
        dicom_seg_instance(out) DISK
    }
    

Setup environment

# Install MONAI and other necessary image processing packages for the application
!python -c "import monai" || pip install --upgrade -q "monai<=1.5.0"
!python -c "import torch" || pip install -q "torch>=1.12.0"
!python -c "import numpy" || pip install -q "numpy>=1.21"
!python -c "import nibabel" || pip install -q "nibabel>=3.2.1"
!python -c "import pydicom" || pip install -q "pydicom>=1.4.2"
!python -c "import highdicom" || pip install -q "highdicom>=0.18.2"
!python -c "import SimpleITK" || pip install -q "SimpleITK>=2.0.0"

# Install MONAI Deploy App SDK package
!python -c "import monai.deploy" || pip install -q "monai-deploy-app-sdk"

Note: you may need to restart the Jupyter kernel to use the updated packages.

Download/Extract input and model/bundle files from Google Drive

Note: Data files are now access controlled. Please first request permission to access the shared folder on Google Drive. Please download zip file, ai_multi_model_bundle_data.zip in the ai_multi_ai_app folder, to the same folder as the notebook example.

# Download ai_spleen_bundle_data test data zip file. Please request access and download manually.
# !pip install gdown
# !gdown https://drive.google.com/file/d/1Iwx-jl7vBu67lMpHwJ2VueAOiTtJF4mL/view?usp=sharing

# After downloading ai_spleen_bundle_data zip file from the web browser or using gdown,
!unzip -o "ai_multi_model_bundle_data.zip"

Set up environment variables

models_folder = "multi_models"
%env HOLOSCAN_INPUT_PATH dcm
%env HOLOSCAN_MODEL_PATH {models_folder}
%env HOLOSCAN_OUTPUT_PATH output

Set up imports

Let’s import necessary classes/decorators to define Application and Operator.

import logging
from pathlib import Path

# Required for setting SegmentDescription attributes. Direct import as this is not part of App SDK package.
from pydicom.sr.codedict import codes

from monai.deploy.conditions import CountCondition
from monai.deploy.core import AppContext, Application
from monai.deploy.core.domain import Image
from monai.deploy.core.io_type import IOType
from monai.deploy.operators.dicom_data_loader_operator import DICOMDataLoaderOperator
from monai.deploy.operators.dicom_seg_writer_operator import DICOMSegmentationWriterOperator, SegmentDescription
from monai.deploy.operators.dicom_series_selector_operator import DICOMSeriesSelectorOperator
from monai.deploy.operators.dicom_series_to_volume_operator import DICOMSeriesToVolumeOperator
from monai.deploy.operators.monai_bundle_inference_operator import (
    BundleConfigNames,
    IOMapping,
    MonaiBundleInferenceOperator,
)
from monai.deploy.operators.stl_conversion_operator import STLConversionOperator

Determining the Input and Output for the Model Bundle Inference Operator

The App SDK provides a MonaiBundleInferenceOperator class to perform inference with a MONAI Bundle, which is essentially a PyTorch model in TorchScript with additional metadata describing the model network and processing specification. This operator uses the MONAI utilities to parse a MONAI Bundle to automatically instantiate the objects required for input and output processing as well as inference, as such it depends on MONAI transforms, inferers, and in turn their dependencies.

Each Operator class inherits from the base Operator base class, and its input/output properties are specified in the setup function (as opposed to using decorators @inputand @output in Version 0.5 and below).

For the MonaiBundleInferenceOperator class, the input/output need to be defined to match those of the model network, both in name and data type. For the current release, an IOMapping object is used to connect the operator input/output to those of the model network by using the same names. This is likely to change, to be automated, in the future releases once certain limitation in the App SDK is removed.

The Spleen CT Segmentation model network has a named input, called “image”, and the named output called “pred”, and both are of image type, which can all be mapped to the App SDK Image. This piece of information is typically acquired by examining the model metadata network_data_format attribute in the bundle, as seen in this [example] (https://github.com/Project-MONAI/model-zoo/blob/dev/models/spleen_ct_segmentation/configs/metadata.json).

Creating Application class

Our application class would look like below.

It defines App class, inheriting the base Application class.

Objects required for DICOM parsing, series selection, pixel data conversion to volume image, model specific inference, and the AI result specific DICOM Segmentation object writers are created. The execution pipeline, as a Directed Acyclic Graph, is then constructed by connecting these objects through self.add_flow().

class App(Application):
    """This example demonstrates how to create a multi-model/multi-AI application.

    The important steps are:
        1. Place the model TorchScripts in a defined folder structure, see below for details
        2. Pass the model name to the inference operator instance in the app
        3. Connect the input to and output from the inference operators, as required by the app

    Required Model Folder Structure:
        1. The model TorchScripts, be it MONAI Bundle compliant or not, must be placed in
           a parent folder, whose path is used as the path to the model(s) on app execution
        2. Each TorchScript file needs to be in a sub-folder, whose name is the model name

    An example is shown below, where the `parent_foler` name can be the app's own choosing, and
    the sub-folder names become model names, `pancreas_ct_dints` and `spleen_model`, respectively.

        <parent_fodler>
        ├── pancreas_ct_dints
        │   └── model.ts
        └── spleen_ct
            └── model.ts

    Note:
    1. The TorchScript files of MONAI Bundles can be downloaded from MONAI Model Zoo, at
       https://github.com/Project-MONAI/model-zoo/tree/dev/models
       https://github.com/Project-MONAI/model-zoo/tree/dev/models/spleen_ct_segmentation, v0.3.2
       https://github.com/Project-MONAI/model-zoo/tree/dev/models/pancreas_ct_dints_segmentation, v0.3.8
    2. The input DICOM instances are from a DICOM Series of CT Abdomen, similar to the ones
       used in the Spleen Segmentation example
    3. This example is purely for technical demonstration, not for clinical use

    Execution Time Estimate:
      With a Nvidia GV100 32GB GPU, the execution time is around 87 seconds for an input DICOM series of 204 instances,
      and 167 second for a series of 515 instances.
    """

    def __init__(self, *args, **kwargs):
        """Creates an application instance."""
        self._logger = logging.getLogger("{}.{}".format(__name__, type(self).__name__))
        super().__init__(*args, **kwargs)

    def run(self, *args, **kwargs):
        # This method calls the base class to run. Can be omitted if simply calling through.
        self._logger.info(f"Begin {self.run.__name__}")
        super().run(*args, **kwargs)
        self._logger.info(f"End {self.run.__name__}")

    def compose(self):
        """Creates the app specific operators and chain them up in the processing DAG."""

        logging.info(f"Begin {self.compose.__name__}")

        app_context = Application.init_app_context({})  # Do not pass argv in Jupyter Notebook
        app_input_path = Path(app_context.input_path)
        app_output_path = Path(app_context.output_path)

        # Create the custom operator(s) as well as SDK built-in operator(s).
        study_loader_op = DICOMDataLoaderOperator(
            self, CountCondition(self, 1), input_folder=app_input_path, name="study_loader_op"
        )
        series_selector_op = DICOMSeriesSelectorOperator(self, rules=Sample_Rules_Text, name="series_selector_op")
        series_to_vol_op = DICOMSeriesToVolumeOperator(self, name="series_to_vol_op")

        # Create the inference operator that supports MONAI Bundle and automates the inference.
        # The IOMapping labels match the input and prediction keys in the pre and post processing.
        # The model_name needs to be provided as this is a multi-model application and each inference
        # operator need to rely on the name to access the named loaded model network.
        # create an inference operator for each.
        #
        # Pertinent MONAI Bundle:
        #   https://github.com/Project-MONAI/model-zoo/tree/dev/models/spleen_ct_segmentation, v0.3.2
        #   https://github.com/Project-MONAI/model-zoo/tree/dev/models/pancreas_ct_dints_segmentation, v0.3.8

        config_names = BundleConfigNames(config_names=["inference"])  # Same as the default

        # This is the inference operator for the spleen_model bundle. Note the model name.
        bundle_spleen_seg_op = MonaiBundleInferenceOperator(
            self,
            input_mapping=[IOMapping("image", Image, IOType.IN_MEMORY)],
            output_mapping=[IOMapping("pred", Image, IOType.IN_MEMORY)],
            app_context=app_context,
            bundle_config_names=config_names,
            model_name="spleen_ct",
            name="bundle_spleen_seg_op",
        )

        # This is the inference operator for the pancreas_ct_dints bundle. Note the model name.
        bundle_pancreas_seg_op = MonaiBundleInferenceOperator(
            self,
            input_mapping=[IOMapping("image", Image, IOType.IN_MEMORY)],
            output_mapping=[IOMapping("pred", Image, IOType.IN_MEMORY)],
            app_context=app_context,
            bundle_config_names=config_names,
            model_name="pancreas_ct_dints",
            name="bundle_pancreas_seg_op",
        )

        # Create DICOM Seg writer providing the required segment description for each segment with
        # the actual algorithm and the pertinent organ/tissue. The segment_label, algorithm_name,
        # and algorithm_version are of DICOM VR LO type, limited to 64 chars.
        # https://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_6.2.html
        #
        # NOTE: Each generated DICOM Seg will be a dcm file with the name based on the SOP instance UID.

        # Description for the Spleen seg, and the seg writer obj
        seg_descriptions_spleen = [
            SegmentDescription(
                segment_label="Spleen",
                segmented_property_category=codes.SCT.Organ,
                segmented_property_type=codes.SCT.Spleen,
                algorithm_name="volumetric (3D) segmentation of the spleen from CT image",
                algorithm_family=codes.DCM.ArtificialIntelligence,
                algorithm_version="0.3.2",
            )
        ]

        custom_tags_spleen = {"SeriesDescription": "AI Spleen Seg for research use only. Not for clinical use."}
        dicom_seg_writer_spleen = DICOMSegmentationWriterOperator(
            self,
            segment_descriptions=seg_descriptions_spleen,
            custom_tags=custom_tags_spleen,
            output_folder=app_output_path,
            name="dicom_seg_writer_spleen",
        )

        # Description for the Pancreas seg, and the seg writer obj
        _algorithm_name = "Pancreas CT DiNTS segmentation from CT image"
        _algorithm_family = codes.DCM.ArtificialIntelligence
        _algorithm_version = "0.3.8"

        seg_descriptions_pancreas = [
            SegmentDescription(
                segment_label="Pancreas",
                segmented_property_category=codes.SCT.Organ,
                segmented_property_type=codes.SCT.Pancreas,
                algorithm_name=_algorithm_name,
                algorithm_family=_algorithm_family,
                algorithm_version=_algorithm_version,
            ),
            SegmentDescription(
                segment_label="Tumor",
                segmented_property_category=codes.SCT.Tumor,
                segmented_property_type=codes.SCT.Tumor,
                algorithm_name=_algorithm_name,
                algorithm_family=_algorithm_family,
                algorithm_version=_algorithm_version,
            ),
        ]
        custom_tags_pancreas = {"SeriesDescription": "AI Pancreas Seg for research use only. Not for clinical use."}

        dicom_seg_writer_pancreas = DICOMSegmentationWriterOperator(
            self,
            segment_descriptions=seg_descriptions_pancreas,
            custom_tags=custom_tags_pancreas,
            output_folder=app_output_path,
            name="dicom_seg_writer_pancreas",
        )

        # NOTE: Sharp eyed readers can already see that the above instantiation of object can be simply parameterized.
        #       Very true, but leaving them as if for easy reading. In fact the whole app can be parameterized for general use.

        # Create the processing pipeline, by specifying the upstream and downstream operators, and
        # ensuring the output from the former matches the input of the latter, in both name and type.
        self.add_flow(study_loader_op, series_selector_op, {("dicom_study_list", "dicom_study_list")})
        self.add_flow(
            series_selector_op, series_to_vol_op, {("study_selected_series_list", "study_selected_series_list")}
        )

        # Feed the input image to all inference operators
        self.add_flow(series_to_vol_op, bundle_spleen_seg_op, {("image", "image")})
        # The Pancreas CT Seg bundle requires PyTorch 1.12.0 to avoid failure to load.
        self.add_flow(series_to_vol_op, bundle_pancreas_seg_op, {("image", "image")})

        # Create DICOM Seg for one of the inference output
        # Note below the dicom_seg_writer requires two inputs, each coming from a upstream operator.
        self.add_flow(
            series_selector_op, dicom_seg_writer_spleen, {("study_selected_series_list", "study_selected_series_list")}
        )
        self.add_flow(bundle_spleen_seg_op, dicom_seg_writer_spleen, {("pred", "seg_image")})

        # Create DICOM Seg for one of the inference output
        # Note below the dicom_seg_writer requires two inputs, each coming from a upstream operator.
        self.add_flow(
            series_selector_op,
            dicom_seg_writer_pancreas,
            {("study_selected_series_list", "study_selected_series_list")},
        )
        self.add_flow(bundle_pancreas_seg_op, dicom_seg_writer_pancreas, {("pred", "seg_image")})

        logging.info(f"End {self.compose.__name__}")


# This is a sample series selection rule in JSON, simply selecting CT series.
# If the study has more than 1 CT series, then all of them will be selected.
# Please see more detail in DICOMSeriesSelectorOperator.
Sample_Rules_Text = """
{
    "selections": [
        {
            "name": "CT Series",
            "conditions": {
                "StudyDescription": "(.*?)",
                "Modality": "(?i)CT",
                "SeriesDescription": "(.*?)"
            }
        }
    ]
}
"""

Executing app locally

We can execute the app in the Jupyter notebook. Note that the DICOM files of the CT Abdomen series must be present in the input folder, the models are already staged, and environment variables are set.

!rm -rf $HOLOSCAN_OUTPUT_PATH
app = App().run()

Once the application is verified inside Jupyter notebook, we can write the whole application as a file, adding the following lines:

if __name__ == "__main__":
    App().run()

The above lines are needed to execute the application code by using python interpreter.

A __main__.py file should also be added, so the application folder structure would look like below:

my_app
├── __main__.py
└── app.py

# Create an application folder
!mkdir -p my_app && rm -rf my_app/*

app.py

%%writefile my_app/app.py
# Copyright 2021-2023 MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#     http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import logging
from pathlib import Path

# Required for setting SegmentDescription attributes. Direct import as this is not part of App SDK package.
from pydicom.sr.codedict import codes

from monai.deploy.conditions import CountCondition
from monai.deploy.core import AppContext, Application
from monai.deploy.core.domain import Image
from monai.deploy.core.io_type import IOType
from monai.deploy.operators.dicom_data_loader_operator import DICOMDataLoaderOperator
from monai.deploy.operators.dicom_seg_writer_operator import DICOMSegmentationWriterOperator, SegmentDescription
from monai.deploy.operators.dicom_series_selector_operator import DICOMSeriesSelectorOperator
from monai.deploy.operators.dicom_series_to_volume_operator import DICOMSeriesToVolumeOperator
from monai.deploy.operators.monai_bundle_inference_operator import (
    BundleConfigNames,
    IOMapping,
    MonaiBundleInferenceOperator,
)


class App(Application):
    """This example demonstrates how to create a multi-model/multi-AI application.

    The important steps are:
        1. Place the model TorchScripts in a defined folder structure, see below for details
        2. Pass the model name to the inference operator instance in the app
        3. Connect the input to and output from the inference operators, as required by the app

    Required Model Folder Structure:
        1. The model TorchScripts, be it MONAI Bundle compliant or not, must be placed in
           a parent folder, whose path is used as the path to the model(s) on app execution
        2. Each TorchScript file needs to be in a sub-folder, whose name is the model name

    An example is shown below, where the `parent_foler` name can be the app's own choosing, and
    the sub-folder names become model names, `pancreas_ct_dints` and `spleen_model`, respectively.

        <parent_fodler>
        ├── pancreas_ct_dints
        │   └── model.ts
        └── spleen_ct
            └── model.ts

    Note:
    1. The TorchScript files of MONAI Bundles can be downloaded from MONAI Model Zoo, at
       https://github.com/Project-MONAI/model-zoo/tree/dev/models
       https://github.com/Project-MONAI/model-zoo/tree/dev/models/spleen_ct_segmentation, v0.3.2
       https://github.com/Project-MONAI/model-zoo/tree/dev/models/pancreas_ct_dints_segmentation, v0.3.8
    2. The input DICOM instances are from a DICOM Series of CT Abdomen, similar to the ones
       used in the Spleen Segmentation example
    3. This example is purely for technical demonstration, not for clinical use

    Execution Time Estimate:
      With a Nvidia GV100 32GB GPU, the execution time is around 87 seconds for an input DICOM series of 204 instances,
      and 167 second for a series of 515 instances.
    """

    def __init__(self, *args, **kwargs):
        """Creates an application instance."""
        self._logger = logging.getLogger("{}.{}".format(__name__, type(self).__name__))
        super().__init__(*args, **kwargs)

    def run(self, *args, **kwargs):
        # This method calls the base class to run. Can be omitted if simply calling through.
        self._logger.info(f"Begin {self.run.__name__}")
        super().run(*args, **kwargs)
        self._logger.info(f"End {self.run.__name__}")

    def compose(self):
        """Creates the app specific operators and chain them up in the processing DAG."""

        logging.info(f"Begin {self.compose.__name__}")

        # Use Commandline options over environment variables to init context.
        app_context = Application.init_app_context(self.argv)
        app_input_path = Path(app_context.input_path)
        app_output_path = Path(app_context.output_path)

        # Create the custom operator(s) as well as SDK built-in operator(s).
        study_loader_op = DICOMDataLoaderOperator(
            self, CountCondition(self, 1), input_folder=app_input_path, name="study_loader_op"
        )
        series_selector_op = DICOMSeriesSelectorOperator(self, rules=Sample_Rules_Text, name="series_selector_op")
        series_to_vol_op = DICOMSeriesToVolumeOperator(self, name="series_to_vol_op")

        # Create the inference operator that supports MONAI Bundle and automates the inference.
        # The IOMapping labels match the input and prediction keys in the pre and post processing.
        # The model_name needs to be provided as this is a multi-model application and each inference
        # operator need to rely on the name to access the named loaded model network.
        # create an inference operator for each.
        #
        # Pertinent MONAI Bundle:
        #   https://github.com/Project-MONAI/model-zoo/tree/dev/models/spleen_ct_segmentation, v0.3.2
        #   https://github.com/Project-MONAI/model-zoo/tree/dev/models/pancreas_ct_dints_segmentation, v0.3.8

        config_names = BundleConfigNames(config_names=["inference"])  # Same as the default

        # This is the inference operator for the spleen_model bundle. Note the model name.
        bundle_spleen_seg_op = MonaiBundleInferenceOperator(
            self,
            input_mapping=[IOMapping("image", Image, IOType.IN_MEMORY)],
            output_mapping=[IOMapping("pred", Image, IOType.IN_MEMORY)],
            app_context=app_context,
            bundle_config_names=config_names,
            model_name="spleen_ct",
            name="bundle_spleen_seg_op",
        )

        # This is the inference operator for the pancreas_ct_dints bundle. Note the model name.
        bundle_pancreas_seg_op = MonaiBundleInferenceOperator(
            self,
            input_mapping=[IOMapping("image", Image, IOType.IN_MEMORY)],
            output_mapping=[IOMapping("pred", Image, IOType.IN_MEMORY)],
            app_context=app_context,
            bundle_config_names=config_names,
            model_name="pancreas_ct_dints",
            name="bundle_pancreas_seg_op",
        )

        # Create DICOM Seg writer providing the required segment description for each segment with
        # the actual algorithm and the pertinent organ/tissue. The segment_label, algorithm_name,
        # and algorithm_version are of DICOM VR LO type, limited to 64 chars.
        # https://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_6.2.html
        #
        # NOTE: Each generated DICOM Seg will be a dcm file with the name based on the SOP instance UID.

        # Description for the Spleen seg, and the seg writer obj
        seg_descriptions_spleen = [
            SegmentDescription(
                segment_label="Spleen",
                segmented_property_category=codes.SCT.Organ,
                segmented_property_type=codes.SCT.Spleen,
                algorithm_name="volumetric (3D) segmentation of the spleen from CT image",
                algorithm_family=codes.DCM.ArtificialIntelligence,
                algorithm_version="0.3.2",
            )
        ]

        custom_tags_spleen = {"SeriesDescription": "AI Spleen Seg for research use only. Not for clinical use."}
        dicom_seg_writer_spleen = DICOMSegmentationWriterOperator(
            self,
            segment_descriptions=seg_descriptions_spleen,
            custom_tags=custom_tags_spleen,
            output_folder=app_output_path,
            name="dicom_seg_writer_spleen",
        )

        # Description for the Pancreas seg, and the seg writer obj
        _algorithm_name = "Pancreas CT DiNTS segmentation from CT image"
        _algorithm_family = codes.DCM.ArtificialIntelligence
        _algorithm_version = "0.3.8"

        seg_descriptions_pancreas = [
            SegmentDescription(
                segment_label="Pancreas",
                segmented_property_category=codes.SCT.Organ,
                segmented_property_type=codes.SCT.Pancreas,
                algorithm_name=_algorithm_name,
                algorithm_family=_algorithm_family,
                algorithm_version=_algorithm_version,
            ),
            SegmentDescription(
                segment_label="Tumor",
                segmented_property_category=codes.SCT.Tumor,
                segmented_property_type=codes.SCT.Tumor,
                algorithm_name=_algorithm_name,
                algorithm_family=_algorithm_family,
                algorithm_version=_algorithm_version,
            ),
        ]
        custom_tags_pancreas = {"SeriesDescription": "AI Pancreas Seg for research use only. Not for clinical use."}

        dicom_seg_writer_pancreas = DICOMSegmentationWriterOperator(
            self,
            segment_descriptions=seg_descriptions_pancreas,
            custom_tags=custom_tags_pancreas,
            output_folder=app_output_path,
            name="dicom_seg_writer_pancreas",
        )

        # NOTE: Sharp eyed readers can already see that the above instantiation of object can be simply parameterized.
        #       Very true, but leaving them as if for easy reading. In fact the whole app can be parameterized for general use.

        # Create the processing pipeline, by specifying the upstream and downstream operators, and
        # ensuring the output from the former matches the input of the latter, in both name and type.
        self.add_flow(study_loader_op, series_selector_op, {("dicom_study_list", "dicom_study_list")})
        self.add_flow(
            series_selector_op, series_to_vol_op, {("study_selected_series_list", "study_selected_series_list")}
        )

        # Feed the input image to all inference operators
        self.add_flow(series_to_vol_op, bundle_spleen_seg_op, {("image", "image")})
        # The Pancreas CT Seg bundle requires PyTorch 1.12.0 to avoid failure to load.
        self.add_flow(series_to_vol_op, bundle_pancreas_seg_op, {("image", "image")})

        # Create DICOM Seg for one of the inference output
        # Note below the dicom_seg_writer requires two inputs, each coming from a upstream operator.
        self.add_flow(
            series_selector_op, dicom_seg_writer_spleen, {("study_selected_series_list", "study_selected_series_list")}
        )
        self.add_flow(bundle_spleen_seg_op, dicom_seg_writer_spleen, {("pred", "seg_image")})

        # Create DICOM Seg for one of the inference output
        # Note below the dicom_seg_writer requires two inputs, each coming from a upstream operator.
        self.add_flow(
            series_selector_op,
            dicom_seg_writer_pancreas,
            {("study_selected_series_list", "study_selected_series_list")},
        )
        self.add_flow(bundle_pancreas_seg_op, dicom_seg_writer_pancreas, {("pred", "seg_image")})

        logging.info(f"End {self.compose.__name__}")


# This is a sample series selection rule in JSON, simply selecting CT series.
# If the study has more than 1 CT series, then all of them will be selected.
# Please see more detail in DICOMSeriesSelectorOperator.
Sample_Rules_Text = """
{
    "selections": [
        {
            "name": "CT Series",
            "conditions": {
                "StudyDescription": "(.*?)",
                "Modality": "(?i)CT",
                "SeriesDescription": "(.*?)"
            }
        }
    ]
}
"""

if __name__ == "__main__":
    logging.info(f"Begin {__name__}")
    App().run()
    logging.info(f"End {__name__}")

%%writefile my_app/__main__.py
from app import App

if __name__ == "__main__":
    App().run()

!ls my_app

At this time, let’s execute the app on the command line. Note the required e.

Note

Since the environment variables have been set with the specific input data and model paths from earlier steps, it is not necessary to provide the command line options on running the application.

!rm -rf $HOLOSCAN_OUTPUT_PATH
!python my_app

!ls $HOLOSCAN_OUTPUT_PATH

Packaging app

Let’s package the app with MONAI Application Packager.

In this version of the App SDK, we need to write out the configuration yaml file as well as the package requirements file, in the application folder.

%%writefile my_app/app.yaml
%YAML 1.2
---
application:
  title: MONAI Deploy App Package - Multi Model App
  version: 1.0
  inputFormats: ["file"]
  outputFormats: ["file"]

resources:
  cpu: 1
  gpu: 1
  memory: 1Gi
  gpuMemory: 10Gi

%%writefile my_app/requirements.txt
highdicom>=0.18.2
monai>=1.0
nibabel>=3.2.1
numpy>=1.21.6
pydicom>=2.3.0
setuptools>=59.5.0 # for pkg_resources
SimpleITK>=2.0.0
torch>=1.12.0

Now we can use the CLI package command to build the MONAI Application Package (MAP) container image based on a supported base image.

Note

Building a MONAI Application Package (Docker image) can take time. Use -l DEBUG option to see the progress.

tag_prefix = "my_app"

!monai-deploy package my_app -m {models_folder} -c my_app/app.yaml -t {tag_prefix}:1.0 --platform x86_64 -l DEBUG

We can see that the Docker image is created.

!docker image ls | grep {tag_prefix}

We can choose to display and inspect the MAP manifests by running the container with the show command. Furthermore, we can also extract the manifests and other contents in the MAP by using the extract command while mapping specific folder to the host’s (we know that our MAP is compliant and supports these commands).

Note

The host folder for storing the extracted content must first be created by the user, and if it has been created by Docker on running the container, the folder needs to be deleted and re-created.

!echo "Display manifests and extract MAP contents to the host folder, ./export"
!docker run --rm {tag_prefix}-x64-workstation-dgpu-linux-amd64:1.0 show
!rm -rf `pwd`/export && mkdir -p `pwd`/export
!docker run --rm -v `pwd`/export/:/var/run/holoscan/export/ {tag_prefix}-x64-workstation-dgpu-linux-amd64:1.0 extract
!ls `pwd`/export

Executing packaged app locally

The packaged app can be run locally through MONAI Application Runner.

# Clear the output folder and run the MAP. The input is expected to be a folder.
!rm -rf $HOLOSCAN_OUTPUT_PATH
!monai-deploy run -i $HOLOSCAN_INPUT_PATH -o $HOLOSCAN_OUTPUT_PATH my_app-x64-workstation-dgpu-linux-amd64:1.0

In the output folder are the DICOM segmentation files.

!ls $HOLOSCAN_OUTPUT_PATH