Tip

Download this tutorial as a Jupyter notebook, or as a python script with code cells. We highly recommend using Visual Studio Code to execute this tutorial.

Tutorial 3: Masking / Segmentation

This tutorial explains how to select a region of interest in a video using optimap. For instance, it is possible to use optimap to automatically select the heart surface, to manually draw a region of interest in the video image, or to ignore parts of the video close to the boundary. The post-processing can then be applied to the masked or segmented part of the video image, for instance. optimap provides several easy-to-use routines for these purposes.

We will first load an example video file using the following code:

import optimap as om
import numpy as np
import matplotlib.pyplot as plt

filepath = om.download_example_data('VF_Rabbit_1_warped.npy')
video_warped = om.load_video(filepath)
om.print_properties(video_warped)

frame = video_warped[0]  # first frame

------------------------------------------------------------------------------------------
array with dimensions: (1000, 390, 300)
datatype of array: uint16
minimum value in entire array: 46
maximum value in entire array: 3847
------------------------------------------------------------------------------------------

Automatic Background Segmentation

We can then use optimap’s background_mask() function to automatically separate much brighter tissue from the dark background:

background_mask = om.background_mask(frame, title='Automatic background mask')
om.save_mask('background_mask.png', background_mask)

Creating mask with detected threshold 401.0

The background_mask() function creates a two-dimensional binary array with True for background (here shown in red) and False for tissue. Here, we used the first frame of the video to create the mask. background_mask() and foreground_mask() automatically estimate an optimal threshold for the separation of foreground and background using the GHT [Barron, 2020] algorithm (see image.detect_background_threshold()). Pixels with a value below this threshold (here 401.0) are considered background, and pixels with a value above this threshold are considered foreground. The threshold can be adjusted manually, if desired:

foreground_mask = om.foreground_mask(frame, threshold=500, title='Manual threshold foreground mask')

Here we specified a threshold value of 500 to generate a foreground mask. Note that both functions only separate foreground from background, i.e. they do not distinguish the ventricles from the atria. For this purpose, we need to use a different approach.

Manual Segmentation using Drawing Tool

We can manually draw a mask and select a region (e.g. the atria) using optimap’s mask drawing tool:

manual_mask = om.interactive_mask(frame, initial_mask=foreground_mask)

The drawing tool can be used to draw one or several arbitrary regions and the tool automatically creates the corresponding binary array as a mask. A drawn mask can be inverted or deleted, edits can be undone or redone, etc. (see Documentation). Simply use the above function and a window will pop up:

The following table lists the available keybindings:

Key/Mouse	Action
Scroll	Zoom in/out
ctrl+z or cmd+z	Undo
ctrl+y or cmd+y	Redo
v	Toggle mask visibility
d	Draw/Lasso mode
e	Erase mode
q	Quit

To edit a mask in different program, e.g. GIMP, save the mask as a PNG file:

om.save_mask("mask.png", background_mask, image=frame)

The image argument is optional, but if provided, the mask will be saved as the alpha channel of the image. For editing the mask in GIMP select the alpha channel and use the paintbrush tool to edit the mask. The mask can then be loaded back into optimap using the following code:

background_mask = om.load_mask("mask.png")

The save_mask() and load_mask() functions support a variety of file formats, including PNG, TIFF, and NPY. See the documentation for more information.

To visualize the mask, use show_mask():

om.show_mask(manual_mask, image=frame, title='Manual mask');

Refining the mask

Not in all cases the mask is perfect, and it may be necessary to adjust the mask. The automatic thresholding may not work well for all videos, and the mask may need to be adjusted. For instance, the mask may be too large or too small, or it may contain holes. We can use image.erode_mask() to shrink the mask by 10 pixels, or image.dilate_mask() to expand the mask by 10 pixels:

mask = np.logical_not(manual_mask)

fig, axs = plt.subplots(1, 3)
dilated = om.image.dilate_mask(mask, iterations=10, show=False)
eroded = om.image.erode_mask(mask, iterations=10, border_value=True, show=False)

om.show_mask(mask, image=frame, ax=axs[0], title='Original')
om.show_mask(eroded, image=frame, ax=axs[1], title='Eroded')
om.show_mask(dilated, image=frame, ax=axs[2], title='Dilated')
plt.tight_layout()
plt.show()

Use image.fill_mask() to fill holes in the mask, to keep only the largest connected component (island) of the mask use image.largest_mask_component(). The image.largest_mask_component() function also has a invert argument to invert the mask before selecting the largest component (e.g. to keep the largest hole in the mask when working with background masks).

mask = om.foreground_mask(frame, threshold=1800, show=False)

fig, axs = plt.subplots(1, 3)
om.show_mask(mask, image=frame, title='Original', ax=axs[0])
om.image.fill_mask(mask, image=frame, title='fill_mask()', ax=axs[1])
om.image.largest_mask_component(mask, image=frame, title='largest_mask_component()', ax=axs[2])
plt.tight_layout()
plt.show()

The opening and closing morphological operations combine erosion and dilation. For instance, image.open_mask() shrinks the mask and then expands it, while image.close_mask() expands the mask by and then shrinks it. By doing several iterations of this it can be used to remove small islands, fill holes, or to smooth the mask. See the scipy documentation for more information on morphological operations.

mask = om.foreground_mask(frame, threshold=1400, show=False)

fig, axs = plt.subplots(1, 3)
om.show_mask(mask, image=frame, title='Original', ax=axs[0])
om.image.open_mask(mask, iterations=10, image=frame, ax=axs[1], title='open_mask()')
om.image.close_mask(mask, iterations=10, image=frame, ax=axs[2], title='close_mask()')
plt.tight_layout()
plt.show()

Working with masks

Warning

This tutorial is currently under development. We will add more information soon.

mask = om.image.dilate_mask(manual_mask, iterations=2, image=frame, show=False)

video_warped = video_warped.astype(np.float32)
video_warped[:, mask] = np.nan

om.show_video(video_warped)