cytoviewer - Interactive multi-channel image visualization in R
Lasse Meyer
Department for Quantitative Biomedicine, University of ZurichInstitute for Molecular Health Sciences, ETH Zurichlasse.meyer@dqbm.uzh.ch
Nils Eling
Department for Quantitative Biomedicine, University of ZurichInstitute for Molecular Health Sciences, ETH Zurichnils.eling@dqbm.uzh.ch
25 June 2024
Source:vignettes/cytoviewer.Rmd
cytoviewer.RmdAbstract
This R/Bioconductor package supports interactive visualization of multi-channel images and segmentation masks generated by imaging mass cytometry and other highly multiplexed imaging techniques using shiny. The cytoviewer interface is divided into image-level (Composite and Channels) and cell-level visualization (Masks). It allows users to overlay individual images with segmentation masks, integrates well with SingleCellExperiment and SpatialExperiment objects for metadata visualization and supports image downloads.
Introduction
This vignette introduces the cytoviewer package for
interactive multi-channel image visualization. Images as well as
corresponding segmentation masks generated by imaging mass cytometry
(IMC) and other highly multiplexed imaging techniques can be
interactively visualized and explored.
The cytoviewer package builds on top of the cytomapper
Bioconductor package (Eling et al. 2020)
and extends the static visualization strategies provided by
cytomapper via an interactive Shiny
application. The cytoviewer package leverages the
image handling, analysis and visualization strategies provided by the
EBImage
Bioconductor package and offers interactive image visualization
strategies in a similar fashion as iSEE for
single-cell data. In addition, building up on SingleCellExperiment,
SpatialExperiment
and cytomapper::CytoImageList classes, the
cytoviewer package integrates into the Bioconductor
framework for single-cell and image analysis.
Read the pre-print here.
Highly multiplexed imaging
Highly multiplexed imaging allows simultaneous spatially and single-cell resolved detection of dozens of biological molecules (e.g. proteins) in their native tissue context. As a result, these technologies allow an in-depth analysis of complex systems and diseases such as the tumor microenvironment (Jackson et al. 2020) and type 1 diabetes progression (Damond et al. 2019).
Imaging-based spatial proteomics methods (Moffitt, Lundberg, and Heyn 2022) can be broadly divided into fluorescent cyclic approaches such as tissue-based cyclic immunofluorescence (t-CyCIF) (Lin et al. 2018) and one-step mass-tag based approaches such as multiplexed ion beam imaging (MIBI) (Angelo et al. 2014) and IMC (Giesen et al. 2014).
Of note, the instructions below will focus on the visualization and
exploration of IMC data as an example. However, data
from other technologies such as t-CyCIF or MIBI, which produce
pixel-level intensities and (optionally) segmentation masks, can be
interactively visualized with cytoviewer as long as they
have the appropriate input format (see Section Data input format).
Imaging mass cytometry
IMC, an advancement of CyTOF, combines antibodies tagged with isotopically pure rare earth metals with laser ablation and mass-spectrometry-based detection to produce high-dimensional images (Giesen et al. 2014). It captures the spatial expression of over 40 proteins in parallel at a sub-cellular resolution of 1 μm. Thus, IMC is able to detect cytoplasmic and nuclear localization of proteins.
Highly multiplexed image analysis
To fully leverage the information contained in IMC and multiplexed imaging data in general, computational tools are of key importance.
The main analysis steps, irrespective of the biological question, include 1) Visual inspection of images for quality control, 2) Image pre-processing and segmentation and 3) Single-cell and spatial analysis (Windhager, Bodenmiller, and Eling 2021).
A comprehensive end-to-end workflow for multiplexed image processing and analysis with detailed information for every analysis step can be found here.
Importantly, the cytoviewer package can support,
simplify and improve any of these analysis steps with its easy-to-use
interactive visualization interface in R.
Below we will showcase an example
workflow that highlights the different functionality and potential
application fields of cytoviewer.
Application overview
The cytoviewer interface is broadly divided into image-level (Composite and Channels) and cell-level visualization (Masks). It allows users
to overlay individual images with segmentation masks, integrates well
with SingleCellExperiment and
SpatialExperiment objects for metadata visualization and
supports image downloads (Figure
2B).
Data input format
The cytoviewer package combines objects of SingleCellExperiment,
SpatialExperiment
and cytomapper::CytoImageList classes (from cytomapper)
to visualize image- and cell-level information.
The cytoviewer function takes up to five
arguments.
Firstly, image refers to a CytoImageList
object containing one or multiple multi-channel images where each
channel represents the pixel-intensities of one marker (proteins in
IMC).
Secondly, mask refers to a CytoImageList
object containing one or multiple segmentation masks. Segmentation masks
are defined as one-channel images containing integer values, which
represent the cell ids or background.
Thirdly, the object entry refers to a
SingleCellExperiment or SpatialExperiment
class object that contains cell-specific metadata in the
colData slot.
Lastly, to match information between the CytoImageList
objects and the
SingleCellExperiment/SpatialExperiment object,
two additional spots can be specified:
img_id: a single character indicating thecolData(of theSingleCellExperiment/SpatialExperimentobject) andelementMetadata(of theCytoImageListobject) entry that contains the image identifiers. These image ids have to match between theSingleCellExperiment/SpatialExperimentobject and theCytoImageListobjects.cell_id: a single character indicating thecolDataentry that contains the cell identifiers. These should be integer values corresponding to pixel-values in the segmentation masks.
For more detailed information on the input objects, please refer to the respective documentation (e.g. the vignettes of the cytomapper or SingleCellExperiment/ SpatialExperiment packages).
In the Read in data section, we provide
example code to directly read in images and masks (e.g. in .tiff format)
into a CytoImageList object and create a
SingleCellExperiment object from them, which we can then
visualize with cytoviewer.
Data input variations
The functionality of cytoviewer depends on which input
objects are user-provided. Below we describe the four use
cases in respect to input objects and functionality.
1. Usage of cytoviewer with images, masks and object
The full functionality of cytoviewer can be leveraged when
image, mask and object are
provided, which is the main intended use case.
This allows image-level visualization (Composite and Channels), cell-level visualization, overlaying images with segmentation masks as well as metadata visualization.
2. Usage of cytoviewer with images only
If only the image object is specified, image-level visualization (Composite and
Channels) is possible.
3. Usage of cytoviewer with images and masks
Image-level visualization (Composite and
Channels), overlaying of images with segmentation masks and cell-level visualization is feasible when
image and mask objects are provided.
4. Usage of cytoviewer with masks and object
If mask and object are specified, cell-level visualization as well as metadata
visualization is possible.
Example workflow
Installation
The cytoviewer package can be installed from
Bioconductor via:
if (!requireNamespace("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install("cytoviewer")The development version of cytoviewer can be installed
from Github via:
if (!requireNamespace("remotes", quietly = TRUE))
install.packages("remotes")
remotes::install_github("BodenmillerGroup/cytoviewer")To load the package in your R session, type the following:
Example dataset
For visualization purposes, we will use a toy dataset provided by the cytomapper package.
The dataset contains 3 images of \(100\mu{m}\) x \(100\mu{m}\) dimensions with 362 segmented cells and pixel-intensities for 5 proteins: H3, CD99, PIN, CD8a, and CDH. It is a small subset from a Type 1 Diabetes dataset (Damond et al. 2019).
Pixel-level intensities for all 5 markers (5 channels) are stored in
the pancreasImages object.
The corresponding segmentation masks are stored in the
pancreasMasks object.
All cell-specific metadata are stored in the colData
slot of the corresponding SingleCellExperiment object:
pancreasSCE.
For more detailed information on the dataset, please refer to the
respective documentation (e.g. via ?pancreasImages or the
vignette of the cytomapper
package).
We also provide example code to directly read in images and masks
(e.g. in .tiff format) into a CytoImageList object and
create a SingleCellExperiment object from them in the Read in data section.
# Load example datasets
library(cytomapper)
data("pancreasImages")
data("pancreasMasks")
data("pancreasSCE")
pancreasImages## CytoImageList containing 3 image(s)
## names(3): E34_imc G01_imc J02_imc
## Each image contains 5 channel(s)
## channelNames(5): H3 CD99 PIN CD8a CDH
pancreasMasks ## CytoImageList containing 3 image(s)
## names(3): E34_mask G01_mask J02_mask
## Each image contains 1 channel
pancreasSCE## class: SingleCellExperiment
## dim: 5 362
## metadata(0):
## assays(2): counts exprs
## rownames(5): H3 CD99 PIN CD8a CDH
## rowData names(4): MetalTag Target clean_Target frame
## colnames(362): E34_824 E34_835 ... J02_4190 J02_4209
## colData names(9): ImageName Pos_X ... MaskName Pattern
## reducedDimNames(0):
## mainExpName: NULL
## altExpNames(0):Function call
Here as an example, we call cytoviewer with
image, mask and object data to
leverage all provided functionality.
This setting allows image-level visualization (Composite and Channels), cell-level visualization, overlaying images with segmentation masks as well as metadata visualization.
For further details, please refer to the ?cytoviewer
manual or the Help page within the shiny application.
# Use cytoviewer with images, masks and object
app <- cytoviewer(image = pancreasImages,
mask = pancreasMasks,
object = pancreasSCE,
img_id = "ImageNb",
cell_id = "CellNb")
if (interactive()) {
shiny::runApp(app, launch.browser = TRUE)
}Interface
The cytoviewer interface is divided into a
Header, Sidebar and
Body section (see Figure below).
The Header includes package version information, access to session information and the help page as well as a dropdown-menu for image downloads.
The Body features a Tabset-Panel layout allowing the user to switch between three image modes: Image-level (Composite and Channels) and Cell-level (Mask). Furthermore, the Composite and Mask tabs have zoom controls.
The Sidebar panel is subdivided into four sections: Sample selection, Image-level, Cell-level and General controls.
Image-level visualization
Image visualization control is split into basic and advanced controls.
Basic controls supports the selection of up to six
markers/channels for image display. Each marker has color
control settings that allow the user to set contrast, brightness, gamma
and select a channel color.
In the advanced controls part, the user can choose to
overlay the displayed images with provided segmentation
masks. Outline color and mask thickness can be adjusted by
the user. Moreover, the masks can be outlined by cell-specific metadata
provided in colData slot of the object.
Of note, for categorical and continuous metadata entries the user can choose between discrete colors and continuous color palettes (viridis, inferno, plasma), respectively.
Cell-level visualization
Cell visualization has basic controls.
Here, the user can choose to display the provided segmentation
masks. If an object is provided, the masks can
be colored by cell-specific metadata.
Please note again that for categorical and continuous metadata entries the user can choose between discrete colors and continuous color palettes (viridis, inferno, plasma), respectively.
General controls
General controls is subdivided into an Image appearance and Image filters part.
In the Image appearance section, the user can adjust the scale bar length and include legend/image titles, while the Image filters section allows to control pixel-wise interpolation (default) and apply a Gaussian filter.
Image download
The cytoviewer package supports fast and uncomplicated
image downloads.
Download controls are part of the Header (see Section Interface).
The user can specify a file name, select the image of interest (Composite, Channels, Mask) and the file format (pdf, png). Upon clicking the download button, a pop-window should appear where the user can specify the download location.
Additional Information
Read in data
To conveniently read in images and segmentation masks into a
CytoImageList object and then visualize these using
cytoviewer, the cytomapper package provides
the loadImages function.
The loadImages function returns a
CytoImageList object containing the multi-channel images or
segmentation masks. Refer to the ?loadImages function to
see the full functionality.
As an example, we will read in multi-channel images and segmentation
masks provided by the cytomapper package.
To correctly scale pixel values of the segmentation masks when
reading them in, we will need to set as.is = TRUE. Users
needs to take care that pixel values are scaled correctly in more
complex cases.
library(cytomapper)
# Data directory that stores images and masks in tiff format
data_path <- system.file("extdata", package = "cytomapper")
# Read in images
cur_images <- loadImages(data_path, pattern = "_imc.tiff")
cur_images## CytoImageList containing 3 image(s)
## names(3): E34_imc G01_imc J02_imc
## Each image contains 5 channel(s)
# Read in masks
cur_masks <- loadImages(data_path, pattern = "_mask.tiff", as.is = TRUE)
cur_masks## CytoImageList containing 3 image(s)
## names(3): E34_mask G01_mask J02_mask
## Each image contains 1 channelAdd metadata
To link images between the two CytoImageList objects and
the corresponding SingleCellExperiment object, the image
ids need to be added to the elementMetadata slot of the
CytoImageList objects.
names(cur_images)## [1] "E34_imc" "G01_imc" "J02_imc"
names(cur_masks)## [1] "E34_mask" "G01_mask" "J02_mask"
mcols(cur_masks)$ImageNb <- mcols(cur_images)$ImageNb <- c("E34", "G01", "J02")Add channel names
To access the correct images in the multi-channel
CytoImageList object, the user needs to set the correct
channel names. For this, the cytomapper package provides
the ?channelNames getter and setter function:
channelNames(cur_images) <- c("H3", "CD99", "PIN", "CD8a", "CDH")Generating the object
Based on the processed segmentation masks and multi-channel images,
cytomapper can be used to measure cell-specific intensities
and morphological features. Here, these features are stored in form of a
SingleCellExperiment object.
cur_sce <- measureObjects(image = cur_images,
mask = cur_masks,
img_id = "ImageNb")
cur_sce## class: SingleCellExperiment
## dim: 5 362
## metadata(0):
## assays(1): counts
## rownames(5): H3 CD99 PIN CD8a CDH
## rowData names(0):
## colnames: NULL
## colData names(8): ImageNb object_id ... m.majoraxis m.eccentricity
## reducedDimNames(0):
## mainExpName: NULL
## altExpNames(0):Run cytoviewer
Next, we can again call cytoviewer with the generated
image, mask and object data and
leverage all provided functionality.
# Use cytoviewer with images, masks and object
app_1 <- cytoviewer(image = cur_images,
mask = cur_masks,
object = cur_sce,
img_id = "ImageNb",
cell_id = "object_id")
if (interactive()) {
shiny::runApp(app_1, launch.browser = TRUE)
}For more detailed information on the input objects, please refer to the respective documentation (the vignettes of the cytomapper or SingleCellExperiment/ SpatialExperiment packages).
Session info
## R version 4.4.1 (2024-06-14)
## Platform: aarch64-apple-darwin20
## Running under: macOS Sonoma 14.5
##
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.0
##
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
##
## time zone: UTC
## tzcode source: internal
##
## attached base packages:
## [1] stats4 stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] cytomapper_1.14.0 SingleCellExperiment_1.24.0
## [3] SummarizedExperiment_1.32.0 Biobase_2.62.0
## [5] GenomicRanges_1.54.1 GenomeInfoDb_1.38.8
## [7] IRanges_2.36.0 S4Vectors_0.40.2
## [9] BiocGenerics_0.48.1 MatrixGenerics_1.14.0
## [11] matrixStats_1.3.0 EBImage_4.44.0
## [13] cytoviewer_1.5.0 BiocStyle_2.30.0
##
## loaded via a namespace (and not attached):
## [1] bitops_1.0-7 gridExtra_2.3 rlang_1.1.4
## [4] magrittr_2.0.3 svgPanZoom_0.3.4 shinydashboard_0.7.2
## [7] compiler_4.4.1 png_0.1-8 systemfonts_1.1.0
## [10] fftwtools_0.9-11 vctrs_0.6.5 pkgconfig_2.0.3
## [13] SpatialExperiment_1.12.0 crayon_1.5.3 fastmap_1.2.0
## [16] magick_2.8.3 XVector_0.42.0 fontawesome_0.5.2
## [19] utf8_1.2.4 promises_1.3.0 rmarkdown_2.27
## [22] ggbeeswarm_0.7.2 ragg_1.3.2 purrr_1.0.2
## [25] xfun_0.45 zlibbioc_1.48.2 cachem_1.1.0
## [28] jsonlite_1.8.8 later_1.3.2 rhdf5filters_1.14.1
## [31] DelayedArray_0.28.0 Rhdf5lib_1.24.2 BiocParallel_1.36.0
## [34] terra_1.7-78 jpeg_0.1-10 tiff_0.1-12
## [37] parallel_4.4.1 R6_2.5.1 bslib_0.7.0
## [40] RColorBrewer_1.1-3 jquerylib_0.1.4 Rcpp_1.0.12
## [43] bookdown_0.39 knitr_1.47 httpuv_1.6.15
## [46] Matrix_1.7-0 nnls_1.5 tidyselect_1.2.1
## [49] viridis_0.6.5 abind_1.4-5 yaml_2.3.8
## [52] codetools_0.2-20 miniUI_0.1.1.1 lattice_0.22-6
## [55] tibble_3.2.1 shiny_1.8.1.1 evaluate_0.24.0
## [58] archive_1.1.8 desc_1.4.3 shinycssloaders_1.0.0
## [61] pillar_1.9.0 BiocManager_1.30.23 generics_0.1.3
## [64] sp_2.1-4 RCurl_1.98-1.14 ggplot2_3.5.1
## [67] munsell_0.5.1 scales_1.3.0 xtable_1.8-4
## [70] glue_1.7.0 tools_4.4.1 colourpicker_1.3.0
## [73] locfit_1.5-9.10 fs_1.6.4 rhdf5_2.46.1
## [76] grid_4.4.1 colorspace_2.1-0 GenomeInfoDbData_1.2.11
## [79] raster_3.6-26 beeswarm_0.4.0 HDF5Array_1.30.1
## [82] vipor_0.4.7 cli_3.6.3 textshaping_0.4.0
## [85] fansi_1.0.6 viridisLite_0.4.2 S4Arrays_1.2.1
## [88] svglite_2.1.3 dplyr_1.1.4 gtable_0.3.5
## [91] sass_0.4.9 digest_0.6.36 SparseArray_1.2.4
## [94] rjson_0.2.21 htmlwidgets_1.6.4 memoise_2.0.1
## [97] htmltools_0.5.8.1 pkgdown_2.0.9 lifecycle_1.0.4
## [100] mime_0.12