Cryo-electron microscopy (cryoEM) refers to the use of transmission electron microscopes to probe the structures of frozen biological molecules embedded in a thin film of amorphous ice. This embedding preserves the native structure of the molecules (usually proteins, or protein/nucleic-acid complexes with sizes in the nanometer range) and allows researchers to recover images of molecular machines at extremely high resolution (Nogales and Scheres, 2015). Furthermore, averaging of particles extracted from each image or image volume allows reconstruction in three dimensions (3D) (according to the central slice theorem, DeRosier and Klug, 1968), significantly improving the resolution and interpretability of structures, and allowing the researcher to derive mechanistic insights based on the positions of amino acid side chains, drug binding sites, and a plethora of other structural elements which may or may not be present. More recently, cryo-electron tomography (cryoET) has emerged as a full-fledged complement to the traditional (called "single-particle") method of using cryoEM: cryoET relies on the acquisition of images of the same object at many different angles such that researchers can reconstruct 3D tomographic volumes of the sample, whereas single-particle cryoEM uses many 2-dimensional projections of different (but identical) objects to reconstruct 3D averages of the sample. These maps are, in reality, maps of the average coulomb potential as incident electrons are scattered by the charges in the sample, generating phase (mostly) and amplitude contrast which can be detected by specialized cameras (Marques et al., 2019, McMullan et al., 2016). While single-particle cryoEM has been used to revolutionize the field of structural biology (Kühlbrandt, 2014) and is of enormous use for purified, homogenous samples, cryoET is much more capable of describing the true context of molecular machinery: the ability to capture 3D volumes allows the researcher to directly image cellular material, capturing unique structures, unexpected assemblies, and molecular machines unsuitable for traditional analysis methods (Beck and Baumeister, 2016). Startlingly, you can actually see what processes are going on inside cells and how macromolecules are interacting (fig. 1). Nonetheless, the enormous complexity of the intracellular environment makes analysis extraordinarily difficult, and extremely low signal-to-noise ratios make things more challenging still. As such, methods to alleviate this analysis problem are an important concern in the field (Frangakis, 2021). Deconvolution of tomograms can provide better interpretability, but great gains are being made with denoising algorithms: this work will briefly aim to introduce the state-of-the-art tools for denoising cryo-electron microscopy data with a focus on implementations of Noise2Noise, a method to denoise based only on independently noisy observations of the same subject.

Figure 1: Examples of denoised cryo-electron tomography data. A: JCVI-Syn3A minimal cell imaged by cryo-electron tomography by Gilbert et al. (2021). Imaged at $4.265\text{Å}/$$4.265\text{Å}/$4.265"Å"//4.265 \AA / pixel. Image was obtained from deposition in EMPIAR-10685. B: Tomogram of an axon from a human cerebral organoid (expressing GFP-ESYT1). Imaged at $3.7\text{Å}$$3.7\text{Å}$3.7"Å"3.7 \AA /pixel by Hoffmann et al. (2021). Image obtained from EMPIAR-10804 deposition. C: Yeast nucleus tomogram from Croxford et al. (2021) imaged at $5.31\text{Å}$$5.31\text{Å}$5.31"Å"5.31 \AA /pixel. Image was obtained from EMPIAR-10762.

The development of the Noise2Noise algorithm by Lehtinen et al. (2018) was a transformative moment for the denoising of cryoET data. CryoET's extremely low signal-to-noise ratios and crowded molecular environments have long been an issue but the inability to obtain clean ground truth data has prevented the useful application of many supervised deep-learning approaches and other similar strategies: since Noise2Noise makes use of images of the same object corrupted independently by noise the lack of clean ground truth data is no longer a problem.

Many contemporary denoising algorithms trained convolutional neural networks (CNNs) with pairs of corrupted images and clean data, optimizing a regression model towards a learning of the noise such that corrupted observations are mapped to the clean versions. Noise2Noise changed the field by making clean ground truth unnecessary. Equation 1 describes the empirical minimization task applied in the Noise2Noise algorithm where both the inputs and the targets are now drawn from a corrupted distribution, conditioned on the unseen clean target $y_i$$y_i$y_(i)y_{i} such that $\mathbb{E}\{{\hat{y}}_i\mid {\hat{x}}_i\}=y_i$$\mathbb{E}\left\{{\widehat{y}}_i\mid {\widehat{x}}_i\right\}=y_i$E{ hat(y)_(i)∣ hat(x)_(i)}=y_(i)\mathbb{E}\left\{\hat{y}_{i} \mid \hat{x}_{i}\right\}=y_{i} for the input-target pairs $(x_i,y_i)$$\left(x_i,y_i\right)$(x_(i),y_(i))\left(x_{i}, y_{i}\right). The algorithm depends upon a property of $L_2$$L_2$L_(2)L_{2} minimization where the estimate, upon maximization, remains unchanged if we replace the targets with random numbers whose expectations match the targets (Lehtinen et al., 2018).

For the $L_2$$L_2$L_(2)L_{2} loss $L(z,y)=(z-y{)}^2$$L(z,y)=(z-y{)}^2$L(z,y)=(z-y)^(2)L(z, y)=(z-y)^{2}, the smallest average deviation from a set of unreliable elements $(y_1,y_2,\dots )$$\left(y_1,y_2,\dots \right)$(y_(1),y_(2),dots)\left(y_{1}, y_{2}, \ldots\right) is found at the arithmetic mean of the observations (equation 2). Based on the property described above, this is true no matter the distribution of $ys$$ys$ysy{s}: This allowed the authors to make the inference that the training targets of a neural network can be corrupted with zero-mean noise without altering what is learned by the network (Lehtinen et al., 2018).

In cryoET, images of the same object are collected in tilt series at defined angles (Hagen et al., 2017): we can use Noise2Noise methods on these 2D images (whose information is interpolated to generate $3D$$3D$3D3 D tomograms) to reduce the effects of noise and improve our ability to visualize molecular machines and cellular ultrastructure. Furthermore, there are some indications that denoising using cryoET movie frames can sometimes provide superior results to denoising based on tilts.

CryoET denoising has been implemented in the Warp refinement pipeline based on the Noise2Noise concept (Tegunov and Cramer, 2019, Tegunov et al. 2020). The authors relied on the separation of odd and even tilts or paired movie frames to use as independently noisy inputs. This implementation has achieved remarkable results in some cases, but it is not trivial to perform and it's often difficult to determine why a particular denoising operation has failed to produce a result that would be called satisfactory. Furthermore, it is theorized that Warp denoising may contribute to blurring outside of cellular environments e.g., for membrane-embedded or secreted proteins (but further investigation would be required to make a more informed conclusion). Warp, and its sister software M, also make special use of Noise2Noise denoising to directly denoise 3D average density maps: the calculation of resolution in cryoEM depends on so-called gold standard Fourier shell correlation (FSC), where random halves of the data are independently refined and the reconstructed maps compared at a FSC of $0.143$$0.143$0.1430.143 (Scheres and Chen, 2012, Rosenthal and Henderson, 2003, Chen et al., 2013), and as such these independent half-maps can be taken as input for Noise2Noise-based denoising (called Noise2Map in Warp). This map denoising has been shown to improve map interpretability (Tegunov et al., 2020).

Bepler et al. (2019) devised a similar approach (Topaz-Denoise) where tomograms are also split into even and odd tilts or paired movie frames to implement Noise2Noise. Their method has been primarily, and to great success, applied to single-particle analysis cryoEM micrographs but the software also includes tomography-specific pretrained networks. Similarly, JANNI (Just Another Noise2Noise Implementation, Wagner et al. 2020) is an implementation designed for cryoEM data applied primarily to SPA, though recent updates to the crYOLO neural network particle picker (Wagner et al. 2019) allow its use to be integrated with picking. Cryo-CARE (Bucholtz et al. 2018) applies a Noise2Noise methodology originally developed by Weigert et al. (2017) for fluorescence microscopy data to cryo-electron tomograms. Finally, Noise2Void, from Krull et al. (2019), is said to take the idea of Noise2Noise a step further and allows denoising of images where only single noisy acquisitions are available.

Despite the number of available packages, no single solution has become dominant and indeed no single solution seems to work perfectly (or even adequately) in every case. Space exists for new Noise2Noise implementations and/or more powerful denoising techniques, and new data processing strategies may allow better denoising based on better representations of the sample. The development of neural radiance fields (Mildenhall et al., 2020), for instance, suggests neural-network based interpolation may be a powerful tool to reconstruct 3D image volumes. Denoising does not yet fulfill its promise and unlock the secrets of cryo-electron tomograms' extraordinarily high resolution, but its potential to accelerate structural biology and visualize how the fundamental molecular machines of our cells interact is unmatched.

Beck, Martin, and Wolfgang Baumeister. "Cryo-Electron Tomography: Can It Reveal the Molecular Sociology of Cells in Atomic Detail?" Trends in Cell Biology 26, no. 11 (November 2016): 825–37.

Bepler, Tristan, Kotaro Kelley, Alex J. Noble, and Bonnie Berger. "Topaz-Denoise: General Deep Denoising Models for CryoEM and CryoET." Nature Communications 11, no. 1 (October 15, 2020): $5208.$$5208.$5208.5208 .

Buchholz, Tim-Oliver, Mareike Jordan, Gaia Pigino, and Florian Jug. "Cryo-CARE: Content-Aware Image Restoration for Cryo-Transmission Electron Microscopy Data." ArXiv, October 15, $2018.$$2018.$2018.2018 .

Chen, Shaoxia, Greg McMullan, Abdul R. Faruqi, Garib N. Murshudov, Judith M. Short, Sjors H.W. Scheres, and Richard Henderson. "High-Resolution Noise Substitution to Measure Overfitting and Validate Resolution in 3D Structure Determination by Single Particle Electron Cryomicroscopy." Ultramicroscopy 135 (December 2013): 24–35.

Croxford, Matthew, Michael Elbaum, Muthuvel Arigovindan, Zvi Kam, David Agard, Elizabeth Villa, and John Sedat. "Entropy Regularized Deconvolution of Cellular Cryo-Transmission Electron Tomograms." bioRxiv, October 7, $2021.$$2021.$2021.2021 .

De Rosier, D. J., and A. Klug. "Reconstruction of Three Dimensional Structures from Electron Micrographs." Nature 217, no. 5124 (January 1968): 130-34.

Frangakis, Achilleas S. "It's Noisy out There! A Review of Denoising Techniques in Cryo-Electron Tomography." Journal of Structural Biology 213, no. 4 (December 1, 2021): $107804.$$107804.$107804.107804 .

Gilbert, Benjamin R., Zane R. Thornburg, Vinson Lam, Fatema-Zahra M. Rashid, John I. Glass, Elizabeth Villa, Remus T. Dame, and Zaida Luthey-Schulten. "Generating Chromosome Geometries in a Minimal Cell From Cryo-Electron Tomograms and Chromosome Conformation Capture Maps." Frontiers in Molecular Biosciences 8 (2021).

Hagen, Wim J.H., William Wan, and John A.G. Briggs. "Implementation of a Cryo-Electron Tomography Tilt-Scheme Optimized for High Resolution Subtomogram Averaging." Journal of Structural Biology 197, no. 2 (February 2017): 191-98.

Hoffmann, Patrick C, Stefano L Giandomenico, Iva Ganeva, Michael R Wozny, Magdalena Sutcliffe, Madeline A Lancaster, and Wanda Kukulski. "Electron Cryo-Tomography Reveals the Subcellular Architecture of Growing Axons in Human Brain Organoids." Edited by Giulia Zanetti and Anna Akhmanova. ELife 10 (October 26, 2021): e70269.

Krull, Alexander, Tim-Oliver Buchholz, and Florian Jug. "Noise2Void - Learning Denoising from Single Noisy Images." ArXiv:1811.10980 [Cs], April 5, $2019.$$2019.$2019.2019 .

Kühlbrandt, Werner. "The Resolution Revolution." Science 343, no. 6178 (March 28, 2014): 1443-44.

Lehtinen, Jaakko, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. "Noise2Noise: Learning Image Restoration without Clean Data." ArXiv:1803.04189 [Cs, Stat], October 29, $2018.$$2018.$2018.2018 .

Marques, Mayra A, Michael D Purdy, and Mark Yeager. "CryoEM Maps Are Full of Potential." Current Opinion in Structural Biology, Biophysical and computational methods - Part B, 58 (October 1 , 2019): 214-23.

McMullan, G., A. R. Faruqi, and R. Henderson. "Direct Electron Detectors." Methods in Enzymology 579 (2016): 1-17.

Mildenhall, Ben, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis." ArXiv:2003.08934 [Cs], August 3, $2020.$$2020.$2020.2020 . Nogales, Eva, and Sjors H. W. Scheres. "Cryo-EM: A Unique Tool for the Visualization of Macromolecular Complexity." Molecular Cell 58, no. 4 (May 21, 2015): 677-89.

Rosenthal, Peter B., and Richard Henderson. "Optimal Determination of Particle Orientation, Absolute Hand, and Contrast Loss in Single-Particle Electron Cryomicroscopy." Journal of Molecular Biology 333, no. 4 (October 31, 2003): 721-45.

Scheres, Sjors H.W., and Shaoxia Chen. "Prevention of Overfitting in Cryo-EM Structure Determination." Nature Methods 9, no. 9 (September 2012): 853-54.

Tegunov, Dimitry, and Patrick Cramer. "Real-Time Cryo-Electron Microscopy Data Preprocessing with Warp." Nature Methods 16, no. 11 (November 2019): 1146-52.

Tegunov, Dimitry, Liang Xue, Christian Dienemann, Patrick Cramer, and Julia Mahamid. "MultiParticle Cryo-EM Refinement with M Visualizes Ribosome-Antibiotic Complex at $3.5\text{Å}$$3.5\text{Å}$3.5"Å"3.5 \AA in Cells." Nature Methods 18, no. 2 (February 2021): 186-93.

Wagner, Thorsten, Felipe Merino, Markus Stabrin, Toshio Moriya, Claudia Antoni, Amir Apelbaum, Philine Hagel, et al. "SPHIRE-CrYOLO Is a Fast and Accurate Fully Automated Particle Picker for Cryo-EM." Communications Biology 2, no. 1 (June 19, 2019): 1-13.

Wagner, Thorsten, and Stefan Raunser. "The Evolution of SPHIRE-CrYOLO Particle Picking and Its Application in Automated Cryo-EM Processing Workflows." Communications Biology 3, no. 1 (December 2020): $61.$$61.$61.61 .

Weigert, Martin, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, et al. "Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy." Nature Methods 15, no. 12 (December 2018): 1090–97.