Simulated Transient Images for Non-Line-of-Sight Imaging and Recognition

Wenzheng Chen (graduate student, University of Toronto), Fangyin Wei (graduate student), Kyros Kutulakos (faculty, University of Toronto), Szymon Rusinkiewicz (faculty), Felix Heide (faculty)

Computer Science

Objects hidden by occluders (other objects), are considered lost in the images acquired by conventional camera systems, making it impossible to see and understand such hidden objects. Non-line-of-sight (NLOS) methods aim to recover information about hidden scenes, which could help make medical imaging less invasive, improve the safety of autonomous vehicles, and potentially enable capturing unprecedented high-definition RGB-D (red, green, blue-depth) data sets that include geometry beyond the directly visible parts.

In the image and videos, we show transient images rendered by our rendering pipeline using hardware-accelerated rasterization to create a 3D image. In the image, our approach renders a car model. These transient images are part of our results published in ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2020) called “Learned Feature Embeddings for Non-Line-of-Sight Imaging and Recognition.” In this paper, we proposed a method that leverages physical models to learn hidden scene feature representations tailored to both reconstruction and recognition tasks such as classification or object detection. Our method, trained on our simulated transient images, can generalize to unseen classes and unseen real-world scenes.

Simulated Bioluminescence

Natalie O’Leary ’21, Theodore Kim (faculty, Yale University)

Computer Science, Music Theater

In nature, glowing waves occur when certain types of algae are stimulated, catalyzing a chemical reaction that briefly emits a bright blue light. Though the light from each individual alga is ephemeral, the chain reaction from the force of a wave can create an effect that makes the entire wave appear to glow. This phenomenon lends itself naturally to CGI water simulation, which already represents fluid as a dynamic container full of tiny particles. In order to simulate the presence of these bioluminescent algae in a fluid simulation, some of these particles can be treated like algae rather than water particles. By calculating the acceleration of each particle at every frame of the simulation, you can derive the force on each particle to determine when any given particle is above the force threshold that would cause it to glow. This force comes from the simulated wave motion, which is created by injecting a velocity grid into the fluid tank that mimics the natural movement of the ocean. Waves break in rolling pipelines like the one shown here when the water further from the shore and closer to the surface has a greater velocity, thus overtaking and crashing over the water in front of it. When it occurs naturally, this difference in velocity comes from the slope of the ocean floor, which naturally slows the pace of incoming water as it becomes shallower. In a computer simulation however, this velocity can be artificially generated by simply applying differing velocities to the correct parts of the tank. Thus, the force of the wave’s acceleration excites the algae particles, which in turn emit a bright blue light before decaying back to darkness.

Development of the Early Embryo

Madeleine Chalifoux (graduate student), Eszter Posfai (faculty), Stanislav Shvartsman (faculty)

Molecular Biology, Chemical and Biological Engineering

We often hear about issues related to embryonic development in the news, but rarely do we stop and think about what the developing embryo actually looks like in the days following fertilization. This timelapse video of a mouse embryo developing from 8 to 32 cells shows the development of an embryo beginning two days after fertilization. Each frame of the movie represents 15 minutes. In the three panels (from left to right), we observe individual cell nuclei, individual cell membranes, and an important protein that will facilitate whether a cell gives rise to the future fetus or to extraembryonic support tissues, such as the placenta. Approximately 30% of cells in this embryo will form the future fetus, while the rest will contribute to extraembryonic tissues. As the embryo grows and divides, notice the bursts of nuclear activity with each cell division in the right-most panel. The embryo shown in this movie is comparable to a human embryo three to five days post-fertilization. Due to the many similarities between mouse and human embryogenesis, the mouse embryo can be used as a model to understand the development of the human embryo.

Gravity Drives the Formation of Beautiful Patterns on Soft Materials

Jonghyun Hwang (graduate student), Malcolm Slutzky ‘22 (graduate student, University of Chicago), Janine K. Nunes (research scholar), Howard A. Stone (faculty)

Mechanical and Aerospace Engineering

When a more dense material lies on top of a less dense material, gravity destabilizes the interface between the two. The less dense material will try to find its way to rise, and the more dense material will fall due to gravity. Such a process is called the Rayleigh-Taylor instability. If the more dense material is “viscoelastic,” meaning that the material shows the characteristics of both solid and liquid, we can observe beautiful patterns that appear over time. The material will flow like a liquid while trying to hold its shape like a solid. Through this competition, we are able to capture a variety of patterns that arise at the interface.

The Sun

Vivek Vijayakumar ’25

Astrophysical Sciences

An animation of a part of the sun’s chromosphere (part of the sun’s atmosphere), taken over 30 minutes with an 80-millimeter telescope and a specialized filter. The animation shows the movement of plasma within the solar disk and in the prominences, large regions of plasma protruding from the sun.

Two Strings of Pearls

Jessica L. Wilson (graduate student), Amir A. Pahlavan (faculty, Yale School of Engineering), Martin A. Erinin (postdoctoral research associate), Camille Duprat (faculty, École Polytechnique) Luc Deike (faculty), and Howard A. Stone (faculty)

Mechanical and Aerospace Engineering

Silicone oil drops on fibers align when exposed to an airflow that is perpendicular to the fibers. In a paper under review, Wilson et al. use an imaging technique called Particle Image Velocimetry (PIV) to explore the aerodynamic interactions between the drops that result in alignment. In short, they discover that there are two mechanisms for alignment. One is short-ranged and impacts drops upstream, while the other is long-ranged and impacts drops downstream.

Sample Asteroid Bennu in 360

James Tralie ’19 (planetary science video producer, NASA), Jonathan North (multimedia specialist, NASA), Walt Feimer (animation manager, NASA), Michael Lentz (art director and artist, NASA), Kel Elkins (data visualizer, NASA’s Scientific Visualization Studio at the Goddard Flight Center)

NASA’s first asteroid sample return mission, OSIRIS-REx, successfully achieved its daring mission of touching the surface of the asteroid Bennu in October, 2020, to TAG (touch-and-go) on Asteroid Bennu. OSIRIS-Rex collected a sample for return to Earth this September. Experience the sample collection event in 360 and watch as OSIRIS-REx contacts the rocky surface of sample site Nightingale on Asteroid Bennu.

Visualizations on a 3D Grid

Eliot Feibush (computational scientist, PPPL), James Stone (faculty)

Princeton Institute for Computational Science & Engineering, Astrophysics

The Athena simulation computes the interaction of two gases with different densities in a nebula. Developed in the Astrophysical Sciences Department at Princeton University, data is calculated on a 3D grid. To explore, verify, and communicate the data we apply numerical thresholding to reveal interior cells of interest. Moving three cutting planes through the geometry is very effective for examining all the cells within the computer grid. These techniques can be applied to any application that produces data on a 3D compute grid. The visualization was produced at the Princeton Institute for Computational Science and Engineering.

Neural Nano-Optics

Ethan Tseng (graduate student), Felix Heide (faculty)

Computer Science

Conventional cameras use bulky refractive lenses to focus and steer light onto a camera sensor. Our research into neural nano-optics allows us to shrink camera lenses by more than 500,000 times. Here, we show a video of blooming flowers that were captured with our nano-camera prototype. The flowers were imaged with fabricated metasurface lenses and the raw images were processed with deep learning.

Frozen Instabilities

Lauren Dreier (graduate student), Yuchen Xi (graduate student), Pierre-Thomas Brun (faculty)

Liquids and Elasticity Lab, Chemical and Biological Engineering

Thermal image of polylactic acid (PLA) extruding from a 3D printer nozzle (0.4 mm diameter). The material is extruded in excess so that the resulting thread buckles. Coils are formed and preserved as the PLA cools down.