Update: Presentation date is pushed back to Week 8 (March 22)
Interpolating through the latent space of a trained StyleGAN2 model creates an effect where images morph seamlessly into one another. Typically, data sets are comprised of consistent examples of some type of subject matter (e.g. faces, wild animals, dogs, cats) where the goal is to get the model to generate accurate versions of the subjects. For these models, the background in the image of the subject isn’t important; the model just needs to learn what the main thing in the photo is. Once the model is trained, traversing the latent space generates virtually infinite examples of whatever the model was trained on.
It’s important to note that although the morphing effect in these videos is really aesthetically interesting, the primary goal for a lot of people is to train models to generate still images that accurately represent the data the model was trained on.
https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/videos/interpolations-ffhq.mp4
https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/videos/interpolations-afhqwild.mp4
https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/videos/interpolations-afhqdog.mp4
https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/videos/interpolations-afhqcat.mp4
As artists working with these tools, we may be less concerned with “accuracy”, and more with the types of imagery these tools afford us. For this assignment, train a StyleGAN2 model not on a specific type of subject, but on a specific scene. Create a data set of images where the scene or background is constant, but the subjects in the foreground are always changing. Once we’ve trained the model, our latent space interpolations should depict an uncanny distortion of time and motion within that scene.
The latent space interpolation below shows a top-down view of my hands typing on my keyboard while objects appear and disappear. In order to create this, I filmed my hands from above for a few minutes while I typed and moved objects around.
interpolation-linear-seeds_21_22_32_37_39_21-30fps.mp4
My goal was not to train a model to accurately synthesize images of my hands on a keyboard. Rather, my goal was to depict an uncanny distortion of time and motion. In the video, my hands morph around the keyboard in a way that sort of resembles typing (but isn’t really typing).
In this example by Kalan Strauss, a model was trained on images of clouds. (Kalan simulated 3D clouds inside Unity and then took screenshots for the data set, but he could also have taken photos of clouds in the actual sky). The shifting clouds sort of resemble the way clouds naturally move and change in the sky.
[https://dougrosman.github.io/cvml-saic/final/kalan/](https://dougrosman.github.io/cvml-saic/final/kalan/video/kalan-clouds-latent-feedback512.mp4)
https://dougrosman.github.io/cvml-saic/final/kalan/