Rewriting a Deep Generative Model: An Overview
Last updated
Last updated
In the words of the authors,
"Deep network training is a blind optimization procedure where programmers define objectives but not the solutions that emerge. In this paper, we ask if deep networks can be created in a different way. Can the rules in a network be directly rewritten?"
The blind optimization procedure is investigated quantitatively in the paper "Deep Ensembles: A Loss Landscape Perspective" by Fort et al. Sayak Paul, and I explored this paper in this report.
The usual recipe for creating a deep neural network is to train such a model on a massive dataset with a defined objective function. This takes a considerable amount of time and is expensive in most cases. The authors of Rewriting a Deep Generative Model propose a method to create new deep networks by rewriting the rule of an existing pre-trained network as shown in figure 1. By doing so, they wish to enable novice users to easily modify and customize a model without the training time and computational cost of large-scale machine learning.
Figure 1: Rewriting GAN without training to remove the watermark, to add people, and to replace the tower with the tree. (Source)
They do so by setting up a new problem statement: manipulation of specific rules encoded by a deep generative model.
If you are unfamiliar with deep generative models, here is my take on the same.
Deep generative models such as a GAN can learn rich semantic and physical rules about a target distribution(faces, etc.). However, it usually takes weeks to achieve state of the art GAN on any dataset. If the target distribution changes by some amount, retraining a GAN would be a waste of resources. However, what if we directly change some trained GAN rules to reflect the target distribution change? Thus by rewriting a GAN:
We can build a new model without retraining, which is a more involved task.
From the perspective of demystifying deep neural nets, the approach to edit a model gives new insight about the model and how semantic features are captured.
It can also provide some insight into the generalization of deep models to unseen scenarios.
Unlike conventional image editing tools where the desired change is applied on a single image, by editing a GAN, one can apply the edit on every image generated.
Using this tool, one can build new generative models without domain expertise, training time, and computational expense.
