Transforming a CNC laser engraver into a plotter to draw artwork generated by means of deep learning
May 2019 - August 2019
(Nov 7, 2024) Overview:
It's been almost 4 years since I first started working on this, for which I only worked a couple months. I am not sure if I will follow through with this fully, since at the time GAN's were in their infancy. Nowadays with the state of generative AI, it would be cool to attempt this again for artistic purposes, but it is less impressive and quite "cliche" to say you worked on an AI project; most people will just think you used a denoising diffusion model with CLiP embeddings and wont consider it "art", and most won't see that you custom-made the CNN + dense layers + GAN using hardware vectorized C because you don't own a GPU, and have to webscrape by hand. That was the original intention, at least at the time :'). It's doubtful that I will re-visit this but if I have absolutely nothing else to do then perhaps.
(Nov 11, 2020) Overview:
Generative adversarial network (GAN) is a type of deep learning architecture in which there are two networks being trained. The first is called a discriminator, which takes a set of data (usually an image) and determines whether it is among the set of data which is real, or among the set of data which is generated (or 'fake'). The second is a generator, who's job it is to generate fake data from random noise. These two networks train with/against one another, so both the discriminator and the generator get increasingly better at their respective tasks. The end result of the GAN is only the generator network, which is a network capable of generating realistic sets of data purely from Gaussian noise.
Fig 1: A general diagram of how a GAN functions
This sort of generative network has many applications, including the non-technical one being art.
Fig 2: The 2D plotter being used
A CNC laser engraver transformed into a 2D plotter (above) is used to plot the edges of some arbitrary GAN's output. The arbitrary GAN is initially trained on landscape portraits. The output (resembling a drawing/painting/photograph of a landscape) will be fed through a series of digital filters and an accompanying MATLAB script to extract edges (of varying degrees) and other features of the image which would seem fit to be plotted. The image is then transformed into a grbl file to be plotted.
Fig 3: Example original image. Of course this is not a landscape, but this was the test image used when I worked on the MATLAB script many years ago (independent of this project)
Fig 4: ONLY MATLAB script used: extract many edges
Fig 5: ONLY MATLAB script used: extract fewer edges
Fig 6: Close up of the previous figure
Currently, the post-processing of the output is complete, the plotter is assembled, and the I am working on programming, training, and tuning the GAN.
Personal Statement:
I have always been fascinated by how deep learning can be applied to art. I gained motivation for this project in 2017, when I met Robbie Barrat at the Intel ISEF in Los Angeles. Barrat, previously a AI researcher at NVIDIA and Stanford, is now an AI artist and fashion designer. He really piqued my interest in deep learning during my time in LA, which motivated me to pursue statistics, deep learning, and computer vision as some of my electrical engineering concentration topics.
The project is still in progress, but on hiatus due to focusing on graduate school. But I will be returning to it in the near future.
Literature:
There are a lot of problems with standard GAN's, like mode collapse, knowing the 'proper' ratio of training iterations for generator vs. discriminator, etc. One partial solution is the Wasserstein GAN, which uses the Wasserstein metric as the loss function as opposed to the standard KL Divergence (used to derive cross-entropy loss) and Jenson-Shannon Divergence.
Media Citation(s):
GAN diagram - https://medium.com/ml-everything/generating-letters-using-generative-adversarial-networks-gans-161b0be3c229
Test image (girl) - http://clipart-library.com/clipart/6Ty5q4Bkc.htm