When we carefully observe the images generated by the neural network, we will often see flaws in the checkerboard pattern. This phenomenon is more obvious in some special scenes, especially in recent models.
More importantly, this kind of checkerboard artifact is more obvious in the brighter the color area. Why is this? Don’t neural networks like bright colors? In fact, the cause of these artifacts is very simple, but it is also very simple to solve.
Deconvolution and overlap
When we use neural networks to generate images, we usually create them from low-resolution, high-level descriptions. The web can represent a rough image and fill in the details.
In order to do this, we need to convert a low-resolution image into a high-resolution image, and this process is performed through a deconvolution operation. Simply put, the deconvolution layer allows the model to use each point in the small image to "draw" a larger square.
However, deconvolution can easily generate "uneven overlap", making some areas have more abstract results than others. Especially when the size of the kernel (the size of the output window) cannot be divisible by the step size (the distance between the top points), the deconvolution will have uneven overlap. Although in theory, the network can avoid this phenomenon through weights, but in reality, it is very difficult for neural networks to completely eliminate this phenomenon.
The overlapping pattern is also two-dimensional. Multiplying uneven overlap on the two axes can create checkerboard artifacts of varying degrees.
In fact, this uneven overlap is more obvious in two dimensions, because when two patterns are multiplied, the unevenness is the original square. For example, in one dimension, the output of a deconvolution with a step size of 2 and a size of 3 is twice the input, but it becomes four times in two dimensions.
Nowadays, neural networks usually use multi-layer deconvolution when creating images, and iteratively create larger images from lower resolutions. Although the deconvolution of these stacks can eliminate artifacts, they are usually mixed and produce checkerboard artifacts at multiple scales.
These artifacts are more conspicuous when outputting special colors. Because neural network layers usually have deviations (a learned value added to the output), it is easy to output average colors. If a color deviates from the average color, the greater the effect of deconvolution.
Overlap and learn
Theoretically, our model can learn to carefully write uneven overlapping positions so that the output can be balanced.
But it is difficult to achieve balance, especially when there are multiple channels. To avoid artifacts, it is necessary to strictly limit the filters that may be used, sacrificing part of the performance of the model. In practice, the neural network has to go through a long learning period to completely avoid these artifacts.
In fact, it is not only that models with uneven overlap will not learn to avoid this situation, but those with uniform overlap will often learn kernels and cause similar artifacts. Although this is not the default situation for uneven overlap, it is easy to cause artifacts for average overlap convolution.
For filters, avoiding artifacts completely is still a very important limitation. In fact, artifacts will still appear in these models, although they are not obvious.
There may be many factors that can cause this situation. For example, in the case of generating a confrontation network, a key element may be the discriminator and its gradient, but the main problem is deconvolution. In the best case, deconvolutions are very fragile because they can easily exhibit artificial artifacts, even if they are small in size. In the worst case, the generation of artifacts is the default behavior of deconvolution.
So, are there other upsampling methods that can avoid artifacts?
Better sampling method
In order to avoid these flaws, we want to use another method to standardize the deconvolution. Unlike deconvolution, this upsampling method does not turn the generation of artifacts into the default behavior. Ideally, it will further combat these artifacts.
There is a way to ensure that the kernel size you use is divisible by the step size to avoid overlap. Like "sub-pixel convolution", this technique was recently proposed and has produced great success on super-resolution images. However, although this method is useful, deconvolution can still easily generate artifacts.
Another method is to separate the upsampling and calculate the features on the higher resolution convolution. For example, you might change the image size and then create a convolutional layer. This seems to be natural, and similar methods work well on super-resolution images.
Deconvolution and various methods of adjusting the size of convolution are linear operations, which can be explained by a matrix, which also helps us understand the difference between them.
Adjusting the size of the convolutional layer can be achieved with tf.image, resize_images() on TensorFlow. If you want to get the best result, use tf.pad() before convolution with tf.nn.conv2d() to avoid appearing on the boundary Artifact.
Image generation result
After research, we found that using nearest neighbor size adjustment and a convolution can get better results in most contexts. Among them, GAN performs better. Simply adjusting the deconvolution layer can generate different degrees of artifacts:
In fact, the difference in artifacts can be seen before training. If we look at an image output by a generator, the initial weight is random, and we can see artifacts. This shows that the artifact is related to the method of generating the picture, and has nothing to do with the adversarial training.
Another reason why artifacts are not unique to GAN is that when we put them in another model, we can still find this phenomenon by adjusting the upsampling of the convolution size.
Concluding remarks
Although deconvolution can generate images, it can also cause artifacts. Use other natural alternatives to eliminate these checkerboard artifacts. At the same time, we have also proposed a neural network method that can improve the quality of image generation. We hope to help researchers in many fields.
Budget Laptop For Students is a great gift parents like to prepare for their children. There are too many choices on the market, so how to choose a right one for your children? 10.1 inch Low Budget Laptop For Students is design for kids, like below 10 years; 14.1 inch n4020 64gb Budget Friendly Laptops For Students is the most competitive and hot one for student project; 15.6 inch n5095 128gb Budget Friendly Laptops For Students is the best one for middle or high school student or teacher online teaching; Of course, also have High Quality Cheap Laptops with i3, i5, i7 10th or 11th; 15.6 inch or 16.1 inch i7 Affordable Laptop With Good Specs and dedicated graphics. Other 11.6inch celeron budget friendly laptops in 2 in 1 or yoga type also available.
Therefore, you can share your recent and powerful hardware interest, then we can recommend the right one for you. Of course, can customize as your special requirements.
In fact, the most easiest way is what you share the details, like size, cpu, ram, rom, pcs, oem service, screen, camera, etc.
Budget Laptop For Students,Low Budget Laptop For Students,Budget Friendly Laptops For Students,Affordable Laptop With Good Specs,High Quality Cheap Laptops,Budget Friendly Laptops
Henan Shuyi Electronics Co., Ltd. , https://www.shuyicustomtablet.com