Pokefan531's Posts

[MMD] Downscaling Anti-Aliasing with RAY-MMD

Edit 3/12: Correct Downsampling terms. 4x is 4 pixels coverage and not actually 2x.

This project of doing the best method of Anti-Aliasing is challenging. If you use MMD-RAY shader, any form of Anti-Aliasing, whether it’s from MMD’s default AA, or your driver’s AA, is not being used at all. You only do have post-procesing shaders instead, such as FXAA and SMAA. Although, I do came across artifacts on SMAA. Those post-aa shaders don’t help temporal aliasing or shimmering from missing geometry lines, and Ray-MMD doesn’t have internal supersampling options. I’ve tried many theories to do this approach, and while it’s easier to do with photos since it takes seconds to do snapshots, videos do take a lot longer the higher the resolution is. Ray Shader is a tricky situation for better Anti-Aliasing since it has its own framebuffer.

Here are the comparisons of Anti-Aliasing, and it includes Downsampling:
https://pokefan531.tumblr.com/post/171769349563

Video:

https://youtu.be/5Uc3wkC7Xx8 1080p version for none and 4x.

https://youtu.be/Sqaz_v9IAZs 720p version for none, 4x, and 9x.

The best option right now is to double, triple, or quadruple the target resolution. You do get more pixel samples and it is the only easiest method if you know how to downscale manually. Think of this as Nvidia’s DSR or AMD’s VSR ecept only doing 4x.

In the photo, I’ve shown no downsampling, 4x, and 16x. All downscaled from Photoshop to have more defined anti-aliasing. FXAA isn’t used here to demonstrate the quality of downsampling. 4x have 4 pixel samples, 9x has 9 samples, and 16x has 16 samples. 16x seems to be the best quality, especially for photo editing. If you are doing a non-integer sampling, such as putting a resolution in the middle between 1080p and 4k, the downsampling won’t really help a lot on shimmering and temporal aliasing on lower samples than 4x.

4x Downsampling is really 4 times pixels from your target resolution. As you can see at the image on top, you can see the difference between no downsampling and 4x. Look at the foliage and small lines at the staircases. It even takes care of the shading aliasing as seen on the models. It is a big difference. It is “recommended” for most users to use this quality of downsampling for video rendering to save rendering time to process on both MMD and Virtualdub.
16x Downsampling is the last thing to show from the image above. It is 16 times pixels. The staircase is taken cared well and the lines around the screen are more connected. It is somewhat less of the difference from 9x from the comparison AA post, but comparing the image is noticeable if you look at small lines and missing disconnected lines. It is more noticeable if you compare it with 4x. It is best for Image rendering for Ray Shader.

Instructions how to downsample your video or image.
For Image, load your favorite photo editor like Photoshop or GIMP, and load that file, and downscale it to your target resolution. Downscale it with Bilinear, and Bicubic gives off ringing around the edges. For example, you downscale 3840x2160 to 1920x1080p with bilinear for 4x downsampling. You can save that image.
For Video, since you render it to AVI with UT Video, load it to Virtualdub, and go to filters and use resizer, and make sure you know the video’s resolution to downsample. Use bilinear and put 1920x1080 if it’s from 4k to do 4x downsampling, and go to Compression and select UT Video RGB to compress and then click Files and Save AVI and name your AVI to save, and you are rendering your downsampled video and it’s saved. If you have audio, it will automatically do a direct copy, and you can check at Stream List and see if you have audio.

You can use it with FXAA to give more AA IQ, even at 16x. It’s always best to upscale the resolution by exact integer, like double, triple, and so to have best AA quality. For example, if you are targeting 1080p (1920x1080), then you double it to 4k (3840,2160) for 4x downsampling, (5760,3240) for 9x, or 8k (7680x4320, INSANE!!!) for 16x. UT Video RGB codec is recommended.

Let’s talk specs for 4k rendering to video with Ray shader. A 2GB video card won’t give you reasonable fps for video rendering. For me with GTX 950 OC, it is around 1fps to render in 4k (VRAM bottleneck) and a bit more at 1440p. It has to be at least more VRAM, but 4GB is recommended, but that doesn’t mean that you can’t get or keep a GTX 1060 3GB. 8k right now is pretty insane if your VRAM is lower than 10GB+ for video rendering while using MMD-RAY shader. RayMMD on higher resolutions are intense right now, and if you want to supersample your 1080p video with 16x or a 4k video with 4x, your GPU has to be like GTX 1080 Ti to render it to video with decent fps, 16GB RAM, and a CPU that has at least 8 threads in last few years, like i7s or Ryzen 7. A CPU while using UT Video with lossless compression and use Gradient for the fastest encode than the lowest settings, Left, which has lower compression ratio than Gradient or Median. T2 renders faster with CPUs that have AVX2 or AVX-512, but if you want to play the video, it’s pretty difficult as I’ve tried to get video players to run it, but didn’t show on players or video editors, but loads well on Virtualdub or Avisynth. RAM is best for at least 8GB, and I’ve had 4k rendering hit around 6.5gb and goes around 5gb on 1080p or lower. Image Snapshots are not really worried about for fps. It takes several seconds with MMD-RAY, but if you got a crash for going even higher resolution, then downsample less like 4x for stability. My GPU has no problems taking snapshots at 8k so far, but whether you are doing snapshots or video rendering that bottlenecks your VRAM, you will experiences slowdown on your desktop since it shares the VRAM and can make it almost unusable during the bottleneck. If you don’t mind letting your PC render on higher resolution for downsampling or doing native 4k, you can leave it on, and it does depend on how long it is the project and can depend how much fps you want to render your video to. 30fps vs 60fps will double the rendering time. For Image, it’s best to use PNG since you’re gonna downscale it and JPEGs has lower quality and artifacts, depends of the quality that an applications will encode it to.

My conclusion is this is our alternative to do better anti-aliasing on MMD Ray to have better quality. It currently doesn’t have its internal downsampling or is compatible with traditional MMD AA or Video Driver’s AA. I prefer doing 4x on videos if I have to wait for video rendering. What I noticed in this image is that bloom effects scale differently on higher resolutions, because the bloom on the models are smaller, but the white lights around the screen have larger blooms.

If you are not using Ray shader and either using your driver’s Supersampling, or stuck with MMD’s AA, and if you want to get rid of small aliasing as possible, 4x Downsampling should do the job if you decide to do that, especially for photo editing.