The conundrum of Nikon Z6 ProRes RAW

Zoomed and scaled view of a zone plate test shot (Nikon Z6, photo mode - no video!)

Ever since Nikon and Atomos annonced the availability of the Apple ProRes RAW codec for Nikon Z6 (and Z7) cameras back at CES 2019, the more curious-minded among us were puzzled.

How can that be? Not the part that a RAW codec via external recording over HDMI was made possible. That part of the announcement sounded fairly straighforward. After all, a RAW codec only contains one channel per pixel. So, even at 12 Bit, there should be ample of bandwidth left in the HDMI specification - even when going beyond 4k. To be precise, HDMI 2.0 supports up to 18 Gbps. a raw 30fps 4k (UHD) 12bit signal is 2.78 Gbps. Even 6k raw adds up to 6.26 Gbps. So enough bandwidth indeed.

The puzzling part was that Atomos and Nikon announced the external raw feed would be 4k. Apple ProRes RAW is compressed but unbayered pixel data directly from the sensor. But it is known that the Nikon Z6 does a full ~6k sensor readout to downsample its video feed. The Nikon Z6 has no 4k raw data! How can that be? We're eventually going to answer that now!

Some background consideration:

I had a hope: Maybe, Nikon is actually outputting a 6k raw video feed and only hiding the fact to make the Z6 not outshine the more expensive Z7. This was further feeded by the later Panasonic announcement to support 6k raw with Atomos. Others were less naiv than me and expected Nikon to do line skipping in raw video. Of course, that would be rather bad ... why should one then use the raw option at all if internal recording uses supersampling?

Let's have a quick look at the options Nikon is having in the first place: As far as I know, Nikon is using the Sony IMX410CQK CMOS image sensor (PDF linked). This sensor has 23 readout modes. It has 3 full sensor readout modes, a 14Bit (19fps) and two 12Bit (~39fps, modes 1, 2). And it has 3 4k readout modes, a 14Bit (35fps) and two 12Bit (72fps, mode 4 and 97fps, mode 5). The two 14Bit modes are used for still photos, in FX and DX (cropped) mode, obviously. The public document unfortunately lacks details about cropping and skipping used in the various readout modes. The Nikon Z6 supports 4k in both FX (full frame) and DX (cropped) video. Obviously, two of the remaining readout modes are used for that, say modes 1 and 5. So, there is one more 4k readout mode left, say mode 4 (assuming the faster mode 5 is used for cropping). So theoretically, there can be a full frame 4k readout mode in the Z6 sensor. Of course, such a mode would skip pixels.

Personally, I do not believe that this mode exists: modes 4 and 5 both sample 4160 px width, and with 2:1 row skipping, would need the sensor to be 6240 pixels wide. But it only is 6064 px. Therefore, I believe all full frame 4k sensor readout modes are cropped and any kind of pixel skipping would be done after readout by dropping data.

Testing methodology:

Line skipping, and to a certain extent, other forms of pixel skipping can be measured. The method of choice are zone plate test charts. I drafted my own version, available for download on my site and displayed below:

Zone plate test chart

As the zone plate is designed for HD rather than 6k resolution, it has to be filmed small enough to provide useful information. The header image at the start of the article shows the above chart, photographed (not filmed!) by a Nikon Z6. A fraction of the chart was displayed on a 3840px-wide monitor. The resulting 24MP photo was cropped to 16:9, then downsampled (bilinear) to 4k (3840x2160px), then cropped to 10% (380x216px), then upsampled 400% (nearest neighbor). The same procedure was also applied to every image displayed below, except for the initial downsampling step.

The red lines in the initial image are the spatial frequencies (of the zone plate rings) corresponding to 6.048k (6048px was also the width of the original still image out of camera). Also shown is the UHD region of spatial frequencies (the inner part of the green rectangle).

As you can see, even the still image shows artifacts, the green/magenta rings at the crossings of the red lines! This is a "standard" color moiré pattern as expected from any sensor without an AA anti-alias low-pass filter and a Bayer color filter. The pattern is strongest at a 45° diagonal angle (1:1 slope). Actually and unlike the Z7, the Z6 does have an AA filter. But it isn't strong enough to completely suppress color moiré and my testing methodology can still measure it. Note that this does also provide us with a precise measurement of the Z6 pixel pitch - whatever Nikon and Sony are telling us. Actually, they tell us the truth here ;)

What then would artifacts from line skipping look like? Well, here is an example from the D800:
->  LumoLabs: Nikon D800 video function demystified.
The D800 only samples every 3rd line and there are blue/yellow artifact rings at 1/3 the sensor pitch spatial frequency; but only at 90°/270° on the zone plates. The still image from the Z6 has no such artifacts.

Nikon Z6 test results

Actual testing has been made possible by both, the Nikon release of the raw feature (a paid in-the-Nikon-center firmware upgrade) and a wonderful collaborator: Marc Jungbluth from Germany (IG @marc_jungbluth, Youtube Marc Jungbluth); make sure to visit his channel - he has some wonderful content about working with Nikon and ProRes RAW. Marc went through the tedious work to videograph the test chart and find the optimal scale and setup. Marc, thank You so much!

First, as a point of reference, let's look at footage from in-the-camera debayering:

Internal recording; FX 4k video frame from Z6, cropped, scaled, graded

You see three things:

• The debayering moiré artifacts at 6.048k frequency at 1:1 slopes
• Same at 2:1/1:2 slopes (the secondary green/magenta rings)
• Ever so slightly blue/yellow debayering moiré artifacts at 1:0/0:1 slopes

The strong contrast is due to the color profile used. The visibility of secondary green/magenta rings is an indicator that the debayering process used for video is simpler than that used in stills mode, leaving out some twirks to prevent color moiré. As we will see, the same applies to debayering when done by the Apple ProRes codecs (FCPX, Compressor). There are no additional artifacts, meaning that the video frame is sampled from a full sensor readout indeed.

External recording 10Bit NLog; FX 4k video frame from Z6, cropped, scaled, graded

The video frames recorded externally using Nlog share the same charasteristics as those recorded internally, except for contrast. I tried to color grade all video frames to look about the same, as good as I could. I did try not match sharpness as that wasn't my aim. There is no indication that another sensor readout mode would have been used here.

Note: the frames show slightly different scales. The shooting distance or cropping of frames is constant though, what has changed is the actual region of the sensor being used for readout. More on that later. The important point here is that artifact frequencies didn't change, i.e., the differences in scale are due to cropping rather than skipping.

ProRes RAW

To process ProRes RAW HQ, I used Apple Compressor to produce a video clip, shown below, and said video frames. Compressor can produce 16Bit Tiff HDR frame images directly from ProRes RAW footage.

The video looks like this: Test video
The video shows a short disturbence, where motion blur temporarily destroys artifact rings - great for analysis :) The video also gives you a better idea how small the artifacts really are.

External recording 12Bit ProRes RAW HQ; FX 4k video frame from Z6, cropped, scaled, graded

You see four things:

• The debayering moiré artifacts at 6.048k frequency at 1:1 slopes
• Same at 2:1/1:2 slopes (the secondary green/magenta rings)
• Strong blue/yellow artifacts at 6.048k frequency at 1:0/0:1 slopes
• Secondary blue/yellow artifacts at ~4k frequency at 1:0/0:1 slopes

The blue/yellow artifact rings actually look exactly as line skipping artifacts. Actually line and row skipping as they are at 0° and 180° positions as well. Except for their frequency: you can't skip every line! Moreover, the secondary blue/yellow artifact rings seem to indicate that some sampling is happening at a ~4k frequency too (possibly 4.16k as these are the sensor's 4k readout modes, or 3.84k when skipping 1/3 of rows).

Therefore, I conclude that for ProResRAW, the sensor uses a readout mode or data pipeline which skips pixels. I am not going to call it line or row skipping yet. Skipping, e.g., every 3rd line and 3rd row doesn't end up to produce a feed which can be debayered easily. However, you can read 4, and then drop 2, lines and rows - and still end up with a Bayer pattern. And it should indeed lead to both, 6k and 4k artifacts. That's line and row skipping and I personally believe that this is what is going on. However, I am not ready yet to draw that conclusion officially. Officially I say though that Nikon applies pixel skipping in ProRes RAW.

Scaling

As already mentioned above, the video frame looks scaled depending on the kind of recording. Doing a quick calculation, I come up with the following approximative figures:

Mode: Sensor-width read
Photo: 6048 px (JPG)
Internal: 6000 px
Ext. NLog: 5400 px
Ext. PRraw: 5760 px
with an error margin of ~100px.

The fact that ext. NLog uses fewer pixels than internal may be an indication that Nikon is working on the edge of feasibility here. The ext. ProRes RAW mode may use an exact number of 5760 px, or 1.5x the produced 3840 feed. Another sign for 4:2 row skipping.

Conclusion

The Nikon Z6 is able to output a full frame ProRes RAW 12 Bit feed via HDMI to an Atomos Ninja V recorder. Which the Z7 and most other cameras can't. However and sadly, this feed isn't created from a ~6k internal full sensor readout, like the internal and external NLog feeds are. Rather, it seems to be created from pixel-skipping to 4k when reading out the sensor. Possibly skipping double lines and rows - although that is a speculation.

Therefore, external footage using ProRes RAW should exhibit extra color moiré (skipping artefacts), extra line flickering, a bit less acuity (texture resolution) and a bit more noise (1 stop less light captured). On the other hand, raw footage has more latitude (dynamic range to work with), possibly less compression and a bit wider field of view than NLog.

I haven't yet have the chance to evaluate all this in practice. It may turn out to be totally irrelevant in practice and ProRes RAW is the way to go. Or it may be a show stopper because NLog, all things considered, actually outperforms ProRes RAW. It probably is something in between :) For a practical evaluation, I would look at line/edge/roof-tile flickering in a motion film first.

Petition

Nikon, your ProRes RAW should have been a no brainer. Esp. as a paid upgrade. Unfortunately, it isn't. Please, add your dropped pixel data back into the prores raw pipeline and make it a 6k feed. All pieces should be there already, no need to protect the Z7, rather think of the Panas ... Thanks for listening :)