Lumolabs: Welcome and testing methodology

Welcome to Lumolabs.

I never was really satisfied with the big lab tests (dxomark.com and dpreview.com) and really was disappointed by print magazine and other ezine tests. So, I decided to set up my own testing laboratory, based on experience as a photographer, insight as a physicist, software writing and image processing skills and patience.

For the time being, I am not challenging the big labs and confine my ambitions to the following areas:

1. RAW file image quality, in particular noise, dynamic range and resolution.
2. Video frame image quality, in particular resolution and artifacts.
3. Pentax.

I am not looking into color accuracy (not even chrominance noise...) or the camera's JPG engine. Or autofocus, ergonomy, build quality, etc. I may do lens tests and other vendors later as well.


Let me provide you with a short summary of my testing methodology:


RAW file treatment

I use the RAW file from the camera (DNG preferred) and use Adobe Lightroom 2.4 (LR) to do the demosaicing. The standard settings must be altered in the follwing ways to get a neutral demosaiced 16Bit sRGB TIFF file:

- Gray card calibrated white balance (I use 2900°K halogen tungsten light).
- Blacks 0 (changed from 5)
- Brightness 0 (changed from 50)
- Contrast 0 (changed from 25)
- Sharpening 0 (changed from 25)
- Noise reduction Color 0 (changed from 25)

I then read the resulting 16Bit sRGB TIFF file into my own lumolabs software. I properly convert the sRGB response curve (which isn't exactly a gamma curve) into linear colors and 18% gray is RGB 117.6/255.


Taking test chart photos

I then take test chart photos with controlled manual settings, including manual focus. About the slowest shutter speed is 1/15s.

Some think that noise test shots must be taken in low tungsten light. I agree for the tungsten part and I use it. I disagree for the low light part, though. A short exposure is a perfect approximation to low light because the read-out time of the sensor is longer than 0.1 seconds anyway whatever fast be the shutter. So, only if I wanted to study exposure time longer than, say 1/5s would low light be able to have an influence. In consequence, I don't publish exposure times shorter than 1/10s. Additionally, except for initial focus, I avoid using live view in noise tests to not overheat the sensor.

One of my test charts is the zone plate chart useful to evaluate anti alias filter and Bayer mosaic filter artifacts. It is shown in the beginning of the article. It is allowed to download it for own testing purposes, providing proper credit is given.

Other testing charts are an ISO-12233 resolution chart which I modified to include inner parts of 4x the resolution. Another chart is the noise testing chart with defined gray and color patches. All charts are printed on A2 paper using a high end photo printer. There are minimal printing artifacts visible at and beyond 4000 LW/PH (line widths per picture height). They do not disturb the testing results though.

The density range of the printed noise test chart is about 7 EV. So, I take a second shot at -5EV and obtain a combined test chart of about 12 EV dynamic range (luminance range 0.02% to 100%). Which immediateley turned out to be outperformed by the recent Pentax K-x ;)

My software will search for patches of minimal variance to exclude the effect of scratches and the like which may be present on the printed test chart. I am not sure though that there aren't some invisible defects below the -40dB range. They do not disturb the testing results though, except maybe for very bright patches.


SNR, noise and dynamic range

I am unrelated to DxO Labs, Boulogne, France. However, whereever I found their methodology to be appropriate (cf. dxomark.com), I adopted it. I disagree on some of their methodologies but some are best practise to be followed. My biggest concern with DxO is their lack of examination of influence of spatial frequency dependencies. They only "correct" their results for resolution effects as to be expected from pure photon shot noise. IMHO, this isn't good enough. I am currently working at a new testing methodology to overcome this. For the time being though, I am happy to adopt DxO's pixel-level measure methodologies.

It is therefore important to understand the SNR [dB] figure. It is explained here: SNR.

Another important detail is the print normalization: Noise, dynamic range etc. are all improved by downsizing an image. Still, measures take place at the varying resolution of the sensor. Like DxO, I use a "print-normalization" calibrated to a 8 MPixel resolution. The normalization formulae are here: print-normalization.

SNR [dB] can be easily understood by using the follwing hints:
- A patch of basically random pixels has an SNR of 0 dB.
- A decrease of light by 1 EV, doubling the ISO, going to a 50% darger region in an image or doubling the number of pixels all amount to a change of SNR by -3dB.
- Going from 1.53 crop APS-C to full frame amounts to a change of SNR by +3.7dB. So, 3dB is a lot ...

My full SNR curves can be directly compared with the DxO mark tab "Full SNR". Note however that DxO uses noise patches made from glass, has a different light source most importantly at a different color temperature and uses a different raw converter. Also, in derived measures, I don't include ISO sensitivity variance and possible raw data noise reduction (smoothing).

The dynamic range is the luminance level where SNR reaches 0dB, converted to f-stops and normalized for resolution. Note that the 0dB level is arbitrarily choosen by DxO and is resolution dependent. They formula-correct for resolution dependency but fail to understand that the full dynamic range must be measured at a much lower resolution. I plan to address this issue.


Video frame tests

The video frames studied in the video section are extracted using the Quicktime Pro "export as an image" feature. Because the video uses the camera's JPG engine, here are the parameters:

- Pentax: WB manual (2900°K), "Natural color profile preset", all other parameters at default values (like noise reduction and sharpness -1).


Enjoy the tests :)