Metaraw

Metadata is information that is stored inside your digital images on top of the data that makes up the photo. For example EXIF metadata records the shutter speed and aperture you used when you created the image. Once the images are on your computer you can save more information into the photo as IPTC metadata including your name, copyright, and contact details.

	<- IPTC <- EXIF
	<- IMAGE

Fig 1. Metadata is stored as a “layer” inside the digital image.

All this metadata travels with the images when you copy, move and edit them. Metadata can and should be saved inside the digital image file so it is not orphaned and you don’t need to keep adding it whenever you make another version. To show metadata really is inside a digital image find a test file and open it in a text editor such as Windows Notepad or Mac TextEdit. Use a test image in the unlikely event the image is damaged by accidently saving it in the text editor. Inside you will see familiar words like camera brands and dates that represent the capture time. The gargled code is the image data. For more information on metadata read our tutorial here.

Fig 2. Actual metadata inside a JPEG image as shown in Windows Notepad.

MetaRAW?
A new type of metadata has begun to show up that will greatly improve the digital photography workflow, make digital asset management more intuitive and less expensive plus help ensure the work you put into RAW conversion is not lost. I call this type of metadata MetaRAW and it is a very easy concept to understand.

MetaRAW saves the adjustments you make in the RAW converter with the RAW file. When you adjust exposure, white balance, etc, in Adobe Photoshop your settings will be stored along with the RAW file itself. Adjust the same image in Phase One Capture One and it will store its settings alongside Adobe’s. Open the RAW file in DxO Optics and it will do the same.

Now copy, backup or migrate that RAW file to another computer system and all your RAW adjustments from all the RAW converters will follow.

As well as the conversion settings Photoshop, Capture One and DxO could also embed a JPEG preview in the file. If you open the RAW file in an image browser like Photo Mechanic you can select which RAW conversion (JPEG preview) to view. Select the version you require and send it to the appropriate RAW converter. It gets even better.

I mentioned “version” above. We already do versioning whenever we save a file as something else – a black and white TIFF version, a low res version, and interpolated version, etc – all adding up to more megabytes of storage space. With MetaRAW you are only saving small sets of instructions for the RAW converter that are less than 10 Kilobytes in size compared to 50+ megabytes. It is also be possible to store a history or snapshot of edits.

With versioning built into the RAW file using any image browser and you can select the not just the appropriate RAW converter but also the version type: black and white, high saturation, cross processed, etc. Along with the instruction sets the RAW Converter saves a JPEG preview. It is conceivable that each RAW file could save dozens of settings with matching JPEG preview, all of which is easily viewed by image browsers. This should also work at the operating system level so you can view versions in the Mac Finder and Windows Explorer.

Science Fiction?
MetaRAW isn’t wishful thinking. Many photographers have been lobbying companies to use this sort of technology. We already have it in Apple Aperture, Adobe Photoshop Lightroom and Nikon Capture NX. These RAW Converters let you make virtual copies of images and display them without having to save a new version. Currently these instruction sets are proprietary to each RAW converter.

Fig 3. Lightroom (above) and Aperture both let you create versions using metadata.

Sidecar Hell
At present most RAW converters save your adjustments in a proprietary database and/or a secondary file called a sidecar file.

Proprietary camera RAW files are undocumented. Camera manufacturers will not tell us how they save information into RAW files such as writing metadata. Saving metadata the wrong way will corrupt a RAW file making it unreadable. This is one reason that software vendors refuse to reverse engineer every RAW file format to find a way to write metadata. Instead they write metadata to sidecar files or store the information in a database.

Fig 4. RAW file (left) and its sidecar file (right) which contains the metadata

Some camera manufacturers claim their RAW file format is inextricably linked to their software and adopting a common RAW format will inhibit quality and innovation. That argument is easily dismissed because most third-party RAW converters already do a better job than the camera manufacturers own software. What camera manufacturers are really afraid of is competing on a level playing field. Having a fully documented RAW camera file format means third-party RAW converters will have access to more information thereby increasing quality. Not having to reverse engineer each new RAW file format means more efficient software development, reduced R&D costs with the savings passed onto consumers. For more information visit www.openraw.org.

DNG
Adobe created a fully documented RAW format called DNG (Digital Negative). They provide all the necessary information and license a software development kit (SDK) for camera manufacturers and software developers to read and write a DNG file. Adobe supplies a free DNG Converter so we can convert our proprietary camera RAW files from Nikon and Canon to DNG. Pentax, Samsung, Ricoh and Leica can write DNG as their native RAW format. Adobe Photoshop, Lightroom, DxO Optics Pro, and VueScan already read/write DNG with Phase One Capture One 4 soon to follow.

Adobe is due to update DNG to version 1.2. It is anticipated DNG1.2 will have versioning including the option of multiple, full size JPEG previews. Many applications including Photo Mechanic and iView MediaPro use these previews for instant display of RAW files without the need to slowly decode the RAW data. You can view RAW files at 100% instantly and even extract the full size JPEG preview. This is like converting RAW to JPEG in under a second.

For political and business reasons some manufacturers refuse to adopt DNG. Adobe created DNG so eventually they could save the costs of reverse engineering every RAW format that inevitably comes with new camera models and to store their own Adobe Camera Raw (ACR) metadata. DNG is owned and controlled by Adobe and not surprisingly works best with their software such as ACR in Photoshop and Lightroom.

Right now the International Standards Organisation (ISO) is developing a standard for the camera RAW file format based on DNG. This will de-politicise DNG making it more acceptable for manufacturers and leaving no excuse for not adopting it. Proprietary file formats have no place when it comes to storing our images and intellectual property.

The ISO TC42 (technical committee for photography) working group 18 (electronic imaging) standards will update the existing TIFF/EP (ISO 12234) to include the camera RAW file. Most camera RAW files are already based on TIFF/EP into which manufacturers hide their proprietary secret sauce. The new TIFF/EP will define the fields camera manufacturers should use to store RAW data, just like EXIF.

Fig 5. The ISO are reviewing the TIFF/EP standard to include camera RAW.

The TIFF/EP RAW file is documented making it safe to write metadata into them such as copyright, keywords, ratings, etc. To improve efficiency you will only need to enter IPTC once to the RAW file and all derivative files will have it copied.

It’s all Good
The future is bright for digital photographers. We will see the end of proprietary RAW camera file formats. This will give us more choice. Our digital camera RAW files can be opened in any application with edits saved to the file – just like a TIFF. Better than TIFF a RAW file uses non-destructive editing. We can safely enter IPTC and only need do it once.

A fully documented ISO standard for RAW files means your MetaRAW data is shared by all applications and operating systems. You will have a much larger selection of RAW converters, image browsers and digital asset management applications. No longer will you have to worry about whether your RAW images will be accessible in the next RAW converter, operating system, or computer. Relax safe in the knowledge your precious digital images can be archived in the highest quality, most flexible file format possible.

MetaRAW means we can reduce storage space by saving edits as tiny sets of instructions rather than duplicating images. Instruction sets will tell RAW converters how to convert the images. MetaRAW lets you save versions and snapshots for different purposes and multiple RAW converters. These versions will be displayed stacked as JPEG previews in any image browser. You choose the version from the preview and send the RAW to the best RAW converter for the job. Life is going to get a lot easier for the digital photographer.