This page covers changes in the file formats between µManager 1.4 and µManager 2.0 (hereafter referred to as “1.4” and “2.0” for conciseness). Unless otherwise indicated, all comments apply to the “multipage TIFF” format, not the “single plane TIFF series” format.
Compatibility
2.0 is backwards-compatible with 1.4 – that is, it is able to load files that were generated by 1.4, and all image data and metadata should be faithfully preserved. However, 1.4 is not forwards-compatible with 2.0. That is, there is no guarantee that 1.4 will be able to read a dataset that was generated by 2.0.
Compatibility with SCIFIO and BioFormats
1.4 and 2.0 both generate valid OME TIFFs which can be loaded by third-party programs using the OME libraries.
The BioFormat’s importer uses heuristics to determine if a dataset is a µManager dataset, and uses a custom loader in that situation to try to load additional metadata (such as device state). This loader is not perfect and may be prone to failing to properly load files, both 1.4 and 2.0. We are working with SCIFIO to resolve issues as they become known; the “MicroManagerReader” used by SCIFIO should be considered a work-in-progress.
Implementation Details: JSON
Both Metadata and SummaryMetadata are converted to JSON for storage, and the DefaultMetadata and DefaultSummaryMetadata classes have methods for generating new instances from a JSON representation. The JSON representations in general are “augmented” versions of the 1.4 representation (to the extent that 1.4 had a consistent spec for its JSON, which it didn’t, really).
In 1.4, all metadata fields were stored “flat” in a TaggedImage’s tags. Some of these tags were treated specially by the code, but most were just recorded in case they were useful. 2.0 adds the notion of “official” properties, “scopeData” properties, and “userData” properties. Official properties are strongly-typed and stored directly in a Metadata or SummaryMetadata object; they include things like the image’s UUID or the profile name of the user. “scopeData” properties are metadata that 2.0 is able to recognize as pertaining to the microscope state. Everything else is stored in the userData.
In the Metadata Panel in the GUI, all metadata properties are displayed “flat”, with no differentiation between official, scopeData, and userData. However, behind the scenes, these different fields are stored separately.
userData and Miscellaneous Fields
When 2.0 writes a field, for compatibility purposes (attempting to not completely break if 1.4, or third-party loaders, tries to load a 2.0 file) it stores all metadata fields “flat”, like in 1.4. However, it also writes a “scopeDataKeys” list, which identifies all properties that belong in the scopeData construct. Everything that is not recognized as an official property, and that is not listed in the scopeDataKeys list, is assumed to be a userData property.
1.4 did not generate this scopeDataKeys list; consequently, when 2.0 loads a 1.4 file, it will store (in memory) all non-official properties in userData. It recognizes this situation by examining the metadata version property – if it’s missing or less than 11, then all fields that were not recognized by the Metadata are stored in userData.
Thus in 1.4, the JSON representation of a given piece of metadata looks like:
- Official fields (in general, any field that has a getter/setter method in MDUtils)
- Everything else
In 2.0, the JSON representation looks like this:
- Official fields (anything specifically recognized by the Metadata/SummaryMetadata; in some cases these still make use of MDUtils)
- scopeDataKeys list
- Fields whose names are in the scopeDataKeys list; these are turned into the scopeData PropertyMap
- Other fields; these are turned into the userData PropertyMap
Official fields
The official fields can be determined by examining the DefaultMetadata.toJSON() and DefaultSummaryMetadata.toJSON() methods. The StorageMultipageTIFF and related classes augment these fields with other information as part of storing them in the file (e.g. FileSet.java adds a “FileName” to each image metadata) – I tried to minimize the degree to which I modified how these objects worked, due to fears of breaking things.
Other structural changes
In 1.4, comments were stored in a specific section of the TIFF (and consequently, could only be modified while the file was still open for writing, prior to completion of acquisition). In 2.0, comments are now stored external to the file, as an “Annotation” object. 2.0 will still correctly load comments from a 1.4 dataset; it will then create an Annotation for the dataset which it will use in preference from then on. Modifications to the comments do not touch the original TIFF.
Display settings are also stored external to the file, in a displaySettings.txt file which is generated by the DisplayWindow(s) that are showing the file at the time saving finishes. If no display settings are found when loading a file (including when loading a 1.4 file), then the user’s profile is used to supply default settings. This does mean that when you load a 1.4 file for the first time in 2.0, you will not get whatever display settings are stored in that file. 2.0 does also store display settings directly in the TIFF; however, the settings used a) do not have the same JSON format as the settings in 1.4, and b) are the default display settings for the user’s profile, not the settings specific to the display.
OME metadata has not been changed, though we are mooting a change to always generate external OME XML, rather than saving it in one of the TIFF files. Coordination with LOCI is needed for changes that involve OME.
The overall structure of each TIFF file has not changed; we still use the same logic to decide when to split a file, tags are at the same offsets, et cetera.
File names for the “single plane TIFF series” format have changed; the new format is intended to be more flexible for alternate data axes (though currently, the format has logic that prevents you from using any axis other than the standard 4, apparently solely for backwards-compatibility reasons). The format stores the image coordinates in the filename as (axis name)(axis position), separated by underscores (e.g. “img_c000_p000_t000000000_z000.tif”). There’s a specific hack to assign greater precision to the time axis to account for long-running timeseries.