1.1   Per-instrument collections¶

Raw and calibration data associated with a particular real-data instrument is organized into collections that start with the “short” instrument name, e.g. “HSC” or “LSSTCam-imSim”, followed by a slash. The naming conventions for these collections are codified by the lsst.obs.base.Instrument class’s make*Name methods. The highest-level collections are always defined as CHAINED collection “pointers” to other versioned collections.

In the case of raw data (including both science observations and raw calibrations), we propose three levels of collections:

• <instrument>/raw/all: the RUN collection into which all raws for that instrument are originally ingested.
• <instrument>/raw/<ticket>: a TAGGED collection containing a curated subset of all raws that do not contain problems (e.g. tracking issues, airplanes, etc.), named according to the ticket (e.g. DM-98765) on which the curation work was done.
• <instrument>/raw: a CHAINED collection that points to the current-best */<ticket> collection for this instrument.

The collections for master calibrations follow a similar pattern, but because master calibration datasets are produced by our own pipelines, not ingested, [1] there is no single RUN collection that holds these all of these datasets directly. As described further in 1.7   Calibration production, each processing run generates a new RUN collection.

Certifying these calibration datasets - marking them as acceptable for use in calibrating observations taken in a certain temporal validity range - involves adding them to a CALIBRATION collection. These should have names of the form <instrument>/calib/<ticket>, and we will use single-element CHAINED collections of the form <instrument>/calib as pointers to the current best set of calibrations for each instrument.

Finally, for convenience, we will define per-instrument CHAINED collections with names of the form <instrument>/defaults that aggregate:

• the recommended raws for that instrument (<instrument>/raw/good),
• the recommended calibrations for that instrument (<instrument>/calib),
• and cross-instrument auxiliary collections (refcats, skymaps, and pretrained_models; see 1.2   Reference catalogs, 1.3   SkyMap definitions, and 1.8   Pretrained models, respectively).

1.2   Reference catalogs¶

External reference catalogs reformatted and sharded by DM code are written to refcats/<ticket> RUN collections, where <ticket> is the ticket on which the reformatting and sharding work was performed. After a reference catalog has been validated, its RUN is added to the overall refcats CHAINED collection.

Different collections for different reference catalogs are not necessary, as the name of a reference catalog (e.g. ps1_pv3_3pi_20170110) is used directly as its dataset type name (note that this was not the case in Gen2, where the reference catalog name was instead part of the data ID).

1.3   SkyMap definitions¶

All skymaps must have a globally unique name in Gen3, which is used as part of the data ID for any dataset that is defined on tracts. The skymap definition datasets (i.e. lsst.skymap.BaseSkyMap subclass instances in Python) also include this globally unique name in their data IDs, and hence can also all go in a single skymaps collection. This is a RUN collection that holds skymap definition datasets directly.

The existence of different skymap definition datasets for different coadd types (goodSeeingCoadd_skyMap, etc.) is a relic of Gen2 that will soon be removed entirely from Gen3; all skymap definition datasets will just use the skyMap dataset type. The new globally-unique skymap data ID names are both necessary and sufficient for uniqueness in Gen3.

SkyMap registration is something we expect to be rare in Gen3 - much more rare than running makeSkyMap.py was in Gen2 - because we almost always use one of a few standard SkyMaps, and in Gen3 a SkyMap (a combination of a lsst.skymap.BaseSkyMap class and its configuration) can only be registered once. Discrete SkyMaps, which typically cover only a small part of the sky and are conceptually a bit more per-user, may be less rare, but our data model currently does not treat these any differently, and until we can identify the patterns and use cases for creating new SkyMaps (even discrete ones), we propose that any new SkyMap registration in a shared repository be preceded by an RFC.

1.4   Source injection datasets¶

Input catalogs containing synthetic source parameters required for source injection are written to injection/catalogs/<ticket> RUN collections, where <ticket> is the ticket on which the catalog was created. After a catalog has been ingested, its RUN is added to the overall injection/catalogs CHAINED collection. All required input data for source injection, including input catalogs, are aggregated in the injection/defaults CHAINED collection.

1.5   Shared/official processing outputs¶

Processing runs overseen by production operators should produce output collections of the form <instrument>/runs/<target>/<release>/<ticket>, or runs/<target>/<release>/<ticket> in the (rare) case of processing that includes science data from multiple instruments and none of them can be considered the “primary” instrument. <target> is a human-meaningful name for the set of data IDs being processed, and <release> is some kind of DM software release version, so examples of complete processing-output collection names might include HSC/runs/RC2/w_2020_50/DM-75643 or DECam/runs/HiTS-2015/d_2021_90/DM-80000. These versions are intended to make it easy for users to browse collections and understand what is in them at a glance; formal provenance for software versions actually used in the processing will be automatically stored in the data repository itself. Of course, the version in the collection name should differ as little as possible from the versions actually used to reduce confusion.

These names should always correspond to a “public” CHAINED collection that aggregates both all RUN collections that directly hold outputs and all collections used as inputs. The organization of those “private” output RUN collections (if there is more than one) is completely at operator discretion (they may correspond to e.g. different tracts, different stages of processing, different attempts), but these collections should start with the same prefix as the umbrella CHAINED collection, followed by a slash.

In cases where one or more private RUN collections contain datasets that should not be considered part of the final public outputs (e.g. because they are superceded by datasets in other private RUN collections), a TAGGED collection can be used to screen and aggregate these. That TAGGED collection would then be a direct child of the final public CHAINED collection, instead of any RUN collections it references. For example, instead of the following chain involving two processing-output RUN collections (first and second) as well as the input (HSC/defaults):

HSC/runs/w_2021_50/DM-20000              CHAINED
  HSC/runs/w_2021_50/DM-20000/second    RUN
  HSC/runs/w_2021_50/DM-20000/first     RUN
  HSC/defaults                          CHAINED
    (nested input collections)

we would redefine the chain to include a TAGGED collection (filtered) that references (at the level of individual datasets) the first and second runs, but still include the inputs directly:

HSC/runs/w_2021_50/DM-20000              CHAINED
  HSC/runs/w_2021_50/DM-20000/filtered  TAGGED
  HSC/defaults                          CHAINED
    (nested input collections)

1.6   Developer processing outputs¶

Processing initiated by DM developers that are intended primarily for personal or small-group use must start with u/<user> (e.g. u/jbosch), and are strongly encouraged to start with u/<user>/<ticket> (e.g. u/jbosch/DM-56789) whenever possible. Names and structure after this prefix are at user discretion, but we strongly recommend using a combination of CHAINED collections and RUN collections to distinguish between “inputs and outputs” collections and “output only” collections, as in 1.5   Shared/official processing outputs. The pipetask tool will automatically take care of this if the --output option is used with or instead of the --output-run option.

1.7   Calibration production¶

Calibration production runs intended for broad use (i.e. outputs will be at least candidates for membership in the recommended calibration collection for this instrument) should output to collections with names that start with <instrument>/calib/<ticket>/. Those produced for experimental or development purposes should start with u/<user>/<ticket>/.

In either case, the RUN collections that hold output datasets directly will usually require another disambiguating term, mapping roughly to the expected validity range epoch. Our current plan is to use our initial estimate of the start date of the validity range; this will rarely change (though if it does, the date in the RUN collection - which cannot be changed after datasets have been written - will not reflect the actual validity range start date). We could consider using only dates (possibly with human-incremented integer suffixes, as necessary) for RUN terms while always using full (e.g. second-precision) date-times for actual validity values in order to reduce both confusion and verbosity. Actual validity ranges are not assigned until datasets are certified (i.e. added to CALIBRATION collections), and until then, the usual dataset type + data ID constraint applies (i.e. there can only be one bias for each detector in a particular RUN collection).

As noted in 1.1   Per-instrument collections, certified calibration products intended for broad use should go in CALIBRATION collections named just <instrument>/calib/<ticket>. CALIBRATION collections can also of course be nested under u/<user>/<ticket>, but may not always be necessary for development work, because a RUN or CHAINED collection directly containing e.g. new bias datasets can also be used as an input to a processing run that generates new flat datasets (as long as only one calibration epoch is in play).

1.8   Pretrained models¶

Pretrained neural network models for meas.transiNet.RbTransiNetTask (and similar tasks that may be developed in the future) are stored in pretrained_models/<model> RUN collections, where <model> is the model package name. Once a model is considered suitable for general use, its RUN is added to the overall pretrained_models CHAINED collection. It is not yet clear whether pretrained_models should act as a single-element “pointer” to the current best model, or whether it can contain multiple RUN collections in priority order (or for different tasks). We will finalize this as we gain more experience with using these models for real data processing.

Models cannot be explicitly selected through task connections, but only implicitly by including a specific run in the execution inputs. It is expected that most users will use pretrained_models to use the latest recommended model. Those who wish to override the default will need to include the corresponding RUN collection ahead of pretrained_models or <instrument>/defaults.

Collections disambiguated by ticket number are not necessary, as the name of a model (e.g. rbResnet50-DC2) may include any versioning information, just as refcats are labeled by conversion date.

2.1   Current paths and migrations¶

3.1   New paths and access controls¶

5.1   “Collections” of data IDs¶

Collections that represent fields of particular interest or regularly-reprocessed test datasets are not described here, because those are conceptually more groups of data IDs than groups of datasets (e.g. not just raw exposures, but tracts on which to combine them as well). As in Gen2, we will continue to record the definitions of these groups outside the data repository itself, though we may add support for in-repository storage of data IDs to Gen3 in the future. It is also worth noting that exposure or visit metadata can sometimes be used to help select some of these data IDs (e.g. visit.target_name='SSP-Wide'), and these selections are automatically combined with the selection of a <instrument>/raw/good input collection.

5.2   Naming conventions for dataset types¶

The names for nearly all dataset types in Gen3 have been inherited directly from Gen2, and while these are sorely in need of standardization and cleanup, we have no plans to change to new names until Gen2 has been fully retired. Naming conventions for new dataset types would be welcome before then, but are still beyond the scope of this proposal.

In the meantime, users should be aware that dataset types are global entities with no implicit namespacing, and hence new dataset types should be created with care. The pipetask tool’s --register-dataset-types option is a non-default option for exactly this reason: in a long-lived repository, re-executions of the same pipeline will eventually outnumber executions of new pipelines (especially new pipelines with new datasets), and hence --register-dataset-types should rarely be needed. Passing it all the time as a matter of habit is an antipattern, because it makes it easy for a typo to result in long-lived, hard-to-clean-up garbage (dataset types can be removed, but only if there are no datasets of that type).

5.3   Provenance and Reproducibility¶

The plan for provenance in the Gen3 butler is centered around storing the directed acyclic graph (DAG) of datasets and processing “quanta” that is used to drive PipelineTask execution, after updating it with the unique identifiers of the datasets actually produced and annotating it with information about which input datasets are actually used by the (rare) PipelineTasks that may not use all predicted inputs. While some provenance information (e.g. software versions and configurations) are currently associated directly with RUN collections (and this information, at least, may always be), and CHAINED collections provide some information about what datasets were used as inputs when creating others (see 1.5   Shared/official processing outputs), these do not carry sufficiently detailed information about the relationships between datasets to meet our needs.

Using fine-grained provenance information to exactly reprocess a DAG will actually be quite different from starting a new run “from scratch”, as it doesn’t involve providing collections or data IDs as inputs - the input datasets are already fully resolved, so there is no need to search for them in collections, and the data IDs are intrinsic to those datasets. We will also need to provide ways to almost exactly reprocess a DAG, of course - e.g. replacing the initial resolved datasets with new collection + data ID searches, modifying Task configuration in a way that does not change the DAG (or changes it only in a limited sense), and probably more.

All of this fine-grained provenance is not yet implemented, however, and at present the only way to guarantee reproducibility is for all input collections to have exactly the same state they had when the original run was performed. The standard collections defined in this document are poorly suited for this role, however; we consider it more important for these to track the “current best” (or in the case of raws, recent observations) than it is for them remain immutable. Users should thus be aware that repeated processing runs using the same input collections (and everything else held constant) are not intended to always produce the same results (and this is a feature, not a bug).