DataType Testing

The data type information is stored in two places in the MS domain objects. Originally, we just had a mapping of column (data or corrected) to data type for all sources and spws. This dictionary is MeasurementSet::data_column with keys being a data type from the DataType enum and values being the column name.

Later we saw the need for the per source/spw information. This is stored in the MeasurementSet::data_types_per_source_and_spw dictionary. Its keys are tuples of source name and real spw ID. The values are lists of data types that exist for the given selection in this MS (at most two because there are just the data and corrected columns). This dictionary is not intended to be manipulated directly, but only via calls to MeasurementSet::set_data_column method. The importdata tasks use it to set a data type for the column. This also sets all source/spw entries in data_types_per_source_and_spw if no special source/spw selection is given (which is the case for the importdata tasks).

The self calibration task is using set_data_column always with a source/spw selection. In that case MeasurementSet::data_column is set, but MeasurementSet::data_types_per_source_and_spw just gets an entry for the particular selection.

The two methods MeasurementSet::get_data_column and MeasurementSet::get_data_type retrieve the corresponding data type for the given column or vice versa, optionally restricting the query to a subset of sources and/or spws.

To arrive at a setup that might be generated by hif_selfcal one needs to mimic some set_data_column calls.

For PIPE-1474, for example, two MSes were used that were mentioned in PIPE-1209 which are not too large and also exercise the virtual spws. From a normal PL run one can take uid___A002_Xed4607_Xfbf7_targets.ms and uid___A002_Xed4607_Xfd64_targets.ms and manually copy the DATA column to CORRECTED just to have some data in both columns. The first MS has science spw IDs 13, 15, 17, 19, the second MS has 5, 7, 9, 11.

Now one can import these MSes in a new PL session and modify the lookup tables manually:

from pipeline.domain import DataType
ctx=h_init()
hifa_importdata(['uid___A002_Xed4607_Xfbf7_targets.ms',
                 'uid___A002_Xed4607_Xfd64_targets.ms'],
                 datacolumns={'data': 'regcal_contline_science'})
# First MS
ctx.observing_run.measurement_sets[0].set_data_column(DataType.SELFCAL_CONTLINE_SCIENCE, 'CORRECTED_DATA', 'HL_Tau', '17')
ctx.observing_run.measurement_sets[0].set_data_column(DataType.SELFCAL_CONTLINE_SCIENCE, 'CORRECTED_DATA', 'HL_Tau', '19')
# Second MS
ctx.observing_run.measurement_sets[1].set_data_column(DataType.SELFCAL_CONTLINE_SCIENCE, 'CORRECTED_DATA', 'HL_Tau', '7')
ctx.observing_run.measurement_sets[1].set_data_column(DataType.SELFCAL_CONTLINE_SCIENCE, 'CORRECTED_DATA', 'HL_Tau', '9')
ctx.observing_run.measurement_sets[1].set_data_column(DataType.SELFCAL_CONTLINE_SCIENCE, 'CORRECTED_DATA', 'HL_Tau', '11')
h_save('c1')

This would set up just regular calibration data for virtual spw 13, one MS with selfcal in virtual spw 15 (real spw ID 7 in MS2) and two MSes with selfcal in virtual spw IDs 17 and 19.

Now one has a setup to test various combinations of data types per source/spw, including virtual spw lookups. One would keep resuming the saved context, run some commands and save to a new file (h_save('c2'), etc.).

For hif_makeimlist and hif_editimlist the query involves a list of MSes, so there is another method to deliver that information. It is ObservingRun::get_measurement_sets_of_type. It takes a list of data types in the order of possible fallbacks. This list is provided by a function get_specmode_datatypes from makeimlist.py. For example, for specmode mfs the list is

[DataType.SELFCAL_CONTLINE_SCIENCE, DataType.REGCAL_CONTLINE_SCIENCE, DataType.REGCAL_CONTLINE_ALL, DataType.RAW]

for science targets and

[DataType.REGCAL_CONTLINE_ALL, DataType.RAW]

for calibrators. This means for science targets that selfcal is preferred over regcal and if the split into the _targets MSes has not yet been done, the regcal from the original MSes is used and so on. The method delivers a list MSes where to find the data. Optionally, one can again specify a source and spw.