QA Score Design¶
Pipeline QA scores are represented by the
pipeline/infrastructure/pipelineqa.py::QAScore class. The basic setup features
a normalized numerical score (“score”) between 0.0 and 1.0 with the following
numerical ranges, meanings and colors:
Score range |
Color |
QA level |
|---|---|---|
0.00 <= score <= 0.33 |
red |
error |
0.33 < score <= 0.66 |
yellow |
warning |
0.66 < score <= 0.90 |
blue |
suboptimal |
0.90 < score <= 1.00 |
green |
optimal |
Optionally, score can be set to -0.1 if the score calculation encounters any unhandled errors. A score of -0.1 is considered an error score.
A long message (longmsg) describes the reason for the given score in detail.
It is rendered in weblog QA score tables. A short message (shortmsg) gives
a concise summary for the “By Task” weblog page. It is shown next to the task
name and should thus be short to render well.
The scores are usually calculated from unnormalized metrics using the actual
data. The metric is coded via the QAOrigin entity class defined in
pipelineqa.py. It has metric_name, metric_score (which is really the
metric value and not any normalized quantity) and “metric_units”. In the
QA score objects the metrics are stored as the origin attribute.
Furthermore there is a data selection object stored as applies_to attribute
in the main QA score object. The data selection is coded in the
TargetDataSelection via the sessions, vis list, scans, spws, fields, intents,
antennas and polarizations that were used to compute the particular metric.
A QA score can be None, presented as “N/A”. You can assign an “N/A” QA score
with custom messages. If no score is assigned to the task results, a “N/A” QA
score with a “No QA” message will be attached as a fallback.
The last attribute is the “weblog_location” which is an enum with possible values “BANNER”, “ACCORDION”, “HIDDEN” and “UNSET”. “BANNER” is the top of a weblog page, “ACCORDION” used to be the QA score table at the bottom of the page. PIPE-1481 requested moving this up to the banner area. Previously, the banner scores were supposed to highlight the main QA findings, in particular error and warning scores. This will be replaced with display of the task score and link to all subscores to be shown in the weblog (all except “HIDDEN”). The difference between “BANNER” and “ACCORDION” will probably no longer be needed. “UNSET” is for legacy scores from the time before introducing the banner concept. They were handled like “BANNER” scores.
Every task result has a QA object called “qa” with a “pool” attribute listing all subscores. The task score is defined by the “representative_score” attribute. It is often the minimum of all subscores, but it could also be computed differently, e.g. as mean of subscores. In case “representative_score” is unset, the pipeline assumes the minimum of the pool scores.
Pipeline QA “Handler” Class Registration¶
Pipeline QA “handler” classes rely on the class inheritance relationship from the QAPlugin base class to register and load correctly (see the QARegistry.do_qa method).
Individual task QA modules need to be imported into the namespace early so their *QAHandler and *ListQAHandler classes can be defined and registered. For this reason, it is essential to have import . qa inside h*/tasks/*/__init__.py for any Pipeline task with a qa.py module.
To check the currently registered *QAHandler classes, you can try:
CASA <2>: from pipeline.infrastructure.pipelineqa import QAPlugin
CASA <3>: QAPlugin.__subclasses__()
Out[3]:
[pipeline.h.tasks.exportdata.qa.ExportDataQAHandler,
pipeline.h.tasks.exportdata.qa.ExportDataListQAHandler,
pipeline.h.tasks.applycal.qa.ApplycalQAHandler,
pipeline.h.tasks.applycal.qa.ApplycalListQAHandler,
pipeline.h.tasks.importdata.qa.ImportDataQAHandler,
pipeline.h.tasks.importdata.qa.ImportDataListQAHandler,
# ...
]