ALICE O2 Project
Search

Run 1 and 2 DQM and QA

DQM

Online QA, automatic monitoring using AMORE

  • Extracting plots from detector algorithms (DA)
  • QA for experts and shifter

Quality Assurance framework (running during reconstruction)

The QA framework provides a procedure to assess the quality of the data at various level of MC simulation and reconstruction

Requirements: check the correctness of the data produced at various steps of the data processing, stay simple, provide a quality check

Design considerations

  • possibility to have QA per event species (calibration, laser, cosmic, physics, etc.);
  • splitting per data type (hits, sdigits, digits, raw, clusters, tracks);
  • splitting per trigger class;
  • splitting per detector + global QA;
  • access to the full information during simulation and reconstruction;
  • provides correlations between the detectors;
  • only objects that inherit from TH1;
  • produces bit status;
  • possibility for automatic alarms.

Problems:

  • no reference data provided by detectors, not so light, , hard to maintain, makes DQM dependent on offline
  • merging is memory consuming in THnSparse-like objects are used;
  • most of the possibilities are not used.

Offline QA

Complex reconstruction QA for MC and real data, using the analysis framework

  • Central productions, using Revision tags
  • Running on ESD‘s, using friend information and rec points

The offline QA train is run at the end of the simulation+reconstruction and calibration+reconstruction cycles to assess as fast as possible the quality of the reconstructed data. The train is using the task model of the analysis framework and is composed by over 30 different functional modules and about 50 analysis wagons overall. These modules are testing the quality of different features like: the offline trigger-based physics selection, the global and per-detector tracking and matching efficiencies, the centrality selection, the vertex or the particle identification.

The outputs of the QA tasks are in general histograms which are organized per folder and written to the same output file. In the AliEn file catalog, the QA output per chunk is always written in the same sub-folder as the ESD files (e.g. run_no/cpass1/chunk_number/QAresults.root). The QA outputs are being merged per run using a multi-stage procedure and the merged output written in the production folder (e.g. run_no/cpass1/QAresults.root). The QA tasks can post-process the merged data in the Terminate phase during the last merging step, producing trending information per run which is added into a second file trending.root. The typical size of the QA output per chunk is about 10 Mbytes for p-p and less than 20 Mbytes for Pb-Pb, while the merged output depends on the run size due to the presence of few small trees and TNnSparse histograms. The maximum size of the merged QA file is about 100 Mbytes for p-p and 350 Mbytes for Pb-Pb. The trending file is smaller than 5 Mbytes.

The processing time of the QA train per chunk is typically few minutes and is negligible with respect to the time to simulate and/or reconstruct the chunk. The merging procedure is based on successive stages which split the total number of files by a tunable reduction factor. The start for the merging procedure is triggered by the monitoring system when reaching 90% of completion for the reconstruction jobs. The submission is handled by the LPM system.The QA train supports a local testing mode which emulates the actual grid jobs. Due to the fast development cycle of QA algorithms and inclusion of new modules, testing the QA train on the production Revision tag is a very important activity for the stability of the process. This requires each QA responsible to follow the standard procedure of submitting documented porting requests in case of changes. In case of observed bugs, the fixes are centrally requested via the Savannah system.

Problems:

  • Memory: about 1.4 GBytes resident memory and 3.5 Mbytes virtual memory in case of simulated data and a bit less for real data. The QA train can also be run on demand on multiple chunks using the alien plug-in, in case some corrections are needed. This mode consumes more time and memory (can reach 2.8 GBytes RSS for 1-2 hours jobs) but is used only exceptionally.
  • Merging: introduces problems of memory and time, specially for runs with many chunks
  • This QA it is not so fast and need human intervention.

QA for Run3

Specialised for each relevant operation/data type in the processing chain: e.g. QA “sensors” can be added to any processing component which may give wrong results: clusterization on FLP's, tracklet reconstruction, calibration, …

Handles different info than the one for the monitoring and control system (status of the node, links, traffic), but QA may report also to the monitoring system (e.g. QA info that may concern the sanity of the data taking chain)

Will run on the processing nodes of the new cluster (FLP's and EPN's). On FLP's we will need a light QA with very fast response for cluster finding and first level calibration. Preferably running as separate process in the stream of clusters (coming from FPGA cluster finding)), and during the calibration procedure. On EPN's, probably will look closer to today's offline QA. The fast response component is still needed to assess fast (and automatic?) the quality that may influence data taking.

Problems to discuss:

  • QA output formats, services and reporting procedure
  • Relation between QA and the monitoring&control system
  • Categories of QA responses (fast, slow, having reference data, semi-automatic, …)
  • Discuss with detectors on what they can provide (and what is important for them) on different QA levels
  • QA for calibration – how to do, dependencies
  • How big can be the output, how fast., where to store.