wiki:DM/ReferenceDesign/MiddlewareStatus
Last modified 8 years ago Last modified on 07/01/2011 09:40:12 PM

Middleware Reference Design Status

MW Developers,

Update the table for the packages/use cases for which you are responsible. The 'Functional / Algorithmic Description' will be used in the EA Description of the object.

DM/ReferenceDesign/MiddlewareUpdate contains a (radical) KTL rewrite of this information.

If you note a missing object, please insert its details appropriately.

Control and Management
"WBS Code" "UML Package" "UML UseCase?" "Module.Class" "Guru" " Functional / Algorithmic Description " "Start in DC?" "Implementation Maturity {Empty: no DC, Basic: 1 DC, Alternates: 2+ DC, Complete: done}" "Comparative Status" "Science Validation"
"02.07.06" "Control and Management" "Control DM System" "Publish Data Products" "After each assessment of Data Product quality, the DMS makes an automated assessment of the need or desirability of reprocessing LSST Data. This can occur due to an increase in the volume or quality of available raw data, or the appearance of improved processing algorithms. Depending on the amount of re-processing, the DMS may initiate the re-processing automatically, or request approval from the Pipeline Operator to initiate the re-processing. If authorized, the re-processing starts"
"02.07.06" "Control and Management" "Control DM System" "Preserve/Retire? Data Product"
"02.07.06" "Control and Management" "Control DM System" "Publish in Archive Center" "After review of the Data Product Quality Reports, Observatory Operations determines whether a new Data Products Release is warranted. If so, Observatory Operations initiates the process to release the Data Products. The release process creates the Data Product Release along with associated descriptive documentation in the Archive. "
"02.07.06" "Control and Management" "Control DM System" "Register in VO"
"02.07.06" "Control and Management" "Control DM System" "Replicate in Data Centers"
"02.07.06" "Control and Management" "Control DM System" "Reprocess Observations"
"02.07.06" "Control and Management" "Control DM System" "Analyze Re-processing Needed"
"02.07.06" "Control and Management" "Control DM System" "Create Re-processing Strategy"
"02.07.06" "Control and Management" "Control DM System" "Monitor DMS Status" "At all times during DMS operation, the DMS collects system health and status across all facilities, and records this into the Event Database. All significant state changes of the DMS are recorded, including pipeline Jobs started and completed, equipment or software failure and recovery Events, data transmissions from facility to facility. Continuous and perriodic evaluations are made of system resource use and availability, including CPU/cluster, disk and secondary storage, and network delivered bandwidth and errors. Any occurrence of these statistics being out of pre-defined tolerances is also recorded and Observatory Operations are notified according to pre-defined risk level"
"02.07.06" "Control and Management" "Control DM System" "Initialize DM Subsystem" "On command from the DMS Administrator, each DM facility initiates an automated startup and initializiation procedure independently (as documented in system administration procedures). This starts the DMCS at each facility. The DMCS initializes and runs diagnostics on all equipment. The DMS then establishes connectifvity between the facilities and with the"
"02.07.06" "Control and Management" "Control DM System" "Produce Nightly DMS Summary" "After the data is reduced by the Nightly Pipelines and Analysis Pipelines at the Archive Center, the DMS generates Nightly Data Product Summaries and transfers them back to the Base Facility for subsequent Observing Nights scheduling"
"02.07.06" "Control and Management" "Control DM System" "Fetch Nightly Startup Products from Archive" "After the data is reduced by the Nightly Pipelines and Analysis Pipelines at the Archive Center, the updated Orbit Catalog and any required Template Images are transferred to the Base Facility for subsequent Observing Night"
"02.07.06" "Control and Management" "Control DM System" "Send Raw Image Data to Archive" "Capture data transfered from Base Camp and stage in the appropriate place for subsequent processing"
Data Access Services
"02.06.01" "Data Access Services" "Construct Catalog" "cat" "Construct database tables used in Production" "Alternates"
"02.06.01" "Data Access Services" "Create Catalog"
"02.06.01" "Data Access Services" "Create Data Type"
"02.06.05" "Data Access Services" "Querying" "Querying" "qserv"
"02.06.05" "Data Access Services" "Querying" "Log Queries" "qserv"
"02.06.05" "Data Access Services" "Querying" "Formulate and Submit Queries" "qserv"
"02.06.05" "Data Access Services" "Querying" "Process Query" "qserv"
"02.06.05" "Data Access Services" "Querying" "Catch Hostile Queries"
"02.06.05" "Data Access Services" "Querying" "Estimate Query Cost"
"02.06.05" "Data Access Services" "Querying" "Access Database Systems" "qserv"
"02.06.05" "Data Access Services" "Querying" "Access File Systems"
"02.06.05" "Data Access Services" "Querying" "Process Query Results"
"02.06.03" "Data Access Services" "Data Ingest" "Data Ingest"
"02.06.03" "Data Access Services" "Data Ingest" "Run Data Ingest" "dbserv"
"02.06.03" "Data Access Services" "Data Ingest" "Run Data Ingest Converter" "dbserv"
"02.06.03" "Data Access Services" "Data Access Framework" "LsstData? and Citizen"
"02.06.01" "Data Access Services" "Data Access Framework" "Configure LsstData? Support" "daf_base" "Change the support (persistence, security,etc) contained in the LsstData? object" "DC2" "Alternates"
"02.06.01" "Data Access Services" "Data Access Framework" "Obtain LsstData? Realization" "daf_base" "Create a specific LsstData? realization." "DC2" "Alternates"
"02.06.01" "Data Access Services" "Data Access Framework" "Persistence"
"02.06.01" "Data Access Services" "Data Access Framework" "Define Persistence Policies" "daf_persistence"
"02.06.01" "Data Access Services" "Data Access Framework" "Persist Data from Pipeline" "daf_persistence daf_butlerUtils" "DC2" "Alternates"
"02.06.01" "Data Access Services" "Data Access Framework" "Persist Persistable Object" "daf_persistence daf_butlerUtils" "DC2" "Alternates"
"02.06.01" "Data Access Services" "Data Access Framework" "Obtain and Configure Persistence Object" "daf_persistence daf_butlerUtils" "DC2" "Alternates"
"02.06.01" "Data Access Services" "Data Access Framework" "Obtain and Configure Storage Object for Persistence" "daf_persistence daf_butlerUtils" "DC2" "Alternates"
"02.06.01" "Data Access Services" "Data Access Framework" "Specify Additional Object Metadata" "daf_persistence daf_butlerUtils" "DC2" "Alternates"
"02.06.01" "Data Access Services" "Data Access Framework" "Format and send Persistable Object to Storage" "daf_data" "DC2" "Alternates"
"02.06.01" "Data Access Services" "Data Access Framework" "Retrieve Persistable Object" "daf_ingest" "DC2" "Alternates"
"02.06.01" "Data Access Services" "Data Access Framework" "Provenance"
"02.06.01" "Data Access Services" "Data Access Framework" "Re-Create Science Exposure" "ctrl_provenance ???"
"02.06.01" "Data Access Services" "Data Access Framework" "daf_butlerutils" "These are implementation centric...need to map to UML centric above" "DC3PT1.0" "Alternates"
"02.06.01" "Data Access Services" "Data Access Framework" "daf_data" "DC3a"
"02.06.01" "Data Access Services" "Data Access Framework" "daf_ingest" "DC2" "Alternates"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Prepare Data Access for Pipelines"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Configure Object Catalog for Association" "<Ray Scripts>"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Initialize catalogs" "<Ray Scripts>"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Generate List of Sky Patches" "<Ray Scripts>"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Stage Input Data" "<Ray Scripts>"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Setup access to Image Collection" "<Ray Scripts>"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Setup Access to Co-add/Template Collection" "<Ray Scripts>"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Retrieve Template/Co?-Add covering an area" "<Ray Scripts>"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Retrieve Image from Image Collection" "<Ray Scripts>"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Retrieve Image Fragments from Image Collection" "<Ray Scripts>"
"02.06.01" "Data Access Services" "Data Access For Pipelines" "Stage a Named Collection" "<Ray Scripts>"
""
Distributed Processing Services
"02.07.01" "Distributed Processing Services" "Event Handling" "Record Event"
"02.07.01" "Distributed Processing Services" "Event Handling" "Establish Event Transmitter in Event System" "ctrl_events ctrl_evmon" "DC2" "Alternates"
"02.07.01" "Distributed Processing Services" "Event Handling" "Publish Event Using Event System" "ctrl_events ctrl_evmon" "DC2" "Alternates"
"02.07.01" "Distributed Processing Services" "Event Handling" "Subscribe to Event Topic"
"02.07.01" "Distributed Processing Services" "Event Handling" "Establish Event Receiver in Event System" "ctrl_events ctrl_evmon" "DC2" "Alternates"
"02.07.01" "Distributed Processing Services" "Event Handling" "Retrieve Event Using Event System" "ctrl_events ctrl_evmon" "DC2" "Alternates"
"02.07.01" "Distributed Processing Services" "Event Handling" "Retrieve Matching Event using Event System" "ctrl_events ctrl_evmon" "DC2" "Alternates"
"02.07.01" "Distributed Processing Services" "Event Monitoring" "Run Event Monitor"
"02.07.01" "Distributed Processing Services" "Event Monitoring" "Initialize Event Monitor" "ctrl_events ctrl_evmon" "DC2" "Alternates"
"02.07.01" "Distributed Processing Services" "Event Monitoring" "Create Timer" "ctrl_events ctrl_evmon" "DC2" "Alternates"
"02.07.01" "Distributed Processing Services" "Event Monitoring" "Process Incoming Event" "ctrl_events ctrl_evmon" "DC2" "Alternates"
"02.07.01" "Distributed Processing Services" "Event Monitoring" "Publish Event Using Event System" "ctrl_events ctrl_evmon" "DC2" "Alternates"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Create Component"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Define Component interface"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Create Component Algorithm"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Pipeline Construction"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Componentize and Add to Component Library"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Create Pipeline"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Define Execution environment" "ctrl_orca"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Define Processing Steps" "pex_harness pex_mpiharness"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Define Pipelines/Tools? Associations"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Save Pipeline Configuration"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Publish Pipeline Configuration"
"02.07.02" "Distributed Processing Services" "Pipeline Construction" "Target to Execution Environment"
"02.07.03" "Distributed Processing Services" "Pipeline Execution" "Run a Pipeline"
"02.07.03" "Distributed Processing Services" "Pipeline Execution" "Record Pipeline Execution Status" "pex_logging" "DC2" "Alternates"
"02.07.03" "Distributed Processing Services" "Pipeline Execution" "Configure Pipeline" "ctrl_orca pex_policy" "DC2" "Alternates"
"02.07.03" "Distributed Processing Services" "Pipeline Execution" "Stage input Data" "<Ray Scripts>"
"02.07.03" "Distributed Processing Services" "Pipeline Execution" "Pipeline Execution and Monitoring" "ctrl_sched, ctrl_orca, pex_harness, pex_mpiharness, pex_exceptions" "DC2" "Alternates"
"02.07.03" "Distributed Processing Services" "Pipeline Execution" "Monitor Pipeline Execution"
"02.07.03" "Distributed Processing Services" "Pipeline Execution" "Cleanup after Execution"
"02.07.03" "Distributed Processing Services" "Pipeline Execution" "Checkpoint / Restart Pipeline" "ctrl_provenance"
"02.07.03" "Distributed Processing Services" "Pipeline Execution" "Stop Pipeline Execution"
"02.07.03" "Distributed Processing Services" "Security" "Configure Security Profiles and Policies"
"02.07.03" "Distributed Processing Services" "Security" "Administer Certificates"
"02.07.03" "Distributed Processing Services" "Security" "Administer Groups and Users"
"02.07.03" "Distributed Processing Services"
""
Miscellaneous
""
"Security" "Enforce Security Policies"
"02.07.05" "User Interface Services" "Dynamic display of Data & Metadata"
"02.07.05" "User Interface Services" "Static display of Data & Metadata"
"02.07.07.10" "System Admin and Operations Services" "Admin and Maintain Applications Software"
"02.07.07.10" "System Admin and Operations Services" "Administer and Maintain Data" "<Ray/David? scripts>"
"02.07.07.10" "System Admin and Operations Services" "Administer and Maintain Systems"
"02.07.07.10" "System Admin and Operations Services" "System Health and Status"
"02.07.07.01" "Software Development Tools" "<not represented in UML>" "devenv_build devenv_codecheck devenv cpptests" "Basic"
"02.07.07.08" "Test Harnesses and Scripts" "<not represented in UML>" "devenv_buildbot" "Alternates"