wiki:Dc3SvnReconfiguration
Last modified 10 years ago Last modified on 11/09/2009 09:39:28 AM

DC3 SVN and C++ Namespace Reorganization

from DC3Management -> DC3StartupTasks.

References

The Data Management Work Breakdown Structure (WBS) guided the following layouts defined for both the SVN source repository and the C++ namespace. Refer to the following DOCUSHARE documents:

  • Document-984 DM WBS Dictionary with element descriptions.
  • Document-4687 DM WBS Responsiblities

DC2 transition to DC3

# means svn '/trunk/' will be installed at that point in tree.

        DC2             DC3                 Namespace           WBS
        =============   ==============      ================    ===============================
        DC2             DMS
Done    DC2/imageproc   DMS/ip/diffim#       lsst::ip::diffim     02.05.1.1.1  Difference Image Processing
        DC2/imageproc   DMS/ip/isr   #       lsst::ip::isr        02.05.1.1.1  Image Subtraction Processing
Done    DC2/associate   DMS/associate#       lsst::ap             02.05.1.1.2  Association
Done    DC2/detection   DMS/detection#       lsst::detection      02.05.1.1.3  Detection
Done    DC2/movingobj   DMS/mops#            lsst::mops (nightmops) 02.05.1.1.4  Night Moving Object
        DC2/movingobj   DMS/mops#            lsst::mops (daymops) 02.05.1.1.4  Day Moving Object
                        DMS/alert#           lsst::alert          02.05.1.1.5  Alert
                        DMS/photocal#        lsst::photocal       02.05.1.2.1  Photometric Calibration
                        DMS/astrocal#        lsst::astrocal       02.05.1.2.2  Astrometric Calibration
                        DMS/coadd#           lsst::coadd          02.05.1.2.3  Image Coaddition
                        DMS/classify#        lsst::classify       02.05.1.2.4  Image Classification
                        DMS/deepdet#         lsst::deepdet        02.05.1.2.5  Deep Detection
                        DMS/dqa#             lsst::dqa            02.05.1.2.6  Data Quality Assurance

Done    DC2/fw          DMS/afw#             lsst::afw            02.05.1.4    Application Framework
"       DC2/fw/image*   DMS/afw#/image       lsst::afw::image
"       DC2/fw/WCS*     DMS/afw#/image/Wcs*  lsst::afw::image
"       DC2/fw/formatters DMS/afw#/formatters lsst::afw::formatters
"       DC2/fw/function* DMS/afw#/math       lsst::afw::math
"       DC2/fw/Kernel*  DMS/afw#/math        lsst::afw::math
"       DC2/fw/DiaSource* DMS/afw#/detection/ lsst::afw::detection

Done    DC2/database    DMS/cat#             lsst::cat::srccat    02.05.1.5.1 Catalogs     
                       
                        DMS/imagerep#        lsst::imagerep       02.05.1.5.2 Image Archive
                        DMS/dui#             lsst::dui            02.05.1.5.3 Data User Interfaces
                        DMS/opctrl#          lsst::opctrl         02.05.1.6   Operat'l Control and Monitoring System
                        DMS/calib#           lsst::calib          02.05.1.7   Calibration Products Pipeline

moved  DC2/dc2pipe      DMS/ctrl/dc2pipe#    lsst::ctrl::dc2pipe  02.05.2.1.1 Control and Management Service
                        DMS/ctrl/orch#       lsst::ctrl::orch     02.05.2.1.1 Control and Management Service
Done   DC2/events       DMS/ctrl/events#     lsst::ctrl::events   02.05.2.1.1 Control and Management Service
                        DMS/pcons#           lsst::pcons          02.05.2.1.2 Pipeline Construction Toolkit
                        DMS/pex              lsst::pex            02.05.2.1.3 Pipeline Execution Services
Done   DC2/dps          DMS/pex/harness#     lsst::pex::harness   02.05.2.1.2 Pipeline Construction Toolkit
Done   DC2/mwi/exceptions  DMS/pex/exceptions#  lsst::pex::exceptions  02.05.2.1.3 Pipeline Execution Services
Done   DC2/mwi/trace    DMS/pex/logging#     lsst::pex::logging   02.05.2.1.3 Pipeline Execution Services
Done   DC2/mwi/logging  DMS/pex/logging#     lsst::pex::logging   02.05.2.1.3 Pipeline Execution Services
Done   DC2/mwi/policy   DMS/pex/policy#      lsst::pex::policy    02.05.2.1.3 Pipeline Execution Services
Done   DC2/mwi/data/Security  DMS/security   lsst::security       02.05.2.1.4 Security and Access Control 

                        DMS/dbserv#          lsst::dbserv         02.05.2.2.1 Database Services         
                        DMS/fsserv#          lsst::fsserv         02.05.2.2.2 Filesystem Services
                        DMS/qserv#           lsst::qserv          02.05.2.2.3 Query Services
                        DMS/catf#            lsst::catf           02.05.2.2.4 Catalog Construction Services
                        DMS/vo#              lsst::vo             02.05.2.2.5 Virtual Observatory Interfaces
                        DMS/daf              lsst::daf            02.05.2.2.6 Data Access Framework
Done    DC2/mwi/data    DMS/daf/data#        lsst::daf::data      02.05.2.2.6 Data Access Framework
Done    DC2/mwi/data    DMS/daf/base#        lsst::daf::base      Citizen, DataProperty, Persistable only
Done    DC2/mwi/persistence DMS/daf/persistence# lsst::daf::persistence   02.05.2.2.6 Data Access Framework
Done    DC2/mwi/utils   DMS/utils#           lsst::utils          (note: only the non-trace/logging routine here)
Done    DC2/mwi/include/lsst/tr1  DMS/utils/include/lsst/tr1
Done    DC2/mwi/python/lsst/mwi/p_lsstSwig.i  DMS/utils/python/lsst/utils/p_lsstSwig.i

                        DMS/devenv#                               02.05.4.1  Development Environment and Tools   
moved   DC2/deploy      DMS/devenv/build#
Done    DC2/editors     DMS/devenv/build#/editors  
Done    DC2/BuildFiles  DMS/devenv/BuildFiles#
Done    DC2/SConsUtils  DMS/devenv/sconsutils#   
                   
Done    DC2/data/fwData DMS/testdata/afwdata#                       02.05.4.2.3 Test Data

Done    DC2/core        DMS/base#
 
Done    DC2/report      DMS/report/dc2report

Moved   DC2/analysis    DMS/analysis#

Miscellaneous DC2 Directories

Existing DC2 directories which need to be included or abandoned in DC3
------------------------------------------------------------------------
DC2/community  -> DMS/community  (community outreach -- empty) 
DC2/support    -> DC1/support   (3rd party support: PySextractor, PyWCSlib;   deadweight?)
DC2/das        -> DC1/das       (DC1 data access service: proto_ds; possibly deadweight)
DC2/dbingest   -> DC1/dbingest  (DC1 possibly deadweight)
DC2/consumer   -> DC1/consumer  (DC1 deadweight)

Include Transition from DC2 to DC3

........... lsst::dps    -->  lsst::pex::dps
........... 02.05.2.1.2  Pipeline Execution
dps/include/lsst/dps                --> pex/include/lsst/pex
dps/include/lsst/dps/InputQueue.h
dps/include/lsst/dps/Queue.h
dps/include/lsst/dps/Slice.h
dps/include/lsst/dps/Pipeline.h
dps/include/lsst/dps/Stage.h
dps/include/lsst/dps/Clipboard.h
dps/include/lsst/dps/OutputQueue.h

........... lsst::events  -->  lsst::ctrl::events
........... 02.05.2.1.1 Control and Management Services
events/include/lsst/events          --> ctrl/include/lsst/ctrl/events
events/include/lsst/events/EventSystem.h
events/include/lsst/events/Events.h
events/include/lsst/events/EventLog.h
events/include/lsst/events/EventTransmitter.h
events/include/lsst/events/EventReceiver.h
events/include/lsst/events/EventFormatter.h

........... lsst::imageproc
........... 02.05.1.1.1 Image Processing Pipeline
imageproc/include/lsst/imageproc
imageproc/include/lsst/imageproc/MaskPlanes.h
imageproc/include/lsst/imageproc/wcsMatch.h
imageproc/include/lsst/imageproc/ImageSubtract.h
imageproc/include/lsst/imageproc/PCA.h

........... lsst::detection
........... 02.05.1.1.2 Detection Processing Pipeline
detection/include/lsst/detection
detection/include/lsst/detection/Measure.h
detection/include/lsst/detection/Peak.h
detection/include/lsst/detection/BCircle.h
detection/include/lsst/detection/Measure.cc
detection/include/lsst/detection/Footprint.h

........... lsst::ap
........... 02.05.1.1.3 Association Pipeline
associate/include/lsst/ap
associate/include/lsst/ap/ZoneTypes.cc
associate/include/lsst/ap/Fifo.h
associate/include/lsst/ap/EllipseTypes.h
associate/include/lsst/ap/ChunkManager.h
associate/include/lsst/ap/Results.h
associate/include/lsst/ap/ChunkToNameMappings.h
associate/include/lsst/ap/Stages.h
associate/include/lsst/ap/ChunkManagerImpl.cc
associate/include/lsst/ap/Chunk.h
associate/include/lsst/ap/Chunk.cc
associate/include/lsst/ap/RectangularRegion.h
associate/include/lsst/ap/Match.h
associate/include/lsst/ap/DataTraits.h
associate/include/lsst/ap/Time.h
associate/include/lsst/ap/ZoneTypes.h
associate/include/lsst/ap/ChunkManagerImpl.h
associate/include/lsst/ap/CircularRegion.h
associate/include/lsst/ap/Bitset.h
associate/include/lsst/ap/Condition.h
associate/include/lsst/ap/SpatialUtil.h
associate/include/lsst/ap/CustomExceptions.h.m4
associate/include/lsst/ap/ScopeGuard.h
associate/include/lsst/ap/io
associate/include/lsst/ap/io/FileIo.h
associate/include/lsst/ap/io/ResultFormatters.h
associate/include/lsst/ap/Common.h
associate/include/lsst/ap/Mutex.h
associate/include/lsst/ap/Exceptions.h
associate/include/lsst/ap/Object.h
associate/include/lsst/ap/EllipseTypes.cc
associate/include/lsst/ap/Random.h
associate/include/lsst/ap/Utils.h

........... lsst::mwi:persistence  --> lsst::daf::persistence
........... 02.05.2.2.6 Data Access Framework
mwi/include/lsst/mwi/persistence        --> daf/include/lsst/daf/persistence/
mwi/include/lsst/mwi/persistence/DbStorageImpl.h
mwi/include/lsst/mwi/persistence/XmlStorage.h
mwi/include/lsst/mwi/persistence/LogicalLocation.h
mwi/include/lsst/mwi/persistence/Formatter.h
mwi/include/lsst/mwi/persistence/DbAuth.h
mwi/include/lsst/mwi/persistence/DateTime.h
mwi/include/lsst/mwi/persistence/Storage.h
mwi/include/lsst/mwi/persistence/FormatterRegistry.h
mwi/include/lsst/mwi/persistence/FitsStorage.h
mwi/include/lsst/mwi/persistence/FormatterImpl.h
mwi/include/lsst/mwi/persistence/StorageRegistry.h
mwi/include/lsst/mwi/persistence/Persistence.h
mwi/include/lsst/mwi/persistence/DbTsvStorage.h
mwi/include/lsst/mwi/persistence/DbStorageLocation.h
mwi/include/lsst/mwi/persistence/Persistable.h
mwi/include/lsst/mwi/persistence/BoostStorage.h
mwi/include/lsst/mwi/persistence/DbStorage.h

........... lsst::mwi:data -> lsst::daf::data
........... 02.05.2.2.6 Data Access Framework
mwi/include/lsst/mwi/data.h         --> daf/include/lsst/daf/
mwi/include/lsst/mwi/data           --> daf/include/lsst/daf/data/
mwi/include/lsst/mwi/data/SupportFactory.h
mwi/include/lsst/mwi/data/Provenance.h
mwi/include/lsst/mwi/data/LsstImpl_DC2.h  -> LsstImpl_DC3.h
mwi/include/lsst/mwi/data/FitsFormatter.h
mwi/include/lsst/mwi/data/LsstDataConfigurator.h
mwi/include/lsst/mwi/data/support.h
mwi/include/lsst/mwi/data/Citizen.h
mwi/include/lsst/mwi/data/LsstBase.h
mwi/include/lsst/mwi/data/ReleaseProcess.h
mwi/include/lsst/mwi/data/DataPropertyFormatter.h
mwi/include/lsst/mwi/data/DataProperty.h
mwi/include/lsst/mwi/data/LsstData.h

........... lsst::mwi::data -> lsst::security
........... 02.05.2.1.4 Security and Access Control
mwi/include/lsst/mwi/data/Security.h    --> security/include/lsst/security/

........... lsst::mwi:policy  -> lsst::pex::policy
........... 02.05.2.1.2  Pipeline Execution
mwi/include/lsst/mwi/policy                 --> pex/include/lsst/pex/policy
mwi/include/lsst/mwi/policy/PolicySource.h
mwi/include/lsst/mwi/policy/any.h
mwi/include/lsst/mwi/policy/PolicyFile.h
mwi/include/lsst/mwi/policy/PolicyParser.h
mwi/include/lsst/mwi/policy/xml
mwi/include/lsst/mwi/policy/Policy.h
mwi/include/lsst/mwi/policy/Dictionary.h
mwi/include/lsst/mwi/policy/PolicyParserFactory.h
mwi/include/lsst/mwi/policy/parserexceptions.h
mwi/include/lsst/mwi/policy/json
mwi/include/lsst/mwi/policy/json/JSONParser.h
mwi/include/lsst/mwi/policy/json/JSONParserFactory.h
mwi/include/lsst/mwi/policy/exceptions.h
mwi/include/lsst/mwi/policy/SupportedFormats.h

............lsst::mwi:utils  -> lsst::utils
........... 02.05.2.2.6 Data Access Framework
mwi/include/lsst/mwi/utils          --> utils/include/lsst/utils
mwi/include/lsst/mwi/utils/Demangle.h
mwi/include/lsst/mwi/utils/Utils.h
mwi/python/lsst/mwi/p_lsstSwig.i  --> utils/python/lsst/utils/p_lsstSwig.i
mwi/include/lsst/tr1/unordered_map.h  --> utils/include/lsst/tr1/

........... lsst::mwi:exceptions  ->  lsst::pex::exceptions
........... 02.05.2.1.3 Pipeline Execution Services
mwi/include/lsst/mwi/exceptions.h   --> pex/exceptions/include/lsst/pex/exceptions/
mwi/include/lsst/mwi/exceptions  --> pex/exceptions/include/lsst/pex/exceptions/
mwi/include/lsst/mwi/exceptions/Runtime.h.m4
mwi/include/lsst/mwi/exceptions/Exception.h

........... lsst::mwi:logging  -> lsst::pex::logging
........... 02.05.2.1.3 Pipeline Execution Services
mwi/include/lsst/mwi/logging        --> pex/logging/include/lsst/pex/logging/
mwi/include/lsst/mwi/logging/LogClient.h
.......... lsst::mwi:utils(trace)  -> lsst::pex::logging
mwi/include/lsst/mwi/utils/Trace.h   --> pex/logging/include/lsst/pex/trace/
mwi/include/lsst/mwi/utils/Component.h  --> pex/logging/include/lsst/pex/Component????
mwi/include/lsst/mwi/logging/LogRecord.h
mwi/include/lsst/mwi/logging/LogDestination.h
mwi/include/lsst/mwi/logging/ScreenLog.h
mwi/include/lsst/mwi/logging/LogFormatter.h
mwi/include/lsst/mwi/logging/Log.h
mwi/include/lsst/mwi/logging/DualLog.h

........... lsst::fw --> lsst::afw
........... 02.05.1.4  Application Framework
fw/include/lsst/fw/Framework.h      --> afw/include/lsst/afw/afw.h

........... lsst::fw --> lsst::mops
........... 02.05.1.1.4  Nightly Moving Object Pipeline
fw/include/lsst/fw/MovingObjectPrediction.h --> mops/include/lsst/mops/

........... lsst::fw --> lsst::detection
........... 02.05.1.2  Detection Pipeline
fw/include/lsst/fw/DiaSource.h      --> detection/include/lsst/detection/Source.h

........... lsst::fw --> delete (should not have been in repository)
fw/include/lsst/fw/testmpa.cc 
fw/include/lsst/fw/testmpa.h
fw/include/lsst/fw/vwimage.cc

........... lsst::fw --> lsst::afw::image
........... 02.05.1.4  Application Framework
fw/include/lsst/fw/fwExceptions.h.m4  --> afw/include/lsst/afw/ImageExceptions.h.m4
fw/include/lsst/fw/PixelAccessors.h      --> afw/include/lsst/afw/image/
fw/include/lsst/fw/MaskedImage.cc
fw/include/lsst/fw/MaskedImage.h
fw/include/lsst/fw/Image.cc
fw/include/lsst/fw/Image.h
fw/include/lsst/fw/Mask.h
fw/include/lsst/fw/Mask.cc
fw/include/lsst/fw/ImageUtils.h
fw/include/lsst/fw/Filter.h
fw/include/lsst/fw/Exposure.h
fw/include/lsst/fw/LSSTFitsResource.cc
fw/include/lsst/fw/LSSTFitsResource.h
fw/include/lsst/fw/DiskImageResourceFITS.h
fw/include/lsst/fw/WCS.h  -> afw/include/lsst/afw/image/Wcs.h

........... lsst::fw  --> lsst::afw::formatters
........... 02.05.2.2.6  Application Framework
fw/include/lsst/fw/formatters       -->afw/include/lsst/afw/formatters/
fw/include/lsst/fw/formatters/ImageFormatter.h
fw/include/lsst/fw/formatters/MaskFormatter.h
fw/include/lsst/fw/formatters/ExposureFormatter.h
fw/include/lsst/fw/formatters/MovingObjectPredictionFormatters.h
fw/include/lsst/fw/formatters/WcsFormatter.h
fw/include/lsst/fw/formatters/MaskedImageFormatter.h
fw/include/lsst/fw/formatters/Utils.h
fw/include/lsst/fw/formatters/DiaSourceFormatters.h _ afw/include/lsst/afw/formatters/SourceFormatters.h

........... lsst::fw  -->  lsst::afw::math
........... 02.05.1.4 Application Framework
fw/include/lsst/fw/Kernel                     --> afw/include/lsst/afw/math
fw/include/lsst/fw/Function.h       --> afw/include/lsst/afw/math/
fw/include/lsst/fw/FunctionLibrary.h
fw/include/lsst/fw/minimize.cc
fw/include/lsst/fw/minimize.h
fw/include/lsst/fw/Kernel.h
fw/include/lsst/fw/KernelFunctions.h

note: the following are automatically moved from Kernel to math when Kernel is renamed
fw/include/lsst/fw/Kernel/KernelFunctions.cc
fw/include/lsst/fw/Kernel/Kernel.cc

Comments Prior to Reconfiguration to DMS

Comments by KTL:

  • Do we want to group DC3/dbserv, DC3/qserv, and DC3/catf, as they may in the end be provided by the same underlying software?
  • Should we still be prefixing everything with "DC3"? Doesn't it make more sense for DC3 to be a tag or version, eventually with its own branch, rather than a top-level structuring element? (Done)
  • DC2/mwi/policy and DC2/mwi/utils (which is mostly Trace) should probably go under DC3/pexecute. (Installed by RAA)
  • DC2/database is a set of schemas, not a set of database services. It corresponds to the catalogs in 02.05.1.5.1.*, but it was not broken down by catalog in DC2. (Installed by RAA)
  • "DC2/mwi/*" should be "DC2/mwi/exceptions" as mapped to DC3/pexecute/exceptions. (installed by RAA)
  • DC2/mwi should not be mapped to DC3/daf, as different parts go to pcontrol, pexecute, and daf. (installed by RAA)
  • The whole DC2/mwi/policy -> DC3/daf/policy entry should not exist (this is in pexecute). (installed by RAA)
  • DC2/mwi/tri -> DC3/daf/tr1 should really be DC2/mwi/tr1 (one, not i) -> DC3/devenv/tr1. (installed by RAA)

Comments by JPK:

  • I would change pcontrol to control, as it will more than likely end up with more control services than just pipelines (e.g. Data transfers) (installed by RAA)
  • We may wish to divide the pipelines (e.g DC3/Imageproc) between nightly and dataRelease (per the WBS), check with apps folks (see Comments by RAA below)
  • All of KT's comments are fine, except I am not sure dbserv and qserv will only be based on one set of software. We may enable some additional query services that go beyond SQL stuff.

Comments by RAA:

We might have a chicken and egg issue wrt dependencies:

  • daf(persistence) depends on pex(policy) and pex(exceptions) depends on daf(data)
    • KTL replies:

      This may not be an issue as they are separate subcomponents of each package. On the other hand, it could be indicating that daf(persistence) and daf(data) really should be split up.

  • Jeff and I talked about his comment on dividing the pipelines between nightly and dataRelease. He is OK with maintaining the layout as specified above. However, we need to decide where and how the policy files (.paf) for Telescope (mountain top), Nightly (base), and dataRelease (archive) will be located for development, build, and operation. I think they should be included in the appropriate pipeline module, for example (names to be decided):
    • imageproc/policies/nightly/*
    • imageproc/policies/archive/*
    • imageproc/policies/mountain/*
  • The current component known as DC2 should morph into an enduring software component stack (named DM); this component would retain its top level structuring characteristic. A final tag should be created for DC2 and thence for each subsequent data challenge stack. The DM structuring element serves to isolate LSST generated code from the other elements in the Root level of the SVN repository. (Done)

Comments by SRP:

  • Agree about using tags for DC3 rather than prefixing with DC3. Generally in SVN trees this is how versioning is done. (OK)
  • Suggestion: When I've done this to source trees in other projects, I've found it to be beneficial to go in stages to be sure things still worked. Make the change to the existing structure, get that to build, and tag it before merging in new DC3 code/changes. That way we'll start DC3 from a known to be working base under the new structure, rather than trying to change the structure and merge changes all at once.

Comments by RayPlante:

I appreciate the attempt to align the packages with the WBS structure, but I feel there are areas where this is to the day-to-day disadvantage of the developer; my detailed comments below try to highlight these areas. I've never had the experience that the architectural organization set down at the beginning of a project survived completely intact; consequently, I expect that the divide between the WBS organization and what actually makes sense programmatically will grow with time. Rather than suffer from this, we should consider (1) adapting the UML organization over time, and/or (2) loosening the one-to-one correspondence between code packages and the WBSes.

Here are some detailed comments/questions:

  • At the start of DC2, we said that we felt it was important to encourage the use of short-ish namespace component names. I think the change from mwi to pexecute goes against this notion. I like the more explanatory length of the third level (e.g. logging), but shorter names above that are nice.
    • Robyn replies:

      Install your short-form names into the document.

  • The dps package is not conceptually part of Pipeline Construction; it is clearly part of Pipeline Execution. Pipeline Construction refers to components that automatically assemble and configure pipelines for particular purposes. In particular, smart components are envisioned for assembling pipelines on the fly that reconstruct previously calculated data products based on provenance. Currently, we have no software that falls in the WBS of Pipeline Construction. (Done)
  • One thing that is not clear to me is the level at which code is versioned and released as a package. Is it assumed that each path under the DC3 column contains a "trunk" subdirectory?
    • KTL replies:

      For DC2, this was always at the top level below DC2. Presumably in the future this would always be below DM[S]. That's one reason to go to a finer-grained split at this level.

    • Robyn replies:

      Yes. Modules at this level were felt to be an integrated entity. See another reply lower.

  • If the answer to the previous question is "yes", does it really make sense to have a separate package for each kind of catalog? Do we envision any commonly shared bits? (Would that be part of "DC3/catf"?)
  • Are we specifically suggesting we split mwi and fw each into smaller packages? This will certainly give us more flexibility in versioning their capabilities. Are there overhead issues? Will it complicate our dependency management?
    • Robyn replies:

      MWI has been parcelled out to 4 or 5 major components; fw remains intact. Further division should be discussed. We need to be concerned with potential for dependency circular loop.

    • Ray replies:

      Based on the replies from KT and Robyn, I'm still confused as to where I expect to find the trunk directories for the separately versioned packages.

  • I think the existence of our mwi package suggests that we need another WBS entry that is analogous to 02.05.1.4 (Application Framework) but called "Middleware Interface". This is consistent with the recent work on the project organization.
  • I believe we proposed at our DC3 post mortem to merge trace and logging. (Done)
  • I recommend that we do not create package directories until we actually have things to put in them. They only clog/confuse our view and maintenance of the software stack, and our notion of the best organization my change over time. (Second that)

I would like to float the idea of grouping our packages together more without necessarily a corresponding change in the namespace. For example, under the root software directory we would have:

apps                 applications (WBS 02.05.1)
  apps/imageproc
  apps/associate
  ...
  apps/dqa
  apps/afw
  apps/db               (short for database)
mw                   middleware (WBS 02.05.2)
  mw/pipeexec           packages related to pipeline execution
     mw/pipeexec/pfw    (formerly dps)
  mw/pipeconst
  mw/control
  mw/daf
     mw/daf/data
     mw/daf/persistence
  mw/mwi                packages exposing middleware capabilities to the pipeline implementations
     mw/mwi/policy
     mw/mwi/logging
     mw/mwi/utils
hwi                  infrastructure (pnemonic: hardware infrastructure; WBS 02.05.3)
devenv               development environment (WBS 02.05.4.1)
     devenv/build
     devenv/sconsUtils

The idea is to reduce the number of items in a directory, thereby giving a less cluttered, functionally organized view of the software stack. (We may want to group apps packages more, but I leave that to the preference of the app developers.) The namespaces, however, can remain much as they are now. One exception is that I would consider inserting a "db" field in each of the catalog packages.

  • KTL replies:

    I think it's useful to have the namespaces (both C++ and Python) reflect the directory hierarchy. Having an additional grouping level above the current package level, while perhaps nice organizationally, would make it more difficult to navigate and to find source files.

  • Ray replies:

    That is, if I'm interpreting your comment correctly, if you are staring at a piece of code that is using classes from some other namespace, you know exactly where the code for those classes are located. If we insert other directories into the directory structure that are not reflected in the namespace, then the location becomes a bit more mysterious.

    • Robyn comments:

      It all comes down to a choice of where the SVN package boundaries are placed and hence, how easily it is for a developer to extract and work on a particular SVN package. Early on we found that having the entire software stack in a single CVS module caused problems with coordination between developers. The division of packages for DC2 was generally OK except for FW and MWI --which bundled too many disparate entities within a single SVN package. The visual layout of the SVN repository should not drive our organization model.

    • Robyn notes:

      Remember changes to the C++ namespace need to maintain consistency with the DCx model being implemented. During reverse engineering, we create an EA package for each namespace grouping; the reverse engineering process is much simpler if the EA packages align with the C++ namespace.

Finally, on the subject of the root directory, I agree that we should make our working branch generic--I would recommend "DMS". When a data challenge is complete, we make a copy of DMS and call it DCn. (Done)

Comments by Russell Owen:

I would like the fw layout to look more like vw (because it seems logical and we might as well emulate it). Thus:

  • Use namespace math instead of function (I think function is too narrow). Then it obviously includes the minimizer. (Done)
  • Add a core namespace for things such as the header file that includes the types we are explicitly instantiating and defines our convention for the position of the center of a pixel.
    • Robyn comments:

      Tell me the explicit headers to which you are referring.

    • Russell replies: There is no such header at present but we need it. Some of the data is in other headers and some does not yet exist. Contents will include typedefs for the supported types of mask (presently in Mask.h), image (does not exist) and kernel (does not exist) and a constant for our definition of the position of the center of the 0,0 pixel (presently in ImageUtils?.h).
    • Russell adds: another possible place for this header is afw/trunk/includes/Image.h (the header file that brings in *all* the contents of afw/trunk/includes/image, not the header file that defines the Image class, which unfortunately will probably also be named Image.h but will be one level down). Then we would not need any extra "core" directory. I lean towards this.
  • Perhaps ditch namesapce kernel and put kernels under image. I don't have strong feelings about this, but in mild support: kernels are basically just images and it reduces ambiguity as to where the convolve and apply functions should go (they could also go under math, but in vw they are in image). (Done)

Also, should the ds9 interface continue to go into display? (Well it is currently Display but now is the time to switch to lowercase.) It seems a bit specialized but I've not thought of anything better.

Comments by TSA:

  • I don't think it's useful to split up DC2/database. What's in there are the schema definitions as .sql files,

and a few utility scripts. There are no namespaces to split up. (Done)

  • Shouldn't there be a module for the proposed middleware orchestration layer? Have I missed it? (placed in DMS/ctrl/orch)
  • I think kernel should go into math with function, not into image (Done)
    • Russell replies: why this departure from VW's layout? I guess it's no big deal but my inclination is to either use image/ (like vw) or put Kernel into a kernel subdirectory. The latter is probably justified by the complexity of the kernel classes.

Comments by RHL

  • I agree with Robyn's comments on nightly and dataRelease pipelines --- the code is basically the same so a split would be counterproductive. Her proposed Policies subdirectories seems fine to me (but shouldn't it be policies?) (Done)
  • Russell asks, "should the ds9 interface continue to go into display?". I'd say no, that's basically history. We might want to rename it to display at the same time (import lsst.fw.display as display; display.zoom(4)). I assume that Catalog will be simply deleted.
    • Russell comments: Just to be sure I understand: you are proposing that there be a package named lsst.fw.display and it will contain the various classes and subroutines that permit sending images to ds9. If so this means renaming python/.../Display to python/.../display and then making the init.py file import all the "good stuff", right?
  • The database namespace seems a little shallow; I'd prefer lsst::catalog::srcca (or something) to lsst::srcca; (Fixed editing error)
  • in fact, maybe all the ex-mwi namespaces should be moved down a level into a small number of namespaces (e.g. do we really want lsst::dbserv and lsst::fsserv). Why don't I say the same about e.g. lsst::imageproc? I think that it's because a developer will probably only be working on e.g. imageproc, whereas I expect a (different) developer to be working more-or-less at once on all of the services. Maybe it's too early to be making these distinctions, but I'd rather over-divide than vice-versa.
  • KT writes, "If we insert other directories into the directory structure that are not reflected in the namespace, then the location becomes a bit more mysterious." I agree (Pan-STARRS did this and it confuses me to this day). You can always build tags files, but why make things hard? This implies that if the previous notes on namespaces are adopted we should change the svn layout too.
  • Can we get rid of directories called core? It scares me that there'll be a cron script out there that'll delete them, and at the very least it's confusing. (Renamed it 'base'; can be changed to something else)
    • Russell replies: could you expand on this? What "core" directories are deleted by cron jobs? (I am curious because I proposed "core" for afw as a place to put the header file that contained the chosen types for Image, etc. and it never occurred to me that directories would be at risk. In any case I'm leaning towards not having that core directory anymore.)

Comments by SRP

  • all references to "DMS/ctrl/orch" should be "DMS/ctrl/orca"