Last modified 3 years ago Last modified on 03/20/2014 11:57:34 AM

Winter2014 Planning and Development


  • 2013-10-09: Plan prioritization discussed at the apps meeting and summarized in the minutes.
  • 2014-01-30: "Report from Camera-Team Visualization Meeting at Harvard, 1/27-1/28"; Tie-in for: DM:cameraGeom : Camera Team needs : DS9 Team support.

Winter 2014 development cycle plans

Please edit this page to discuss W'14 plans. To seed the discussion, I'm attaching a few e-mails on this topic:

* Planning for Winter 2014

  * Goals for FY'14: prepare the codebase for construction (i.e.,
    get it to a state where a) it's easy to bring in new people
    to work on it and b) the fundamentals are sound so new work
    can build on them (much of the code is already in that state,
    but we should prioritize and fix the parts that aren't)).

  * Winter2014 -- planning page on the wiki (to be opened by MJ)
    (note: the list below is *NOT* priority ordered)

    1. measurement needs to be redone (redesign): rewrite the
       measurement framework, make it work with Python, add an
       option to fit objects simultaneously, enable simultaneous
       multi-band measurements.
         - should be independent of any Task framework rework

    2. overall code design and Python/C++ may need to be rethought
       - document existing tasks first, look at redesign then
       - Russell: may need to refactor some tasks (Astrometry,
         Calibrate are on the top of JFBs list)
       - PP: This would be nice, but isn't getting in the way

    3. CameraGeom needs to be redone
       - JFB: Needs to be more tightly integrated w. Butler.
       - KTL & RHL: mildly disagree
       - JFB will send a note explaining his ideas

    4. need major code audit (decide what was good, what needs a
       - question of how much can we take with us into construction

    5. rethink to see if we can integrate with AstroPy world
       - import it at least into the distro server

    6. Butler
       - KTL: daf_persistence C++ portion needs to go away. PP: Needs
         to be done before construction.
       - PP: Everything should be on the table for the butler.
       - (strong feelings that butler work should be prioritized)

    7. logging needs a rewrite
       - KTL: OK as it is for now (we could live with it)
       - something similar is being written for qserv, should make it
         usable stack-wide.
       - Need to ensure no duplication occurs

    8. Source association/matching refactored into a callable and
       scalable module
       - Serge's source-association module.

    * Planning discussion to continue tomorrow at the apps call. MJ,
      KTL and GPDF will be absent due to the CCB meeting.

* Dick points out someone should look over the tickets for requested
features/fixes that were postponed (Dick may be a good candidate!). RHL:
That's what we did the last time around.

* JFB: We should have a 1-2 day virtual "refactoring and planning
workshop", via videocon. MJ strongly agrees, proposal to set it up as
early as possible (likely after the FDR, but it's possible we may
squeeze it in even earlier, with reduced participation of those who are
preparing for FDR).
Here's my list as of a few months ago, with a ~year worth of work. The
goal is to get the codebase ready for construction.

R4 (by Sep 1st 2014):
Staffing: no change (doc writer?)

  * code refactored [2]
    - measurement needs to be redone (redesign)
    - overall code design and Python/C++ may need to be rethought
    - CameraGeom needs to be redone
    - Policy needs to be retired
    - need major code audit (decide what was good, what needs a rewrite)
    - rethink to see if we can integrate with AstroPy world
    - butler needs enhancement   * [wiki:Winter2014/Meetings/CameraGeom/]
    - logging needs a rewrite
    - Source association/matching refactored into a callable and
      scalable module

  * Object Characterization v1 [1]
    - greedy optimizer
    - improved galaxy photometry (Sersic model fits)
    - improved star-galaxy separation

  * Initial R4 stack documentation written/updated [1]
    - Also, terminology updated/harmonized (Source vs. Detection, etc.)

  * SDQA Python visualization toolkit
    - pipeQA update (?)
    - debug plotting   * [wiki:Winter2014/Meetings/CameraGeom/]
    - get rid of DS9

  * Upgrade dev tools [1]
    * package and distribution tools updated w. required
      functionallity (EUPS)
    * buildbot updated / new buildbot set up
    * move development to stash, mirror on github
    * move to Confluence/Jira

RHL: Under upgrade dev tools add

  • Move to C++11 
  • Move to e.g. sphinx + doxygen for integrated python + C++ docs [I think that this is possible]

Russell Owen: another task for the list:

  • Unit tests for tasks. We need to make it easier to write such tests and then write them.

Bosch's Refactoring Thoughts

Various scribblings (requirements, complaints, strawmen) from Jim Bosch on how we should modify various parts of our current codebase. All of these are under Winter2014/Bosch.

ACB's data analysis toolkit / QA Page

Database, including Qserv

SRP’s Comments

  • We need to have a better way of capturing errors and reporting them to HTCondor/Pegasus. Both HTCondor/Pegasus register success or failure based on exit codes, and act on those accordingly (rescheduling, etc).
  • In order to use Pegasus effectively, we need to know all input and output file names while the DAX is being constructed. That means we need to know how each task in the DAX is going to name the output files that we feed into later tasks. So we have to come up with a good naming convention, or we need to be able to query the tasks to find this out, or come up with some other kind of mechanism to get this information.
  • We need to evaluate how we’re going to use the Event Monitor. The current version is written in Java with a Jython interface to use it in Python scripts. We ran into issues for some of the package imports that Jython didn’t support. This should be re-written in Python to be able to fully utilize all packages.

W14 Measurement Analysis Pages