wiki:db/MergingObjectCats
Last modified 6 years ago Last modified on 02/01/2013 01:39:38 PM

Merging DR Object catalog with Alert Prod Object catalog

We are updating the up-to-date Object catalog at the Base Camp every night. Independently we are generating a new Object catalog each time we build a new Data Release. It would be very useful to merge these two catalogs.

Suggestion how to merge:

  • the up-to-date catalog is the "master" -- it is updated by the Alert Pipeline every night.
  • after each Data Release, transfer the Object Catalog generated by the DR to the Base Site and place it "near" the up-to-date catalog.
  • flag all objects in the up-to-date catalog as "not-merged"
  • perform "lazy", "on-the-fly" merge. How? When we run Alert Production we run a spatial association that matches just-detected objects with the up-to-date Object catalog. So, immediately after running that association, run additional association that tries to associate just added/updated objects which are still flagged as "non-merged" with the corresponding piece of the DR-based Object catalog. When a match is found:
    • update appropriate fields of the matched object in the master up-to-date catalog
    • flag the object as "merged"
    • delete (or flag as "merged") the corresponding object in the DR Object catalog.

If that is too costly to run in real time inside Alert Production pipeline, we can run it offline during the day. Of course, the merge has to finish for all objects when a new DR Object Catalog becomes ready.