Ticket #2533 (deferred new functionality)

Opened 7 years ago

Last modified 6 years ago

PipeQA and user queries via qserv

Reported by: jbecla Owned by: jbecla
Priority: normal Milestone: qserv-usr2013
Component: qserv Keywords:
Cc: danielw, jbecla Blocked By: #1848, #2056
Blocking: Project: LSST
Version Number:
How to repeat:

not applicable

Description (last modified by jbecla) (diff)

We need to be able to run PipeQA and selected user queries via qserv. Planning to use DC_W13_Stripe82 data set, planning to do tests on lsst-db2. Need to:

  • come up with a list of PipeQA queries [Jacek]
  • find how the PipeQA queries change overtime [Jacek]
  • prepare space on lsst-db2 [Jacek]
  • dump the database and transfer to lsst-db2 [Jacek]
  • split RunDeepForcedSource into ~100-million-row files [Jacek]
  • load RunDeepForcedSource [Douglas]
  • partition/load all other tables as needed, Match tables will stay unpartitioned until #1848 is done [Douglas]
  • prepare micro-version of DC_W13_Stripe82 (~1000 RunDeepSource rows + corresponding data from other tables) [Jacek]
  • load the micro-version into qserv [Douglas]
  • run the tests and document [Jacek]
  • fix critical issues discovered [all]
  • enable access to few selected users [Jacek]

Change History

comment:1 Changed 7 years ago by jbecla

  • Status changed from new to assigned

comment:2 Changed 7 years ago by jbecla

List of queries run by PipeQA - see dbPipeQAQueries

comment:3 Changed 7 years ago by jbecla

  • Blocked By 2056 added

comment:4 Changed 7 years ago by jbecla

  • Description modified (diff)

comment:5 Changed 7 years ago by jbecla

Added comment about how the PipeQA queries are likely to change over time, see dbPipeQAQueries page.

comment:6 Changed 7 years ago by jbecla

  • Description modified (diff)

comment:7 Changed 7 years ago by jbecla

  • Description modified (diff)

comment:8 Changed 7 years ago by jbecla

  • Status changed from assigned to inTicketWork

comment:9 Changed 7 years ago by jbecla

  • Owner changed from jbecla to douglas
  1. The DC_W13_Stripe82 data (as csv) files are transfered, see /u1/DC_W13_csv/.
  2. RunDeepForcedSource is divided into 100-million-row files, see /u1/DC_W13_csv/RunDeepForcedSource.
  3. The micro-version of DC_W13_Stripe82 (1000 RunDeepSource rows + corresponding data from other tables) is in /u1/DC_W13_1K/ (as output from mysqldump).
  4. Let's put everything on /u1/, I created directory called "qserv", owner: douglas. This file system is now formatted as xfs, so it should handle very large files.

Douglas - reassigning to you. Let me know if there is anything I can help with.

Last edited 7 years ago by jbecla (previous) (diff)

comment:10 Changed 7 years ago by jbecla

  • Cc jbecla added
  • Milestone changed from qserv-fdr to qserv-usr2013

comment:11 Changed 7 years ago by jbecla

  • Owner changed from douglas to jbecla
  • Description modified (diff)

The last two bullets still to-do, all above done

comment:12 Changed 6 years ago by jbecla

  • Priority changed from critical to normal
  • Status changed from inTicketWork to deferred
Note: See TracTickets for help on using tickets.