Changes between Version 9 and Version 10 of db/DC2/PartitioningTests


Ignore:
Timestamp:
05/03/2007 01:52:47 AM (13 years ago)
Author:
smm
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • db/DC2/PartitioningTests

    v9 v10  
    1414    The responsibility of this phase is to make sure that changes to __Object__ (inserts, updates, possibly deletes), __DIASource__ (inserts only), and __Alert__ (inserts only) are present on disk. There is some (TODO: what is this number) amount of time during which we are guaranteed not to revisit the same FOV.   
    1515 
    16 Note that LSST has a requirement to send out alerts within 60 seconds of performing an observation. Of the 3 AP phases, only '''compare-and-update''' is in the critical timing path. This is illustrated by the following (greatly simplified) diagram: 
     16Note that LSST has a requirement to send out alerts within 60 seconds of image capture (there is a stretch goal of 30 seconds). Of the 3 AP phases, only '''compare-and-update''' is in the critical timing path. The telescope will be taking data for 1 FOV every 37 seconds: 15 sec exposure, 2 sec readout, 15 sec exposure, 5 sec readout and slew. 
     17 
     18This is illustrated by the following (greatly simplified) diagram: 
    1719 
    1820[[Image(APPhases.png)]] 
    1921 
    20 In this diagram, processing of a FOV starts right after observation with the image processing pipeline (IPP) which is followed by the DP, and finally the AP. The yellow and red boxes together represent processing which must happen in the 60 second window. Please note the boxes are not drawn to scale - IPP and DP are likely to take up more of the 60 second window than the diagram suggests. Also note that interaction with the moving object pipeline (MOPS) is omitted, but that there is some planned interaction between AP and MOPS.    
     22In this diagram, processing of a FOV starts right after observation with the image processing pipeline (IPP) which is followed by the DP, and finally the AP. The yellow and red boxes together represent processing which must happen in the 60 second window. Please note the boxes are not drawn to scale - IPP and DP are likely to take up more of the 60 second window than the diagram suggests. Also note that interaction with the moving object pipeline (MOPS) is omitted, but that there is some planned interaction between AP and MOPS (notably when a ''DIA source'' is mis-classified as an ''object'' rather than a ''moving object''). 
    2123 
    2224The database tests are currently focused on how to partition __Object__ and __DIASource__ such that the '''prepare phase''' is as fast as possible, and on how to perform the distance based crossmatch of the '''compare-and-update phase'''. Tests of database updates, inserts, and of how quickly such changes can be moved from in-memory tables to disk based tables will follow at a later date. 
     
    6769So with coarse chunks, each logical table (__Object__, __DIASource__) is split into 13218 physical tables. With fine chunks, 335482 physical tables are needed. Due to OS filesystem limitations, chunk tables will have to be distributed among multiple databases (this isn't currently implemented).  
    6870 
    69 == Testing == 
     71= Testing = 
    7072 
    7173=== Hardware === 
    7274 
     75 * SunFire V240 
     76 * 2 UltraSPARC IIIi CPUs, 1503 MHz 
     77 * 16 GB RAM 
     78 * 2 Sun StoreEdge T3 arrays, 470GB each, configured in RAID 5, sustained sequential write speed (256KB blocks) 150 MB/sec, read: 146 MB/sec 
     79 * OS: Sun Solaris sun4x_510 
     80 * MySQL: version 5.0.37 
     81 
    7382=== General Notes === 
    7483 
    75 Each test is run with both "skinny" rows (USNO-B with minimal additions, ~100bytes), and "fat" rows (USNO-B with 200 DOUBLE columns set to random values added) that match the expected row size (including overhead) of the LSST __Object__ table. Any particular test is always run twice in row: this should shed some light on how OS caching of files affects the results. Between sets of tests that touch the same tables, the `bustcache` program is run in an attempt to flush the operating system caches (it does this by performing random 1MB reads from the USNO-B data until 16GB of data have been read). 
     84Each test is run with both "skinny" ''objects'' (USNO-B with some additions, ~100bytes), and "fat" ''objects'' (USNO-B plus 200 DOUBLE columns set to random values) that match the expected row size (including overhead) of the LSST __Object__ table. Any particular test is always run twice in a row: this should shed some light on how OS caching of files affects the results. Between sets of tests that touch the same tables, the `bustcache` program is run in an attempt to flush the operating system caches (it does this by performing random 1MB reads from the USNO-B data until 16GB of data have been read). 
    7685 
    77 === Test description === 
     86=== Test descriptions === 
    7887 
    79  * Reading the test FOVs into in-memory table(s) with no indexes whatsoever. 
    80  * Reading the test FOVs into in-memory table(s) that have the required indexes created before data is loaded. These indexes are: 
     88 * Read __Object__ data from the test FOVs into in-memory table(s) with no indexes whatsoever. 
     89 * Read __Object__ data from the test FOVs into in-memory table(s) that have the required indexes created before data is loaded. These indexes are: 
    8190   * a primary key on ''id'' (hash index) and 
    8291   * a B-tree index on ''zoneId'' for fine chunks, or (for all others) a composite index on ''(zoneId, ra)''. 
    83  * Reading the test FOVs into in-memory table(s) with no indexes whatsoever, then creating the same indexes as above after loading is complete. 
     92 * Read __Object__ data from the test FOVs into in-memory table(s) with no indexes whatsoever, then create indexes (the same ones as above) after loading finishes. 
    8493 
     94Note that all tests except the fine chunking tests place ''objects'' into a single __InMemoryObject__ table (and ''DIA sources'' into a single __InMemoryDIASource__ table) which are then used to for cross-matching tests. The fine chunking tests place each on-disk table into a separate in-memory table. This complicates the crossmatch implementation somewhat, but allows for reading many chunk tables in parallel without contention on inserts to a single in-memory table. It allows crossmatch to be parallelized by having different clients call the matching routine for different sub-regions of the FOV (each client is handled by a single thread on the MySQL server). 
    8595 
     96The following variations on the basic crossmatch were run (all on in-memory tables): 
     97 
     98 * Test performance using both match-orders: ''objects'' to ''DIA sources'' and ''DIA sources'' to ''objects''  
     99 * Test both slim and wide matching: 
     100   * a slim match stores results simply as pairs of keys by which ''objects'' and ''DIA sources'' can be looked up 
     101   * a wide match stores results as a key for a ''DIA source'' along with the values of all columns for the matching ''object''. This is important for the fine chunk case, since looking up objects becomes painful when they can be in one of 100+ in-memory tables. 
     102   
    86103= Performance Results = 
    87104