wiki:MopsRunlog
Last modified 9 years ago Last modified on 03/23/2010 11:18:47 PM

DayMOPS Runs

A fairly comprehensive list of every MOPS run which has been performed

2008 One-lunation Run With Noise

Performed on mops64 using PS-SSM, an LSST limiting magnitude.

The first lunation contained 13 days of observation. Detections were split by "field" via PS-MOPS, resulting in yielding 21,134,266 detections in all fields (note that some detections may be non-unique due to overlap in fields.)

Each field was processed using Kubica's command-line findTracklets with max_obs 2. Deep stacks (fields observed >= 4 times) were post-processed by collapseTracklets, purifyTracklets, and removeSubets --keepOnlyLongest=true. Unfortunately, the max velocity settings as well as the time spent processing are lost.

This yielded 7,600,724 tracklets.

Processed with linkTracklets, 185458 tracks were generated in 52 min 55 sec.

Re-running with eval true generated these results:

Pairwise: Per. Corr. = 0.822181  Per. Found = 0.000000  Per. Found2 = 0.000000
Full Track: There are 13654 true tracks with 6 or more obs on 3 or more days.
Full Track: We found 94937 tracks with 6 or more obs on 3 or more days.
Full Track: Per. Corr. = 0.131719      Per. Found = 0.915849
Full Track: AAC (at 0.90) = 344378279.000000
Full Track: AAC (at 0.75) = 209062464.000000
Full Track Sum: 0.131719 0.915849 344378279.000000 209062464.000000
Exact Match: Number = 2642,  Percent Found = 0.193496, Percent Correct = 0.027829
3166.60user 65.36system 1:24:09elapsed 64%CPU (0avgtext+0avgdata 0maxresident)k
25365704inputs+159776outputs (376563major+8363782minor)pagefaults 0swaps 

Jan. 2010 Partial Sky, No-Noise Runs

Data set one: The cadence used was LSST Op-Sim output, but only observations within a 15 degree x 15 degree square centered on 0,0 were kept (equivalent to DC3b, PT1). These op-sim images were used to simulate observation of objects in the SSM with an LSST limiting magnitude (5 sigma above the limiting mags reported by opsim for each observation). The resulting detections are here: http://www.astro.washington.edu/users/ljones/ephems/obsidaaei.tar.gz http://www.astro.washington.edu/users/ljones/ephems/obsidabfj.tar.gz http://www.astro.washington.edu/users/ljones/ephems/obsidacg.tar.gz and http://www.astro.washington.edu/users/ljones/ephems/obsidadh.tar.gz

There are 14075373, 7897987, 7604525 + 8533019 = 38,110,904 total diasources in these datasets. These are moving objects only .. no diasources from transients.

Per-Night findTracklets runs

Using the first night of data from the second data set (no longer linked above, as it was 'as processed for imsim' and thus not suitable for further testing - was a stopgap only. the second data set did NOT include deep drilling fields). Conducted using Kubica's C implementation of findTracklets, using various settings:

max obs 2, no PHT, max velocity .5, maxt .5:
0:58, 436026 tracklets


max obs 2, no PHT, max velocity .75, maxt .5:
2:59, 473922 tracklets


max obs 2, no PHT, max velocity 1.0, maxt .5:
6:58, 494266 tracklets


max obs 2, no PHT, max velocity 1.25, maxt .5:
15:05, 506354 tracklets


max obs 2, no PHT, max velocity 1.5, maxt .5:
26:36, 511436 tracklets





max obs 2, no PHT, max velocity .5, maxt .0625:
0:54, 436026 tracklets

max obs 2, no PHT, max velocity 1.0, maxt .0625:
7:08, 494266 tracklets






max obs 4, no PHT, max velocity .5, maxt .5:
0:30, 494724 tracklets

max obs 4, no PHT, max velocity .75, maxt .5:
1:42,  537919 tracklets

max obs 4, no PHT, max velocity 1, maxt .5:
4:15, 562491 tracklets

Note that the time to run grows more-than-linearly with the max velocity setting; the impact of changing the max time setting is not that significant. Increasing the max_obs setting to 4 actually seems to reduce the total run-time.

FindTracklets + LinkTracklets on Second Data Set, one month

In the first 30 days of observation, the patch of sky considered was visited on the 1st, 12th, 21st, 29th, and 30th days.

FindTracklets was run on each night's data, with a max velocity limit of .5 deg/day and a maximum time limit between detections of .0626 days (90 min)

Night     Input detections (#)       Output tracklets (#)      Runtime (sec)
1         47269                      300694                     9.3
12        6440                       16176                      0.6
21        3196                       90469                      2.2
29        25380                      603567                    24.6
30        14389                      1239469                   94.1

linkTracklets was run with an upper bound on acceleration of .0002 deg/(day2) and a vtree error threshold of .0005 deg. The linkTracklets completed in 12 hours, 43 minutes, generating 21691607 tracks. A second run was performed, and this one terminated in 11 hours, 38 minutes. An identical number of tracks were generated.

Another linkTracklets run was attempted with the default acceleration limit of .02 deg/day/day and a vtree error threshold of .0002 deg; this run lasted over 90 hours without completion.

Using debug output, it was noted that linking was very slow with endpoint trees on night 1 and night 29. This is much more noticeable on the second run, when acceleration was high; note that with acceleration of .02 deg/day/day and 28 days, a tracklet on the first night would be capable of changing velocity +/- .56 deg/day, meaning that virtually all prograde tracklets would be compatible in velocity, and the feasible change in (ra, dec) would be between 7.8 and 21.8 degrees, meaning that any two tracklets from the same field of view would be compatible (RA0, dec0) as well as velocity. As a result, it is quite possible that for any field visited on both nights, linkTracklets would essentially attempt brute-force linking of all endpoint tracklets, but with added overhead of a tree-walk, resulting in painfully slow execution and an explosion in resulting tracks.

Another linkTracklets run was attempted using acceleration of .0002 deg/day2, vtree error thresh of .0005 deg, but only the first 20 nights of observation, a total of . This run took 43 hours, 43 minutes and generated 27366830 tracks. This is highly suspicious when compared to the results of the first run, with identical acceleration and vtree thresh, but more days, yet the run was conspicuously faster!

FindTracklets on Second Data Set, One Year

Using the second data set, one year of data was considered. All observations from a single night were lumped together and sent to findTracklets with PHT mode enabled, a max velocity setting of .5 deg/day, and a maxt setting of .0625. Nights were numbered such that the first night containing observations was night 1.

Processing on nights 160, 67, 123, 125, 127, 131 terminated with seg faults; all other nights were processed successfully.

Full results:

Day # diasources Elapsed time # tracklets
day 1 47269 0:09.30 300694
day 12 6440 0:00.68 16176
day 21 3196 0:02.45 90469
day 29 25380 0:24.69 603567
day 30 14389 1:36.68 1239469
day 34 2209 0:00.14 1424
day 40 93637 14:10.62 8864868
day 41 36692 1:00.29 1596556
day 42 59279 2:32.70 3294136
day 49 83859 12:43.58 8919508
day 58 50734 3:42.46 2784612
day 59 1363 0:00.03 0
day 60 58238 18:18.49 9899104
day 67 176892 2:36:34 CRASHED
day 68 207818 2:57:21 CRASHED
day 69 5675 0:08.90 346734
day 70 37809 3:52.04 5120444
day 87 9817 0:00.41 6152
day 88 48308 0:30.38 728364
day 89 4383 0:00.15 1069
day 90 2333 0:00.06 0
day 91 18404 0:31.63 1002872
day 93 67842 1:05:27 22681955
day 94 157127 1:28:29 27350538
day 95 5340 0:00.79 16254
day 96 18828 0:02.40 74419
day 104 2691 0:00.13 194
day 105 1638 0:00.02 0
day 106 2902 0:00.05 239
day 107 15299 0:00.50 2983
day 108 1949 0:00.03 0
day 109 9270 0:05.66 222923
day 114 163633 26:56.93 17493384
day 116 98561 22:20.40 12696754
day 117 59574 3:46.67 3361532
day 118 1540 0:00.34 163
day 120 23714 0:23.11 463745
day 121 90004 8:10.33 6248493
day 123 319182 3:22:26 CRASHED
day 125 219353 4:48:15 CRASHED
day 127 374243 12:24:11 CRASHED
day 129 100973 13:22.81 13457974
day 131 284791 30:31:40 CRASHED
day 135 76649 13:28.96 10723392
day 138 68597 19:03.89 12608729
day 141 1903 0:00.04 0
day 143 112796 4:07:22 42816210
day 144 82619 15:11.53 10777934
day 148 5736 0:05.00 180763
day 149 8032 0:01.21 42865
day 150 69856 7:26.51 5612246
day 152 15190 1:30.00 1880990
day 156 117180 55:07.35 24470279
day 157 36031 0:01.43 14152
day 158 167016 48:33.02 25305752
day 159 18663 0:01.10 19712
day 160 302535 4:54:05 CRASHED
day 163 16735 1:14.10 2166848
day 167 80536 13:16.76 7930304
day 170 13972 0:20.80 678569
day 175 8456 0:40.50 827584
day 178 2341 0:00.09 235
day 180 18036 0:01.63 51811
day 181 18784 0:22.23 523624
day 183 5804 0:00.46 1625

LSST Day-MOPS run

(redo this after re-running to get more info)

Attachments