wiki:db/DC2/CrossmatchPerf
Last modified 6 years ago Last modified on 02/01/2013 02:23:10 PM

Crossmatch performance

Zone height is 1 arc-minute and match distance is 3 arcseconds for all tests.

A single in-memory table of objects matched against a single in-memory table of DIA sources (and vice versa) :

FOV Row sizematch-width match direction Time - 1st run Time - 2nd run Time - 3d run
High density Skinny slim objects vs. DIA sources 0m51.989s 0m52.081s 0m52.169s
DIA sources vs. objects 0m35.692s 0m35.371s 0m35.442s
wide objects vs. DIA sources 0m52.489s 0m52.498s 0m52.521s
DIA sources vs. objects 0m36.069s 0m36.180s 0m35.805s
Fat slim objects vs. DIA sources 1m4.088s 1m4.085s 1m3.906s
DIA sources vs. objects 0m41.543s 0m41.763s 0m41.480s
wide objects vs. DIA sources 1m19.124s 1m20.902s 1m11.180s
DIA sources vs. objects 0m57.569s 1m0.377s 0m48.830s
Low density Skinny slim objects vs. DIA sources 0m1.284s 0m1.278s 0m1.278s
DIA sources vs. objects 0m0.947s 0m0.933s 0m0.945s
wide objects vs. DIA sources 0m1.307s 0m1.285s 0m1.283s
DIA sources vs. objects 0m0.980s 0m0.961s 0m0.950s
Fat slim objects vs. DIA sources 0m1.582s 0m1.596s 0m1.567s
DIA sources vs. objects 0m1.109s 0m1.108s 0m1.104s
wide objects vs. DIA sources 0m1.995s 0m2.000s 0m1.883s
DIA sources vs. objects 0m1.568s 0m1.580s 0m1.459s
Avg. density Skinny slim objects vs. DIA sources 0m5.891s 0m5.887s 0m5.926s
DIA sources vs. objects 0m4.085s 0m4.174s 0m4.148s
wide objects vs. DIA sources 0m5.991s 0m5.951s 0m5.939s
DIA sources vs. objects 0m4.211s 0m4.199s 0m4.161s
Fat slim objects vs. DIA sources 0m7.392s 0m7.662s 0m7.356s
DIA sources vs. objects 0m4.831s 0m4.870s 0m4.871s
wide objects vs. DIA sources 0m9.118s 0m9.155s 0m8.587s
DIA sources vs. objects 0m6.807s 0m6.811s 0m6.234s

For the fine chunking scheme, the match is processed one stripe at a time (for stripes overlapping the FOV) yielding 11 units of work that are currently executed in serial fashion, but could be run in parallel.

FOV Row sizematch-width match direction Time - 1st run Time - 2nd run
High density Skinny slim objects vs. DIA sources 1m20.287s 1m20.58s
DIA sources vs. objects 0m56.065s 0m56.5s
wide objects vs. DIA sources 1m21.169s 1m21.151s
DIA sources vs. objects 0m57.72s 0m59.873s
Fat slim objects vs. DIA sources 2m9.347s 2m9.122s
DIA sources vs. objects 1m18.949s 1m18.181s
wide objects vs. DIA sources 2m26.441s 2m26.048s
DIA sources vs. objects 1m37.277s 1m38.656s
Low density Skinny slim objects vs. DIA sources 0m10.592s 0m10.595s
DIA sources vs. objects 0m8.498s 0m8.44s
wide objects vs. DIA sources 0m11.056s 0m11.019s
DIA sources vs. objects 0m9.024s 0m9.228s
Fat slim objects vs. DIA sources 0m12.064s 0m12.01s
DIA sources vs. objects 0m9.141s 0m9.14s
wide objects vs. DIA sources 0m16.527s 0m16.656s
DIA sources vs. objects 0m13.04s 0m13.156s
Avg. density Skinny slim objects vs. DIA sources 0m15.764s 0m15.812s
DIA sources vs. objects 0m13.407s 0m13.378s
wide objects vs. DIA sources 0m16.19s 0m16.135s
DIA sources vs. objects 0m14.016s 0m14.033s
Fat slim objects vs. DIA sources 0m21.451s 0m21.437s
DIA sources vs. objects 0m16.337s 0m16.392s
wide objects vs. DIA sources 0m26.95s 0m26.814s
DIA sources vs. objects 0m21.757s 0m21.92s


Effect of zone height on cross-match performance

Varying the zone height doesn't have a huge effect on performance - smaller zone heights are slightly faster than larger ones, until one reaches 20 arcseconds. At a zone height of 15 arcseconds, performance suddenly drops by a factor of 12 to 16. One possible explanation for this would be that the in-memory zone tables involved in the match pass a size threshold below which table scans are used instead of b-tree range lookups.

2 arcminute zone height

FOV Row sizematch-width match direction Time
High density Skinny slim objects vs. DIA sources 0m54s
DIA sources vs. objects 0m36s
wide objects vs. DIA sources 0m55s
DIA sources vs. objects 0m36s

45 arcsecond zone height

FOV Row sizematch-width match direction Time
High density Skinny slim objects vs. DIA sources 0m51.203s
DIA sources vs. objects 0m34.112s
wide objects vs. DIA sources 0m51.615s
DIA sources vs. objects 0m34.511s

30 arcsecond zone height

FOV Row sizematch-width match direction Time
High density Skinny slim objects vs. DIA sources 0m50.535s
DIA sources vs. objects 0m33.615s
wide objects vs. DIA sources 0m50.989s
DIA sources vs. objects 0m34.261s

20 arcsecond zone height

FOV Row sizematch-width match direction Time
High density Skinny slim objects vs. DIA sources 0m54.990s
DIA sources vs. objects 0m35.113s
wide objects vs. DIA sources 0m55.579s
DIA sources vs. objects 0m35.657s

15 arcsecond zone height

FOV Row sizematch-width match direction Time
High density Skinny slim objects vs. DIA sources 11m18.951s
DIA sources vs. objects 8m25.763s
wide objects vs. DIA sources 11m20.142s
DIA sources vs. objects 8m28.906s