wiki:DM/buildbot/Daily_Status
Last modified 8 years ago Last modified on 06/08/2011 03:23:26 PM

DM/buildbot/Daily_Status

Status commentary on the routine buildbot runs.

For more detailed discussion of buildbot's Weekly Production Runs (and misses) see Weekly Production.

Date Triggered Trunk Vs CurrentRelease Pipeline Release Incremental Full Trunk Vs Trunk Full Trunk Vs Trunk NoTestDatarel Trunk Vs Trunk
2011
June
8TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: Yesterday's run to test galaxy Modeling fix, worked.
7TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: Yesterday's run to test Galaxy Modeling hung during processing of 1st record in each slice.
6TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: No production run
5TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: No production run
4TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: No production run
3TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: No production run
2TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: Trunk production run start evening of 1 June completed successfully
1TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: No production run
May
31TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: No production run
30TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: No production run
29TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: Wrapping up from successful trunk run yesterday.
28TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: Successful TAG run mid-day; started trunk run later.
25TvC: N/A RP: OK RI: OK TvT: Updated benchmark suite yesterday; nightly run totally successful. TvTNT:N/A DTvTNT: No production run
24TvC: N/A RP: OK RI: OK TvT: Back to successful build and test except for datarel/endToEnd.py TvTNT:N/A DTvTNT: Rebuilt stack at end of yesterday; no production run
23TvC: N/A RP: OK RI: OK TvT: Failed due to HW problems on DB server TvTNT:N/A DTvTNT: No production run
20TvC: N/A RP: OK RI: OK TvT: Back to datarel only failing due to old benchmark suite TvTNT:N/A DTvTNT: Trunk Weekly Run: marginally successful; Tagged Weekly Run: unsuccessful; see DM/buildbot/Weekly_Production
19TvC: N/A RP: OK RI: OK TvT: Auto build failed last night due to eups lock hold; removed lock and rerun is progressing. TvTNT:N/A DTvTNT: 3 Failed production runs. First two code issues (potentially resolved), third was hardware issue. Harware issue resolved; second code issue not resolved as shown in new run which is progressing but generating no 'src'.
18TvC: N/A RP: OK RI: OK TvT: meas_utils still failing in test. TvTNT:N/A DTvTNT: Gambled on last minute fix to source Measurement policy setup however Tuesday night production run failed.
17TvC: N/A RP: OK RI: OK TvT: Lots of code development yesterday: meas_utils and obs_lsstSim are both failing. Developers notified. TvTNT:N/A DTvTNT: No production run. Test Build terminated on test error even though tests should be noted but ignored. Will probably install the new version which has been working on fbot for a week.
16TvC: N/A RP: OK RI: OK TvT: Same as yesterday; just fixed bug and updated benchmarks; retesting; benchmark still failed... TvTNT:N/A DTvTNT: Production run on Friday night; successful.
13TvC: N/A RP: OK RI: OK TvT: datarel now failing due to bad comparison operation against an unexpected type. TvTNT:N/A DTvTNT: No production run; rebuild was OK.
12TvC: N/A RP: OK RI: OK TvT: MOPS was removed from build list; datarel benchmarks were updated; trunkVsTrunk script was updated TvTNT:N/A DTvTNT: No production run
11TvC: N/A RP: OK RI: OK TvT: Same dayMOPS and datarel errors, as expected. TvTNT:N/A DTvTNT: Production run on late Tuesday completed successfully
10TvC: N/A RP: OK RI: OK TvT: Same as yesterday; except afwdata is missing from eups DB rebuild; reinstalled, retesting now. Daterel now fails in endToEnd.py benchmark verification of PSF. TvTNT:N/A DTvTNT: No production run
9TvC: N/A RP: OK RI: OK TvT: Same as Friday for dayMops. datarel failing due to bad eups DB; wiped bad entry and retesting- which later failed the night's test. TvTNT:N/A DTvTNT: Production build on Saturday OK
6TvC: N/A RP: OK RI: OK TvT: Same as yesterday TvTNT:N/A DTvTNT: No production run
5TvC: N/A RP: OK RI: OK TvT: Same as yesterday TvTNT: N/A DTvTNT: No production run
4TvC: N/A RP: OK RI: OK TvT: Same as yesterday TvTNT: N/A DTvTNT: OK
3TvC: N/A RP: OK RI: OK TvT: Failing on mops_daymops and datarel. Mops_daymops still needs slalib installed; datarel failed in endToEnd.py due to changes in psf persistence format; benchmark has been updated. TvTNT: N/A DTvTNT: Wait on DB ingest of new star+galaxy dataset for SrcAssoc processing.
2TvC: N/A RP: OK RI: OK TvT: Failing on mops_daymops, meas_multifit, datarel. Mops_daymops is asking TCT for 3rd party pkg approval; meas_multifit was modified and hopefully fixed this am; datarel had the benchmark suite updated to reflect the new psf algorithm. A full rebuild has been initiated; rebuild complete, only mops_daymops fails - as expected. TvTNT: N/A DTvTNT: The SVN upgrade on the cluster caused build failures this weekend. Restarting the local buildbot slave solved the problem.
Apr
29TvC: N/A RP: OK RI: OK TvT: Failed in mops_daymops due to missing dependencies (slalib, eigen >= 3.0). TvTNT: N/A DTvTNT: OK;
28TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: OK; no test failures
27TvC: N/A RP: OK RI: OK TvT: Failed on datarel unit test; waiting on Ticket #1645 TvTNT: N/A DTvTNT: OK; No longer stopping on unit test failures, only reporting them. Fail only on compile failures.
26 TvC: N/A RP: OK RI: OK TvT: datarel:endToEnd.py failed in butler (input vs outout) comparison of icSrc; had just updated benchmark suite 22 hrs earlier; determine if recent changes could impact benchmark. Later run...ran out of disk space. TvTNT:N/A DTvTNT: No build yesterday. Benchmark test failed due to access permissions on cfhtlsDeep. Next run failed in datarel:endToEnd.py icSrc benchmark comparison #1645. Split compilation of datarel&dependencies from their unit tests; exit on compilation failures but report any&all unit test failures at end of compilation.
25TvC: N/A RP: OK; auto update of Release worked fine RI: Failed due to out-of-date Release installed; performed complete re-Release (lsstpkg install -C LSSTPipe); All OK TvT: Ran out of disk space and Buildslave died; needed hand restart; cleared disk space; now testing; test failed; rebuilding sandbox from scratch (again*2; need to clear hidden cache in /tmp/eups). TvTNT: N/A DTvTNT: Still failing due to mis-match of benchmark data and actual results. RHL just OK'ed upgrade of datarel benchmark values; data updated, datarel passes.
21TvC: RP: RI: TvT: mops_dayMops failed due to eups lock problem. Rerun will probably solve this. TvTNT: DTvTNT:
20TvC: N/A RP: Failed for same reason; ip_diffim and ap Release packages have been built and will be tested tonight. RI: Same as RP TvT: OK TvTNT: N/A DTvTNT: Build OK; discovered bug in handling of failed tests which didn't stop build. Solved bug but now need to update datarel test since code generating new & better PSF values doesn't match benchmark PSF value.
19TvC: N/A RP: The failures now are due to failed builds for ap and ip_diffim which still need to be integrated with the new afw interface. RI: Same as RP TvT: OK TvTNT: N/A DTvTNT: Build OK; waiting for astrometry_net_data fix....Fixed and Test in progress.
18TvC: N/A RP: Ray tracked down problem to use of gcc 4.5 instead of 4.4; fix in buildbot master is being tested....Fix worked so ndarray is compiling correctly. The failures now are due to failed builds for ap and ip_diffim which still need to be integrated with the new afw interface. RI: Same as RP TvT: Major merge of changes resulting from AFW mods occurred. Only AP's dependency list remains to be fixed so when AP and its dependencies are built in isolation, ap-tests find all required libraries. Datarel (& tests) builds OK. TvTNT: N/A DTvTNT: Generating a full stack for new weekly run.
15TvC: N/A RP: FAIL: afw-4.3.1.0 build failed due to RedHat include discrepancy; see http://dev.lsstcorp.org/buildlogs/willy.ncsa.illinois.edu/2011/04/14/14.33.39/EupsBuildDir/Linux64/afw-4.3.1.0/afw-4.3.1.0/build.log. eigen-2.0.15 build failed for unknown reason; see http://dev.lsstcorp.org/buildlogs/willy.ncsa.illinois.edu/2011/04/14/14.33.39/Linux64/external/eigen/2.0.15/include/Eigen/build.log. RI: FAIL same as RP TvT: OK TvTNT: N/A DTvTNT: Still waiting for new astrometry_net_data + associated SrcAssoc file; will rerun Production using version with bad headers (but good data) when lsst10 (DB host) returns to service. After DB returned to service, able to successfully run the Weekly Production run.
14TvC: N/A RP: Slave ceased functioning last night; had to restart the slave. Web-Forced an RP build; Failing on 'lsstpkg --tags=current LSSTPipe' when compiling daf_data 4.1.0.0+2. daf_data issue resolved. Realized the change in testbed may have caused issue. Reverted change and trying again. RI: Slave ceased functioning last night; had to restart the slave. TvT: slave ceased functioning; there were 4 buildbot jobs in the queue but they did not startup; had to restart the slave. Web-forced a TvT build; Rebuld OK TvTNT: N/A DTvTNT: Ready and waiting
13TvC: N/A RP: OK RI: OK TvT: Failed due to lock contention during 'utils' build; finished build and in eups-install TvTNT: N/A DTvTNT: Performed Tuesday run; single sensor failed when host ran out of memory
12TvC: N/A RP: OK RI: OK TvT: OK TvTNT: N/A DTvTNT: Ready for Tuesday run
11TvC: N/A RP: OK RI: OK TvT: OK TvTNT: N/A DTvTNT: Rebuild of code stack failed due to compilation errors from morning's check-ins; 11 am and the stack built OK.
10TvC: N/A RP: OK RI: OK TvT: OK TvTNT: N/A DTvTNT: No activity
9TvC: N/A RP: OK RI: OK TvT: Failed with compilation error in meas_algorithms #1616 TvTNT: NA DTvTNT: no run planned until TVT run succeeds (new stack) and developers want to exercise new algorithms.
8TvC: N/A RP: OK RI: OK TvT: OK TvTNT: N/A DTvTNT: New astrometry_net_data installed, new weekly run in progress
7TvC: N/A RP: OK RI: OK TvT: OK, Disk space cleared; Run time changed to midnight instead of mid-day; Midafternoon run failed when mops_daymops failed due to eups lock contention. Hand recompilation of that package succeeded. TvTNT: N/A DTvTNT: All support files installed, weekly run completed, determined that astrometry_net_data indices were incorrect so need rerun after new indices are installed.
6TvC: N/A RP: OK RI: OK TvT: Ran out of disk space during mid-day run. TvTNT:N/A DTvTNT: waiting on ancillary files supporting the new data to be installed
5TvC: N/A RP: OK RI: OK TvT: OK TvTNT:N/A DTvTNT: 1st attempt at DB unification failed; will try again once new data installed
4TvC: N/A RP: Failed when unscheduled rebooot failed to restore necessary daemons; retry OK RI: Failed for same reason as RP; retry OK TvT: OK TvTNT: N/A DTvTNT: need to put under Buildbot Management but first a bit of DB unification per K-T
3TvC: N/A RP: OK RI: OK TvT: OK TvTNT: N/A DTvTNT: Successfully ran Weekly Production Run script as user buildbot...now need to put under automatic Buildbot Management
2TvC: N/A RP: OK RI: OK TvT:OK TvTNT: N/A DTvTNT: Still working on issues related to performing a production run at user buildbot
1TvC: N/A RP: OK RI: OK TvT: OK TvTNT: N/A DTvTNT: Integration issues with new trunk apps/MW
Mar
31 TvC: N/A RP: OK RI: OK TvT: OK TvTNT: N/A DTvTNT: access issues resolved; non-buildbot-automated test runs failing so buildbot-automation is delayed
30 TvC: N/A RP: OK RI: OK TvT: OK TvTNT: N/A DTvTNT: waiting on access to account resources
29 TvC: removed from display until active debug starts RP: OK RI:OK TvT:OK TvTNT:tbd DTvTNT: will be converted to weekly Production Run asap
28 TvC: broken RP: OK RI: OK TvT: OK TvTNT: tbd DTvTNT: OK
27 TvC: broken RP:failed due to eups cache corruption; reinitialized cache; all OK RI: same as RP since on same host TvT: OK TvTNT: TBD DTvTNT: OK
25 TvC: broken RP: OK RI: OK TvT: OK TvTNT: tbd DTvTNT: installed and working; under development today so expect multiple runs; ...now stable
24 TvC: buildbot process broken RP: OK RI: OK TvT: Recent errors due to lock access timout have been worked around. Now OK TvTNT: Needs to be moved onto new buildbot slave DTvTNT: Needs to be retested on cluster prior to installation