wiki:AstronomicalTablesUsingPython
Last modified 10 years ago Last modified on 10/13/2009 09:50:36 AM

Notes on Python Package ATpy

Set Up

Download and installation instructions at

ATpy

ATpy authors: Thomas Robitaille (robitaille\@users.sourceforge.net) and Eli Bressert (elibre\@users.sourceforge.net)

The latest ATpy changes can be downloaded from the following SVN repository:

https://atpy.svn.sourceforge.net/svnroot/atpy

In order to test ATpy, I installed or have installed the following on my MacBookPro? (Mac OS X 10.5.7, 2.53 GHz Intel Core 2 Duo):

  • Python 2.5.1
  • mysql Ver 14.14 Distrib 5.1.39, for apple-darwin9.5.0 (i386) using readline 5.1
  • ATpy 0.9.2 (latest from SVN)
  • numpy-1.3.0-py2.5-macosx-10.5-i386.egg
  • pyfits 2.2
  • vo-0.4-py2.5.egg
  • setuptools-0.6c9-py2.5.egg
  • MySQL_python-1.2.3c1-py2.5-macosx-10.5-i386.egg

Summary

I first installed ATpy v. 0.9.0, ran some tests, and uncovered a bug associated with handling a MySQL table with an 8-byte integer data type (see Example 6 below). I e-mailed the ATpy authors and they fixed the bug (along with others), and delivered ATpy v. 0.9.1. I also suggested that it would be more natural to specify database tables by name rather than by index, and they responded by deprecating the "tid" input argument and replacing it with the "table" argument in ATpy v. 0.9.1.

I wrote a unit test for ATpy v. 0.9.1, and it revealed limited-data-precision problems with IPAC, VO, and MySQL tables. These were largely fixed in the latest source code (v. 0.9.2) checked out from their SVN repository. See unit-test results below. MySQL tables have only 15 digits of precision for double-precision data. Thomas Robitaille ran my unit test against a PostgreSQL database and found similar data-precision problems, which he is now investigating.

My plan now is to expand the unit test to cover a broader range of data possibilities, including NaNs?, Infs, etc.

Example 1

Read in a 12-column binary FITS table with 5316 rows and write it out as an IPAC table:

import atpy
tbl = atpy.Table()
tbl.read(' mops.fits')  
tbl.write('mops.tbl') 
$ python atpytest.py
Auto-detected input type: FITS table
Auto-detected input type: IPAC table

Example 2

Read in a 12-column binary FITS table with 5316 rows and write it out as a MySQL database table:

import atpy
tbl = atpy.Table('mops.fits')
tbl.write('mysql', db='test', overwrite=True, verbose=False)
$ python atpytest2.py
Auto-detected input type: FITS table

Since EXTNAME = 'IPAC_PTF_MOPS' is defined in the header of the binary FITS table, database table IPAC_PTF_MOPS is created. The CFITSIO library treats EXTNAME as a character string and does not restrict its value from having spaces; however, ATpy will choke if the string does not contain a MySQL-compliant table name.

Here is a description of the newly created database table:

mysql> desc IPAC_PTF_MOPS;
+---------+------------+------+-----+---------+-------+
| Field   | Type       | Null | Key | Default | Extra |
+---------+------------+------+-----+---------+-------+
| SID     | bigint(20) | YES  |     | NULL    |       |
| RA_DEG  | double     | YES  |     | NULL    |       |
| RA_SIG  | double     | YES  |     | NULL    |       |
| DEC_DEG | double     | YES  |     | NULL    |       |
| DEC_SIG | double     | YES  |     | NULL    |       |
| MAG     | double     | YES  |     | NULL    |       |
| MAG_SIG | double     | YES  |     | NULL    |       |
| STARPSF | double     | YES  |     | NULL    |       |
| ANG     | double     | YES  |     | NULL    |       |
| ANG_SIG | double     | YES  |     | NULL    |       |
| LEN     | double     | YES  |     | NULL    |       |
| LEN_SIG | double     | YES  |     | NULL    |       |
+---------+------------+------+-----+---------+-------+
12 rows in set (0.00 sec)

Example 3

Read in a 12-column binary FITS table with 5316 rows and write it out as a SQLite database table:

import atpy
tbl = atpy.Table('mops.fits')
tbl.write('sqlite', 'russ.db', overwrite=True, verbose=True)
$ python atpytest3.py
Auto-detected input type: FITS table
Auto-detected input type: SQL table

This created a SQLite database called "russ.db".

Example 4

Read in a 12-column binary FITS table with 5316 rows and write it out as a VO table:

import atpy
tbl = atpy.Table('mops.fits')
tbl.write('mops.xml')   
mucha:Downloads rlaher$ python atpytest4.py 
Auto-detected input type: FITS table
Auto-detected input type: VO table

Example 5

Read in the following simple table from MySQL database "test" and write it out as an IPAC table:

mysql> desc mops;
+-------+---------+------+-----+---------+-------+
| Field | Type    | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+-------+
| mid   | int(11) | NO   |     | NULL    |       |
| x     | double  | NO   |     | NULL    |       |
+-------+---------+------+-----+---------+-------+
2 rows in set (0.00 sec)

mysql> select * from mops;
+-----+--------+
| mid | x      |
+-----+--------+
|  66 | 77.881 |
|  77 | 77.882 |
|  88 | 77.883 |
|  99 | 77.884 |
+-----+--------+
4 rows in set (0.01 sec)
import atpy
tbl = atpy.Table('mysql', table='mops', db='test', verbose=True)
tbl.write('mops2.tbl')
$ python atpytest5.py
Auto-detected input type: SQL table
Auto-detected input type: IPAC table

Here is the resulting IPAC table:

$ more mops2.tbl
|       mid|               x|
|       int|          double|
         66  7.788100000e+01 
         77  7.788200000e+01 
         88  7.788300000e+01 
         99  7.788400000e+01 

Example 6

This example is slightly more stressing than Example 5. Read in the MySQL database table created in Example 2 and write it out as an IPAC table:

import atpy
tbl = atpy.Table('mysql', table='IPAC_PTF_MOPS', db='test', verbose=True)
tbl.write('mops3.tbl')
mucha:Downloads rlaher$ python atpytest6.py 
Auto-detected input type: SQL table
Auto-detected input type: IPAC table

The first column of the 12-column MySQL database table is an 8-byte integer data type. This test failed because of the bigint with ATpy v. 0.9.0, but passed with ATpy v. 0.9.1.

Unit Test (v. 0.9.1)

I wrote a unit test that covers only IPAC, VO, FITS binary, and MySQL tables. The test consists of a manually constructed 10-row, two-column table, where the data type of the first column is int64 and the second column is float64. There was no issue for the FITS-binary-table case, where the table data were compared with another instantiation of the table data after writing it out to a FITS binary table and then reading it back in. In order to make the test run successfully for the cases involving IPAC, VO, and MySQL tables, I had to use the "assertAlmostEqual" unittest function for the second column of table data. For the MySQL table, the agreement was within 13 decimal places, whereas the IPAC table had agreement to within 8 decimal places, and the VO table had agreement only to within 4 decimal places. I e-mailed the ATpy authors with this information.

Here is the unit-test source:

#! /usr/bin/env python

# File: ATpyUnitTest.py
#
# Created: 12 October 2009, Russ Laher (laher@ipac.caltech.edu).  
#
# Unit test for ATpy.  Covers only IPAC, VO, FITS binary, and MySQL tables.

import os
import atpy
import unittest
from numpy import *

class ATpyUnitTest(unittest.TestCase):
    """A unit test for ATpy."""

    def setUp(self):
        self.t=atpy.Table(name='mops_ut')
        i = 987654321
        x=arange(i, i+10, dtype=int64)
        j = arange(10)
        y = empty(10, dtype=float64)
        y = j * 3.141592651111111111
        self.t.add_column('x', x, dtype=int64)
        self.t.add_column('y', y, dtype=float64)
        self.assertEqual(self.t, self.t)
        if (not os.path.exists('testATpy')):
            os.mkdir('testATpy')

    def tearDown(self):
        if (os.path.exists('testATpy/mops_ut.fits')):
            os.remove('testATpy/mops_ut.fits')
        if (os.path.exists('testATpy/mops_ut.tbl')):
            os.remove('testATpy/mops_ut.tbl')
        if (os.path.exists('testATpy/mops_ut.xml')):
            os.remove('testATpy/mops_ut.xml')
        os.rmdir('testATpy')

    def testFITS(self):
        self.t.write('testATpy/mops_ut.fits')             # Write binary FITS table
        t_fits=atpy.Table('testATpy/mops_ut.fits')        # Read binary FITS table
        for i in range(10):
            print "-->", self.t.data['x'][i], self.t.data['y'][i], t_fits.data['x'][i], t_fits.data['y'][i]
            self.assertEqual(self.t.data['x'][i], t_fits.data['x'][i])
            self.assertEqual(self.t.data['y'][i], t_fits.data['y'][i])

    def testMYSQL(self):
        # Write MySQL table
        self.t.write('mysql', db='test', overwrite=True, verbose=False) 
        # Read MySQL table
        t_mysql=atpy.Table('mysql', table='mops_ut', db='test', verbose=True)              
        for i in range(10):
            print "-->", self.t.data['x'][i], self.t.data['y'][i], t_mysql.data['x'][i], t_mysql.data['y'][i]
            self.assertEqual(self.t.data['x'][i], t_mysql.data['x'][i])
            self.assertAlmostEqual(self.t.data['y'][i], t_mysql.data['y'][i], 13)

    def testVO(self):
        self.t.write('testATpy/mops_ut.xml')        # Write VO table
        t_vo=atpy.Table('testATpy/mops_ut.xml')     # Read VO table        
        for i in range(10):
            print "-->", self.t.data['x'][i], self.t.data['y'][i], t_vo.data['x'][i], t_vo.data['y'][i]
            self.assertEqual(self.t.data['x'][i], t_vo.data['x'][i])
            self.assertAlmostEqual(self.t.data['y'][i], t_vo.data['y'][i], 4)

    def testTBL(self):
        self.t.write('testATpy/mops_ut.tbl')         # Write IPAC table
        t_tbl=atpy.Table('testATpy/mops_ut.tbl')     # Read IPAC table        
        for i in range(10):
            print "-->", self.t.data['x'][i], self.t.data['y'][i], t_tbl.data['x'][i], t_tbl.data['y'][i]
            self.assertEqual(self.t.data['x'][i], t_tbl.data['x'][i])
            self.assertAlmostEqual(self.t.data['y'][i], t_tbl.data['y'][i], 8)

if __name__ == '__main__':
    unittest.main()

Here is the output from running the unit test:

$ ./ATpyUnitTest.py > test.out
....
----------------------------------------------------------------------
Ran 4 tests in 0.167s

OK

Here is the STDOUT redirected to test.out:

$ cat test.out
Auto-detected input type: FITS table
Auto-detected input type: FITS table
--> 987654321 0.0 987654321 0.0
--> 987654322 3.14159265111 987654322 3.14159265111
--> 987654323 6.28318530222 987654323 6.28318530222
--> 987654324 9.42477795333 987654324 9.42477795333
--> 987654325 12.5663706044 987654325 12.5663706044
--> 987654326 15.7079632556 987654326 15.7079632556
--> 987654327 18.8495559067 987654327 18.8495559067
--> 987654328 21.9911485578 987654328 21.9911485578
--> 987654329 25.1327412089 987654329 25.1327412089
--> 987654330 28.27433386 987654330 28.27433386
Auto-detected input type: SQL table
--> 987654321 0.0 987654321 0.0
--> 987654322 3.14159265111 987654322 3.14159265111
--> 987654323 6.28318530222 987654323 6.28318530222
--> 987654324 9.42477795333 987654324 9.42477795333
--> 987654325 12.5663706044 987654325 12.5663706044
--> 987654326 15.7079632556 987654326 15.7079632556
--> 987654327 18.8495559067 987654327 18.8495559067
--> 987654328 21.9911485578 987654328 21.9911485578
--> 987654329 25.1327412089 987654329 25.1327412089
--> 987654330 28.27433386 987654330 28.27433386
Auto-detected input type: IPAC table
Auto-detected input type: IPAC table
--> 987654321 0.0 987654321 0.0
--> 987654322 3.14159265111 987654322 3.141592651
--> 987654323 6.28318530222 987654323 6.283185302
--> 987654324 9.42477795333 987654324 9.424777953
--> 987654325 12.5663706044 987654325 12.5663706
--> 987654326 15.7079632556 987654326 15.70796326
--> 987654327 18.8495559067 987654327 18.84955591
--> 987654328 21.9911485578 987654328 21.99114856
--> 987654329 25.1327412089 987654329 25.13274121
--> 987654330 28.27433386 987654330 28.27433386
Auto-detected input type: VO table
Auto-detected input type: VO table
--> 987654321 0.0 987654321 0.0
--> 987654322 3.14159265111 987654322 3.14159
--> 987654323 6.28318530222 987654323 6.28319
--> 987654324 9.42477795333 987654324 9.42478
--> 987654325 12.5663706044 987654325 12.5664
--> 987654326 15.7079632556 987654326 15.708
--> 987654327 18.8495559067 987654327 18.8496
--> 987654328 21.9911485578 987654328 21.9911
--> 987654329 25.1327412089 987654329 25.1327
--> 987654330 28.27433386 987654330 28.2743

Unit Test (v. 0.9.2)

Thomas Robataille delivered changes into version 0.9.2 of ATpy to their SVN repository, which completely fixes the limited-data-precision problems associated with IPAC and VO tables. He also noted that the unittest.assertAlmostEqual function considers only decimal places, not significant digits, and delivered a new function called "assertAlmostEqualSig" to remedy this. I incorporated it into the unit test for MySQL, which now passes with 15 significant digits. I also added a unit test for the SQLlite-table functionality. Here is the latest unit-test source code:

#! /usr/bin/env python

# File: ATpyUnitTest.py
#
# Created: 12 October 2009, Russ Laher (laher@ipac.caltech.edu).  
#
# Unit test for ATpy.  Covers only IPAC, VO, FITS binary, SQLite and MySQL tables.

import os
import atpy
import unittest
from numpy import *

class ATpyUnitTest(unittest.TestCase):
    """A unit test for ATpy."""

    def setUp(self):
        self.t=atpy.Table(name='mops_ut')
        i = 987654321
        x = arange(i, i+10, dtype=int64)
        j = arange(10)
        y = empty(10, dtype=float64)
        y = j * 3.141592651111111111
        self.t.add_column('x', x, dtype=int64)
        self.t.add_column('y', y, dtype=float64)
        self.assertEqual(self.t, self.t)
        if (not os.path.exists('testATpy')):
            os.mkdir('testATpy')

    def tearDown(self):
        if (os.path.exists('testATpy/mops_ut.fits')):
            os.remove('testATpy/mops_ut.fits')
        if (os.path.exists('testATpy/mops_ut.tbl')):
            os.remove('testATpy/mops_ut.tbl')
        if (os.path.exists('testATpy/mops_ut.xml')):
            os.remove('testATpy/mops_ut.xml')
        if (os.path.exists('testATpy/mops_ut.db')):
            os.remove('testATpy/mops_ut.db')
        os.rmdir('testATpy')

    def testFITS(self):
        self.t.write('testATpy/mops_ut.fits')             # Write binary FITS table
        t_fits=atpy.Table('testATpy/mops_ut.fits')        # Read binary FITS table
        for i in range(10):
            print "-->", self.t.data['x'][i], self.t.data['y'][i], t_fits.data['x'][i], t_fits.data['y'][i]
            self.assertEqual(self.t.data['x'][i], t_fits.data['x'][i])
            self.assertEqual(self.t.data['y'][i], t_fits.data['y'][i])

    def testMYSQL(self):
        # Write MySQL table
        self.t.write('mysql', db='test', overwrite=True, verbose=False) 
        # Read MySQL table
        t_mysql=atpy.Table('mysql', table='mops_ut', db='test', verbose=True)              
        for i in range(10):
            print "-->", self.t.data['x'][i], self.t.data['y'][i], t_mysql.data['x'][i], t_mysql.data['y'][i]
            self.assertEqual(self.t.data['x'][i], t_mysql.data['x'][i])
            # The following function, written by Thomas Robataille, is similar to unittest.assertAlmostEqual, 
            # except that it accounts for significant digits, rather than just decimal places.  It is
            # set up to check for 15 significant digits (it fails for 16, but passes for 15).
            self.assertAlmostEqualSig(self.t.data['y'][i], t_mysql.data['y'][i], 15)

    def testSQLITE(self):
        # Write SQLite table
        self.t.write('sqlite', 'testATpy/mops_ut.db', overwrite=True, verbose=True)
        # Read SQLite table
        t_sqlite=atpy.Table('sqlite', 'testATpy/mops_ut.db', table='mops_ut', verbose=True)              
        for i in range(10):
            print "-->", self.t.data['x'][i], self.t.data['y'][i], t_sqlite.data['x'][i], t_sqlite.data['y'][i]
            self.assertEqual(self.t.data['x'][i], t_sqlite.data['x'][i])
            self.assertEqual(self.t.data['y'][i], t_sqlite.data['y'][i])

    def testVO(self):
        self.t.write('testATpy/mops_ut.xml')        # Write VO table
        t_vo=atpy.Table('testATpy/mops_ut.xml')     # Read VO table        
        for i in range(10):
            print "-->", self.t.data['x'][i], self.t.data['y'][i], t_vo.data['x'][i], t_vo.data['y'][i]
            self.assertEqual(self.t.data['x'][i], t_vo.data['x'][i])
            self.assertEqual(self.t.data['y'][i], t_vo.data['y'][i])

    def testTBL(self):
        self.t.write('testATpy/mops_ut.tbl')         # Write IPAC table
        t_tbl=atpy.Table('testATpy/mops_ut.tbl')     # Read IPAC table        
        for i in range(10):
            print "-->", self.t.data['x'][i], self.t.data['y'][i], t_tbl.data['x'][i], t_tbl.data['y'][i]
            self.assertEqual(self.t.data['x'][i], t_tbl.data['x'][i])
            self.assertEqual(self.t.data['y'][i], t_tbl.data['y'][i])

    def assertAlmostEqualSig(self, first, second, significant=7, msg=None):
        format = "%%.%ig" % significant
        if format % first <> format % second:
            raise self.failureException, \
            (msg or '%r != %r within %r significant digits' % (first, second, significant))

if __name__ == '__main__':
    unittest.main()

Here is the output from running the unit test:

$ ./ATpyUnitTest.py > test.out
.....
----------------------------------------------------------------------
Ran 5 tests in 0.491s

OK

Here is the STDOUT redirected to test.out:

$ cat test.out
Auto-detected input type: FITS table
Auto-detected input type: FITS table
--> 987654321 0.0 987654321 0.0
--> 987654322 3.14159265111 987654322 3.14159265111
--> 987654323 6.28318530222 987654323 6.28318530222
--> 987654324 9.42477795333 987654324 9.42477795333
--> 987654325 12.5663706044 987654325 12.5663706044
--> 987654326 15.7079632556 987654326 15.7079632556
--> 987654327 18.8495559067 987654327 18.8495559067
--> 987654328 21.9911485578 987654328 21.9911485578
--> 987654329 25.1327412089 987654329 25.1327412089
--> 987654330 28.27433386 987654330 28.27433386
Auto-detected input type: SQL table
--> 987654321 0.0 987654321 0.0
--> 987654322 3.14159265111 987654322 3.14159265111
--> 987654323 6.28318530222 987654323 6.28318530222
--> 987654324 9.42477795333 987654324 9.42477795333
--> 987654325 12.5663706044 987654325 12.5663706044
--> 987654326 15.7079632556 987654326 15.7079632556
--> 987654327 18.8495559067 987654327 18.8495559067
--> 987654328 21.9911485578 987654328 21.9911485578
--> 987654329 25.1327412089 987654329 25.1327412089
--> 987654330 28.27433386 987654330 28.27433386
Auto-detected input type: SQL table
Auto-detected input type: SQL table
--> 987654321 0.0 987654321 0.0
--> 987654322 3.14159265111 987654322 3.14159265111
--> 987654323 6.28318530222 987654323 6.28318530222
--> 987654324 9.42477795333 987654324 9.42477795333
--> 987654325 12.5663706044 987654325 12.5663706044
--> 987654326 15.7079632556 987654326 15.7079632556
--> 987654327 18.8495559067 987654327 18.8495559067
--> 987654328 21.9911485578 987654328 21.9911485578
--> 987654329 25.1327412089 987654329 25.1327412089
--> 987654330 28.27433386 987654330 28.27433386
Auto-detected input type: IPAC table
Auto-detected input type: IPAC table
--> 987654321 0.0 987654321 0.0
--> 987654322 3.14159265111 987654322 3.14159265111
--> 987654323 6.28318530222 987654323 6.28318530222
--> 987654324 9.42477795333 987654324 9.42477795333
--> 987654325 12.5663706044 987654325 12.5663706044
--> 987654326 15.7079632556 987654326 15.7079632556
--> 987654327 18.8495559067 987654327 18.8495559067
--> 987654328 21.9911485578 987654328 21.9911485578
--> 987654329 25.1327412089 987654329 25.1327412089
--> 987654330 28.27433386 987654330 28.27433386
Auto-detected input type: VO table
Auto-detected input type: VO table
--> 987654321 0.0 987654321 0.0
--> 987654322 3.14159265111 987654322 3.14159265111
--> 987654323 6.28318530222 987654323 6.28318530222
--> 987654324 9.42477795333 987654324 9.42477795333
--> 987654325 12.5663706044 987654325 12.5663706044
--> 987654326 15.7079632556 987654326 15.7079632556
--> 987654327 18.8495559067 987654327 18.8495559067
--> 987654328 21.9911485578 987654328 21.9911485578
--> 987654329 25.1327412089 987654329 25.1327412089
--> 987654330 28.27433386 987654330 28.27433386