Missing APASS data?
evertrol opened this issue · comments
There seem to be entries for the "apassdr9_main" table for certain sky areas.
For example,
wsdb=> select count(*) from apassdr9_main where ra between 10 and 20 and "dec" between 35 and 40;
count
---------
0
(1 row)where my locally installed APASS catalogue shows 51838 entries.
I haven't checked for any other areas.
My local installation has 61 176 401 entries total, while in the wsdb I find 49 696 593 entries.
For what it's worth: I had a go at a local installation on my machine of the WSDB, and which ended with an error.
Here's what I get from grepping for headers through the unzipped catalogue files (grep RA *.sum files):
zm05_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm10_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm15_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm20_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm25_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm30_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm35_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm45_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm55_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm60_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm65_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm70_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm75_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm80_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm85_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zm90_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp00_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp05_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp10_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp15_9.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp20_8.sum:1544049:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp25_8.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp35_8.sum:1639453:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp40_8.sum:1883587:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp45_8.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp50_8.sum:1656581:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp55_8.sum:1375649:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp65_8.sum:696844:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp75_8.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp80_8.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
zp85_8.sum:1:# Name RA(J2000) raerr DEC(J2000) decerr nobs mobs filt mag err
I'm not sure what went wrong, but several DR8 files are messed up, by having their header at the end of the file (as tail -n1 zp*_8.sum will show).
For now, I've worked around it by changing ingest.py:
...
fixed_lines = []
N = len(contents)
if 'RA' in contents[0]:
contents = contents[1:]
else: # Assume header at the last line of the file
contents = contents[:-1]
for i, line in enumerate(contents):
...Now my local APASS-WSDB copy does contain 61 176 401 entries.
I've fixed the ingest.py script using @evertrol's workaround, dropped the apassdr9_main table, and re-ingested everything.