Difference between revisions of "Calibration Server"

From Gaia Science Alerts Working Group
Jump to navigationJump to search
 
(19 intermediate revisions by 4 users not shown)
Line 1: Line 1:
The main address of the server at the moment is:
+
The main address of the server since 2022 is:
http://gaia020.ast.cam.ac.uk:5000/
+
[https://cpcs.astrolabs.pl/| https://cpcs.astrolabs.pl/]
 +
The old server (http://gsaweb.ast.cam.ac.uk/followup) has been replaced with the new one listed above.
  
== Autorization ==
 
  
* In order to access the server you need a so called "hashtag" which is an autorization key. In order to get one you need to ask Lukasz Wyrzykowski and give him also
+
== The manual document ==
the lon,lat of your observatory and list of filters you have.
+
Below you can find a quick manual. For more detailed pdf version, see [http://www.astrouw.edu.pl/~wyrzykow/manualCPCS2016.pdf here].
 +
 
 +
== Authorization ==
 +
In order to access the server you need a so-called "hashtag" which is an authorization key.
 +
 
 +
* ''' The easiest way is to register an account in our new system, BHTOM [bhtom.space], which offers way more than the CPCS. But it also generates a hashtag for your CPCS use.'''
 +
* Alternative option (old way) is to ask Lukasz Wyrzykowski and give him also the lon,lat of your observatory and list of filters you have.
 
* If you have the hashtag, you can use the login form of the server to authorize yourself. Or if you are doing some scripting, you will need to specify hashtag option in the GET, POST requests.  
 
* If you have the hashtag, you can use the login form of the server to authorize yourself. Or if you are doing some scripting, you will need to specify hashtag option in the GET, POST requests.  
 +
 +
== Preparing your data ==
 +
In order to obtain the SExtractor file with [Ra, Dec] coordinates required by the photometric server, the following steps have to be performed.
 +
 +
* Obtain the Equatorial coordinates for you photometry fits file by using [http://nova.astrometry.net/ Astrometry.net].
 +
* Download the Astrometry.net output file (''new-image.fits'' by default) from the site and use it as the sextractor input.
 +
* Run sextractor:
 +
sex new-image.fits -c /path_to/default.sex
 +
* make sure SExtractor outputs those : MAG_AUTO, MAGERR_AUTO, ALPHA_J2000, DELTA_J2000 and your configuration file should state that you want the output to be ASCII file with header (CATALOG_TYPE    ASCII_HEAD)
 +
* Upload the SExtractor output catalogue (by default ''test.cat'') to the Cambridge Photometry Calibration Server.
  
 
== Calibration of your photometric data ==
 
== Calibration of your photometric data ==
  
The main url for calibration of your sextractor catalogs is  
+
The main url for calibration of your SExtractor catalogs is :
http://gaia020.ast.cam.ac.uk:5000/uploader/
+
[https://cpcs.astrolabs.pl/| https://cpcs.astrolabs.pl/]
 +
 
 +
You first need to login with your hashtag in order to be able to upload the data and to view the observations. Without login you can still see the lightcurves of the events.
 +
 
 +
==== Data format ====
 +
The allowed formats for the sextractor catalogs are plain ascii and FITS. We also read FITS_LDAC. The fits should have all necessary columns defined.
 +
The columns which are required (specified in ''default.params'':
 +
MAG_AUTO
 +
MAGERR_AUTO
 +
ALPHA_J2000
 +
DELTA_J2000
  
 +
If outputing as ASCII make sure the header is also outputed. Set it in ''default.sex'' as:
 +
CATALOG_TYPE    ASCII_HEAD
 +
 +
If using different photometric software than SExtractor, then please add the following header to the ASCII output:
 +
#  1 MAG_AUTO        Kron-like elliptical aperture magnitude        [mag]
 +
#  2 MAGERR_AUTO    RMS error for AUTO magnitude                    [mag]
 +
#  3 ALPHA_J2000    Right ascension of barycenter (J2000)          [deg]
 +
#  4 DELTA_J2000    Declination of barycenter (J2000)              [deg]
 +
and make sure the columns are as described in the header (i.e. 1st is mag, 2nd is error, and so on). Also, note the separator in the data file should be space, not tab.
 +
 +
==== Uploading your observations ====
 +
Before uploading, please make sure your event you have observed is on the list of alerts. You can see all alerts or only those for which there is any data already.
 +
If your event is not on the list you will need to add it manually, see below.
 +
If your event is there, then please note its ivorn (ID), because you will need to provide it while uploading the data - this helps identify the right object on your image.
 +
 +
The upload form is straightforward. You need to provide the IVORN (the unique id of the alert), MJD of observations (note that MJD=JD-2400000.5), exposure time in seconds, optional comment on the datapoint, Sextractor catalog: browse the file here from your computer, matching radius: how accurate is your astrometry with respect to e.g. SDSS (typically 2'' is good, but you might want to try larger radii if calibration is poor). Force Filter = no (default) indicates that the filter will be determined automatically for your data - the best matching one will be used. If you know which filter is the closest to your then you can force it by choosing from the list.
 +
Dry Run: if ticked, it means this is just a test submission and nothing gets stored. You can then repeat the insertion as soon as you are happy with it. Then, switch off Dry Run and your data will get stored. Note, you will not be able to submit the same data point twice if not in Dry Run.
 +
 +
==== New event ====
 +
It is important that you can upload the data for the alerts which are in the database (e.g. which are listed here https://cpcs.astrolabs.pl/list_of_alerts ).
 +
If the alert is not in our DB, you can insert it https://cpcs.astrolabs.pl/newevent/.
 +
IVORN is the unique Virtual Observatory event name, e.g. ivo://nvo.caltech/voeventnet/catot#1306271320054132125. If it is unknown the format should be, e.g. ivo://asassn/ASASSN-13ax, for ASAS-SN survey object 13ax.
 +
 +
Ra,Dec are given in decimal degrees.
 +
 +
URL: link to the transient's page, ideally to individual event, but if not available, then to the main page of the feeding survey, e.g. http://ogle.astrouw.edu.pl/ogle4/transients/
 +
 +
== Automated submission ==
 
You can also do that from the command line using HTTP POST protocol  
 
You can also do that from the command line using HTTP POST protocol  
  curl -F matchDist=2 -F EventID='ivo://110610' -F sexCat="@path_to_your_sex_catalog;filename=test.cat"  
+
  curl -F matchDist=2 -F EventID='ivo://110610' -F sexCat="@path_to_your_sex_catalog_with_filename;filename=test.cat"  
       -F "hashtag=XXXX" -F "MJD=2" -F "matchDist=2"
+
       -F "hashtag=XXXX" -F "MJD=2" -F expTime=1 -F noPlot=1
      -F  EventID='ivo://110610' -F expTime=1 -F noPlot=1
+
       -F forceFilter=no -F dryRun=1 -F outputFormat=json "https://cpcs.astrolabs.pl/cgi/upload"
       -F forceFilter=no -F dryRun=1 -F outputFormat=json "http://gaia020.ast.cam.ac.uk:5000/cgi/upload"
+
Parameters of the  https://cpcs.astrolabs.pl/cgi/upload
Parameters of the  http://gaia020.ast.cam.ac.uk:5000/cgi/upload
 
 
* matchDist -- matching radius in arcseconds
 
* matchDist -- matching radius in arcseconds
 
* hashtag -- your authorization key
 
* hashtag -- your authorization key
Line 30: Line 83:
 
* outputFormat -- at the moment the only allowed values are json and html  
 
* outputFormat -- at the moment the only allowed values are json and html  
  
It is important that you upload the data for the alerts which are in the database (e.g. which are listed here http://gaia020.ast.cam.ac.uk:5000/list_of_alerts ).
 
If the alert is not in our DB, you can insert it http://gaia020.ast.cam.ac.uk:5000/newevent/
 
 
The allowed formats for the sextractor catalogs are plain ascii and FITS. Obviously those catalogs must contain
 
RA, DEC and MAG columns.
 
  
 
== Review of the data in the Database ==
 
== Review of the data in the Database ==
  
You can view the list of followup_data which are in the DB http://gaia020.ast.cam.ac.uk:5000/list_of_followups
+
You can view the list of followup_data which are in the DB https://cpcs.astrolabs.pl/list_of_followups
  
You can view the list of alerts which are in the DB http://gaia020.ast.cam.ac.uk:5000/list_of_alerts
+
You can view the list of alerts which are in the DB https://cpcs.astrolabs.pl/list_of_alerts
 
, and the subset of them which have followup data  
 
, and the subset of them which have followup data  
http://gaia020.ast.cam.ac.uk:5000/list_of_alerts?observed_only=1.
+
https://cpcs.astrolabs.pl/list_of_alerts?observed_only=1.
 
From these pages you can also see the light curves for those alerts which do have followup data.
 
From these pages you can also see the light curves for those alerts which do have followup data.
 
Here is the example of the light curve:
 
Here is the example of the light curve:
http://gaia020.ast.cam.ac.uk:5000/get_alert_lc?alert_id=2999
+
https://cpcs.astrolabs.pl/get_alert_lc?alert_id=2999
 +
 
 +
== Source code ==
 +
 
 +
The calibration server is written in python using Flask framework. If you'd like to help/implement something, ask Sergey Koposov for the access to the git repository
 +
on http://bitbucket.org
 +
 
 +
= Data Policy =
 +
If you use any of the data from the Calibration Server in your publication, please acknowledge it with the following text:
 +
 
 +
'''We acknowledge the use of the Cambridge Photometric Calibration Server (https://cpcs.astrolabs.pl), developed by Sergey Koposov and maintained by Lukasz Wyrzykowski, Pawel Zielinski, funded by the European Union's Horizon 2020 research and innovation programme under grant agreement No 101004719 (OPTICON-RadioNET Pilot).'''
 +
 
 +
<big>Also note that if the data from the CPCS is essential for the paper, please include the observers as co-authors.</big>

Latest revision as of 15:58, 22 April 2022

The main address of the server since 2022 is: https://cpcs.astrolabs.pl/ The old server (http://gsaweb.ast.cam.ac.uk/followup) has been replaced with the new one listed above.


The manual document

Below you can find a quick manual. For more detailed pdf version, see here.

Authorization

In order to access the server you need a so-called "hashtag" which is an authorization key.

  • The easiest way is to register an account in our new system, BHTOM [bhtom.space], which offers way more than the CPCS. But it also generates a hashtag for your CPCS use.
  • Alternative option (old way) is to ask Lukasz Wyrzykowski and give him also the lon,lat of your observatory and list of filters you have.
  • If you have the hashtag, you can use the login form of the server to authorize yourself. Or if you are doing some scripting, you will need to specify hashtag option in the GET, POST requests.

Preparing your data

In order to obtain the SExtractor file with [Ra, Dec] coordinates required by the photometric server, the following steps have to be performed.

  • Obtain the Equatorial coordinates for you photometry fits file by using Astrometry.net.
  • Download the Astrometry.net output file (new-image.fits by default) from the site and use it as the sextractor input.
  • Run sextractor:
sex new-image.fits -c /path_to/default.sex
  • make sure SExtractor outputs those : MAG_AUTO, MAGERR_AUTO, ALPHA_J2000, DELTA_J2000 and your configuration file should state that you want the output to be ASCII file with header (CATALOG_TYPE ASCII_HEAD)
  • Upload the SExtractor output catalogue (by default test.cat) to the Cambridge Photometry Calibration Server.

Calibration of your photometric data

The main url for calibration of your SExtractor catalogs is : https://cpcs.astrolabs.pl/

You first need to login with your hashtag in order to be able to upload the data and to view the observations. Without login you can still see the lightcurves of the events.

Data format

The allowed formats for the sextractor catalogs are plain ascii and FITS. We also read FITS_LDAC. The fits should have all necessary columns defined. The columns which are required (specified in default.params:

MAG_AUTO
MAGERR_AUTO
ALPHA_J2000
DELTA_J2000

If outputing as ASCII make sure the header is also outputed. Set it in default.sex as:

CATALOG_TYPE     ASCII_HEAD

If using different photometric software than SExtractor, then please add the following header to the ASCII output:

#   1 MAG_AUTO        Kron-like elliptical aperture magnitude         [mag]
#   2 MAGERR_AUTO     RMS error for AUTO magnitude                    [mag]
#   3 ALPHA_J2000     Right ascension of barycenter (J2000)           [deg]
#   4 DELTA_J2000     Declination of barycenter (J2000)               [deg]

and make sure the columns are as described in the header (i.e. 1st is mag, 2nd is error, and so on). Also, note the separator in the data file should be space, not tab.

Uploading your observations

Before uploading, please make sure your event you have observed is on the list of alerts. You can see all alerts or only those for which there is any data already. If your event is not on the list you will need to add it manually, see below. If your event is there, then please note its ivorn (ID), because you will need to provide it while uploading the data - this helps identify the right object on your image.

The upload form is straightforward. You need to provide the IVORN (the unique id of the alert), MJD of observations (note that MJD=JD-2400000.5), exposure time in seconds, optional comment on the datapoint, Sextractor catalog: browse the file here from your computer, matching radius: how accurate is your astrometry with respect to e.g. SDSS (typically 2 is good, but you might want to try larger radii if calibration is poor). Force Filter = no (default) indicates that the filter will be determined automatically for your data - the best matching one will be used. If you know which filter is the closest to your then you can force it by choosing from the list. Dry Run: if ticked, it means this is just a test submission and nothing gets stored. You can then repeat the insertion as soon as you are happy with it. Then, switch off Dry Run and your data will get stored. Note, you will not be able to submit the same data point twice if not in Dry Run.

New event

It is important that you can upload the data for the alerts which are in the database (e.g. which are listed here https://cpcs.astrolabs.pl/list_of_alerts ). If the alert is not in our DB, you can insert it https://cpcs.astrolabs.pl/newevent/. IVORN is the unique Virtual Observatory event name, e.g. ivo://nvo.caltech/voeventnet/catot#1306271320054132125. If it is unknown the format should be, e.g. ivo://asassn/ASASSN-13ax, for ASAS-SN survey object 13ax.

Ra,Dec are given in decimal degrees.

URL: link to the transient's page, ideally to individual event, but if not available, then to the main page of the feeding survey, e.g. http://ogle.astrouw.edu.pl/ogle4/transients/

Automated submission

You can also do that from the command line using HTTP POST protocol

curl -F matchDist=2 -F EventID='ivo://110610' -F sexCat="@path_to_your_sex_catalog_with_filename;filename=test.cat" 
     -F "hashtag=XXXX" -F "MJD=2" -F expTime=1 -F noPlot=1
     -F forceFilter=no -F dryRun=1 -F outputFormat=json "https://cpcs.astrolabs.pl/cgi/upload"

Parameters of the https://cpcs.astrolabs.pl/cgi/upload

  • matchDist -- matching radius in arcseconds
  • hashtag -- your authorization key
  • MJD -- mjd of the observations
  • expTime -- exposure time
  • comment -- comments
  • EventID -- the ivorn of the alert
  • dryRun -- the value of 1 allows you to check the results of the calibration without inserting anything into our DB
  • forceFilter -- "no" means that the calibration will be fully automated. But You can also specify APASS/V if you want to calibrate using a particular survey/filter
  • sexCat -- that's the Sextractor catalog you are trying to submit
  • outputFormat -- at the moment the only allowed values are json and html


Review of the data in the Database

You can view the list of followup_data which are in the DB https://cpcs.astrolabs.pl/list_of_followups

You can view the list of alerts which are in the DB https://cpcs.astrolabs.pl/list_of_alerts , and the subset of them which have followup data https://cpcs.astrolabs.pl/list_of_alerts?observed_only=1. From these pages you can also see the light curves for those alerts which do have followup data. Here is the example of the light curve: https://cpcs.astrolabs.pl/get_alert_lc?alert_id=2999

Source code

The calibration server is written in python using Flask framework. If you'd like to help/implement something, ask Sergey Koposov for the access to the git repository on http://bitbucket.org

Data Policy

If you use any of the data from the Calibration Server in your publication, please acknowledge it with the following text:

We acknowledge the use of the Cambridge Photometric Calibration Server (https://cpcs.astrolabs.pl), developed by Sergey Koposov and maintained by Lukasz Wyrzykowski, Pawel Zielinski, funded by the European Union's Horizon 2020 research and innovation programme under grant agreement No 101004719 (OPTICON-RadioNET Pilot).

Also note that if the data from the CPCS is essential for the paper, please include the observers as co-authors.