Difference between revisions of "Calibration Server"
(10 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
− | The main address of the server | + | The main address of the server since 2022 is: |
− | http://gsaweb.ast.cam.ac.uk/followup | + | [https://cpcs.astrolabs.pl/| https://cpcs.astrolabs.pl/] |
+ | The old server (http://gsaweb.ast.cam.ac.uk/followup) has been replaced with the new one listed above. | ||
+ | |||
+ | |||
+ | == The manual document == | ||
+ | Below you can find a quick manual. For more detailed pdf version, see [http://www.astrouw.edu.pl/~wyrzykow/manualCPCS2016.pdf here]. | ||
== Authorization == | == Authorization == | ||
+ | In order to access the server you need a so-called "hashtag" which is an authorization key. | ||
− | * | + | * ''' The easiest way is to register an account in our new system, BHTOM [bhtom.space], which offers way more than the CPCS. But it also generates a hashtag for your CPCS use.''' |
− | the lon,lat of your observatory and list of filters you have. | + | * Alternative option (old way) is to ask Lukasz Wyrzykowski and give him also the lon,lat of your observatory and list of filters you have. |
* If you have the hashtag, you can use the login form of the server to authorize yourself. Or if you are doing some scripting, you will need to specify hashtag option in the GET, POST requests. | * If you have the hashtag, you can use the login form of the server to authorize yourself. Or if you are doing some scripting, you will need to specify hashtag option in the GET, POST requests. | ||
== Preparing your data == | == Preparing your data == | ||
− | In order to obtain the | + | In order to obtain the SExtractor file with [Ra, Dec] coordinates required by the photometric server, the following steps have to be performed. |
* Obtain the Equatorial coordinates for you photometry fits file by using [http://nova.astrometry.net/ Astrometry.net]. | * Obtain the Equatorial coordinates for you photometry fits file by using [http://nova.astrometry.net/ Astrometry.net]. | ||
Line 15: | Line 21: | ||
* Run sextractor: | * Run sextractor: | ||
sex new-image.fits -c /path_to/default.sex | sex new-image.fits -c /path_to/default.sex | ||
− | * make sure SExtractor outputs those : MAG_AUTO, MAGERR_AUTO, ALPHA_J2000, DELTA_J2000 and your | + | * make sure SExtractor outputs those : MAG_AUTO, MAGERR_AUTO, ALPHA_J2000, DELTA_J2000 and your configuration file should state that you want the output to be ASCII file with header (CATALOG_TYPE ASCII_HEAD) |
* Upload the SExtractor output catalogue (by default ''test.cat'') to the Cambridge Photometry Calibration Server. | * Upload the SExtractor output catalogue (by default ''test.cat'') to the Cambridge Photometry Calibration Server. | ||
== Calibration of your photometric data == | == Calibration of your photometric data == | ||
− | The main url for calibration of your | + | The main url for calibration of your SExtractor catalogs is : |
− | + | [https://cpcs.astrolabs.pl/| https://cpcs.astrolabs.pl/] | |
You first need to login with your hashtag in order to be able to upload the data and to view the observations. Without login you can still see the lightcurves of the events. | You first need to login with your hashtag in order to be able to upload the data and to view the observations. Without login you can still see the lightcurves of the events. | ||
Line 27: | Line 33: | ||
==== Data format ==== | ==== Data format ==== | ||
The allowed formats for the sextractor catalogs are plain ascii and FITS. We also read FITS_LDAC. The fits should have all necessary columns defined. | The allowed formats for the sextractor catalogs are plain ascii and FITS. We also read FITS_LDAC. The fits should have all necessary columns defined. | ||
− | The columns which are required (specified in | + | The columns which are required (specified in ''default.params'': |
MAG_AUTO | MAG_AUTO | ||
MAGERR_AUTO | MAGERR_AUTO | ||
Line 33: | Line 39: | ||
DELTA_J2000 | DELTA_J2000 | ||
− | If outputing as ASCII make sure the header is also outputed. Set it in | + | If outputing as ASCII make sure the header is also outputed. Set it in ''default.sex'' as: |
CATALOG_TYPE ASCII_HEAD | CATALOG_TYPE ASCII_HEAD | ||
+ | |||
+ | If using different photometric software than SExtractor, then please add the following header to the ASCII output: | ||
+ | # 1 MAG_AUTO Kron-like elliptical aperture magnitude [mag] | ||
+ | # 2 MAGERR_AUTO RMS error for AUTO magnitude [mag] | ||
+ | # 3 ALPHA_J2000 Right ascension of barycenter (J2000) [deg] | ||
+ | # 4 DELTA_J2000 Declination of barycenter (J2000) [deg] | ||
+ | and make sure the columns are as described in the header (i.e. 1st is mag, 2nd is error, and so on). Also, note the separator in the data file should be space, not tab. | ||
==== Uploading your observations ==== | ==== Uploading your observations ==== | ||
Line 41: | Line 54: | ||
If your event is there, then please note its ivorn (ID), because you will need to provide it while uploading the data - this helps identify the right object on your image. | If your event is there, then please note its ivorn (ID), because you will need to provide it while uploading the data - this helps identify the right object on your image. | ||
− | The upload form is straightforward. You need to provide the IVORN (the unique id of the alert), MJD of observations (note that MJD=JD-2400000. | + | The upload form is straightforward. You need to provide the IVORN (the unique id of the alert), MJD of observations (note that MJD=JD-2400000.5), exposure time in seconds, optional comment on the datapoint, Sextractor catalog: browse the file here from your computer, matching radius: how accurate is your astrometry with respect to e.g. SDSS (typically 2'' is good, but you might want to try larger radii if calibration is poor). Force Filter = no (default) indicates that the filter will be determined automatically for your data - the best matching one will be used. If you know which filter is the closest to your then you can force it by choosing from the list. |
Dry Run: if ticked, it means this is just a test submission and nothing gets stored. You can then repeat the insertion as soon as you are happy with it. Then, switch off Dry Run and your data will get stored. Note, you will not be able to submit the same data point twice if not in Dry Run. | Dry Run: if ticked, it means this is just a test submission and nothing gets stored. You can then repeat the insertion as soon as you are happy with it. Then, switch off Dry Run and your data will get stored. Note, you will not be able to submit the same data point twice if not in Dry Run. | ||
==== New event ==== | ==== New event ==== | ||
− | It is important that you can upload the data for the alerts which are in the database (e.g. which are listed here | + | It is important that you can upload the data for the alerts which are in the database (e.g. which are listed here https://cpcs.astrolabs.pl/list_of_alerts ). |
− | If the alert is not in our DB, you can insert it | + | If the alert is not in our DB, you can insert it https://cpcs.astrolabs.pl/newevent/. |
IVORN is the unique Virtual Observatory event name, e.g. ivo://nvo.caltech/voeventnet/catot#1306271320054132125. If it is unknown the format should be, e.g. ivo://asassn/ASASSN-13ax, for ASAS-SN survey object 13ax. | IVORN is the unique Virtual Observatory event name, e.g. ivo://nvo.caltech/voeventnet/catot#1306271320054132125. If it is unknown the format should be, e.g. ivo://asassn/ASASSN-13ax, for ASAS-SN survey object 13ax. | ||
Line 57: | Line 70: | ||
curl -F matchDist=2 -F EventID='ivo://110610' -F sexCat="@path_to_your_sex_catalog_with_filename;filename=test.cat" | curl -F matchDist=2 -F EventID='ivo://110610' -F sexCat="@path_to_your_sex_catalog_with_filename;filename=test.cat" | ||
-F "hashtag=XXXX" -F "MJD=2" -F expTime=1 -F noPlot=1 | -F "hashtag=XXXX" -F "MJD=2" -F expTime=1 -F noPlot=1 | ||
− | -F forceFilter=no -F dryRun=1 -F outputFormat=json " | + | -F forceFilter=no -F dryRun=1 -F outputFormat=json "https://cpcs.astrolabs.pl/cgi/upload" |
− | Parameters of the | + | Parameters of the https://cpcs.astrolabs.pl/cgi/upload |
* matchDist -- matching radius in arcseconds | * matchDist -- matching radius in arcseconds | ||
* hashtag -- your authorization key | * hashtag -- your authorization key | ||
Line 73: | Line 86: | ||
== Review of the data in the Database == | == Review of the data in the Database == | ||
− | You can view the list of followup_data which are in the DB | + | You can view the list of followup_data which are in the DB https://cpcs.astrolabs.pl/list_of_followups |
− | You can view the list of alerts which are in the DB | + | You can view the list of alerts which are in the DB https://cpcs.astrolabs.pl/list_of_alerts |
, and the subset of them which have followup data | , and the subset of them which have followup data | ||
− | + | https://cpcs.astrolabs.pl/list_of_alerts?observed_only=1. | |
From these pages you can also see the light curves for those alerts which do have followup data. | From these pages you can also see the light curves for those alerts which do have followup data. | ||
Here is the example of the light curve: | Here is the example of the light curve: | ||
− | + | https://cpcs.astrolabs.pl/get_alert_lc?alert_id=2999 | |
== Source code == | == Source code == | ||
Line 86: | Line 99: | ||
The calibration server is written in python using Flask framework. If you'd like to help/implement something, ask Sergey Koposov for the access to the git repository | The calibration server is written in python using Flask framework. If you'd like to help/implement something, ask Sergey Koposov for the access to the git repository | ||
on http://bitbucket.org | on http://bitbucket.org | ||
+ | |||
+ | = Data Policy = | ||
+ | If you use any of the data from the Calibration Server in your publication, please acknowledge it with the following text: | ||
+ | |||
+ | '''We acknowledge the use of the Cambridge Photometric Calibration Server (https://cpcs.astrolabs.pl), developed by Sergey Koposov and maintained by Lukasz Wyrzykowski, Pawel Zielinski, funded by the European Union's Horizon 2020 research and innovation programme under grant agreement No 101004719 (OPTICON-RadioNET Pilot).''' | ||
+ | |||
+ | <big>Also note that if the data from the CPCS is essential for the paper, please include the observers as co-authors.</big> |
Latest revision as of 15:58, 22 April 2022
The main address of the server since 2022 is: https://cpcs.astrolabs.pl/ The old server (http://gsaweb.ast.cam.ac.uk/followup) has been replaced with the new one listed above.
The manual document
Below you can find a quick manual. For more detailed pdf version, see here.
Authorization
In order to access the server you need a so-called "hashtag" which is an authorization key.
- The easiest way is to register an account in our new system, BHTOM [bhtom.space], which offers way more than the CPCS. But it also generates a hashtag for your CPCS use.
- Alternative option (old way) is to ask Lukasz Wyrzykowski and give him also the lon,lat of your observatory and list of filters you have.
- If you have the hashtag, you can use the login form of the server to authorize yourself. Or if you are doing some scripting, you will need to specify hashtag option in the GET, POST requests.
Preparing your data
In order to obtain the SExtractor file with [Ra, Dec] coordinates required by the photometric server, the following steps have to be performed.
- Obtain the Equatorial coordinates for you photometry fits file by using Astrometry.net.
- Download the Astrometry.net output file (new-image.fits by default) from the site and use it as the sextractor input.
- Run sextractor:
sex new-image.fits -c /path_to/default.sex
- make sure SExtractor outputs those : MAG_AUTO, MAGERR_AUTO, ALPHA_J2000, DELTA_J2000 and your configuration file should state that you want the output to be ASCII file with header (CATALOG_TYPE ASCII_HEAD)
- Upload the SExtractor output catalogue (by default test.cat) to the Cambridge Photometry Calibration Server.
Calibration of your photometric data
The main url for calibration of your SExtractor catalogs is : https://cpcs.astrolabs.pl/
You first need to login with your hashtag in order to be able to upload the data and to view the observations. Without login you can still see the lightcurves of the events.
Data format
The allowed formats for the sextractor catalogs are plain ascii and FITS. We also read FITS_LDAC. The fits should have all necessary columns defined. The columns which are required (specified in default.params:
MAG_AUTO MAGERR_AUTO ALPHA_J2000 DELTA_J2000
If outputing as ASCII make sure the header is also outputed. Set it in default.sex as:
CATALOG_TYPE ASCII_HEAD
If using different photometric software than SExtractor, then please add the following header to the ASCII output:
# 1 MAG_AUTO Kron-like elliptical aperture magnitude [mag] # 2 MAGERR_AUTO RMS error for AUTO magnitude [mag] # 3 ALPHA_J2000 Right ascension of barycenter (J2000) [deg] # 4 DELTA_J2000 Declination of barycenter (J2000) [deg]
and make sure the columns are as described in the header (i.e. 1st is mag, 2nd is error, and so on). Also, note the separator in the data file should be space, not tab.
Uploading your observations
Before uploading, please make sure your event you have observed is on the list of alerts. You can see all alerts or only those for which there is any data already. If your event is not on the list you will need to add it manually, see below. If your event is there, then please note its ivorn (ID), because you will need to provide it while uploading the data - this helps identify the right object on your image.
The upload form is straightforward. You need to provide the IVORN (the unique id of the alert), MJD of observations (note that MJD=JD-2400000.5), exposure time in seconds, optional comment on the datapoint, Sextractor catalog: browse the file here from your computer, matching radius: how accurate is your astrometry with respect to e.g. SDSS (typically 2 is good, but you might want to try larger radii if calibration is poor). Force Filter = no (default) indicates that the filter will be determined automatically for your data - the best matching one will be used. If you know which filter is the closest to your then you can force it by choosing from the list. Dry Run: if ticked, it means this is just a test submission and nothing gets stored. You can then repeat the insertion as soon as you are happy with it. Then, switch off Dry Run and your data will get stored. Note, you will not be able to submit the same data point twice if not in Dry Run.
New event
It is important that you can upload the data for the alerts which are in the database (e.g. which are listed here https://cpcs.astrolabs.pl/list_of_alerts ). If the alert is not in our DB, you can insert it https://cpcs.astrolabs.pl/newevent/. IVORN is the unique Virtual Observatory event name, e.g. ivo://nvo.caltech/voeventnet/catot#1306271320054132125. If it is unknown the format should be, e.g. ivo://asassn/ASASSN-13ax, for ASAS-SN survey object 13ax.
Ra,Dec are given in decimal degrees.
URL: link to the transient's page, ideally to individual event, but if not available, then to the main page of the feeding survey, e.g. http://ogle.astrouw.edu.pl/ogle4/transients/
Automated submission
You can also do that from the command line using HTTP POST protocol
curl -F matchDist=2 -F EventID='ivo://110610' -F sexCat="@path_to_your_sex_catalog_with_filename;filename=test.cat" -F "hashtag=XXXX" -F "MJD=2" -F expTime=1 -F noPlot=1 -F forceFilter=no -F dryRun=1 -F outputFormat=json "https://cpcs.astrolabs.pl/cgi/upload"
Parameters of the https://cpcs.astrolabs.pl/cgi/upload
- matchDist -- matching radius in arcseconds
- hashtag -- your authorization key
- MJD -- mjd of the observations
- expTime -- exposure time
- comment -- comments
- EventID -- the ivorn of the alert
- dryRun -- the value of 1 allows you to check the results of the calibration without inserting anything into our DB
- forceFilter -- "no" means that the calibration will be fully automated. But You can also specify APASS/V if you want to calibrate using a particular survey/filter
- sexCat -- that's the Sextractor catalog you are trying to submit
- outputFormat -- at the moment the only allowed values are json and html
Review of the data in the Database
You can view the list of followup_data which are in the DB https://cpcs.astrolabs.pl/list_of_followups
You can view the list of alerts which are in the DB https://cpcs.astrolabs.pl/list_of_alerts , and the subset of them which have followup data https://cpcs.astrolabs.pl/list_of_alerts?observed_only=1. From these pages you can also see the light curves for those alerts which do have followup data. Here is the example of the light curve: https://cpcs.astrolabs.pl/get_alert_lc?alert_id=2999
Source code
The calibration server is written in python using Flask framework. If you'd like to help/implement something, ask Sergey Koposov for the access to the git repository on http://bitbucket.org
Data Policy
If you use any of the data from the Calibration Server in your publication, please acknowledge it with the following text:
We acknowledge the use of the Cambridge Photometric Calibration Server (https://cpcs.astrolabs.pl), developed by Sergey Koposov and maintained by Lukasz Wyrzykowski, Pawel Zielinski, funded by the European Union's Horizon 2020 research and innovation programme under grant agreement No 101004719 (OPTICON-RadioNET Pilot).
Also note that if the data from the CPCS is essential for the paper, please include the observers as co-authors.