ocd.shots_db – Store and manage shots

This module contains:

• SQLite3 database to store and manage the list of shots. The database interface is done using peewee
• functions load and dump the database content
• an interface with a MySQL database to obtain the shot number

ocd.shots_db.database(fn) = <peewee.SqliteDatabase object>

Database where the information will be stored

ocd.shots_db.connect = <pyhetdex.tools.db_helpers.SQLiteConnector object>

Context manager to open and close the database connection, unless it’s in memory

ocd.shots_db.SHOT_NAMES = ['shotid', 'ra', 'dec', 'track', 'priority', 'n_obs', 'forced_az']

Name of the columns in the input and output shot file that define a shot and its priority, number of observations, …

ocd.shots_db.METADATA_NAMES = ['QIDX', 'QPROG', 'QOBJECT', 'QIFU', 'QRA', 'QDEC', 'QEQUINOX', 'QPMRA', 'QPMDEC', 'QEPOCH']

Name of the columns in the input shot file that contains metadata to pass to TCS

ocd.shots_db.COLUMN_NAMES = ['shotid', 'ra', 'dec', 'track', 'priority', 'n_obs', 'forced_az', 'QIDX', 'QPROG', 'QOBJECT', 'QIFU', 'QRA', 'QDEC', 'QEQUINOX', 'QPMRA', 'QPMDEC', 'QEPOCH']

Name of the mandatory columns of the input shot file

class ocd.shots_db.BaseModel(*args, **kwargs)

Bases: peewee.Model

Base Model class for the tables representing shots

shotid = <CharField: BaseModel.shotid>

DoesNotExist

alias of BaseModelDoesNotExist

id = <AutoField: BaseModel.id>

class ocd.shots_db.Shots(*args, **kwargs)

Bases: ocd.shots_db.BaseModel

Table representing the shots

ra = <FloatField: Shots.ra>

dec = <FloatField: Shots.dec>

track = <IntegerField: Shots.track>

priority = <IntegerField: Shots.priority>

n_obs = <IntegerField: Shots.n_obs>

forced_az = <FloatField: Shots.forced_az>

DoesNotExist

alias of ShotsDoesNotExist

id = <AutoField: Shots.id>

shotid = <CharField: Shots.shotid>

class ocd.shots_db.ShotMetadata(*args, **kwargs)

Bases: ocd.shots_db.BaseModel

Shot metadata. For each entry in Shots there must be an entry here

QIDX = <IntegerField: ShotMetadata.QIDX>

QPROG = <CharField: ShotMetadata.QPROG>

QOBJECT = <CharField: ShotMetadata.QOBJECT>

QIFU = <CharField: ShotMetadata.QIFU>

QRA = <CharField: ShotMetadata.QRA>

QDEC = <CharField: ShotMetadata.QDEC>

QEQUINOX = <FloatField: ShotMetadata.QEQUINOX>

QPMRA = <FloatField: ShotMetadata.QPMRA>

QPMDEC = <FloatField: ShotMetadata.QPMDEC>

QEPOCH = <FloatField: ShotMetadata.QEPOCH>

DoesNotExist

alias of ShotMetadataDoesNotExist

id = <AutoField: ShotMetadata.id>

shotid = <CharField: ShotMetadata.shotid>

ocd.shots_db.init(conf)

Initialize the database and the Shots.

Parameters:

conf : pyhetdex.tools.configuration.ConfigParser

configuration. It uses the options database_name and drop_existing_tables of the [database] section. If database_name is :tmpfile: a temporary file is used

ocd.shots_db.store_shot_file(conf)

Read the file and add all the elements to the database.

Parameters:

conf : pyhetdex.tools.configuration.ConfigParser

configuration. It uses the following options of the [shots] section:

• shot_file: name of the shot file;
• clear_shot_table: if found and true, clear the database table before inserting the new shot file;
• update_existing_ids: if a shotid already exists in the database update it with the values from the shot file

ocd.shots_db.create_shot_file(conf)

Create a shot file from the database.

Parameters:

conf : pyhetdex.tools.configuration.ConfigParser

configuration. It uses the following options of the [shots] section:

• out_shot_dir: directory where to store the shot files; see load_shot_file() for output file description;
• out_shot_file_template: template for the output shot file name
• all_shots: if found and true, save all the shots, otherwise saves only the shots with non null n_obs;

ocd.shots_db.update_shot(shots_dict, success=True)

Update the entry for the given shot id, decreasing the n_obs.

Todo

This function is not in its final form. The current functionality is very limited and will be expanded. See issue #2056, issue #2057 and issue #2080.

Parameters:

shots_dict : dict

dictionary with the parameters used to run the shot. Relevant entries:

• shotid (string): id of the shot
• azimuth (float): commanded azimuth; if Shots.forced_az is negative, replace it with this value
• track (int): commanded track; if Shots.track is 2, replace it with this value

success : bool, optional

whether the shot is successful or not

Raises:

ShotDoesNotExist

if the shot id is not found

ocd.shots_db.get_metadata(shotid)

Get the metadata dictionary for the give shotid.

Parameters:

shotid : string

SHOTID to search

Returns:

metadata : dict

dictionary of metadata

Raises:

ShotDoesNotExist

if the shotid is not found in the tables

ocd.shots_db.bulk_insert(shot_records, metadata_records)

Do a bulk insert of the shot and the metadata records.

Parameters:

shot_records : iterable of dicts

records corresponding to the Shots table

metadata_records : iterable of dicts

records corresponding to the ShotMetadata table

ocd.shots_db.insert_or_update(shot_records, metadata_records, update_exising)

Go through all the records and try to add them to the database. If a shot id already exists and update_exising is True, use the record to update the database, otherwise skip the record

Parameters:

shot_records : list of dicts

records corresponding to the Shots table

metadata_records : iterable of dicts

records corresponding to the ShotMetadata table

update_exising : bool

whether existing shot ids must be updated by a corresponding record

ocd.shots_db.load_shot_file(shot_file)

Load the shot file.

The file must contain at least the columns given by COLUMN_NAMES:

• shotid (string): the ID of the shot; must be unique;
• ra (float): ra of the shot in decimal hours;
• dec (float): dec of the shot in decimal degrees;
• track (integer): must be one of 0, 1, 2; 0: force EAST track; 1: force WEST track; 2: let the software decide the optimal track;
• priority (int): priority for the shot; 1 is the highest priority and decreases as the number increases; must be equal or larger than 1;
• n_obs (int): number of times the shot must be observed; if not present defaults to 1;
• forced_az (float): if -1, the scheduling algorithm decides the best azimuth to use, if positive forces the azimuth to the given value.
• Q* (from METADATA_NAMES): extra column that connect each HETDEX shot to the global HET queue

The shot file can be in any format supported by astropy ascii.

If file must an header with all the COLUMN_NAMES columns.

Parameters:

shot_file : string

name of the file to load

Returns:

table : astropy.table.Table instance

relevant part of the shot file

Raises:

ShotColumnError

if a mandatory column is missing

ocd.shots_db.table_to_records(table, columns=None)

Generator that yields the rows of the input astropy.table.Table to as dictionaries.

Parameters:

table : astropy.table.Table instance

relevant part of the shot file

columns : list of strings, optional

if given, returns only the record for the given columns

Yields:

record : dictionary

dictionary representation of a table row

ocd.shots_db.query_to_table(query)

Convert a query to an astropy Table

Parameters:

query : peewee.SelectQuery

table query to convert

all_shots : boolean

if False, skip rows with no remaining observations

Returns:

table : astropy.table.Table

table representation of the query

ocd.shots_db.OBSNUM_TABLE_NAME = 'vl_obsnum'

name of the table in the mysql database containing the observation number

ocd.shots_db.mysql_connection(conf)

Context manager yielding a mysql connection object. The connection is closed after the context returns.

Parameters:

conf : pyhetdex.tools.configuration.ConfigParser

configuration. It uses the following options of the [database] section:

• mysql_host, mysql_port: IP/host name and port of the MySQL server
• mysql_database: name of the database
• mysql_user, mysql_password: name and password of the user that does the queries

Returns:

conn : pymysql.connections.Connection

MySQL connection

ocd.shots_db.get_obsnumber(conf)

For the current UTC date and the “virus” instrument, get the highest obsnum from MySQL database, increase it by one, insert it back in the database and return it.

If no obsnum is found, returns 1.

Parameters:

conf : pyhetdex.tools.configuration.ConfigParser

configuration. It uses the following options of the [database] section:

• same as mysql_connection()
• mysql_update_obsnum: if true, the new obsnum is inserted in the database, otherwise the insertion is skipped.

Returns:

next_obsnum : int

next observation number

ocd.shots_db.create_obsnum_table(conf)

Create the table OBSNUM_TABLE_NAME

Parameters:

conf : pyhetdex.tools.configuration.ConfigParser

configuration. It uses the following options of the [database] section:

• same as mysql_connection()

ocd.shots_db.fill_obsnum_table(conf, date=None)

Add one line in the table OBSNUM_TABLE_NAME

Parameters:

conf : pyhetdex.tools.configuration.ConfigParser

configuration. It uses the following options of the [database] section:

• same as mysql_connection()

date : datetime.datetime, optional

if given, use when for the ts and obsdate fields, otherwise use the current utc time