======================================== STIS Pixel-Based CTI-Correction Tutorial ======================================== Please note that AURA/STScI is not responsible for any damage resulting from the use of this software. Please send any feedback to the STIS team at help@stsci.edu. **Update:** See the `Known Issues`_ section. Introduction ============ The ``stis_cti`` package implements the Anderson & Bedin (`PASP 2010, 122: 1035-1064 `_) pixel-based Charge Transfer Inefficiency (CTI)-correction on HST/STIS CCD data. It performs bias corrections on the data, calls ``StisPixCteCorr()`` on the intermediate products, and finishes running the remaining ``CalSTIS`` processing steps. The code also creates CTI-corrected super-dark reference files to use in this processing from component dark files. When run, the user will be prompted to retrieve these component darks from the MAST archive if they have not already been retrieved. The code makes use of the ``refstis`` package to generate super-darks from the component darks. Note that not all observing modes are currently supported. The code will not reprocess data taken in subarray or binned modes, with a non-D amplifier, in ACQ mode, or from before the STIS recovery of May 2009. System Requirements =================== This package is designed to run on top of the AstroConda channel (with legacy IRAF support) of the Anaconda Python environment on the UNIX/Linux and MacOS X operating systems. It currently supports Python 2.7. AstroConda is available at http://astroconda.readthedocs.io. Installation ============ First, launch the AstroConda (with legacy IRAF support) environment: :: source activate astroconda Then, install the required packages within the AstroConda environment: :: pip install stis_cti Or, to upgrade from a previous installation: :: pip install --upgrade --no-deps stis_cti pip install stis_cti pip install --upgrade --no-deps refstis pip install refstis To optionally install a local CRDS cache in a common location on your system, please see: https://hst-crds.stsci.edu/docs/cmdline_bestrefs/ To do this, you will need to define the environmental variables ``$CRDS_SERVER_URL``, ``$CRDS_PATH``, and ``$oref`` appropriately. Directory Structure =================== To run the correction, you will need directories to hold dark files, pipeline reference files, reference files generated by the script, and science data. Many of these directories may be shared between projects, thus reducing redundant computations. Here's one possible layout: * base_dir/ - science/ — contains at least the pipeline's "uncalibrated" products that are to be corrected - darks/ — component darks (``_flt.fits``); their CTI-corrected counterparts (``_cte.fits``) will be placed in here - ref/ — location for CTI-corrected super-darks produced by the ``stis_cti`` script - oref/ — pipeline reference files (specifying the ``--crds_update`` option creates this directory at ``ref/references/hst/oref/``) For example:: mkdir stis_cti_corrected mkdir stis_cti_corrected/my_science mkdir stis_cti_corrected/darks mkdir stis_cti_corrected/ref By default, the command-line interface assumes that there are ``ref/`` and ``darks/`` directories parallel to the ``science/`` directory (of any name), though alternate configurations may be specified. Output files in the ``science/`` directory from a previous run of ``stis_cti`` will need to be deleted or moved, unless the ``--clean`` option is specified. Applicable corrected super-darks and darks may be reused for other science data by specifying the appropriate ``darks/`` and ``ref/`` directories. Specifying ``--clean_all`` will force the script to recreate super-darks and re-run the CTI-correction on all needed component darks. If the ``--crds_update`` option is not selected, then the ``$oref`` shell variable must be set before launching the script (including the trailing ``'/'``). In Bash:: export oref="/path/to/oref/" This directory must contain the needed reference files specified in the science and component dark headers, as is typical for running ``CalSTIS``. Command-line Usage ================== The easiest way to invoke the correction is from the ``UNIX`` shell. Note that running this script requires an internet connection to the MAST archive site. The Archive status may be checked at http://archive.stsci.edu/help/archive_status.html. :: source activate astroconda stis_cti --help :: usage: stis_cti [-h] [-d DARK_DIR] [-r REF_DIR] [-n NUM_PROCESSES] [-p PCTETAB] [--crds_update] [--clean] [--clean_all] [--ignore_missing] [-v VERBOSE_LEVEL] [SCIENCE_DIR] Run STIS/CCD pixel-based CTI-correction on data specified in SCIENCE_DIR. Uncorrected component darks are read from DARK_DIR, and corrected component darks are written there too. Corrected super-darks are read from and stored to REF_DIR. See documentation at http://pythonhosted.org/stis_cti/ positional arguments: SCIENCE_DIR directory containing RAW science data (default="./") optional arguments: -h, --help show this help message and exit -d DARK_DIR directory of dark FLT data (default="[SCIENCE_DIR]/../darks/") -r REF_DIR directory of CTI-corrected reference files (default="[SCIENCE_DIR]/../ref/") -n NUM_PROCESSES maximum number of parallel processes to run (default=15); number of available CPU cores on your system = 40 -p PCTETAB name of PCTETAB to use in pixel-based correction (default="[REF_DIR]/[MOST_RECENT]_pcte.fits" or package's default PCTETAB) --crds_update update and download $oref files --clean remove intermediate and final products from previous runs of this script ('*.txt' files are skipped and clobbered) --clean_all '--clean' + remove previous super-darks and CTI-corrected component darks --ignore_missing process data even with an incomplete set of dark FLTs -v VERBOSE_LEVEL verbosity ({0,1,2}; default=1) Author: Sean Lockwood; Version: 1.1 The script is designed to run the pixel-based correction in parallel on the component darks, and in parallel on the science files. The maximum number of processes may be specified via the ``-n #`` option. A typical call looks like:: stis_cti -v 2 -n 15 --crds_update my_science_dir/ (Assuming that there are ``ref/`` and ``darks/`` directories parallel to ``my_science_dir/``.) .. Warning:: For recent STIS observations (new data taken in last 30 to 60 days) optimal dark reference files will not yet be available. This will affect the selection of data being used to generate the CTI-corrected super-darks. To get the most accurate calibration, please re-reduce your data after the pipeline's new super-biases and super-darks have been delivered by deleting the relevant old CTI-corrected super-darks in the ``ref/`` directory and running ``stis_cti`` with the ``--clean`` and ``--crds_update`` options specified. You may need to download additional component darks from MAST. To receive updates when STIS reference files are delivered to CRDS, go to https://maillist.stsci.edu and subscribe to the ``stis_reffiles_upd`` mailing list. You can also check the status of super-dark and super-bias files by going to https://hst-crds.stsci.edu and clicking on STIS-->darkfile and STIS-->biasfile. Sort by USEAFTER to see if the week corresponding to your science data has been delivered yet. ---------------------------------------------------------------------------------------- While ``stis_cti`` will tell you what component darks are needed, you can also determine this ahead of time: .. parsed-literal:: archive_dark_query data/\*_raw.fits Querying MAST archive for dark and anneal program IDs... Querying MAST archive for darks... Parsing archive results... Download darks via this link: http://archive.stsci.edu/hst/search.php?sci_instrume=STIS&sci_instrument_config=STIS%2FCCD&sci_targname=DARK&sci_aec=C&resolve=don%27tresolve&sci_data_set_name=OC4W6XH3Q%2COC4W6YHBQ%2COC4W6ZP2Q%2COC4W70PCQ%2COC4W71TEQ%2COC4W72TOQ%2COC4W73X8Q%2COC4W74XJQ%2COC4W75D0Q%2COC4W76DCQ%2COC4W77HHQ%2COC4W78I0Q%2COC4W79A5Q%2COC4W7AADQ%2COC4W7BFGQ%2COC4W7CF9Q%2COC4W7DJNQ%2COC4W7EJRQ%2COC4W7FOAQ%2COC4W7GO4Q%2COC4W7HSNQ%2COC4W7ISUQ%2COC4W7JXEQ%2COC4W7KXAQ%2COC4W7LGRQ%2COC4W7MGWQ%2COC4W7NA1Q%2COC4W7OA8Q%2COC4W7PM6Q%2COC4W7QMDQ%2COC4W7RTJQ%2COC4W7STNQ%2COC4W7TX4Q%2COC4W7UXDQ%2COC4W7VIKQ%2COC4W7WIRQ%2COC4W7XNJQ%2COC4W7YNRQ%2COC4W7ZSZQ%2COC4W80TMQ%2COC4W81A4Q%2COC4W82AGQ%2COC4W83NMQ%2COC4W84O1Q%2COC4W85SRQ%2COC4W86SZQ%2COC4W87XWQ%2COC4W88YHQ%2COC4W89D6Q%2COC4W8ADJQ%2COC4W8BHWQ%2COC4W8CI2Q%2COC4W8DNUQ%2COC4W8EOAQ%2COC4W8FBPQ%2COC4W8GBTQ&max_records=50000&max_rpp=5000&ordercolumn1=sci_start_time&action=Search Place these darks in the ``DARK_DIR`` directory. Alternatively, a list of the component darks comprising each annealing period is maintained at http://www.stsci.edu/~STIS/monitors/anneals/anneal_periods.html. Note that it does not list the most recent annealing period until it is complete. Python Usage ============ If you wish to run the ``stis_cti`` script from within Python (say from within another script), it may be called via: .. code:: python import stis_cti # Explicit directories must be specified when run through the Python interface. stis_cti.stis_cti('science_dir/', 'dark_dir/', 'ref_dir/', 5, verbose=True) # where 5 is num_processes Note that there are other options available via the Python interface, but these have not yet been fully vetted. Output Products =============== The following data products are output by ``stis_cti``: +----------------------------------+------------------+-----------------------+ | **Product** | **Original Ext** | **CTI-Corrected Ext** | +==================================+==================+=======================+ | Bias- and CTI-corrected science | n/a | CTE [#cte]_ | | (intermediate product) | | | +----------------------------------+------------------+-----------------------+ | Cosmic ray rejected, flat- | CRJ | CRC | | fielded science | | | +----------------------------------+------------------+-----------------------+ | Flat-fielded science | FLT | FLC | +----------------------------------+------------------+-----------------------+ | 1-D extracted spectra for | X1D | X1C | | individual imsets: | | | | | | | | * Aperture extracted, background | | | | subtracted, flux and | | | | wavelength calibrated spectra | | | +----------------------------------+------------------+-----------------------+ | 2-D spectral and direct images | X2D | X2C | | for individual imsets: | | | | | | | | * Rectified, wavelength and | | | | flux calibrated first order | | | | spectra or | | | | * Geometrically corrected | | | | imaging data. | | | +----------------------------------+------------------+-----------------------+ | 1-D extracted spectra from | SX2 | S2C | | from summed (REPEATOBS) or | | | | cosmic ray rejected (CRSPLIT) | | | | images. | | | +----------------------------------+------------------+-----------------------+ | 2-D rectified direct or spectral | SX1 | S1C | | images from summed (REPEATOBS) | | | | or cosmic ray rejected (CRSPLIT) | | | | images. | | | +----------------------------------+------------------+-----------------------+ .. [#cte] Where ``CTE`` files are bias- and CTI-corrected intermediate products. .. figure:: obr101010_comparison.png :width: 100 % :alt: obr101010_crj comparison :align: center A comparison of part of a STIS cosmic-ray rejected image. Note the CTI trails are removed in both the science and super-dark data used to generate the ``_crc`` file. Advanced Topics =============== Custom Super-Darks ------------------ For detailed text on how to create and apply a custom super-dark on your system, run: .. code:: python import stis_cti stis_cti.custom_superdark_info() The ``stis_cti`` script first determines if the ``DARKFILE`` specified in each science file's header is already CTI-corrected (assuming it exists) by checking that the ``ext=0`` header keyword ``PCTECORR=='COMPLETE'``. If it is, then the script will not attempt to replace it. This allows users the flexibility to create their own super-darks via the ``refstis`` package with their own parameters and/or input data (e.g. herringbone-corrected_ data files). .. _herringbone-corrected: http://stis2.sese.asu.edu/ The ``stis_cti`` package will ordinarily create a CTI-corrected super-dark automatically, assuming updated super-dark files have been applied to pipeline data (this is typically done in the months following an observation). To create your own super-dark from component darks of your own choosing, you may follow the procedure outlined below. .. code:: python import refstis import glob from astropy.io import fits # First, populate the _flt.fits dark file headers with the PCTETAB reference file # location and name. # Then, run stis_cti.StisPixCteCorr.CteCorr() on the _flt.fits dark files to produce # CTI-corrected _cte.fits dark files. # Make the monthly basedark, which is used in making the weekdark: # (Assuming only the annealing month's darks are selected below.) month_files = glob.glob('annealing_month/*_cte.fits') refstis.basedark.make_basedark(month_files, refdark_name='basedark_drk.fits') # (This produces basedark_drk.fits, which is used as an input below.) # Make the weekdark, which is applied to the science data: # (Assuming we have moved the appropriate _cte files for the week to my_week/) week_files = glob.glob('my_week/*_cte.fits') refstis.weekdark.make_weekdark(week_files, refdark_name='weekdark_drk.fits', thebasedark='basedark_drk.fits') # (This produces weekdark_drk.fits, which will be used in calibrating our science data.) # You must mark the new weekdark(s) as being CTI-corrected: fits.setval('weekdark_drk.fits', 'PCTECORR', value='COMPLETE') # Point the science files at the new weekdark: # Define $stisref to point to the directory containing the weekdark in the (Bash) shell. # export stisref='/path/to/my_dir/' # # Then, on each science file: fits.setval('science/filename_raw.fits', 'DARKFILE', value='stisref$weekdark_drk.fits') Now when ``stis_cti`` is run on the science directory, it won't try to recreate the super-dark, but will still CTI-correct the science images and apply the new super-dark. Be sure not to run ``stis_cti`` with the ``--crds_update`` option, as this will override the custom super-dark specified above. CRDS Updates ------------ Oftentimes, the reference files specified in a dataset get replaced within the pipeline. This is especially true of super-biases and super-darks produced in the months following the execution of an observation. When new reference files are available, you may re-retrieve the data from MAST. Alternatively, the CRDS ``bestrefs`` script now supports updating header keywords and downloading required reference files automatically. When run with the ``--crds_update`` option, the script will update header keywords and download reference files to the ``$oref`` directory nested properly within the ``$CRDS_PATH`` directory, if it is writable. If this is not set up, one will be created within the ``ref/`` directory. Note that this option is not currently compatible with specifying one's own super-dark, as the user-specified ``DARKFILE`` keyword will be over-written. As a workaround, you may run the CRDS bestref script manually and then override the ``DARKFILE`` keyword as desired. To run CRDS bestref manually, see https://hst-crds.stsci.edu/docs/cmdline_bestrefs/ or re-retrieve your data and best reference files from MAST. You may also wish to setup a common local CRDS cache of reference files to avoid redundancy and save disk space. PCTETAB Updates --------------- The ``stis_cti`` package includes the ``v0.1`` ``PCTETAB`` reference file, which specifies the parameters necessary to run the pixel-based correction on STIS data. If this file is updated, or if an advanced user wishes to modify the file to run the correction differently, the new version may be placed in the ``ref/`` directory. (If multiple ``PCTETAB`` files are present, the one with the last alphabetical name will be used.) The location of the current package ``PCTETAB`` can be found by running ``stis_cti.custom_superdark_info()`` or checking log files. To completely re-run the CTI-correction, you can delete any needed basedarks/weekdarks in the ``ref/`` directory and any needed ``_cte.fits`` files in the ``darks/`` directory by specifying the ``--clean_all`` option. Known Issues ============ .. Warning:: If you use ``stistools.x1d.x1d()`` to manually extract your spectra, we recommend using the argument ``ctecorr="OMIT"`` for pixel-based CTI-corrected data. - The ``--crds_update`` option breaks with ``CRDS v7.0.10``. Until this is fixed, users may manually downgrade to the earlier ``CRDS v7.0.7`` package with the command: ``conda install crds=7.0.7`` - Some annealing months contain non-standard amplifier=A dark files (typical observations are taken with amp=D). These files do not produce ``FLT`` files in the Archive, but are still expected by ``stis_cti`` (even though they are excluded from any amp=D super-darks). As of ``v1.1``, users may bypass the missing file check by specifying the ``--ignore_missing`` flag. Care should be taken that only intended dark files are excluded. - The primary FITS header keyword ``FILENAME`` does not get updated in CTI-corrected output products. - Running ``stistools.x1d.x1d()`` -- A tool to manually extract 1D spectra from ``FLT`` / ``FLC`` files: - The older empirical CTI flux correction is incorrectly run by default, even if the FITS primary header keyword is set to ``CTECORR`` = ``OMIT``. To properly run ``stistools.x1d.x1d()`` on ``FLC`` files, specify ``ctecorr="OMIT"`` in the ``stistools.x1d.x1d()`` argument list. - The output product names when running ``stistools.x1d.x1d()`` do not match those output by ``stis_cti``: +---------------+---------------------+--------------------+----------------------+ | CR-Corrected? | Standard Products | Output of stis_cti | Output of x1d() from | | | (Non-CTI-Corrected) | | stis_cti 2D product | +===============+=====================+====================+======================+ | No | ``_x1d.fits`` | ``_x1c.fits`` | ``_flc_x1d.fits`` | +---------------+---------------------+--------------------+----------------------+ | Yes | ``_sx1.fits`` | ``_s1c.fits`` | ``_crc_x1d.fits`` | +---------------+---------------------+--------------------+----------------------+