========================================= SHOW Level 0 to Level 1A Processing Steps ========================================= The Level 0 to Level 1A processing chain converts raw SHOW images into Level 1A interferograms. The processing chain generates a product that is sub-windowed, corrected for dark current and has a relative calibration applied. The processing chain consists of the following processing steps ================================== ===================================================== Processing Step Description ================================== ===================================================== Mask bits 14 and 15 Bits 14 and 15 are undefined so mask them out Initialize readout error Assign the readout error Mask saturated and bad pixels Set all saturated and bad pixels to NaN Bias Remove the DC bias. Apply Shot Noise Apply shot noise Remove Dark Current Sub Window Selection Select the useful part of the SHOW detector NUC Correction Apply the NUC correction + error Interpolate Bad Pixels Possible interpolation of bad pixels. Flat-Field correction Apply the flat field correction. AnglesFromBoresight The angle of each interferogram row from bore-sight. Apply ER2 pointing ================================== ===================================================== Configuration Settings ^^^^^^^^^^^^^^^^^^^^^^ The SHOW configuration is stored across the following files: 1) software module: `showcdb.yaml_files.er2_2017.yaml` 2) software module: `showcdb.yaml_files.er2_2017_site_settings.yaml` 3) calibration database: the location of the base directory of the CDB is located in `er2_2017_site_settings.yaml` Calibration Database Format ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The calibration database is a directory structure with the follwing format: BadPixelMap Darks FilterPassband Flat_Field iFOV LittrowandMagnification NUCcorrection Mask bits 14 and 15 ------------------- The Xiphos frame-grabber occassionally sets bits 14 and 15 of the OWL 640 camera producing values greater than the allowed 14 bit range, 0-16383. The OWL640 level 0 software actively sets the upper two bits to zero in every image it reads. Test For Normal Readout Mode ---------------------------- Each image is tested to see if it is Normal readout mode. This is easily tested by ensuring pixels are above 1500 DN. Test For Bad Images ------------------- NOT YET IMPLEMENTED. This procedure will test to ensure that the images are good. Bad images are discarded from the processing pipeline. 1. Detector has not reset as indicated by bad FPA temperature 2. Detector has not jittered as indicated by location of known hot-spots Initialize readout error ------------------------ Initializes the error bar of the data product. The initial error bar is set to the read-out error of each pixel. The format of the read-out is t.b.d. but will either be a single scalar value used for all pixels or will be a 2-D array with one value for each pixel. Teh readout-error will be specifie in A/D units. **Configuration Settings:** 1. ``level0/dn_saturated_value``, the maximum value a pixel can have before it is assumed it is saturated, ``default=16383`` 2. ``level0/bad_pixel_file``, the name of a file containing a list of known bad pixels. Mask saturated and bad pixels ----------------------------- This procedure sets to NaN all pixels which are either known to be bad or are saturated upon readout. The bad pixels are determined during calibration and are written into a text file as a sequence zero based [X,Y] locations, one entry per line. The text file is stored in the same directory as the yaml registry configuration file. In addition a second entry in the registry is used to determine the raw digital number that defines a saturated pixel. Any raw level 0 pixel with a number greater than or equal to this value is deemed to be saturated and is set to NaN. **Configuration Settings:** 1. ``level0/dn_saturated_value``, the maximum value a pixel can have before it is assumed it is saturated, ``default=16383`` 2. ``level0/bad_pixel_file``, the name of a file containing a list of known bad pixels. Dark Current Removal -------------------- Removes dark current and DC bias from the raw level 0 image. The dark current tables are dependent upon FPA temperature, exposure time and frame rate. Dark current values have been measured in 10 ms increments from 10 to 500 ms and 1 Hz frame rate and nominal FPA temperature of 0 Celsius. Dark current is generally very small for the range of SHOW exposure times. **Configuration Settings:** 1. ``level0/dcbias_darkcurrent_file``, the name of a file containing the dc bias and dark current parameters. Apply Shot Noise ---------------- Shot noise is applied. Convert DN to electrons , take square root and convert back to DN. Add shot noise error to current error. This process initializes the error propagation. The code considers the following three sources of error, * Shot noise * Readout error * DC bias error. The readout error tabulates the accuracy of the detector readout circuitry specification and specifies the error reading out N electrons from any given pixel. The DC bias error is the accuracy of removing the DC bias in a previous step. The shot noise is driven by Poisson counting statistics on the number of electrons in each pixel converted back to digital numbers. It is given by, .. math:: \Delta N_{Poisson} = \sqrt{\frac{N}{N_e}} where, :math:`N` is the raw digital number readout from the detector corrected for DC bias, :math:`\Delta N_{Poisson}` is the error in :math:`N` due to shot noise and :math:`N_e` is the number of electrons per DN. The three errors are added using standard propagation of errors, .. math:: \Delta N_{Total} = \sqrt{ \Delta N^2_{Poisson} + \Delta N^2_{readout} + \Delta N^2_{DCbias}} **Configuration Settings:** 1. ``level0/electrons_per_DN``, the number of electrons per DN of readout. This may start as a single value but may be adapted to 2-D images if calibration justifies the change. Typically 39.6 e\ :sup:`-` per DN. 2. ``level0/detector_readout_noise``, the readout error in electrons, typically, 150 e\ :sup:`-` 3. ``level0/dcbias_darkcurrent_file``, contains the error in the DC bias Remove Dark Current ------------------- This processing step removes the dark current signal from the Level 0 image generated from the previous steps. The dark current is a function of both detector **FPA** temperature and integration time. The interplation algorithm for the dark current algorithm in both exposure time and FPA temperature is TBD. We note that we have fine control over the exposure time and they can be easily constrained to fixed values but we do not have fine control over the FPA temperature and it may drift by ~0.1C when it is stable. It is unknown at the current time if these drifts are true drifts in temperature or drifts in temperature readout. **Possible idea:** we could use the signal in non-imaged areas of the detector, well away from scattered light signals, as measurements of dark current and consequently as a proxy for temperature. **NOTE:** There are periods of time when the detector resets that the FPA temperature is grossly unreliable. These may require filtering and elimination if they are a problem. The dark current algorithm will load its dark current settings from a netcdf file specified in the registry settings. The netcdf file will be generated by the calibration process. The dark current code will propagate errors through the system using the values in the netcdf file. **Configuration Settings:** 1. ``level0/dcbias_darkcurrent_file``, contains the dark current data and error analyss **Configuration Settings:** 1. ``level0/dark_region_filename``, the name of a file containing the dark region parameters. Sub Window Selection -------------------- This procedure will cut the raw level 0 image to the sub-window size. The sub-window is a rectangle on the original image that selects the good imaging area of the detector. The sub-window is defined as part of the calibration process and is written into the instrument configuration registry. The following keys are read from the instrument configuration file and define the sub-window, **Configuration Settings:** 1. ``shs_design/fft_window_start``, the included, left x coordinate, start pixel of the sub-window in interferogram space, ``default 0``. 2. ``shs_design/fft_window_end``, the excluded, right x coordinate, end pixel of the sub-window in interferogram space, ``default 512``. 3. ``shs_design/height_window_start``, the included, bottom y coordinate, start pixel of the sub-window in height space, ``default 0``. 4. ``shs_design/height_window_end``, the excluded, upper y coordinate, end pixel o fthe sub-window in height space, ``default 640`` The raw image is reduced to the sub-window dimensions and the sub-window dimensions are stored in the L1A header. NUC Correction -------------- Blah Blah Interpolate Bad Pixels ---------------------- Blah Blah Flat-Field Correction --------------------- Blah Blah AnglesFromBoresight ------------------- Blah Blah Latest Changes -------------- 1.0.1 2017-03-27 - Initial development