# Data Processing: Command Line Utilities¶

The showapi exposes several command line utilities used in standard data processing:

show_raw_to_l0 input_directory output_filename

Convert directories of raw images to a single netcdf Level 0 files. This is a console script that can be executed from a standard windows or Linux shell. The command writes raw images in sub-folders of directory input_directory to netcdf file output_filename. The directories of raw images can be either downloaded from the SHOW instrument or collected the SHOW GSE. A new output file is generated each time the function is executed. It is not possible to append images to an existing file.

The code reads in both raw images and temperature data from the input_directory and linearly interpolates the temperature data to the instant of each exposure before writing to the output_file. All level 0 images stored in Level 0 files are rotated or transposed so they have the same physical orientation regardless of framegrabber. The showapi file I/O routines will return numpy arrays where the first index is along the height dimension and the 2nd index is along the interfrogram dimension.

The input directories must be organized according to SHOW instrument downloads or GSE formats otherwise it will not work properly.

The SHOW level 0 file generated by show_raw_to_l0 conatins all images found in the sub-directories. Images are sorted inside the netcdf file according to SHOW science description and time. Each unique science description generates a netcdf group within the file, think of it as a folder within the file, with the same name. All records within each netcdf group are sorted into chronological order.

This console script does not access the SHOW calibration database.

show_l0_to_l1a input_filename <options>

Convert images in a single Level 0 netcdf file to Level 1A images stored in the Level 1A directory structure. This is a console script that can be executed from a standard windows or Linux shell. The command creates SHOW level 1A interferogram images from SHOW Level 0 netcdf files.

--output_folder=foldername

The base directory of the level 1A directory structure. If this option is not set then the default value is extracted from the showapi site-settings configuration file.

--group=groupname

Only extract and process images from group groupname within the Level 0 netcdf file. If not specified it will extract and process images in all groups in the Level 0 netcdf file.

--verbose

If this option is provided it will send all verbose output to both logfile and screen. The default setting directs all verbose output only to the log file.

The code currently generates a level 1A file structure as follows:

<base_directory>                                                        # Level 1A Base directory from site-settings yaml file
+
+ --- <group name 1>                                              # Name of group taken from Level 0 netcdf file
|
|--- <yyyymmdd>                                        # Date in format year, month, day
|
|----  l1a_yyyymmdd-HHMM_vxx.xx.xx.nc         # Level 1A files for the given date grouped into 1 minute intervals
l1a_yyyymmdd-HHMM_vxx.xx.xx.nc         # Each filename has the following attributes
l1a_yyyymmdd-HHMM_vxx.xx.xx.nc         #    yyyymmdd = year, month, day
l1a_yyyymmdd-HHMM_vxx.xx.xx.nc         #    HHMM     = hour, minute
l1a_yyyymmdd-HHMM_vxx.xx.xx.nc         #    xx.xx.xx = version as  major.minor.build


This directory structure allows mutliple level 0 files to add level 1A files to the same directory structure.

show_l1a_to_l1b groupname <options>

Convert SHOW interferograms stored in the Level 1A directory structure to Level 1B spectra stored in the Level 1B directory structure. This is a console script that can be executed from a standard windows or Linux shell. The processing will include the attitude calculation if ER2 navigation information is provided through the iwg1 option.

The Level 1B data are written to a Level 1B directory structure that is similar to the Level 1A structure.

groupname

Process Level 1A images stored in this group within the Level 1A directory structure. There is currently no support to process multiple groups

--output_folder=foldername

Override the default output folder used as the base of the Level 1B directory structure. By default the Level 1B output folder is taken from the site settings yaml configuration file.

--input_folder=foldername

Override the default input folder used as the base of the Level 1A directory structure. By default the Level 1A input folder is taken from the site settings yaml configuration file.

--l1a_version=xx.yy.zz

Override the default version of level 1A files read from the Level 1A directory structure. Must be of the form xx.yy.zz. By default the default Level 1A version is taken from the site settings yaml configuration file.

--version=xx.yy.zz

Override the default version of level 1B files generated by the code. Must be of the form xx.yy.zz. By default the default Level 1B version is taken from the site settings yaml configuration file.

--startutc="yyyy-mm-dd HH:MM"

Select the start time used select input Level 1A interferograms from the Level 1A directory structure. Thius option will reject all images collected before this time. The endutc option can be used to reject images collected after a specified time. The time comparison is always truncated to the starting minute and any seconds entries are ignored. Useful when you want to quickly process a limited time period.

--endutc="yyyy-mm-dd HH:MM"

Select the end time used select input Level 1A interferograms from the Level 1A directory structure. Thius option will reject all images collected after this time. The startutc option can be used to reject images collected before a specified time. The time comparison is always truncated to the starting minute and any seconds entries are ignored; this has the slightly odd effect that “yyyy-mm-dd HH:MM:59.999” is considered less than or equal to “yyyy-mm-dd HH:MM” but means that the code will always process all the images a single 1 minute level 1A netcdf file. Useful when you want to quickly process a limited time period.

--iwg1=er2_attitude_filename

Provides attitude information for the Level 1B data records. The er2_attitude_filename is the file containing ER2 IWG1 format attitude records downloaded from the NASA ER2 website. The records in this file are linearly interpolated and used to fill out the appropriate attitude fields in the Level 1B data structures

--verbose

If this option is provided it will send all verbose output to both logfile and screen. The default setting directs all verbose output only to the log file.

The code currently generates a level 1B file structure as follows:

<Level 1B base_directory>                                               # Level 1B Base directory from site-settings yaml file
+
+ --- <group name 1>                                              # Name of group taken from Level 1A directory structure
|
|--- <yyyymmdd>                                        # Date in format year, month, day
|
|----  l1b_yyyymmdd-HHMM_vxx.xx.xx.nc         # Level 1B files for the given date grouped into 1 minute intervals
l1b_yyyymmdd-HHMM_vxx.xx.xx.nc         # Each filename has the following attributes
l1b_yyyymmdd-HHMM_vxx.xx.xx.nc         #    yyyymmdd = year, month, day
l1b_yyyymmdd-HHMM_vxx.xx.xx.nc         #    HHMM     = hour, minute
l1b_yyyymmdd-HHMM_vxx.xx.xx.nc         #    xx.xx.xx = version as  major.minor.build


The Level 1A and Level 1B directory structures can co-exist with the same base directory. It is the users choice whether they write Level 1A and Level 1B to the same directory structure or two separate directories.