gautoedit "package" for "automatic adcp" editing Matlab 5.3 or higher Jules Hummon; 12/2000 contents of this file: (I) Introduction (II) Directions (easy start) (III) program descriptions (IV) details of parameters to set up ======================================================================= =========================== (I) Introduction ========================== ======================================================================= This editing package arose from a need to apply some new editing parameters to a large number of cruises, many of which contained varying (but generally large) amounts of bad data. This editing scheme was patterned after the original waterfall editing used in conjuction with codas processing: view a time range with some editing criteria, list the ascii flag information to disk, and move on. With both kinds of editing, the results are not seen in the data until you apply the editing to the database. You should already be familiar with the original interactive editing which used waterfall plots in matlab before using autoedit. One reason for this is that that you can (from autoedit) jump back into the old waterfall editing and zap profiles or bins and it is very easy to get confused. The other reason is that the scheme is the same: data are tagged for rejection using editing criteria which are similar between the two schems, and the approach is the same: view, list, view next group, list, ..... apply. There are three additional useful tools which can make life much easier in setting up a cruise for processing. Although they are not part of the autoedit package, they were developed at the same time were designed to work together. These are all located in the programs tree, under codas3/bin/perl. They are: linkping.prl, scanping.prl, and quick_adcp.prl. Add the programs/codas3/bin/perl directory to your path, and type the name of the file on the command line for help. There is a mini-tutorial available with quick_adcp.prl if you type "quick_adcp.prl -readme". ======================================================================== =================== (II) directions (easy start) ==================== ======================================================================== chdir to your cruiseid/edit subdirectory and copy: cp /home/noio/programs/matlab/autoedit/asetup.m.template asetup.m cp /home/noio/programs/matlab/autoedit/aflagit_setup.m.template aflagit_setup.m You either have good data and want to automate the old-fashioned editing (eg. Kaimimoana), or you have lousy data and want to cavalierly eliminate anything of dubious character (eg. N.B. Palmer). The example is for decent data. (Examples of aflagit_setup.m for good and bad cruises are in goodcruise_ex/ and badcruise_ex/, respectively) EDIT in a text editor --> edit asetup.m --> edit aflagit_setup with your data type in mind (see info in the program or at the bottom of aREADME for details) --> edit setup.m (for the original interactive editing programs) (overview) --> asetup; run_aplottopog (to look at the cruise track) MATLAB - GUI APPROACH (recommended) setup %if you want to use old-fashioned interactive editing as well gautoedit %to start the gui tool This gui runs asetup (to get path info, dates,...) and aflagit_setup (sets the editing parameters) and runs a gui tool to let you view the effects of the editing on a range of days, adjust the parameters and/or go into the old-fashioned editing scheme to trim bins or profiles, allows you to list the chosen flags, and move on. This tool was developed as a way to view and screen entire datasets for gross abnormalities and identify issues to deal with later. suggested iteration: - adjust the parameters which control bin and profile flagging - "show now" to see the data - "show next" to see the next group in time - "list flags to disk" to add the flags from the current panel to the ascii (a*.asc) files. tweaking: - "find PG [blk, prf]" will let you identify time, block, profile, and bin from the Percent Good data (for your notes) - "zap edit" lets you identify a [block, profile] location in the Percent Good data and puts you into the older interactive editing scheme. You can access commands from the command line **NOTE: Make sure you "list" on the command line so your flags to to update the appropriate ascii files **NOTE: You must edit setup.m before you try to use "zap" editing so it can find your database NOTE that the reference layer and percent good cutoff can be changed in this program to let you choose what value will work best for a given time range. HOWEVER in general we have used one PG and one reference layer for an entire cruise. These parameters are set in other control files that are run when applying the editing and running the nav steps) MATLAB - MANUAL APPROACH (but you should use the gui) --> asetup; run_alookit and observe what the editing will do... repeat: [tweak aflagit_setup and run again until satisfied --> asetup; run_aedit will run through your cruise in big chunks, writing "aflag*.mat" files with flagging information, then writing three ascii files (abottom.asc, abadbin.asc, abadprf.asc) from the existing aflag*.mat files. APPLY THE RESULTS: --> apply the editing: dbupdate ../adcpdb/dbname abottom.asc >& edit.log dbupdate ../adcpdb/dbname abadprf.asc >>& edit.log badbin ../adcpdb/dbname abadbin.asc >>& edit.log dbupdate ../adcpdb/dbname bottom.asc >& edit.log dbupdate ../adcpdb/dbname badprf.asc >>& edit.log badbin ../adcpdb/dbname badbin.asc >>& edit.log set_lgb ../adcpdb/dbname >>& edit.log setflags setflags.cnt >>& edit.log --> rerun the nav steps: cd ../nav (should have already run ubprint) ubprint ubprint.cnt >& nav.log adcpsect as_nav.cnt >>& nav.log refabs refabs.cnt >>& nav.log smoothr smoothr.cnt >>& nav.log putnav putnav.cnt >>& nav.log cd ../edit examine the results, reiterate if necessary WARNING: (*) be sure to remove all previous or unwanted aflag*.mat files before running run_aedit (or you may apply more than you anticipated ======================================================================== ==================== (III) - program descriptions ====================== ======================================================================== PROGRAMS YOU EDIT: aflagit_setup.m script: set up editing (flagging) parameters (see below) asetup.m script: set up info for getmat: (database path, name, decimal day (start and end), yearbase PROGRAMS THAT RESIDE IN /home/noio/programs/matlab/autoedit %%% engines -------------------- agetmat.m function: uses structure "config" and varargin to set up information to call getmat, call getmat, load the data and calculate additional values, return everything in structures ("data" and updated "config") aplotit.m function: make a plot in the given axes (uses "data" and "config" structures to make the plot, can override "config" with varargin) aflagit.m function: use "data" and "config" structures to return "flagged" (profile flags at bad points) and "pfc1" (profile flags configuration, masks, flag values) alookit.m script: (called by run_alookit): look at the effects of the flagging without saving to disk. aedit.m script: (called by run_aedit): run through the whole cruise saving matlab files with "flagged" and dday to disk; (reminder to apply editing to database; reminder to rerun nav steps) alistit.m function: lists to ascii (like "list" in standard matlab editing of shipboard adcp data) - makes three files (so far): abottom.asc, abadprf.asc, abadbin.asc %%% drivers (for running manually) -------------------- run_aplottopog.m (this one is still useful) script: plot a cruisetrack over topography (helps with interpretation of currents and bottom locations) (it's kind of nice but isn't necessary) run_aplotscat.m script: makes a plot of shear vs/ AGC scatter (a diagnostic to help with along-track bias) (development diagnostic - unnecessary for gen user) run_alookit.m script: show the effect of editing graphically but do not save. (calls aflagit_setup) run_aedit.m script: use the flagging parameters set in aflagit_setup to find bad data and build big a big profile_flags matrix: do not show grahically; write matlab files with flag info to disk, list those to standard ascii files (abottom.asc, abadbin.asc, abadprf.asc) (calls aflagit_setup) %%% others (subsidiary tools) %%% apuv.m function: plots navigation info (called by run_alookit.m) %%% nifty tools to use while doing regular shipboard ADCP processing: app.m script: in the context of old-fashioned waterfall-plot shipboard adcp editing, load the currently existing getmat output and plot it similarly to alookit. cl_blkprf.m function: choose a field, click on block and profile and it will output two arrays: [block profile] and dday. (NOTE this exists in gautoedit as a button) ======================================================================== ================ (IV) details of parameters to set up ================ ======================================================================== The structure "config" contains information used by agetmat (to get the data) and aplotit (to plot the data). Each function starts with defaults for "config" which can be overridden by the structure passed in, and those values can be overridden by varargin: any field of "config" can be overridden by varargin {name,value} pairs. asetup.m: %set these parameters: they get stored in the structure called %"config", which gets passed to most engines % --> usually run all this from edit/ subdirectory % %start_dd = 240; % start decimal day %end_dd = 275; % beginning of last decimal day %yearbase = 2000; % year base %comment = 'ka0007'; % comment %dbpath = '../adcpdb/'; % path to database (from here: INCLUDE "/") %dbname = 'a0007'; % dbname not including "???.blk" %reflayer_range = [4 12]; % should match what you did with smoothr %numbins = 40; % number of bins to extract from database %longitude_split = 0; % split longitude here (eg. 0 if in the % % Pacific, 180 if in the Atlantic %prefix = 'a_' % getmat prefixes %navfile = ''; % read a *_xy.mat file from adcpsect for % navigation - this overrides the navigation % created with getmat (which is the original % user buffer navigation). NOTE: if you use % this option, the jitter criteria will % be nearly useless agetmat: % [data, cfg] = agetmat(config, varargin) % two modes exist: % (1) set up configuration to call getmat and do so % usage: [data, cfg] = agetmat(config, name1, val1, name2, val2) % (2) just read the output of a previous getmat call (eg during % shipboard adcp editing). % usage: [data, cfg] = agetmat; % % % --- mode: run getmat ---- % config has lots of stuff in it. % varargin is name,value pairs which can override the values in config % data is getmat data; cfg is titles, clips, fields % % required fields in config are: % start_dd % ddrange % yearbase % comment % dbpath % dbname % % usage: % [data, cfg] = agetmat(config, name1, val1, name2, val2,...) % % --- mode: just load it --- % suitable for loading getmat data which were generated by shipboard editing % data comes out the same; cfg gets the following: % % usage: % [data, config] = agetmat; % % NOTE: % units of diagnostics are all directly from getmat % abs water speeds are cm/s, ship and relative water speeds are m/s % e and w are X1000 % % --------------------- agetmat: additional details -------------------- % if only loading, % comment = ''; % rungetmat = 0; %generally want to run getmat % os_cutoff = 2; %on-station is under 2m/s % rl_range = [5 12]; %reference layer range % start_dd %from DAYS (loaded) % ddrange %from DAYS (loaded) % yearbase = -9999; comment = ''; dbname = ''; dbpath = ''; % % % % optional fields in config default to: % % nbins = 128; %number of bins to extract using getmat % cleanerparams = [200,1]; %start with [2,3] for more cleaner % comment = ''; % no additional comment % rungetmat = 1; %generally want to run getmat % os_cutoff = 2; %on-station is under 2m/s (to show fore/port ) % rl_range = [5 12]; %reference layer range (should match smoothr) % refstd_window = 5; %running std window length (for jitter calc) % % % substructures containing plotting information defaults, accessed by % field name in structure "data" % % title %structure of titles % minclipval %structure of minimum value to show % maxclipval %structure of maximum value to show aplotit: % cfg = aplotit(data, config, varargin); % plots data vs/ dday as a line or pcolor (w/ colorbar) in current axes. % if just one argument, it assumes abcissa, ordinate, and field are x,y,z. % optional: 'config' holds plotting nformation, varargin will override % %% ========================================================================== % In this context, structure "data" comes from agetmat: % % % ---------- structure "data": % % dday %1xN or Nx1 % depth %NBINSxNPRFS % [fieldname] %pg, sw, amp, u_meas, (see below for a detailed list) % % ---------- structure "config": % fn % plot this field name (defaults to '') % abc_name % fieldname to use as abcissa % % (if one argument, defaults to 'x') % % (if two arguments, defaults to 'dday') % % can also set 'abcissa_title' for xlabel (default: empty) % ord_name % fieldname to use as ordinate (defaults to data.bins) % % can also set 'ordinate_title' for ylabel % % (line plot (1xN), or pcolor (nbinsxN)) % yearbase % should have been in config structure used in agetmat % cur_ax % handle to figure axes (defaults to new axes in gca) % reverse_ydir % reverse direction of ordinate % % (defaults to 1 for bins or depth); otherwise 0 % index % how to subsample abcissa : % % - empty (use it all) % % - scalar (subsample by this many) % % - length 2 vector (range in dday) % % - length > 2 vector (indices of abcissa to use) % xlim % limits of abcissa (default is autoscale) % ylim % limits of ordinate (default is autoscale) % var_range % variable range to use for field: [minval maxval] % % defaults to substructures data.minclipval and % % data.maxclipval (values from fieldname) % cmap % colormap to match var_range (sets the number of colors) % % defaults to jet(20) % flagged % matrix of flags for excluding data in plots. same scheme % % as PFLAGS, i.e. 0 is OK (not flagged), otherwise % % type of flagging is indicated by value in bin % % (default is all zeros, i.e. none flagged) % % (see aflagit_setup or below for parameters to set) % fcolor % color of figure label (defaults to black) % comment2 % additional comment; follows config.comment, specifically % % final_comment is [comment, (comment)] % % % examples: % aplotit(data, config, 'flagged', flagged); % % % dd.x = data.dday; dd.y = data.bins; dd.z = data.pg; aplotit(dd); % % % data.z = something;aplotit(data, config, 'fn', 'z'); % % ANNOYING FACT: latest cmap will override all cmaps % can get around this using 'direct' colormaps instead of 'scaled' % surface(x,y,scaled_z,cmap_for_scaled_z,... % 'CDataMapping, 'direct',... % 'edgecolor','none'); aflagit_setup: %This is a setup script to set the structure values used by aflagit.m % "pfc" is "profile flags config" % substructures are: % flag (the value of the bad data) % cfg (configuration information) % % You either have good data and want to automate the old-fashioned editing, or % you have lousy data and want to cavalierly eliminate anything of dubious % character. Here is a table with suggested settings for a "good" or "bad" % dataset. The defaults for this file will start out for "good" data so you % don't lose too much right at the start. % % %% field of structure "pfc" good dataset bad dataset %% (eg. Kaimimoana) (eg. NBPalmer) %% ----------------------- ------------ ----------- %% pfc.refl_startbin config.rl_range(1) config.rl_range(1) %% pfc.refl_endbin config.rl_range(2) config.rl_range(2) %% pfc.yearbase config.yearbase config.yearbase %% %% pfc.cfg.bigtarget_amp 25 25 %% pfc.cfg.bigtarget_offset 5 5 %% pfc.cfg.bigtarget_avg 2 2 %% pfc.cfg.pgcutoff 30 50 (data dependent) %% pfc.cfg.maxpgbin_fudge 0 3 (data dependent) %% pfc.cfg.badpgrefl 30 50 %% pfc.cfg.badpgrefl_nbins (maybe 70% of the number of bins in the %% reference layer? easier to code it: %% round(.7*(pfc.refl_endbin - pfc.refl_startbin)) %% pfc.cfg.trimtopbins 0 1 (data dependent) %% pfc.cfg.amp_cutoff_atbin1 0 130 (data dependent) %% pfc.cfg.ampdecrperbin 10 10 (data dependent) %% pfc.cfg.ecutoff 100 100 (data dependent) %% pfc.cfg.wire_ecutoff 70 70 (data dependent) %% pfc.cfg.wire_onstation 2 2 (data dependent) %% pfc.cfg.wire_binrange [1 20] [1 20](data dependent) %% pfc.cfg.jitter_cutoff 5 5 (data dependent) %% pfc.cfg.orphan_numbins 0 pfc.cfg.badpgrefl_nbins %% pfc.cfg.orphan_slide 0 3 (data dependent) % % ---------- initialize some useful information ---------- % pfc.refl_startbin = ... % ref layer should match what was % config.rl_range(1); % used in smoothr (for consistency) % pfc.refl_endbin = ... % (get it from ../nav/as_nav.tmp or % config.rl_range(2); % ( ../nav/adcpsect.cnt) % pfc.yearbase = config.yearbase; % for labelling % % % ---------- for the bottom ---------- % looking for a high amplitude signal which SHOULD be the bottom. % these values might have to be changed if we end up flagging % scattering layers. These values are good for the bottom being deeper than % about bin 5. If the bottom gets really shallow, try no smoothing % (pfc.cfg.bigtarget_avg = 0) and a small window (pfc.cfg.bigtarget_offset = 2) % and decrease the threshold (pfc.cfg.bigtarget_amp = 10) % %pfc.cfg.bigtarget_avg = 2; % A value of 1, 2, or 3 is for smoothing % % the vertical amplitude profile with a % % a running average of 3, 5, or 7 points % % respectively. A value of zero is no % % smoothing. %pfc.cfg.bigtarget_amp = 25; % an amplitude bump bigger than this is % % identified as the bottom (like % % AMP_THRESHOLD in the zap editing) %pfc.cfg.bigtarget_offset = 5; % calculate the value of the amplitude % % bump by comparing the value at a given % % bin to the value shallower and deeper % % by this number of bins. % % ---------- percent good ---------- % % percent good is used in three ways: % % % % (1) low percent good: eliminate all % % bins with pg less than % % pfc.cgf.pgcutoff % % % (2) ringing: identified by a % % near-surface maximum (caused by a % % decrease in PG near the surface, % % then PG returns to its normal value % % deeper down). We locate the % % shallowest bin where PG attains its % % maximum and chop of everything % % shallower than that. Because PG % % may hover near its maximum for % % several bins, we back off the % % strict requirement of "the % % shallowest bin with the maximum pg" % % and instead look for "the % % shallowest bin within % % pfc.cfg.maxpgbin_fudge of the the % % maximum" % % % % (3) reference layer: eliminate % % profiles if there are too few % % reference layer bins with % % sufficiently good PG. % % % % pfc.cfg.pgcutoff = 30; % (1) low percent good (LEAVE THIS) % pfc.cfg.maxpgbin_fudge = 3; % (2) ringing (allow this much slop % % in identifying the "maximum" PG % % (set to 0 to disable) % pfc.cfg.badpgrefl = 30; % (3) reference layer PG cutoff % pfc.cfg.badpgrefl_nbins = 6; % (nbins required to accept profile) % % (set both to 0 to disable) % % NOTE: should be a large fraction % % of the number of bins in the % % reference layer, say 6 for a % % reference layer with 9 bins % % ---------- trim the top ---------- % % There are two editing criteria in this % % section eliminate bad data at the top. % % % % (1) remove top N bins from all profiles % % % % (2) remove bins where low amplitude % % may be artificially inflating % % signal/noise ratio. At the % % moment this is a linear amplitude % % cutoff with bin. % % % % pfc.cfg.trimtopbins = 0; % (1) exclude top N bins anyway % %%%% DISABLE THE NEXT PARAMETERS %%%% % % disable both by setting % % pfc.cfg.amp_cutoff_atbin1 = 0 % pfc.cfg.amp_cutoff_atbin1 = 0; % (2) amplitude cutoff at bin 1. % pfc.cfg.ampdecrperbin = 10; % Decrease amplitude cutoff by % % this many amplitude counts ber bin % % ---------- error velocity ---------- % % error velocity is used in two ways: % % (1) plain old error velocity cutoff % % % % (2) wire interference: different % % criterion (?), only on station, % % remove an additional bin left, % % right, up, and down from flagged % % bins (disable by setting cutoff to % % a large value (eg 10^6)) % % pfc.cfg.ecutoff = 100; % (1) plain old error velocity cutoff % pfc.cfg.wire_ecutoff = 70; % (2) wire interference cutoff % pfc.cfg.onstation = 2; % (only look when speed is less) % pfc.cfg.binrange = [1 20]; % (only look in this bin range) % % % % ---------- jitter flag ---------- % flag data where the measured data jitter % % is too high: "jitter" is a running % % 5-point std of the difference between % % measured velocity using the original % % navigation and smoothed navigation. % % % pfc.cfg.jitter_cutoff = 5; % flag profiles as bad if jitter % % exceeds this value in cm/s % % % ---------- flag isolated profiles ---------- % % % If the data are mostly bad, we want % % to eliminate isolated profiles as % % suspect. We do this by requiring a % % profile to have a few good close % % friends. % % % NOTE: to disable, must set both of these % % to zero % % % pfc.cfg.orphan_numbins = 0; % number of unflagged bins in ref layer % % required to accept the profile % % (before looking for neighbors) % % (to disable, set to 0) % % (for good data, try 70% of the number % % of bins in the reference layer % % pfc.cfg.orphan_slide = 0 ; % half-width of isolation profile % % i.e. slide left and right by this, % % and accept the center of a run of 3 % % or middle 3 if a run of 5 % % (for really bad data, set to 2) % % for moderately bad data, try 1 % % % the values of the flags are currently: % pfc.flag.pgcutoff = 2^0; % (written to abadbin.asc) % pfc.flag.trimtopbins = 2^1; % (written to abadbin.asc) % pfc.flag.badpgrefl = 2^2; % (written to badprf.asc) % pfc.flag.maxampbin = 2^3; % (written to bottom.asc) % pfc.flag.bottom = 2^4; % (only used in graphics as a mask) % pfc.flag.ecutoff = 2^5; % (written to abadbin.asc) % pfc.flag.orphan = 2^6; % (written to abadprf.asc) % pfc.flag.wire = 2^7; % (written to abadbin.asc) Notes on nav steps for the Palmer: % 1) should probably use something like 15-25 for ref layer % 2) smoothr: % filter_hwidth= .0104 /* 15 min: less smoothing than 0.0208 */ % min_filter_fraction= .75 /* more stringent than 0.5 */ default values in substructure of data: % % NBINSxNPRFS loaded variables and their titles % title.pg = 'PERCENT GOOD'; % title.depth = 'DEPTH'; % title.pflag = 'PROFILE_FLAGS'; % title.amp = 'AMPLITUDE'; % title.sw = 'SPECTRAL WIDTH'; % title.umeas = 'MEASURED U'; % title.umeas_refl = 'MEASURED REFL (U)'; % title.uship_sm = 'SHIP U (smoothr)'; % title.vmeas = 'MEASURED V'; % title.vmeas_refl = 'MEASURED REFL (V)'; % title.vship_sm = 'SHIP V (smoothr)'; % title.w = 'W'; % title.wmean = 'REFL W'; % title.e = 'ERROR VELOCITY'; % % NBINSxNPRFS calculated variables and their titles % title.uabs_sm = 'UABS (smoothr)'; % title.uabs_g = 'UABS (gps nav)'; % title.uabs_refl_sm = 'UABS (REFL, smoothr)'; % title.uabs_refl_g = 'UABS (REFL, gps nav)'; % % title.vabs_sm = 'VABS (smoothr)'; % title.vabs_g = 'VABS (gps nav)'; % title.vabs_refl_sm = 'VABS (REFL, smoothr)'; % title.vabs_refl_g = 'VABS (REFL, gps nav)'; % % title.fabs_sm = 'FORE ABS (smoothr)'; % title.fabs_g = 'FORE ABS (gps nav)'; % title.fabs_refl_sm = 'FORE ABS (REFL, smoothr)'; % title.fabs_refl_g = 'FORE ABS (REFL, gps nav)'; % % title.pabs_sm = 'PORT ABS (smoothr)'; % title.pabs_g = 'PORT ABS (gps nav)'; % title.pabs_refl_sm = 'PORT ABS (REFL, smoothr)'; % title.pabs_refl_g = 'PORT ABS (REFL, gps nav)'; % % % NPRFSx1 loaded or calculated variables and their titles % title.smmps = 'SHIP SPEED (smoothr)'; % title.uship_g = 'SHIP SPEED (U: gps)'; % title.vship_g = 'SHIP SPEED (U: gps)'; % title.lon = 'LONGITUDE'; % title.iblkprf = 'BLOCK AND PROFILE'; % title.lat = 'LATITUDE'; % title.dday = 'DECIMAL DAY'; % title.heading = 'SHIP HEADING'; % title.clmps = 'SHIP SPEED (gps)'; % title.bins = 'bins'; % % % default minimum and maximum clip values % % minclipval.pg = 30; maxclipval.pg = 100; % minclipval.amp = 10; maxclipval.amp = 230; % minclipval.sw = 30; maxclipval.sw = 260; % minclipval.e = -70; maxclipval.e = 70; % minclipval.uv = -1; maxclipval.uv = 1; % minclipval.mps = 0; maxclipval.mps = 7; % %