<?xml version="1.0"?>
<!DOCTYPE texinfo PUBLIC "-//GNU//DTD TexinfoML V5.0//EN" "http://www.gnu.org/software/texinfo/dtd/5.0/texinfo.dtd">
<texinfo xml:lang="en">
<filename file="nco.xml"></filename>
<preamble>\input texinfo @c -*-texinfo-*- 
</preamble><!-- c \input /home/zender/texinfo_fedora.tex -->
<!-- c \input /home/zender/texinfo_ubuntu.tex -->

<!-- c 19980817: TeX-based systems (texi2dvi, texi2dvi -pdf) require texinfo.tex -->
<!-- c Hyper-text systems (texi2html, makeinfo) do not require texinfo.tex -->

<ignore>

$Header$

Purpose: TeXInfo documentation for NCO suite

URL: http://nco.sf.net/nco.texi

Copyright (C) 1995--2014 Charlie Zender
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts. The license is available online at 
http://www.gnu.org/copyleft/fdl.html

The original author of this software, Charlie Zender, seeks to improve
it with your suggestions, contributions, bug-reports, and patches.
Charlie Zender &lt;surname at uci dot edu&gt; (yes, my surname is zender)
Department of Earth System Science
3200 Croul Hall
University of California, Irvine
Irvine, CA 92697-3100

After editing any hyperlink locations, Emacs users update indices with
C-c C-u C-a	texinfo-all-menus-update
C-c C-u C-e	texinfo-every-node-updatexb

Multiple files:
M-x texinfo-multiple-files-update

Usage: 
export TEX='tex --src-specials'
cd ~/nco/doc;texi2dvi nco.texi; makeinfo --html --ifinfo --no-split --output=nco.html nco.texi; makeinfo nco.texi; dvips -o nco.ps nco.dvi;texi2dvi --pdf nco.texi;makeinfo --xml --ifinfo --no-split --output=nco.xml nco.texi;makeinfo --no-headers --output=nco.txt nco.texi
dvips -o nco.ps nco.dvi 
dvips -Ppdf -G0 -o nco.ps nco.dvi 
makeinfo --html --ifinfo --no-split --output=nco.html nco.texi
makeinfo --no-split --output=nco.info nco.texi
makeinfo --no-headers --no-split --output=nco.txt nco.texi
makeinfo --xml --no-split --output=nco.xml nco.texi
makeinfo --docbook --no-split --output=nco.xml nco.texi
pdftotext nco.pdf nco.txt
ps2pdf -dMaxSubsetPct=100 -dCompatibilityLevel=1.2 -dSubsetFonts=true -dEmbedAllFonts=true nco.ps nco.pdf
texi2dvi --output=nco.dvi nco.texi 
texi2dvi --pdf --output=nco.pdf nco.texi
texi2html -monolithic -verbose nco.texi
texi2html -l2h -l2h_tmp=./l2h_tmp -monolithic -verbose nco.texi # Invoke latex2html
cd ~/nco/doc;/usr/bin/scp index.shtml nco_news.shtml ChangeLog TODO README VERSION nco.dvi nco.html nco.info* nco.pdf nco.ps nco.texi nco.txt nco.xml ../data/ncap.in ../data/ncap2.in zender,nco@web.sf.net:/home/project-web/nco/htdocs;cd -
cd ~/nco/doc;      scp -p index.shtml nco_news.shtml ChangeLog TODO README VERSION nco.dvi nco.html nco.info* nco.pdf nco.ps nco.texi nco.txt nco.xml ../data/ncap.in ../data/ncap2.in dust.ess.uci.edu:/var/www/html/nco;cd -
# 20130806 Copy CMIP5 images (takes additional time)
cd ~/nco/doc/xmp;/usr/bin/scp fgr*.png fgr*.eps zender,nco@web.sf.net:/home/project-web/nco/htdocs/xmp;cd -
cd ~/nco/doc/xmp;/usr/bin/scp fgr*.png fgr*.eps                 dust.ess.uci.edu:/var/www/html/nco/xmp;cd -
# 20120203 Copy all nco.html PNG images (takes additional time)
cd ~/nco/doc;/usr/bin/scp nco_*.png zender,nco@web.sf.net:/home/project-web/nco/htdocs;cd -
cd ~/nco/doc;scp -p nco_*.png dust.ess.uci.edu:/var/www/html/nco;cd -
# 20130801 Copy nco.texi to Ubuntu machine for quick build-tests
scp ~/nco/doc/nco.texi givre.ess.uci.edu:nco/doc

NB: @cindex references in footnotes propagate to PDF file, not to
    HTML files, bug-report sent to makeinfo people 20040229

Producing HTML, makeinfo vs. texi2html:
makeinfo: Better format overall 
          Uses node names for cross references and index
          Acronyms look better
          Excludes @ifinfo sections by default (override with --ifinfo)
texi2html: Index sub-divided by first character
           More fancy options (e.g., latex2html), though none very useful 
           Misprints title
           Includes @ifinfo sections by default

Legend (defined in &quot;highlighting&quot; section of TeXInfo manual):
@code{}: Program text, e.g., @code{if(foo) x=y;}
@command{}: Commands, e.g., @command{ncra}
@dfn{}: Define use of term, e.g., @dfn{supercalifragilisticexpialidocious}
@email{}: E-mail address, e.g., @email{surname at uci dot edu}
@env{}: Environment variable, e.g., @env{HOME}
@file{}: Filename, e.g., @file{in.nc}
@html: Text until @end html passed without translation
@ifhtml: Text until @end ifhtml passed with translation
@kbd{}: Keyboard input, e.g., @kbd{ncra in.nc out.nc}
@key{}: Key name, e.g., @key{ESC} (rarely needed)
@option{}: Command-line option, e.g., @option{--dbg}
@samp{}: Extended commands, character sequences, e.g., @samp{ncra in.nc out.nc}
@uref{}: URL with optional alternate text, e.g., @uref{http://nco.sf.net,NCO homepage}
@url{}: URL, synonym for @uref
@var{}: Metasyntactic variable, e.g., @var{}
@verbatim: Anything goes inside environment (no @'s needed to protect special characters like braces)
@example: Quoted environment (@'s needed to protect special characters like braces)
@w{}: Unbreakable text, e.g., @w{of 1}

Use '@*' to force hard carriage return
Use '@:', after periods, questions marks, exclamation marks, or colons
that do not end sentences, e.g., 'vs.@:'
Use '@.', '@!', and `@?' to end sentences that end with single capital letters (e.g., initials)

Resources:
Octave TeXInfo manual shows clean TeXInfo structure
/usr/share/doc/octave-2.1.34/interpreter/octave.texi
</ignore>

<!-- c Start of header -->

<!-- c No variables may be defined before TeXInfo @setfilename header -->
<setfilename file="nco.info" spaces=" ">nco.info</setfilename>

<!-- c Define edition, date, ... -->
<set name="nco-edition" line=" nco-edition 4.4.2">4.4.2</set>
<set name="doc-edition" line=" doc-edition 4.4.2">4.4.2</set>
<set name="copyright-years" line=" copyright-years 1995--2014">1995--2014</set>
<set name="update-year" line=" update-year 2014">2014</set>
<set name="update-date" line=" update-date 19 February 2014">19 February 2014</set>
<set name="update-month" line=" update-month February 2014">February 2014</set>

<settitle spaces=" "><acronym><acronymword>NCO</acronymword></acronym> 4.4.2 User Guide</settitle>

<!-- c Uncomment following line to produce guide in smallbook format -->
<!-- c @smallbook -->
<!-- c Merge function index into concept index -->
<syncodeindex from="fn" to="cp" line=" fn cp"></syncodeindex>

<!-- c 20090226 Add bibliography capabilities as per -->
<!-- c http://lists.gnu.org/archive/html//help-texinfo/2004-12/txtPW9h_VG8ez.txt -->
<!-- c % csz 20100313 remove next line to prevent "\input texinfo" from -->
<!-- c % appearing in printed output -->
<!-- c %% \input texinfo   @c -*-texinfo-*- -->

<!-- c %% my-bib-macros.texi - Texinfo macros providing a crude -->
<!-- c %% bibliography and citation capability. -->

<!-- c % Copyright (C) 2004  Aaron S. Hawley -->

<!-- c % Author: Aaron S. Hawley <ashawley@gnu.uvm.edu> -->
<!-- c % Keywords: docs, texinfo, extensions, bib -->

<!-- c % This file is free software; you can redistribute it and/or modify -->
<!-- c % it under the terms of the GNU General Public License as published by -->
<!-- c % the Free Software Foundation; either version 2, or (at your option) -->
<!-- c % any later version. -->

<!-- c % This file is distributed in the hope that it will be useful, -->
<!-- c % but WITHOUT ANY WARRANTY; without even the implied warranty of -->
<!-- c % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the -->
<!-- c % GNU General Public License for more details. -->

<!-- c % You should have received a copy of the GNU General Public License -->
<!-- c % along with GNU Emacs; see the file COPYING.  If not, write to -->
<!-- c % the Free Software Foundation, Inc., 59 Temple Place - Suite 330, -->
<!-- c % Boston, MA 02111-1307, USA. -->

<!-- c %% Commentary: -->

<!-- c %%% Introduction -->

<!-- c %% Creates references to a ``Bibliography'' or ``References'' -->
<!-- c %% section of a Texinfo document, using Texinfo's -->
<!-- c %% macro system.  Although not as terse a way to cite systems as is -->
<!-- c %% found in document systems like TeX, the format is simpler and -->
<!-- c %% the rendering  is easier for non-academic readers. -->

<!-- c % -->

<!-- c %%% Usage -->

<!-- c %% References to cited works can be put in any section of a document. -->
<!-- c %% The cited works must be put in either a Texinfo table (for -->
<!-- c %% instance with ``@table @asis ... @end table'') or within a -->
<!-- c %% Texinfo list (something like ``@enumerate ... @end enumerate'' -->
<!-- c %% or ``@itemize @bullet ... @end itemize'').  They are created -->
<!-- c %% with the command ``@mybibitem{REF-NAME}''.  To cite a reference -->
<!-- c %% with a @mybibitem use ``@mybibcite{REF-NAME}''. -->

<!-- c %% The beginning of a document must include (using the @include -->
<!-- c %% command) the file my-bib-macros.texi, which should be made -->
<!-- c %% available in the current directory of the parent file.  A single -->
<!-- c %% call should be made to choose to use a list or a table.  The -->
<!-- c %% command to chose is called @mybibuselist{NODE}, where NODE is -->
<!-- c %% the node containing the location where the references are listed. -->

<!-- c % -->

<!-- c %%% Example -->

<!-- c %% \input texinfo   @c -*-texinfo-*- -->
<!-- c %% @comment %**start of header -->
<!-- c %% @setfilename my-file-with-bib.info -->
<!-- c %% @settitle Texinfo with a Bibliography and References -->
<!-- c %% -->
<!-- c %% @include my-bib-macros.texi -->
<!-- c %% @mybibuselist{References} -->
<!-- c %% -->
<!-- c %% @comment %**end of header -->
<!-- c %% -->
<!-- c %% @node Top -->
<!-- c %% @top Top -->
<!-- c %% -->
<!-- c %% @menu -->
<!-- c %% * Introduction:: -->
<!-- c %% * @mybibnode{}:: -->
<!-- c %% -->
<!-- c %% @end menu -->
<!-- c %% -->
<!-- c %% @node Introduction -->
<!-- c %% @chapter Introduction -->
<!-- c %% -->
<!-- c %% The ability of a documentation format to make cross references to a -->
<!-- c %% bibliography, a feature of LaTeX @mybibcite{LaTeX2e}, isn't -->
<!-- c %% currently supported in Texinfo. -->
<!-- c %% -->
<!-- c %% @node @mybibnode{} -->
<!-- c %% @chapter References -->
<!-- c %% -->
<!-- c %% @itemize @asis -->
<!-- c %% -->
<!-- c %% @mybibitem{LaTeX2e} Leslie Lamport, LaTeX User's Guide and -->
<!-- c %% Reference Manual, 2nd edition, Addison-Wesley, Reading, -->
<!-- c %% Massachusetts, 1994. -->
<!-- c %% -->
<!-- c %% @end itemize -->
<!-- c %% -->
<!-- c %% @bye -->

<!-- c %% -->

<!-- c %% This example produces (in Info): -->

<!-- c %%       1 Introduction -->
<!-- c %%       ************** -->
<!-- c %% -->
<!-- c %%       The ability of a documentation format to make cross -->
<!-- c %%       references to a bibliography, a feature of LaTeX (See item -->
<!-- c %%       [LaTeX2e] in *Note LaTeX2e: References.), is not currently -->
<!-- c %%       supported in Texinfo. -->
<!-- c %% -->
<!-- c %% -->
<!-- c %%       2 References -->
<!-- c %%       ************ -->
<!-- c %% -->
<!-- c %%       [LaTeX2e] Leslie Lamport, LaTeX User's Guide and Reference -->
<!-- c %%       Manual, 2nd edition, Addison-Wesley, Reading, -->
<!-- c %%       Massachusetts, 1994. -->

<!-- c %% and (in printed output): -->

<!-- c %%       1 Introduction -->
<!-- c %%       ************** -->
<!-- c %% -->
<!-- c %%       The ability of a documentation format to make cross -->
<!-- c %%       references to a bibliography, a feature of LaTeX (See item -->
<!-- c %%       [LaTeX2e] in Chapter 2 [References], page 3.), is not -->
<!-- c %%       currently supported in Texinfo. -->
<!-- c %% -->
<!-- c %% -->
<!-- c %%       2 References -->
<!-- c %%       ************ -->
<!-- c %% -->
<!-- c %%       [LaTeX2e] Leslie Lamport, LaTeX User's Guide and Reference -->
<!-- c %%       Manual, 2nd edition, Addison-Wesley, Reading, -->
<!-- c %%       Massachusetts, 1994. -->

<!-- c % -->

<!-- c %%% Notes -->

<!-- c %% The pointers to references will be functional in hypertext -->
<!-- c %% documentation (info, HTML, XML and others) and properly rendered -->
<!-- c %% in print documents, because they are implemented with Texinfo's -->
<!-- c %% cross referencing capabilities (using @anchor and @ref).  Failures -->
<!-- c %% by an author to make proper references with ``my-bib-macros'' in -->
<!-- c %% their document will give cross referencing errors by Texinfo -->
<!-- c %% conversion tools. -->

<!-- c %% Only one ``Reference'' section is allowed per document. -->

<!-- c %% An improvement of this system would create cross references -->
<!-- c %% (with @xref) at each cited work to all the originating cross -->
<!-- c %% refererences. -->

<!-- c % -->

<!-- c %% Code: -->

<!-- c % Configuration Options -->

<!-- c %% @mybibsetrefnode : Defines the name of the node to contain -->
<!-- c %% references. -->

<macro name="mybibsetrefnode" line=" mybibsetrefnode{node}"><formalarg>node</formalarg>@set mybibrefnode \node\
</macro>

<!-- c %% @mybibnode{} : Macro to be placed at node containing references -->
<!-- c %% and calls to @mybibcite{} -->

<macro name="mybibnode" line=" mybibnode{}">@value{mybibrefnode}
</macro>

<!-- c %% @mybibusetable : Whether each @mybibitem will be put in a -->
<!-- c %% table. -->

<macro name="mybibusetable" line=" mybibusetable{node}"><formalarg>node</formalarg>@set mybibtable true
@ifset mybiblist
@clear mybiblist
@end ifset
@mybibsetrefnode{\node\}
</macro>

<!-- c %% @mybibuselist : Whether each @mybibitem will be put in a -->
<!-- c %% list. -->

<macro name="mybibuselist" line=" mybibuselist{node}"><formalarg>node</formalarg>@set mybiblist true
@ifset mybibtable
@clear mybibtable
@end ifset
@mybibsetrefnode{\node\}
</macro>


<!-- c %% @mybibcite{REF} : Cites the cross reference REF. -->

<macro name="mybibcite" line=" mybibcite{ref}"><formalarg>ref</formalarg>@ifclear mybibrefnode
@mybibmakeref{mybibsetrefnode was not used, \ref\}
@end ifclear
@c %**else if
@ifset mybibrefnode
@mybibmakeref{@mybibnode{}, \ref\}
@end ifset

</macro>

<macro name="mybibmakeref" line=" mybibmakeref{node, ref}"><formalarg>node</formalarg><formalarg>ref</formalarg>(See item [\ref\] in @ref{\node\, \ref\}.)
</macro>

<!-- c %% @mybibcite{REF} : Creates a cross referenced citation REF. -->

<macro name="mybibitem" line=" mybibitem{ref}"><formalarg>ref</formalarg>@ifclear mybiblist
@ifclear mybibtable
@set mybiblist true
@end ifclear
@end ifclear
@ifset mybiblist
@item
@anchor{\ref\}[\ref\]
@end ifset
@c %**else if
@ifset mybibtable
@item @anchor{\ref\}[\ref\]
@end ifset

</macro>

<!-- c %% my-bib-macros.texi ends here -->
<set name="mybiblist" line=" mybiblist true">true</set>
<set name="mybibrefnode" line=" mybibrefnode References">References</set>

<!-- c end of header -->

<!-- c TeXInfo macros may not appear before TeXInfo @setfilename header -->
<!-- c [idx] Index -->
<macro name="idx" line=" idx {}">i
</macro>
<!-- c [m s-1] Meridional wind speed -->
<macro name="wndmrd" line=" wndmrd {}">v
</macro>
<!-- c [m s-1] Zonal wind speed -->
<macro name="wndznl" line=" wndznl {}">u
</macro>
<macro name="xxx" line=" xxx {}">x
</macro>
<!-- c TeX macros may appear anywhere after line 1 -->
<tex endspaces=" ">
% Define TeX macros to roughly correspond to LaTeX style files
% Use \gdef instead of \def to make definition persistent across TeX blocks
% These should be consistent with any TeXInfo macros
% 1. Primary commands
\gdef\dmn{D} % [dmn] Variable dimension
\gdef\rdr{R} % [dmn] Re-order dimension
\gdef\shr{S} % [dmn] Share dimension
\gdef\dmnidx{n} % [idx] Dimension index
\gdef\dmnnbr{N} % [nbr] Dimension number
\gdef\rdridx{r} % [idx] Re-order index
\gdef\rdrnbr{R} % [nbr] Re-order number
\gdef\shridx{s} % [idx] Share index
\gdef\shrnbr{S} % [nbr] Share number
\gdef\dfr{{\rm d}} % [frc] Math differential fxm: upright
\gdef\idx{i} % [idx] Index
\gdef\iii{i} % [idx] i
\gdef\jjj{j} % [idx] j
\gdef\kkk{k} % [idx] k
\gdef\lmnidx{i} % [idx] Element index
\gdef\outnbr{J} % [nbr] Number of elements in output hyperslab
\gdef\lmnnbr{N} % [nbr] Number of elements in input hyperslab
\gdef\tllnbr{M} % [nbr] Tally (number of valid elements in input hyperslab)
\gdef\me{{\rm e}} % [frc] Math e fxm: upright
\gdef\mi{{\rm i}} % [frc] Math i fxm: upright
\gdef\mpi{\pi} % [frc] Math pi
\gdef\mpp{\cal{M}} % [map] Map
\gdef\mskflg{m} % [flg] Mask flag
\gdef\mssflg{\mu} % [flg] Missing value flag
\gdef\tm{t} % [s] Time
\gdef\prmsbs{\prime} % [sbs] Prime subscript
\gdef\wgt{w} % [frc] Weight
\gdef\wndmrd{v} % [m s-1] Meridional wind speed
\gdef\wndznl{u} % [m s-1] Zonal wind speed
\gdef\xxx{x} % [ltr] x
\gdef\yyy{y} % [ltr] y

% 2. Derived commands
\gdef\dmnvct{{\bf \dmn}} % [vct] Dimension vector
\gdef\dmnprm{\dmn^{\prmsbs}} % [vct] Dimension prime
\gdef\dmnsubnnn{\dmn_{\dmnidx}} % [dmn] Dimension sub nnn
\gdef\shrsubnnn{\shr_{\dmnidx}} % [dmn] Share dimension sub nnn
\gdef\shrsubsss{\shr_{\shridx}} % [dmn] Share dimension sub sss
\gdef\dmnsubnnnprm{\dmn_{\dmnidx}^{\prmsbs}} % [vct] Dimension prime sub nnn 
\gdef\dmnvctprm{{\bf \dmn}^{\prmsbs}} % [vct] Dimension vector prime
\gdef\rdrvct{{\bf \rdr}} % [vct] Re-order vector
\gdef\shrvct{{\bf \shr}} % [vct] Share vector
\gdef\xxxprm{\xxx^{\prmsbs}} % [ltr] x prime

% 3. Doubly derived commands

</tex>

<!-- c install-info installs NCO info into this category -->
<dircategory spaces=" ">netCDF</dircategory>
<direntry endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::        ">NCO</menunode><menudescription><pre xml:space="preserve">User Guide for the netCDF Operator suite
</pre></menudescription></menuentry></direntry>

<!-- c Set smallbook if printing in smallbook format -->
<!-- c Example of smallbook font is actually written using smallbook -->
<!-- c In bigbook, a kludge is used for TeX output -->
<!-- c set smallbook -->
<clear name="smallbook" line=" smallbook"></clear>

<tex endspaces=" ">
% fxm: Try to get thumbnails working with texinfo &textndash;pdf -generated PDF files
% \input thumbpdf.sty
% Experiment with smaller amounts of whitespace between chapters and sections
\global\chapheadingskip = 15pt plus 4pt minus 2pt 
\global\secheadingskip = 12pt plus 3pt minus 2pt
\global\subsecheadingskip = 9pt plus 2pt minus 2pt
</tex>

<!-- c Experiment with smaller amounts of whitespace between paragraphs in the 8.5 by 11 inch format -->
<tex endspaces=" ">
\global\parskip 6pt plus 1pt
</tex>

<!-- c Uncomment next line to remove ugly TeX warning blocks from overfull hboxes -->
<finalout></finalout>

<para>This file documents <acronym><acronymword>NCO</acronymword></acronym>, a collection of utilities to
manipulate and analyze netCDF files.
</para>
<para>Copyright &copyright; 1995&textndash;2014 Charlie Zender
</para>
<para>This is the first edition of the <cite>NCO User Guide</cite>,&linebreak;
and is consistent with <w>version 2</w> of <file>texinfo.tex</file>.
</para>
<para>Permission is granted to copy, distribute and/or modify this document 
under the terms of the <acronym><acronymword>GNU</acronymword></acronym> Free Documentation License, <w>Version 1.3</w>
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts. The license is available online at 
<uref><urefurl>http://www.gnu.org/copyleft/fdl.html</urefurl></uref>
</para>
<para>The original author of this software, Charlie Zender, wants to improve it
with the help of your suggestions, improvements, bug-reports, and patches.&linebreak;
Charlie Zender &lt;surname at uci dot edu&gt; (yes, my surname is zender)&linebreak;
3200 Croul Hall&linebreak;
Department of Earth System Science&linebreak;
University of California, Irvine&linebreak;
Irvine, CA 92697-3100&linebreak;
</para>
<ignore>
Permission is granted to process this file through TeX and print the 
results, provided the printed document carries copying permission
notice identical to this one except for the removal of this paragraph
(this paragraph not being relevant to the printed manual).
</ignore>

<setchapternewpage line=" odd"></setchapternewpage>

<titlepage endspaces=" ">
<html endspaces=" ">
&lt;meta name=&quot;Author&quot; content=&quot;Charlie Zender&quot;&gt;
&lt;meta name=&quot;Keywords&quot; content=&quot;NCO documentation, NCO User Guide,
netCDF, operator, GCM, CCM, scientific data, ncbo, ncfe, ncecat,
ncflint, ncks, ncra, ncrcat, ncrename, ncwa&quot;&gt;
</html>
<html endspaces=" ">
&lt;body text=&quot;#000000&quot; link=&quot;#0000EF&quot; vlink=&quot;#008080&quot; alink=&quot;#FF0000&quot;&gt;
&lt;font face=&quot;Arial&quot;&gt;
</html>
<ignore>
</ignore>
<title spaces=" ">NCO User Guide</title>
<subtitle spaces=" ">A suite of netCDF operators</subtitle>
<subtitle spaces=" ">Edition 4.4.2, for <acronym><acronymword>NCO</acronymword></acronym> Version 4.4.2</subtitle>
<subtitle spaces=" ">February 2014</subtitle>

<author spaces=" ">by Charlie Zender</author>
<author spaces=" ">Department of Earth System Science</author>
<author spaces=" ">University of California, Irvine</author>
<html endspaces=" ">
&lt;p&gt;WWW readers: Having trouble finding the section you want?&lt;/p&gt; 
&lt;p&gt;Search for keywords in the &lt;a href=&quot;#index&quot;&gt;(hyper) index&lt;/a&gt; at the end&lt;/p&gt;
</html>

<!-- c Include Distribution inside titlepage so that headings are turned off -->
<page></page>
<vskip> 0pt plus 1filll</vskip>
<para>Copyright &copyright; 1995&textndash;2014 Charlie Zender.
</para>
<sp value="2" line=" 2"></sp>
<para>This is the first edition of the <cite>NCO User Guide</cite>,&linebreak;
and is consistent with <w>version 2</w> of <file>texinfo.tex</file>.
</para><sp value="2" line=" 2"></sp>

<para>Published by Charlie Zender&linebreak;
Department of Earth System Science&linebreak;
3200 Croul Hall&linebreak;
University of California, Irvine&linebreak;
Irvine, CA 92697-3100 USA&linebreak;
</para>
<para>Permission is granted to copy, distribute and/or modify this document
under the terms of the <acronym><acronymword>GNU</acronymword></acronym> Free Documentation License, <w>Version 1.3</w>
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts. The license is available online at 
<uref><urefurl>http://www.gnu.org/copyleft/fdl.html</urefurl></uref>
</para><sp value="2" line=" 2"></sp>
<para>The original author of this software, Charlie Zender, wants to improve it
with the help of your suggestions, improvements, bug-reports, and patches.&linebreak;
Charlie Zender &lt;surname at uci dot edu&gt; (yes, my surname is zender)&linebreak;
Department of Earth System Science&linebreak;
3200 Croul Hall&linebreak;
University of California, Irvine&linebreak;
Irvine, CA 92697-3100&linebreak;
</para><sp value="2" line=" 2"></sp>
<!-- c Cover art by Robynn Rudel -->
</titlepage>

<!-- c Print table of contents -->
<contents></contents>

<node name="Top" spaces=" "><nodename>Top</nodename><nodenext spaces=" ">Foreword</nodenext><nodeprev spaces=" ">(dir)</nodeprev><nodeup spaces=" ">(dir)</nodeup></node>
<top spaces=" "><sectiontitle>NCO User Guide</sectiontitle>
<!-- c node format: node-name, next, previous, up -->

<para><emph>Note to readers of the NCO User Guide in Info format</emph>: 
<emph>The <uref><urefurl>./nco.pdf</urefurl><urefdesc>NCO User Guide in PDF format</urefdesc></uref>
(also on <uref><urefurl>http://nco.sf.net/nco.pdf</urefurl><urefdesc>SourceForge</urefdesc></uref>)
contains the complete <acronym><acronymword>NCO</acronymword></acronym> documentation.</emph>
This Info documentation is equivalent except it refers you to the 
printed (i.e., DVI, PostScript, and PDF) documentation for description 
of complex mathematical expressions. Also, images appear only in the
PDF document due to SourceForge limitations.
</para>
<para>The netCDF Operators, or <acronym><acronymword>NCO</acronymword></acronym>, are a suite of programs known as 
operators. 
The operators facilitate manipulation and analysis of data stored in the 
self-describing netCDF format, available from
(<uref><urefurl>http://www.unidata.ucar.edu/packages/netcdf</urefurl></uref>).
Each <acronym><acronymword>NCO</acronymword></acronym> operator (e.g., ncks) takes netCDF input
file(s), performs an operation (e.g., averaging, hyperslabbing, or
renaming), and outputs a processed netCDF file. 
Although most users of netCDF data are involved in scientific research,
these data formats, and thus <acronym><acronymword>NCO</acronymword></acronym>, are generic and are equally
useful in fields from agriculture to zoology.
The <acronym><acronymword>NCO</acronymword></acronym> User Guide illustrates <acronym><acronymword>NCO</acronymword></acronym> use with
examples from the field of climate modeling and analysis. 
The <acronym><acronymword>NCO</acronymword></acronym> homepage is <uref><urefurl>http://nco.sf.net</urefurl></uref>, and
there is a mirror at <uref><urefurl>http://dust.ess.uci.edu/nco</urefurl></uref>.
</para>
<para>This documentation is for <acronym><acronymword>NCO</acronymword></acronym> version 4.4.2.
It was last updated 19 February 2014.
Corrections, additions, and rewrites of this documentation are very
welcome.
</para>
<para>Enjoy,&linebreak;
Charlie Zender
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Foreword</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Summary</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Introduction</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Strategies</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Common features</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Operator Reference Manual</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Contributing</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Quick Start</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">CMIP5 Example</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Parallel</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">CCSM Example</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">mybibnode</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">General Index</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<html endspaces=" ">
&lt;a name=&quot;fwd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#fwd &textndash;&gt;
</html>
</top>
<node name="Foreword" spaces=" "><nodename>Foreword</nodename><nodenext spaces=" ">Summary</nodenext><nodeprev spaces=" ">Top</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<unnumbered spaces=" "><sectiontitle>Foreword</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1">foreword</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="2">Charlie Zender</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> is the result of software needs that arose while I worked
on projects funded by <acronym><acronymword>NCAR</acronymword></acronym>, <acronym><acronymword>NASA</acronymword></acronym>, and <acronym><acronymword>ARM</acronymword></acronym>.
Thinking they might prove useful as tools or templates to others,  
it is my pleasure to provide them freely to the scientific community.  
Many users (most of whom I have never met) have encouraged the
development of <acronym><acronymword>NCO</acronymword></acronym>.  
Thanks espcially to Jan Polcher, Keith Lindsay, Arlindo <w>da Silva</w>,
John Sheldon, and William Weibel for stimulating suggestions and
correspondence. 
Your encouragment motivated me to complete the <cite>NCO User Guide</cite>.
So if you like <acronym><acronymword>NCO</acronymword></acronym>, send me a note!
<w>I should</w> mention that <acronym><acronymword>NCO</acronymword></acronym> is not connected to or
officially endorsed by Unidata, <acronym><acronymword>ACD</acronymword></acronym>, <acronym><acronymword>ASP</acronymword></acronym>,
<acronym><acronymword>CGD</acronymword></acronym>, or Nike.&linebreak; 
</para><sp value="1" line=" 1"></sp>
<noindent></noindent>
<para>Charlie Zender&linebreak;
May 1997&linebreak;
Boulder, Colorado&linebreak;
</para>
<sp value="2" line=" 2"></sp>
<para>Major feature improvements entitle me to write another Foreword.
In the last five years a lot of work has been done to refine
<acronym><acronymword>NCO</acronymword></acronym>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="3">open source</indexterm></cindex>
<acronym><acronymword>NCO</acronymword></acronym> is now an open source project and appears to be much
healthier for it. 
The list of illustrious institutions that do not endorse <acronym><acronymword>NCO</acronymword></acronym>     
continues to grow, and now includes <acronym><acronymword>UCI</acronymword></acronym>.&linebreak; 
</para><sp value="1" line=" 1"></sp>
<noindent></noindent>
<para>Charlie Zender&linebreak;
October 2000&linebreak;
Irvine, California&linebreak;
</para>
<sp value="2" line=" 2"></sp>
<para>The most remarkable advances in <acronym><acronymword>NCO</acronymword></acronym> capabilities in the last  
few years are due to contributions from the Open Source community.
Especially noteworthy are the contributions of Henry Butowsky and Rorik 
Peterson.&linebreak; 
</para><sp value="1" line=" 1"></sp>
<noindent></noindent>
<para>Charlie Zender&linebreak;
January 2003&linebreak;
Irvine, California&linebreak;
</para>
<sp value="2" line=" 2"></sp>
<para><acronym><acronymword>NCO</acronymword></acronym> was generously supported from 2004&textndash;2008 by US 
National Science Foundation (<acronym><acronymword>NSF</acronymword></acronym>) grant 
<uref><urefurl>http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0431203</urefurl><urefdesc>IIS-0431203</urefdesc></uref>. 
This support allowed me to maintain and extend core <acronym><acronymword>NCO</acronymword></acronym> code,
and others to advance <acronym><acronymword>NCO</acronymword></acronym> in new directions: 
Gayathri Venkitachalam helped implement <acronym><acronymword>MPI</acronymword></acronym>;
Harry Mangalam improved regression testing and benchmarking;
Daniel Wang developed the server-side capability, <acronym><acronymword>SWAMP</acronymword></acronym>;
and Henry Butowsky, a long-time contributor, developed <command>ncap2</command>.
This support also led <acronym><acronymword>NCO</acronymword></acronym> to debut in professional journals
and meetings.  
The personal and professional contacts made during this evolution have
been immensely rewarding.&linebreak;
</para><sp value="1" line=" 1"></sp>
<noindent></noindent>
<para>Charlie Zender&linebreak;
March 2008&linebreak;
Grenoble, France&linebreak;
</para>
<sp value="2" line=" 2"></sp>
<para>The end of the <acronym><acronymword>NSF</acronymword></acronym> SEI grant in August, 2008 curtailed <acronym><acronymword>NCO</acronymword></acronym>
development.  
Fortunately we could justify supporting Henry Butowsky on other research 
grants until May, 2010 while he developed the key <command>ncap2</command>
features used in our climate research.
And recentely the NASA ACCESS program commenced funding NCO support for
netCDF4 group functionality.&linebreak; 
Thus <acronym><acronymword>NCO</acronymword></acronym> will grow and evade bit-rot for the foreseeable future.
</para>
<para>On a personal level, I continue to receive with gratitude the thanks of
<acronym><acronymword>NCO</acronymword></acronym> users at nearly every scientific meeting I attend. 
People introduce themselves, shake my hand and extol, sometimes rather
effusively, these time-saving tools.
These exchanges lighten me like anti-gravity.
Sometimes I daydream how many hours <acronym><acronymword>NCO</acronymword></acronym> has turned from grunt
work to productive research for researchers world-wide, or from research
into early happy hours. 
It&textrsquo;s a cool feeling.
</para>
<!-- c @uref{http://www.nasa.gov/mission_pages/IPY/main/PolarExploration.html,NNX07AR23G}.@* -->
<sp value="1" line=" 1"></sp>
<noindent></noindent>
<para>Charlie Zender&linebreak;
April, 2012&linebreak;
Irvine, California&linebreak;
</para>
<html endspaces=" ">
&lt;a name=&quot;smr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#smr &textndash;&gt;
</html>
</unnumbered>
<node name="Summary" spaces=" "><nodename>Summary</nodename><nodenext spaces=" ">Introduction</nodenext><nodeprev spaces=" ">Foreword</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<unnumbered spaces=" "><sectiontitle>Summary</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="4">operators</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="5">summary</indexterm></cindex>
<para>This manual describes <acronym><acronymword>NCO</acronymword></acronym>, which stands for netCDF Operators.
<acronym><acronymword>NCO</acronymword></acronym> is a suite of programs known as <dfn>operators</dfn>.
Each operator is a standalone, command line program executed at the
shell-level like, e.g., <command>ls</command> or <command>mkdir</command>.  
The operators take netCDF files (including <acronym><acronymword>HDF5</acronymword></acronym> files
constructed using the netCDF <acronym><acronymword>API</acronymword></acronym>) as input, perform an
operation (e.g., averaging or hyperslabbing), and produce a netCDF file 
as output.  
The operators are primarily designed to aid manipulation and analysis of 
data.
The examples in this documentation are typical applications of the
operators for processing climate model output. 
This stems from their origin, though the operators are as general as
netCDF itself.
</para>
<html endspaces=" ">
&lt;a name=&quot;ntr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ntr &textndash;&gt;
</html>
</unnumbered>
<node name="Introduction" spaces=" "><nodename>Introduction</nodename><nodenext spaces=" ">Strategies</nodenext><nodeprev spaces=" ">Summary</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<chapter spaces=" "><sectiontitle>Introduction</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="6">introduction</indexterm></cindex>

<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Availability</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">How to Use This guide</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Compatability</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Symbolic Links</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Libraries</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">netCDF2/3/4 and HDF4/5 Support</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Help Requests and Bug Reports</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<node name="Availability" spaces=" "><nodename>Availability</nodename><nodenext spaces=" ">How to Use This guide</nodenext><nodeprev spaces=" ">Introduction</nodeprev><nodeup spaces=" ">Introduction</nodeup></node>
<section spaces=" "><sectiontitle>Availability</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="7"><acronym><acronymword>NCO</acronymword></acronym> availability</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="8">source code</indexterm></cindex>
<para>The complete <acronym><acronymword>NCO</acronymword></acronym> source distribution is currently distributed
as a <dfn>compressed tarfile</dfn> from
<uref><urefurl>http://sf.net/projects/nco</urefurl></uref>
and from
<uref><urefurl>http://dust.ess.uci.edu/nco/nco.tar.gz</urefurl></uref>.
The compressed tarfile must be uncompressed and untarred before building
<acronym><acronymword>NCO</acronymword></acronym>.
Uncompress the file with <samp>gunzip nco.tar.gz</samp>. 
Extract the source files from the resulting tarfile with <samp>tar -xvf
nco.tar</samp>.    
<acronym><acronymword>GNU</acronymword></acronym> <code>tar</code> lets you perform both operations in one step
with <samp>tar -xvzf nco.tar.gz</samp>. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="9">documentation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="10">WWW documentation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="11">on-line documentation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="12"><acronym><acronymword>HTML</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="13">&tex;info</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="14">Info</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="15"><cite>User Guide</cite></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="16"><cite>NCO User Guide</cite></indexterm></cindex>
<para>The documentation for <acronym><acronymword>NCO</acronymword></acronym> is called the 
<cite>NCO User Guide</cite>. 
The <cite>User Guide</cite> is available in <acronym><acronymword>PDF</acronymword></acronym>, Postscript,
<acronym><acronymword>HTML</acronymword></acronym>, <acronym><acronymword>DVI</acronymword></acronym>, &tex;info, and Info formats.
These formats are included in the source distribution in the files
<file>nco.pdf</file>, <file>nco.ps</file>, <file>nco.html</file>, <file>nco.dvi</file>,
<file>nco.texi</file>, and <file>nco.info*</file>, respectively.
All the documentation descends from a single source file,
<file>nco.texi</file>
<footnote spaces="   \n"><para>To produce these formats, <file>nco.texi</file> was simply run through the
freely available programs <code>texi2dvi</code>, <code>dvips</code>,
<code>texi2html</code>, and <code>makeinfo</code>.    
Due to a bug in &tex;, the resulting Postscript file, <file>nco.ps</file>,
contains the Table of Contents as the final pages. 
Thus if you print <file>nco.ps</file>, remember to insert the Table of
Contents after the cover sheet before you staple the manual.
</para></footnote>.
Hence the documentation in every format is very similar.
However, some of the complex mathematical expressions needed to describe
<command>ncwa</command> can only be displayed in <acronym><acronymword>DVI</acronymword></acronym>, Postscript, and 
<acronym><acronymword>PDF</acronymword></acronym> formats. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="17">publications</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="18">presentations</indexterm></cindex>
<para>A complete list of papers and publications on/about <acronym><acronymword>NCO</acronymword></acronym> 
is available on the <acronym><acronymword>NCO</acronymword></acronym> homepage.
Most of these are freely available. 
<!-- c 20130526 fxm Replace mybibnode with @mybibnode{} -->
<!-- c 20130526 Doing so, unfortunately, produces error "TeX capacity exceeded, sorry [input stack size=5000]." -->
<!-- c 20130526 Denude document of @mybibcite{} until it works -->
<!-- c The primary refereed publications are @mybibcite{ZeM06} and @mybibcite{Zen08}.  -->
The primary refereed publications are ZeM06 and Zen08. 
These contain copyright restrictions which limit their redistribution,
but they are freely available in preprint form from the <acronym><acronymword>NCO</acronymword></acronym>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="19"><acronym><acronymword>NCO</acronymword></acronym> homepage</indexterm></cindex>
<para>If you want to quickly see what the latest improvements in <acronym><acronymword>NCO</acronymword></acronym>
are (without downloading the entire source distribution), visit the
<acronym><acronymword>NCO</acronymword></acronym> homepage at 
<uref><urefurl>http://nco.sf.net</urefurl></uref>.
The <acronym><acronymword>HTML</acronymword></acronym> version of the <cite>User Guide</cite> is also available 
online through the World Wide Web at <acronym><acronymword>URL</acronymword></acronym>
<uref><urefurl>http://nco.sf.net/nco.html</urefurl></uref>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="20">netCDF</indexterm></cindex>
To build and use <acronym><acronymword>NCO</acronymword></acronym>, you must have netCDF installed.
The netCDF homepage is
<uref><urefurl>http://www.unidata.ucar.edu/packages/netcdf</urefurl></uref>.
</para>
<para>New <acronym><acronymword>NCO</acronymword></acronym> releases are announced on the netCDF list 
and on the <code>nco-announce</code> mailing list 
<uref><urefurl>http://lists.sf.net/mailman/listinfo/nco-announce</urefurl></uref>.
</para>
<ignore>
This tests incorporates an image using the @code{@@image} command.
@image{/data/zender/ps/odxc,6in,}
</ignore>

</section>
<node name="How-to-Use-This-guide" spaces=" "><nodename>How to Use This guide</nodename><nodenext spaces=" ">Compatability</nodenext><nodeprev spaces=" ">Availability</nodeprev><nodeup spaces=" ">Introduction</nodeup></node>
<section spaces=" "><sectiontitle>How to Use This Guide</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="21">contents</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="22">introduction</indexterm></cindex>
<para>Detailed instructions about
<uref><urefurl>http://nco.sf.net/#Source</urefurl><urefdesc spaces=" ">how to download the newest version</urefdesc></uref>, 
and <uref><urefurl>http://nco.sf.net/#bld</urefurl><urefdesc spaces=" ">how to complie source code</urefdesc></uref>,
as well as a <uref><urefurl>http://nco.sf.net/#FAQ</urefurl><urefdesc spaces=" "><acronym><acronymword>FAQ</acronymword></acronym></urefdesc></uref> and 
descriptions of <uref><urefurl>http://nco.sf.net/#bug</urefurl><urefdesc spaces=" ">Known Problems</urefdesc></uref> etc.
are on our homepage 
(<uref><urefurl>http://nco.sf.net/</urefurl></uref>).
</para>
<para>There are twelve operators in the current version (4.4.2).
The function of each is explained in <ref label="Operator-Reference-Manual"><xrefnodename>Operator Reference Manual</xrefnodename><xrefinfoname spaces=" ">Operator Reference Manual</xrefinfoname></ref>.
Many of the tasks that <acronym><acronymword>NCO</acronymword></acronym> can accomplish are described during
the explanation of common <acronym><acronymword>NCO</acronymword></acronym> Features (<pxref label="Common-features"><xrefnodename>Common features</xrefnodename></pxref>).
More specific use examples for each operator can be seen by visiting the
operator-specific examples in the <ref label="Operator-Reference-Manual"><xrefnodename>Operator Reference Manual</xrefnodename></ref>.
These can be found directly by prepending the operator name with the
<code>xmp_</code> tag, e.g., <uref><urefurl>http://nco.sf.net/nco.html#xmp_ncks</urefurl></uref>.
Also, users can type the operator name on the shell command line to 
see all the available options, or type, e.g., <samp>man ncks</samp> to see
a help man-page.
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> is a command-line language.
You can either use an operator after the prompt (e.g., <samp>$</samp> here),
like, 
</para><example endspaces=" ">
<pre xml:space="preserve">$ <command>operator</command> <option>[options]</option> <file>input</file> <file>[output]</file>
</pre></example>
<para>or write all commands lines into a shell script, as in
the <acronym><acronymword>CMIP5</acronymword></acronym> Example (<pxref label="CMIP5-Example"><xrefnodename>CMIP5 Example</xrefnodename></pxref>).
</para>
<para>If you are new to <acronym><acronymword>NCO</acronymword></acronym>, the Quick Start (<pxref label="Quick-Start"><xrefnodename>Quick Start</xrefnodename></pxref>)
shows simple examples about how to use <acronym><acronymword>NCO</acronymword></acronym> on different kinds
of data files.  
More detailed &textldquo;real-world&textrdquo; examples are in the
<ref label="CMIP5-Example"><xrefnodename>CMIP5 Example</xrefnodename><xrefinfoname spaces=" "><acronym><acronymword>CMIP5</acronymword></acronym> Example</xrefinfoname></ref>. 
The <ref label="General-Index"><xrefnodename>General Index</xrefnodename><xrefinfoname spaces=" ">Index</xrefinfoname></ref> is presents multiple keyword entries for
the same subject. 
If these resources do not help enough, please 
<pxref label="Help-Requests-and-Bug-Reports"><xrefnodename>Help Requests and Bug Reports</xrefnodename></pxref>.
</para>
</section>
<node name="Compatability" spaces=" "><nodename>Compatability</nodename><nodenext spaces=" ">Symbolic Links</nodenext><nodeprev spaces=" ">How to Use This guide</nodeprev><nodeup spaces=" ">Introduction</nodeup></node>
<section spaces=" "><sectiontitle>Operating systems compatible with <acronym><acronymword>NCO</acronymword></acronym></sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="23"><acronym><acronymword>OS</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="24"><acronym><acronymword>IBM</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="25"><acronym><acronymword>NEC</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="26"><acronym><acronymword>SGI</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="27"><acronym><acronymword>HP</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="28"><acronym><acronymword>DEC</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="29"><acronym><acronymword>PGI</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="30">Cray</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="31">Digital</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="32">Sun</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="33">Intel</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="34">Comeau</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="35">Compaq</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="36">Macintosh</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="37">Microsoft</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="38">Windows</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="39">PathScale</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="40">QLogic</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="41">compatability</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="42">portability</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="43">installation</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> has been successfully ported and tested and is known to
work on the following 32- and 64-bit platforms:  
<!-- c alphabetize by OS name -->
<acronym><acronymword>IBM AIX</acronymword></acronym> 4.x, 5.x,
FreeBSD 4.x, 
<acronym><acronymword>GNU</acronymword></acronym>/Linux 2.x, LinuxPPC, LinuxAlpha, LinuxARM, LinuxSparc64,
<acronym><acronymword>SGI IRIX</acronymword></acronym> 5.x and 6.x,
<w>MacOS X</w> 10.x, 
<acronym><acronymword>NEC</acronymword></acronym> Super-UX 10.x, 
<acronym><acronymword>DEC OSF</acronymword></acronym>, 
Sun SunOS 4.1.x, Solaris 2.x, 
<acronym><acronymword>Cray UNICOS</acronymword></acronym> 8.x&textndash;10.x,
and MS Windows95 and all later versions.
If you port the code to a new operating system, please send me a note 
and any patches you required.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="44"><acronym><acronymword>UNIX</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="45">Unidata</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="46">UDUnits</indexterm></cindex>
<para>The major prerequisite for installing <acronym><acronymword>NCO</acronymword></acronym> on a particular
platform is the successful, prior installation of the netCDF library
(and, as of 2003, the UDUnits library).
Unidata has shown a commitment to maintaining netCDF and UDUnits on all
popular <acronym><acronymword>UNIX</acronymword></acronym> platforms, and is moving towards full support for 
the Microsoft Windows operating system (<acronym><acronymword>OS</acronymword></acronym>).
Given this, the only difficulty in implementing <acronym><acronymword>NCO</acronymword></acronym> on a
particular platform is standardization of various <w>C</w>-language API
system calls. 
<acronym><acronymword>NCO</acronymword></acronym> code is tested for <acronym><acronymword>ANSI</acronymword></acronym> compliance by
compiling with <w>C99 compilers</w> including those from 
<cindex index="cp" spaces=" "><indexterm index="cp" number="47"><command>CC</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="48"><command>c++</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="49"><command>cc</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="50"><command>clang</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="51"><command>como</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="52"><command>cxx</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="53"><command>gcc</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="54"><command>g++</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="55"><command>icc</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="56"><command>MVS</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="57"><command>pgcc</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="58"><command>pgCC</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="59"><command>pathcc</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="60"><command>pathCC</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="61"><command>xlC</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="62"><command>xlc</command></indexterm></cindex>
<acronym><acronymword>GNU</acronymword></acronym> (<samp>gcc -std=c99 -pedantic -D_BSD_SOURCE -D_POSIX_SOURCE</samp> -Wall)
<footnote spaces="\n"><para>The <samp>_BSD_SOURCE</samp> token is required on some Linux platforms where 
<command>gcc</command> dislikes the network header files like
<file>netinet/in.h</file>).</para></footnote>,
Comeau Computing (<samp>como --c99</samp>),
Cray (<samp>cc</samp>),
<acronym><acronymword>HP</acronymword></acronym>/Compaq/<acronym><acronymword>DEC</acronymword></acronym> (<samp>cc</samp>),
<acronym><acronymword>IBM</acronymword></acronym> (<samp>xlc -c -qlanglvl=extc99</samp>),
Intel (<samp>icc -std=c99</samp>),
<cindex index="cp" spaces=" "><indexterm index="cp" number="63"><acronym><acronymword>LLVM</acronymword></acronym></indexterm></cindex>
<acronym><acronymword>LLVM</acronymword></acronym> (<samp>clang</samp>),
<acronym><acronymword>NEC</acronymword></acronym> (<samp>cc</samp>),
PathScale (QLogic) (<samp>pathcc -std=c99</samp>),
<acronym><acronymword>PGI</acronymword></acronym> (<samp>pgcc -c9x</samp>),
<acronym><acronymword>SGI</acronymword></acronym> (<samp>cc -c99</samp>),
and
Sun (<samp>cc</samp>).
<cindex index="cp" spaces=" "><indexterm index="cp" number="64">C++</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="65"><acronym><acronymword>ISO</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="66"><command>libnco</command></indexterm></cindex>
<acronym><acronymword>NCO</acronymword></acronym> (all commands and the <command>libnco</command> library) and
the C++ interface to netCDF (called <command>libnco_c++</command>) comply with
the <acronym><acronymword>ISO</acronymword></acronym> C++ standards as implemented by
Comeau Computing (<samp>como</samp>),
Cray (<samp>CC</samp>),
<acronym><acronymword>GNU</acronymword></acronym> (<samp>g++ -Wall</samp>),
<acronym><acronymword>HP</acronymword></acronym>/Compaq/<acronym><acronymword>DEC</acronymword></acronym> (<samp>cxx</samp>),
<acronym><acronymword>IBM</acronymword></acronym> (<samp>xlC</samp>),
Intel (<samp>icc</samp>),
Microsoft (<samp>MVS</samp>),
<acronym><acronymword>NEC</acronymword></acronym> (<samp>c++</samp>),
PathScale (Qlogic) (<samp>pathCC</samp>),
<acronym><acronymword>PGI</acronymword></acronym> (<samp>pgCC</samp>),
<acronym><acronymword>SGI</acronymword></acronym> (<samp>CC -LANG:std</samp>),
and
Sun (<samp>CC -LANG:std</samp>).
<cindex index="cp" spaces=" "><indexterm index="cp" number="67"><file>Makefile</file></indexterm></cindex>
See <file>nco/bld/Makefile</file> and <file>nco/src/nco_c++/Makefile.old</file> for
more details and exact settings. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="68"><acronym><acronymword>ANSI</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="69">C89</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="70"><code>printf</code></indexterm></cindex>
<para>Until recently (and not even yet), <acronym><acronymword>ANSI</acronymword></acronym>-compliant has meant
compliance with the 1989 <acronym><acronymword>ISO</acronymword></acronym> C-standard, usually called C89 (with
minor revisions made in 1994 and 1995).
C89 lacks variable-size arrays, restricted pointers, some useful
<code>printf</code> formats, and many mathematical special functions.
<cindex index="cp" spaces=" "><indexterm index="cp" number="71">C99</indexterm></cindex>
These are valuable features of C99, the 1999 <acronym><acronymword>ISO</acronymword></acronym> C-standard. 
<acronym><acronymword>NCO</acronymword></acronym> is C99-compliant where possible and C89-compliant where
necessary. 
Certain branches in the code are required to satisfy the native
<acronym><acronymword>SGI</acronymword></acronym> and SunOS <w>C compilers</w>, which are strictly <acronym><acronymword>ANSI</acronymword></acronym> 
C89 compliant, and cannot benefit from C99 features.
However, C99 features are fully supported by modern <acronym><acronymword>AIX</acronymword></acronym>,
<acronym><acronymword>GNU</acronymword></acronym>, Intel, <acronym><acronymword>NEC</acronymword></acronym>, Solaris, and <acronym><acronymword>UNICOS</acronymword></acronym>
compilers. 
<acronym><acronymword>NCO</acronymword></acronym> requires a C99-compliant compiler as of <acronym><acronymword>NCO</acronymword></acronym>
<w>version 2.9.8</w>, released in August, 2004.
</para>
<para>The most time-intensive portion of <acronym><acronymword>NCO</acronymword></acronym> execution is spent in
arithmetic operations, e.g., multiplication, averaging, subtraction.
These operations were performed in Fortran by default until August,
1999.  
This was a design decision based on the relative speed of Fortran-based
object code vs.&noeos; C-based object code in late 1994.
<w>C compiler</w> vectorization capabilities have dramatically improved 
since 1994.
We have accordingly replaced all Fortran subroutines with <w>C functions</w>.
This greatly simplifies the task of building <acronym><acronymword>NCO</acronymword></acronym> on nominally
unsupported platforms.  
<cindex index="cp" spaces=" "><indexterm index="cp" number="72">C language</indexterm></cindex>
As of August 1999, <acronym><acronymword>NCO</acronymword></acronym> built entirely <w>in C</w> by default.
This allowed <acronym><acronymword>NCO</acronymword></acronym> to compile on any machine with an
<acronym><acronymword>ANSI</acronymword></acronym> <w>C compiler</w>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="73">C99</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="74">C89</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="75"><code>restrict</code></indexterm></cindex>
In August 2004, the first C99 feature, the <code>restrict</code> type
qualifier, entered <acronym><acronymword>NCO</acronymword></acronym> in version 2.9.8. 
<w>C compilers</w> can obtain better performance with C99 restricted 
pointers since they inform the compiler when it may make Fortran-like
assumptions regarding pointer contents alteration.
Subsequently, <acronym><acronymword>NCO</acronymword></acronym> requires a C99 compiler to build correctly
<footnote><para><acronym><acronymword>NCO</acronymword></acronym> may still build with an 
<acronym><acronymword>ANSI</acronymword></acronym> or <acronym><acronymword>ISO</acronymword></acronym> C89 or C94/95-compliant compiler if the
<w>C pre-processor</w> undefines the <code>restrict</code> type qualifier, e.g.,
by invoking the compiler with <samp>-Drestrict=''</samp>.</para></footnote>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="76"><acronym><acronymword>GSL</acronymword></acronym></indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="1" mergedindex="cp">ncap2</indexterm></findex>
<para>In January 2009, <acronym><acronymword>NCO</acronymword></acronym> version 3.9.6 was the first to link
to the GNU Scientific Library (<acronym><acronymword>GSL</acronymword></acronym>).
<acronym><acronymword>GSL</acronymword></acronym> must be <w>version 1.4</w> or later. 
<acronym><acronymword>NCO</acronymword></acronym>, in particular <command>ncap2</command>, uses the <acronym><acronymword>GSL</acronymword></acronym>
special function library to evaluate geoscience-relevant mathematics
such as Bessel functions, Legendre polynomials, and incomplete gamma
functions (<pxref label="GSL-special-functions"><xrefnodename>GSL special functions</xrefnodename></pxref>).
</para> 
<cindex index="cp" spaces=" "><indexterm index="cp" number="77"><var>gamma</var></indexterm></cindex>
<para>In June 2005, <acronym><acronymword>NCO</acronymword></acronym> version 3.0.1 began to take advantage
of C99 mathematical special functions.
These include the standarized gamma function (called <code>tgamma()</code> 
for &textldquo;true gamma&textrdquo;). 
<cindex index="cp" spaces=" "><indexterm index="cp" number="78">automagic</indexterm></cindex>
<acronym><acronymword>NCO</acronymword></acronym> automagically takes advantage of some <acronym><acronymword>GNU</acronymword></acronym>
Compiler Collection (<acronym><acronymword>GCC</acronymword></acronym>) extensions to <w><acronym><acronymword>ANSI</acronymword></acronym> C</w>.
</para>
<para>As of July 2000 and <acronym><acronymword>NCO</acronymword></acronym> <w>version 1.2</w>, <acronym><acronymword>NCO</acronymword></acronym> no
longer performs arithmetic operations in Fortran.
We decided to sacrifice executable speed for code maintainability.
Since no objective statistics were ever performed to quantify 
the difference in speed between the Fortran and <w>C code</w>,
the performance penalty incurred by this decision is unknown.
Supporting Fortran involves maintaining two sets of routines for every
arithmetic operation. 
The <code>USE_FORTRAN_ARITHMETIC</code> flag is still retained in the
<file>Makefile</file>.
The file containing the Fortran code, <file>nco_fortran.F</file>, has been
deprecated but a volunteer (<w>Dr.&noeos; Frankenstein</w>?) could resurrect it.
If you would like to volunteer to maintain <file>nco_fortran.F</file> please 
contact me. 
</para>
<!-- c Following section is obsolete -->
<ignore>
It is still possible to request Fortran routines to perform arithmetic
operations, however.
@cindex preprocessor tokens
@cindex @code{USE_FORTRAN_ARITHMETIC}
This can be accomplished by defining the preprocessor token
@code{USE_FORTRAN_ARITHMETIC} and rebuilding @acronym{NCO}.
@cindex performance
As its name suggests, the @code{USE_FORTRAN_ARITHMETIC} token instructs
@acronym{NCO} to attempts to interface the @w{C routines} with Fortran
arithmetic. 
Although using Fortran calls instead @w{of C} reduces the portability and
and increases the maintenance of the @acronym{NCO} operators, it may
also increase the performance of the numeric operators.
Presumably this will depend on your machine type, the quality of @w{the C}
and Fortran compilers, and the size of the data files
@footnote{If you decide to test the efficiency of the averagers compiled
with @code{USE_FORTRAN_ARITHMETIC} versus the default @w{C averagers} I
would be most interested to hear the results.
Please E-mail me the results including the size of the datasets, the
platform, and the change in the wallclock time for execution.}.
</ignore>

<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Windows Operating System</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<html endspaces=" ">
&lt;a name=&quot;wnd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#wnd &textndash;&gt;
&lt;a name=&quot;windows&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#windows &textndash;&gt;
&lt;a name=&quot;qt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#qt &textndash;&gt;
</html>
<node name="Windows-Operating-System" spaces=" "><nodename>Windows Operating System</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">Compatability</nodeprev><nodeup spaces=" ">Compatability</nodeup></node>
<subsection spaces=" "><sectiontitle>Compiling <acronym><acronymword>NCO</acronymword></acronym> for Microsoft Windows <acronym><acronymword>OS</acronymword></acronym></sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="79">Windows</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="80">Microsoft</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="81">XP (Microsoft operating system)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="82">NT (Microsoft operating system)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="83">Vista (Microsoft operating system)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="84"><acronym><acronymword>MVS</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="85">Microsoft Visual Studio</indexterm></cindex>

<para><acronym><acronymword>NCO</acronymword></acronym> has been successfully ported and tested on most Microsoft  
Windows operating systems including: XP SP2/Vista/7.
Support is provided for compiling either native Windows executables,
using the Microsoft Visual Studio 2010 Compiler, or with Cygwin, the
UNIX-emulating compatibility layer with the GNU toolchain.
The switches necessary to accomplish both are included in the standard
distribution of <acronym><acronymword>NCO</acronymword></acronym>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="86">Qt</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="87">C99</indexterm></cindex>
<para>Using Microsoft Visual Studio (<acronym><acronymword>MVS</acronymword></acronym>), one must build
<acronym><acronymword>NCO</acronymword></acronym> with the C++ compiler since MVS does not support C99.
Qt, a convenient integrated development environment, was used to convert
the project files to <acronym><acronymword>MVS</acronymword></acronym> format.
The Qt files themselves are distributed in the <file>nco/qt</file> directory.
</para>
<para>Using the freely available Cygwin (formerly gnu-win32) development
environment  
<footnote><para>The Cygwin package is available from&linebreak;
<code>http://sourceware.redhat.com/cygwin</code>&linebreak;
<cindex index="cp" spaces=" "><indexterm index="cp" number="88"><code>gcc</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="89"><code>g++</code></indexterm></cindex>
Currently, <w>Cygwin 20.x</w> comes with the <acronym><acronymword>GNU</acronymword></acronym> C/C++
compilers (<command>gcc</command>, <command>g++</command>.
These <acronym><acronymword>GNU</acronymword></acronym> compilers may be used to build the netCDF
distribution itself.</para></footnote>, the compilation process is very similar to
installing <acronym><acronymword>NCO</acronymword></acronym> on a <acronym><acronymword>UNIX</acronymword></acronym> system.  
<cindex index="cp" spaces=" "><indexterm index="cp" number="90">preprocessor tokens</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="91">Cygwin</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="92"><code>gnu-win32</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="93"><code>WIN32</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="94"><file>GNUmakefile</file></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="95"><file>Makefile</file></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="96"><code>f90</code></indexterm></cindex>
Set the <code>PVM_ARCH</code> preprocessor token to <code>WIN32</code>.  
Note that defining <code>WIN32</code> has the side effect of disabling
Internet features of <acronym><acronymword>NCO</acronymword></acronym> (see below). 
<acronym><acronymword>NCO</acronymword></acronym> should now build like it does on <acronym><acronymword>UNIX</acronymword></acronym>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="97"><acronym><acronymword>UNIX</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="98"><code>getuid</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="99"><code>gethostname</code></indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="2" mergedindex="cp"><file>&lt;arpa/nameser.h&gt;</file></indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="3" mergedindex="cp"><file>&lt;resolv.h&gt;</file></indexterm></findex>
<para>The least portable section of the code is the use of standard
<acronym><acronymword>UNIX</acronymword></acronym> and Internet protocols (e.g., <code>ftp</code>, <code>rcp</code>,
<code>scp</code>, <code>sftp</code>, <code>getuid</code>, <code>gethostname</code>, and header
files <file>&lt;arpa/nameser.h&gt;</file> and 
<file>&lt;resolv.h&gt;</file>). 
<cindex index="cp" spaces=" "><indexterm index="cp" number="100"><code>ftp</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="101"><code>sftp</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="102"><code>rcp</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="103"><code>scp</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="104"><acronym><acronymword>SSH</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="105">remote files</indexterm></cindex>
Fortunately, these <acronym><acronymword>UNIX</acronymword></acronym>-y calls are only invoked by the single  
<acronym><acronymword>NCO</acronymword></acronym> subroutine which is responsible for retrieving files
stored on remote systems (<pxref label="Remote-storage"><xrefnodename>Remote storage</xrefnodename></pxref>).
In order to support <acronym><acronymword>NCO</acronymword></acronym> on the Microsoft Windows platforms,
this single feature was disabled (on Windows <acronym><acronymword>OS</acronymword></acronym> only).
This was required by <w>Cygwin 18.x</w>&textmdash;newer versions of Cygwin may
support these protocols (let me know if this is the case).
The <acronym><acronymword>NCO</acronymword></acronym> operators should behave identically on Windows and
<acronym><acronymword>UNIX</acronymword></acronym> platforms in all other respects.
</para>
<html endspaces=" ">
&lt;a name=&quot;sym&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#sym &textndash;&gt;
</html>
</subsection>
</section>
<node name="Symbolic-Links" spaces=" "><nodename>Symbolic Links</nodename><nodenext spaces=" ">Libraries</nodenext><nodeprev spaces=" ">Compatability</nodeprev><nodeup spaces=" ">Introduction</nodeup></node>
<section spaces=" "><sectiontitle>Symbolic Links</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="106">symbolic links</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> relies on a common set of underlying algorithms.
To minimize duplication of source code, multiple operators sometimes
share the same underlying source.
This is accomplished by symbolic links from a single underlying
executable program to one or more invoked executable names.
For example, <command>nces</command> and <command>ncrcat</command> are symbolically linked  
to the <command>ncra</command> executable.
The <command>ncra</command> executable behaves slightly differently based on its
invocation name (i.e., <samp>argv[0]</samp>), which can be 
<command>nces</command>, <command>ncra</command>, or <command>ncrcat</command>.
Logically, these are three different operators that happen to share 
the same executable.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="107">Cygwin</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="108">synonym</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="109">pseudonym</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="110"><code>--pseudonym</code></indexterm></cindex>
<para>For historical reasons, and to be more user friendly, multiple synonyms 
(or pseudonyms) may refer to the same operator invoked with different
switches. 
For example, <command>ncdiff</command> is the same as <command>ncbo</command> and
<command>ncpack</command> is the same as <command>ncpdq</command>.
We implement the symbolic links and synonyms by the executing the
following <acronym><acronymword>UNIX</acronymword></acronym> commands in the directory where the
<acronym><acronymword>NCO</acronymword></acronym> executables are installed.
</para><example endspaces=" ">
<pre xml:space="preserve">ln -s -f ncbo ncdiff    # ncbo --op_typ='+'
ln -s -f ncra ncecat    # ncra --pseudonym='ncecat'
ln -s -f ncra ncrcat    # ncra --pseudonym='ncrcat'
ln -s -f ncbo ncadd     # ncbo --op_typ='+'
ln -s -f ncbo ncsubtract # ncbo --op_typ='-'
ln -s -f ncbo ncmultiply # ncbo --op_typ='*'
ln -s -f ncbo ncdivide   # ncbo --op_typ='/'
ln -s -f ncpdq ncpack    # ncpdq
ln -s -f ncpdq ncunpack  # ncpdq --unpack
# NB: Cygwin executable (and link) names have an '.exe' suffix, e.g.,
ln -s -f ncbo.exe ncdiff.exe
...
</pre></example>
<para>The imputed command called by the link is given after the comment.
As can be seen, some these links impute the passing of a command line
argument to further modify the behavior of the underlying executable.
For example, <command>ncdivide</command> is a pseudonym for
<command>ncbo --op_typ='/'</command>.
</para>
<html endspaces=" ">
&lt;a name=&quot;lbr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#lbr &textndash;&gt;
</html>
</section>
<node name="Libraries" spaces=" "><nodename>Libraries</nodename><nodenext spaces=" ">netCDF2/3/4 and HDF4/5 Support</nodenext><nodeprev spaces=" ">Symbolic Links</nodeprev><nodeup spaces=" ">Introduction</nodeup></node>
<section spaces=" "><sectiontitle>Libraries</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="111">libraries</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="112"><code>LD_LIBRARY_PATH</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="113">dynamic linking</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="114">static linking</indexterm></cindex>
<para>Like all executables, the <acronym><acronymword>NCO</acronymword></acronym> operators can be built using dynamic
linking. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="115">performance</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="116">operator speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="117">speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="118">execution time</indexterm></cindex>
This reduces the size of the executable and can result in significant
performance enhancements on multiuser systems.
Unfortunately, if your library search path (usually the
<env>LD_LIBRARY_PATH</env> environment variable) is not set correctly, or if
the system libraries have been moved, renamed, or deleted since
<acronym><acronymword>NCO</acronymword></acronym> was installed, it is possible <acronym><acronymword>NCO</acronymword></acronym> operators
will fail with a message that they cannot find a dynamically loaded (aka
<dfn>shared object</dfn> or <samp>.so</samp>) library. 
This will produce a distinctive error message, such as
<samp>ld.so.1:&hyphenbreak; /usr/local/bin/nces:&hyphenbreak; fatal:&hyphenbreak; libsunmath.&hyphenbreak;so.1:&hyphenbreak; can't
open&hyphenbreak; file:&hyphenbreak; errno&hyphenbreak;=2</samp>.   
If you received an error message like this, ask your system 
administrator to diagnose whether the library is truly missing
<footnote><para>The <command>ldd</command> command, if it is available on your system,
will tell you where the executable is looking for each dynamically
loaded library. Use, e.g., <code>ldd `which nces`</code>.</para></footnote>, or whether you
simply need to alter your library search path.
As a final remedy, you may re-compile and install <acronym><acronymword>NCO</acronymword></acronym> with all
operators statically linked.  
</para>
</section>
<node name="netCDF2_002f3_002f4-and-HDF4_002f5-Support" spaces=" "><nodename>netCDF2/3/4 and HDF4/5 Support</nodename><nodenext spaces=" ">Help Requests and Bug Reports</nodenext><nodeprev spaces=" ">Libraries</nodeprev><nodeup spaces=" ">Introduction</nodeup></node>
<section spaces=" "><sectiontitle>netCDF2/3/4 and HDF4/5 Support</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="119">netCDF2</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="120">netCDF3</indexterm></cindex>
<para>netCDF <w>version 2</w> was released in 1993.
<acronym><acronymword>NCO</acronymword></acronym> (specifically <command>ncks</command>) began soon after this in 1994.  
<w>netCDF 3.0</w> was released in 1996, and we were not exactly eager to 
convert all code to the newer, less tested netCDF implementation.
One <w>netCDF3</w> interface call (<code>nc_inq_libvers</code>) was added to 
<acronym><acronymword>NCO</acronymword></acronym> in January, 1998, to aid in maintainance and debugging. 
In March, 2001, the final <acronym><acronymword>NCO</acronymword></acronym> conversion to <w>netCDF3</w> 
was completed (coincidentally on the same day <w>netCDF 3.5</w> was
released). 
<acronym><acronymword>NCO</acronymword></acronym> <w>versions 2.0</w> and higher are built with the
<code>-DNO_NETCDF_2</code> flag to ensure no <w>netCDF2</w> interface calls   
are used.
<cindex index="cp" spaces=" "><indexterm index="cp" number="121"><code>NO_NETCDF_2</code></indexterm></cindex>
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="122"><acronym><acronymword>HDF</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="123">Hierarchical Data Format</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="124">Mike Folk</indexterm></cindex>
<para>However, the ability to compile <acronym><acronymword>NCO</acronymword></acronym> with only <w>netCDF2</w>
calls is worth maintaining because <acronym><acronymword>HDF</acronymword></acronym> <w>version 4</w>, 
aka <acronym><acronymword>HDF4</acronymword></acronym> or simply <acronym><acronymword>HDF</acronymword></acronym>, 
<footnote><para>The Hierarchical Data Format, or <acronym><acronymword>HDF</acronymword></acronym>, is another
self-describing data format similar to, but more elaborate than,
netCDF. 
<acronym><acronymword>HDF</acronymword></acronym> comes in two flavors, <acronym><acronymword>HDF4</acronymword></acronym> and <acronym><acronymword>HDF5</acronymword></acronym>. 
Often people use the shorthand <acronym><acronymword>HDF</acronymword></acronym> to refer to the older
format <acronym><acronymword>HDF4</acronymword></acronym>.
People almost always use <acronym><acronymword>HDF5</acronymword></acronym> to refer to <acronym><acronymword>HDF5</acronymword></acronym>.</para></footnote> 
(available from <uref><urefurl>http://hdfgroup.org</urefurl><urefdesc spaces=" ">HDF</urefdesc></uref>)
supports only the <w>netCDF2</w> library calls
(see <uref><urefurl>http://hdfgroup.org/UG41r3_html/SDS_SD.fm12.html#47784</urefurl></uref>).
There are two versions of <acronym><acronymword>HDF</acronymword></acronym>.
Currently <acronym><acronymword>HDF</acronymword></acronym> <w>version 4.x</w> supports the full <w>netCDF2</w>
<acronym><acronymword>API</acronymword></acronym> and thus <acronym><acronymword>NCO</acronymword></acronym> <w>version 1.2.x</w>. 
If <acronym><acronymword>NCO</acronymword></acronym> <w>version 1.2.x</w> (or earlier) is built with only
<w>netCDF2</w> calls then all <acronym><acronymword>NCO</acronymword></acronym> operators should work with 
<acronym><acronymword>HDF4</acronymword></acronym> files as well as netCDF files
<footnote><para>One must link the <acronym><acronymword>NCO</acronymword></acronym> code to the <acronym><acronymword>HDF4</acronymword></acronym>
<acronym><acronymword>MFHDF</acronymword></acronym> library instead of the usual netCDF library. 
Apparently <samp>MF</samp> stands for Multi-file not for Mike Folk.
In any case, until about 2007 the <acronym><acronymword>MFHDF</acronymword></acronym> library only supported 
<w>netCDF2</w> calls. 
Most people will never again install <acronym><acronymword>NCO</acronymword></acronym> 1.2.x and so will
never use <acronym><acronymword>NCO</acronymword></acronym> to write <acronym><acronymword>HDF4</acronymword></acronym> files.
It is simply too much trouble.</para></footnote>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="125"><code>NETCDF2_ONLY</code></indexterm></cindex>
The preprocessor token <code>NETCDF2_ONLY</code> exists
in <acronym><acronymword>NCO</acronymword></acronym> <w>version 1.2.x</w> to eliminate all <w>netCDF3</w>
calls.  
Only versions of <acronym><acronymword>NCO</acronymword></acronym> numbered 1.2.x and earlier have this
capability. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="126">Unidata</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="127"><acronym><acronymword>NCSA</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="128">netCDF4</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="129"><acronym><acronymword>HDF5</acronymword></acronym></indexterm></cindex>
<para><acronym><acronymword>HDF</acronymword></acronym> <w>version 5</w> became available in 1999, but did not
support netCDF (or, for that matter, Fortran) as of December 1999.
By early 2001, <acronym><acronymword>HDF5</acronymword></acronym> did support Fortran90.
Thanks to an <acronym><acronymword>NSF</acronymword></acronym>-funded &textldquo;harmonization&textrdquo; partnership,
<acronym><acronymword>HDF</acronymword></acronym> began to fully support the <w>netCDF3</w> read interface
(which is employed by <w><acronym><acronymword>NCO</acronymword></acronym> 2.x</w> and later). 
In 2004, Unidata and <acronym><acronymword>THG</acronymword></acronym> began a project to implement
the <acronym><acronymword>HDF5</acronymword></acronym> features necessary to support the netCDF API.
<acronym><acronymword>NCO</acronymword></acronym> version 3.0.3 added support for reading/writing
netCDF4-formatted <acronym><acronymword>HDF5</acronymword></acronym> files in October, 2005.
See <ref label="File-Formats-and-Conversion"><xrefnodename>File Formats and Conversion</xrefnodename></ref> for more details.
</para>
<para>HDF support for netCDF was completed with HDF5 version 
<w>version 1.8</w> in 2007. 
The netCDF front-end that uses this <acronym><acronymword>HDF5</acronymword></acronym> back-end 
was completed and released soon after as netCDF <w>version 4</w>.
Download it from the
<uref><urefurl>http://my.unidata.ucar.edu/content/software/netcdf/netcdf-4</urefurl><urefdesc>netCDF4</urefdesc></uref>
website. 
</para>
<html endspaces=" ">
&lt;a name=&quot;nco4&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#nco4 &textndash;&gt;
</html>
<para><acronym><acronymword>NCO</acronymword></acronym> version 3.9.0, released in May, 2007, added support for
all netCDF4 atomic data types except <code>NC_STRING</code>.
Support for <code>NC_STRING</code>, including ragged arrays of strings,
was finally added in version 3.9.9, released in June, 2009.
Support for additional netCDF4 features has been incremental.
We add one netCDF4 feature at a time.
You must build <acronym><acronymword>NCO</acronymword></acronym> with netCDF4 to obtain this support.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="130"><code>NC_UBYTE</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="131"><code>NC_USHORT</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="132"><code>NC_UINT</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="133"><code>NC_INT64</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="134"><code>NC_UINT64</code></indexterm></cindex>
<para>The main netCDF4 features that NCO currently supports are the new 
atomic data types, Lempel-Ziv compression (deflation), and chunking. 
The new atomic data types are <code>NC_UBYTE</code>, <code>NC_USHORT</code>, 
<code>NC_UINT</code>, <code>NC_INT64</code>, and <code>NC_UINT64</code>.
Eight-byte integer support is an especially useful improvement from
netCDF3. 
All <acronym><acronymword>NCO</acronymword></acronym> operators support these types, e.g., <command>ncks</command>
copies and prints them, <command>ncra</command> averages them, and
<command>ncap2</command> processes algebraic scripts with them.
<command>ncks</command> prints compression information, if any, to screen.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="135">deflation</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> version 3.9.1 (June, 2007) added support for netCDF4 
Lempel-Ziv deflation.
Lempel-Ziv deflation is a lossless compression technique.
See <ref label="Deflation"><xrefnodename>Deflation</xrefnodename></ref> for more details.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="136">chunking</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> version 3.9.9 (June, 2009) added support for netCDF4
chunking in <command>ncks</command> and <command>ncecat</command>.
<acronym><acronymword>NCO</acronymword></acronym> version 4.0.4 (September, 2010) completed support for
netCDF4 chunking in the remaining operators.
See <ref label="Chunking"><xrefnodename>Chunking</xrefnodename></ref> for more details.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="137">groups</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> version 4.2.2 (October, 2012) added support for netCDF4
groups in <command>ncks</command> and <command>ncecat</command>.
Group support for these operators was complete (e.g., regular
expressions to select groups and Group Path Editing) as of 
<acronym><acronymword>NCO</acronymword></acronym> version 4.2.6 (March, 2013).
See <ref label="Group-Path-Editing"><xrefnodename>Group Path Editing</xrefnodename></ref> for more details.
Group support for all other operators was finished in the
<acronym><acronymword>NCO</acronymword></acronym> version 4.3.x series completed in December, 2013.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="138">broadcasting groups</indexterm></cindex>
<para>Support for netCDF4 in the first arithmetic operator, <command>ncbo</command>,
was introduced in <acronym><acronymword>NCO</acronymword></acronym> version 4.3.0 (March, 2013).
<acronym><acronymword>NCO</acronymword></acronym> version 4.3.1 (May, 2013) completed this support and
introduced the first example of automatic group broadcasting.
See <ref label="ncbo-netCDF-Binary-Operator"><xrefnodename>ncbo netCDF Binary Operator</xrefnodename></ref> for more details.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="139"><acronym><acronymword>HDF5</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="140"><code>-4</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="141"><code>-3</code></indexterm></cindex>
<para>netCDF4-enabled <acronym><acronymword>NCO</acronymword></acronym> handles netCDF3 files without change.
In addition, it automagically handles netCDF4 (<acronym><acronymword>HDF5</acronymword></acronym>) files:
If you feed <acronym><acronymword>NCO</acronymword></acronym> netCDF3 files, it produces netCDF3 output.
If you feed <acronym><acronymword>NCO</acronymword></acronym> netCDF4 files, it produces netCDF4 output.
Use the handy-dandy <samp>-4</samp> switch to request netCDF4 output from 
netCDF3 input, i.e., to convert netCDF3 to netCDF4.
See <ref label="File-Formats-and-Conversion"><xrefnodename>File Formats and Conversion</xrefnodename></ref> for more details.
</para>
<html endspaces=" ">
&lt;a name=&quot;hdf4&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hdf4 &textndash;&gt;
&lt;a name=&quot;HDF4&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#HDF4 &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="142"><acronym><acronymword>HDF4</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="143"><samp>--hdf4</samp></indexterm></cindex>
<para>When linked to a netCDF library that was built with <acronym><acronymword>HDF4</acronymword></acronym>
support
<footnote><para>The procedure for doing this is documented at
<uref><urefurl>http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html</urefurl></uref>.</para></footnote>,
<acronym><acronymword>NCO</acronymword></acronym> automatically supports reading <acronym><acronymword>HDF4</acronymword></acronym> 
files and writing them as netCDF3/netCDF4/<acronym><acronymword>HDF5</acronymword></acronym> files.
<acronym><acronymword>NCO</acronymword></acronym> can only write through the netCDF <acronym><acronymword>API</acronymword></acronym>, which
can only write netCDF3/netCDF4/<acronym><acronymword>HDF5</acronymword></acronym> files. 
So <acronym><acronymword>NCO</acronymword></acronym> can <emph>read</emph> <acronym><acronymword>HDF4</acronymword></acronym> files, perform
manipulations and calculations, and then <emph>write</emph> the results in
netCDF format. 
</para>
<para>Full support for these features is forthcoming, yet support as of
December, 2013 is quite functional.
For best results install <acronym><acronymword>NCO</acronymword></acronym> versions 4.4.0 or later on top of
netCDF versions 4.3.1 or later. 
Getting to this point has been an iterative effort where Unidata
improved netCDF library capabilities in response to our requests.
<acronym><acronymword>NCO</acronymword></acronym> versions 4.3.6 and earlier do not explicitly support
<acronym><acronymword>HDF4</acronymword></acronym>, yet should work with <acronym><acronymword>HDF4</acronymword></acronym> if compiled with 
a version of netCDF (4.3.2 or later?) that does not unexpectedly die
when probing <acronym><acronymword>HDF4</acronymword></acronym> files with standard netCDF calls.
<acronym><acronymword>NCO</acronymword></acronym> versions 4.3.7&textndash;4.3.9 (October&textndash;December, 2013)
use a special flag to workaround netCDF <acronym><acronymword>HDF4</acronymword></acronym> issues.
The user must tell these versions of <acronym><acronymword>NCO</acronymword></acronym> that an input file is
<acronym><acronymword>HDF4</acronymword></acronym> format by using the <samp>--hdf4</samp> switch. 
</para>
<para>When compiled with netCDF version 4.3.1 (20140116) or later, 
<acronym><acronymword>NCO</acronymword></acronym> versions 4.4.0 (January, 2014) and later more gracefully 
handle <acronym><acronymword>HDF4</acronymword></acronym> files.
In particular, the <samp>--hdf4</samp> switch is obsolete. 
Current versions of <acronym><acronymword>NCO</acronymword></acronym> use netCDF to determine automatically
whether the underlying file is <acronym><acronymword>HDF4</acronymword></acronym>, and then take appropriate
precautions to avoid calls not yet supported by the netCDF4 subset
<acronym><acronymword>HDF4</acronymword></acronym>. 
The <samp>--hdf4</samp> switch is supported (for backwards compatibility) yet
redundant (i.e., does no harm) with current versions of <acronym><acronymword>NCO</acronymword></acronym>
and netCDF. 
</para>
<para>Converting <acronym><acronymword>HDF4</acronymword></acronym> files to netCDF:
Since <acronym><acronymword>NCO</acronymword></acronym> reads <acronym><acronymword>HDF4</acronymword></acronym> files natively, it is now easy  
to convert <acronym><acronymword>HDF4</acronymword></acronym> files to netCDF files directly, e.g.,
</para><example endspaces=" ">
<pre xml:space="preserve">ncks        fl.hdf fl.nc # Convert HDF4-&gt;netCDF4 (NCO 4.4.0+, netCDF 4.3.1+)
ncks --hdf4 fl.hdf fl.nc # Convert HDF4-&gt;netCDF4 (NCO 4.3.7-4.3.9)
</pre></example>
<para>The most efficient and accurate way to convert <acronym><acronymword>HDF4</acronymword></acronym> data to
netCDF format is to convert to netCDF4 using <acronym><acronymword>NCO</acronymword></acronym> as above.
It suffices when the data will only be processed by <acronym><acronymword>NCO</acronymword></acronym>, or
other netCDF4-aware tools. 
However, many tools are not fully netCDF4-aware, and so conversion to
netCDF3 may be desirable.
</para>
<para>Obtaining a netCDF3 file from an <acronym><acronymword>HDF4</acronymword></acronym> is now easy, even though
the <acronym><acronymword>HDF4</acronymword></acronym> file may contain netCDF4 atomic types (e.g., unsigned
bytes, 64-bit integers): 
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -3 fl.hdf fl.nc      # HDF4-&gt;netCDF3 (NCO 4.4.0+, netCDF 4.3.1+)
ncks -7 -L 1 fl.hdf fl.nc # HDF4-&gt;netCDF4 (NCO 4.4.0+, netCDF 4.3.1+)
ncks --hdf4 -3 fl.hdf fl.nc # HDF4-&gt;netCDF3 (netCDF 4.3.0-)
ncks --hdf4 -7 fl.hdf fl.nc # HDF4-&gt;netCDF4 classic (netCDF 4.3.0-)
</pre></example>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.4.0 (January, 2014), these commands work
with netCDF4 atomic types
<footnote spaces="\n"><para>Prior to <acronym><acronymword>NCO</acronymword></acronym> version 4.4.0 (January, 2014), we recommend the
<command>ncl_convert2nc</command> tool to convert <acronym><acronymword>HDF</acronymword></acronym> to netCDF3 when
both these are true: <w>1. You</w> must have netCDF3 and <w>2. the</w>
<acronym><acronymword>HDF</acronymword></acronym> file contains netCDF4 atomic types. 
More recent versions of <acronym><acronymword>NCO</acronymword></acronym> handle this problem fine, so we
no longer recommend <command>ncl_convert2nc</command> because <command>ncks</command>
is faster and more space-efficient.
Both automatically convert netCDF4 types to netCDF3 types, yet
<command>ncl_convert2nc</command> cannot produce full netCDF4 files.
In contrast, <command>ncks</command> will happily convert <acronym><acronymword>HDF</acronymword></acronym> straight
to netCDF4 files with netCDF4 types. 
Hence <command>ncks</command> can and does preserve the variable types.
Unsigned bytes stay unsigned bytes. 
64-bit integers stay 64-bit integers. 
Strings stay strings. 
Hence, <command>ncks</command> conversions often result in smaller files than
<command>ncl_convert2nc</command> conversions.</para></footnote>.  
Many <acronym><acronymword>HDF4</acronymword></acronym> producers (<acronym><acronymword>NASA</acronymword></acronym>!) love to use netCDF4
types, e.g., unsigned bytes. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="144"><acronym><acronymword>RPM</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="145">Debian</indexterm></cindex>
<para>As of 2012, netCDF4 is relatively stable software.
Problems with netCDF4 and <acronym><acronymword>HDF</acronymword></acronym> libraries have mainly been fixed.
Binary <acronym><acronymword>NCO</acronymword></acronym> distributions shipped as <acronym><acronymword>RPM</acronymword></acronym>s and as debs
have used the netCDF4 library since 2010 and 2011, respectively.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="146"><code>NETCDF4_ROOT</code></indexterm></cindex>
<para>One must often build <acronym><acronymword>NCO</acronymword></acronym> from source to obtain netCDF4
support. 
Typically, one specifies the root of the netCDF4
installation directory. Do this with the <code>NETCDF4_ROOT</code> variable.
Then use your preferred <acronym><acronymword>NCO</acronymword></acronym> build mechanism, e.g.,
</para><example endspaces=" ">
<pre xml:space="preserve">export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location
cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
cd ~/nco/bld;./make NETCDF4=Y allinone # Old Makefile mechanism
</pre></example>

<para>We carefully track the netCDF4 releases, and keep the netCDF4 atomic
type support and other features working.
Our long term goal is to utilize more of the extensive new netCDF4
feature set. The next major netCDF4 feature we are likely to utilize
is parallel I/O. We will enable this in the <acronym><acronymword>MPI</acronymword></acronym> netCDF
operators. 
</para>
<html endspaces=" ">
&lt;a name=&quot;help&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#help &textndash;&gt;
&lt;a name=&quot;hlp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hlp &textndash;&gt;
&lt;a name=&quot;bug&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#bug &textndash;&gt;
</html>
</section>
<node name="Help-Requests-and-Bug-Reports" spaces=" "><nodename>Help Requests and Bug Reports</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">netCDF2/3/4 and HDF4/5 Support</nodeprev><nodeup spaces=" ">Introduction</nodeup></node>
<section spaces=" "><sectiontitle>Help Requests and Bug Reports</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="147">reporting bugs</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="148">bugs, reporting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="149">core dump</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="150">help</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="151">features, requesting</indexterm></cindex>
<para>We generally receive three categories of mail from users: help requests,
bug reports, and feature requests.
Notes saying the equivalent of &quot;Hey, <acronym><acronymword>NCO</acronymword></acronym> continues to work
great and it saves me more time everyday than it took to write this
note&quot; are a distant fourth.
</para>
<para>There is a different protocol for each type of request.
The preferred etiquette for all communications is via <acronym><acronymword>NCO</acronymword></acronym>
Project Forums. 
Do not contact project members via personal e-mail unless your request
comes with money or you have damaging information about our personal
lives.
<emph>Please use the Forums</emph>&textmdash;they preserve a record of the questions
and answers so that others can learn from our exchange.
Also, since <acronym><acronymword>NCO</acronymword></acronym> is government-funded, this record helps us
provide program officers with information they need to evaluate our
project. 
</para>
<para>Before posting to the <acronym><acronymword>NCO</acronymword></acronym> forums described below, you might
first <uref><urefurl>https://sf.net/account/register.php</urefurl><urefdesc spaces=" ">register</urefdesc></uref>
your name and email address with SourceForge.net or else all of your
postings will be attributed to &quot;nobody&quot;.
Once registered you may choose to &quot;monitor&quot; any forum and to receive
(or not) email when there are any postings including responses to your
questions.
We usually reply to the forum message, not to the original poster.
</para>
<para>If you want us to include a new feature in <acronym><acronymword>NCO</acronymword></acronym>, check first to 
see if that feature is already on the <uref><urefurl>file:./TODO</urefurl><urefdesc>TODO</urefdesc></uref> list.
If it is, why not implement that feature yourself and send us the patch?
If the feature is not yet on the list, then send a note to the
<uref><urefurl>http://sf.net/projects/nco/forums/forum/9829</urefurl><urefdesc spaces=" ">NCO Discussion forum</urefdesc></uref>.
</para>
<para>Read the manual before reporting a bug or posting a help request.
Sending questions whose answers are not in the manual is the best
way to motivate us to write more documentation.  
We would also like to accentuate the contrapositive of this statement.  
If you think you have found a real bug <emph>the most helpful thing you 
can do is simplify the problem to a manageable size and then report it</emph>.
The first thing to do is to make sure you are running the latest
publicly released version of <acronym><acronymword>NCO</acronymword></acronym>.  
</para>
<para>Once you have read the manual, if you are still unable to get
<acronym><acronymword>NCO</acronymword></acronym> to perform a documented function, submit a help request.
Follow the same procedure as described below for reporting bugs
(after all, it might be a bug).
<cindex index="cp" spaces=" "><indexterm index="cp" number="152">debugging</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="153"><code>-r</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="154"><code>-D</code></indexterm></cindex>
That is, describe what you are trying to do, and include the complete
commands (run with <samp>-D 5</samp>), error messages, and version of
<acronym><acronymword>NCO</acronymword></acronym> (with <samp>-r</samp>).  
Post your help request to the 
<uref><urefurl>http://sf.net/projects/nco/forums/forum/9830</urefurl><urefdesc spaces=" ">NCO Help forum</urefdesc></uref>.
</para>
<para>If you think you used the right command when <acronym><acronymword>NCO</acronymword></acronym> misbehaves,
then you might have found a bug.  
Incorrect numerical answers are the highest priority.
We usually fix those within one or two days.
Core dumps and sementation violations receive lower priority.
They are always fixed, eventually. 
</para>
<para>How do you simplify a problem that reveal a bug?
Cut out extraneous variables, dimensions, and metadata from the
offending files and re-run the command until it no longer breaks.  
Then back up one step and report the problem.
Usually the file(s) will be very small, i.e., one variable with one or
two small dimensions ought to suffice.
<html endspaces=" ">
&lt;a name=&quot;dbg&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dbg &textndash;&gt;
&lt;a name=&quot;-D&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-D &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="155"><code>-r</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="156"><code>--revision</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="157"><code>--version</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="158"><code>--vrs</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="159"><code>-D <var>debug-level</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="160"><code>--debug-level <var>debug-level</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="161"><code>--dbg_lvl <var>debug-level</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="162"><var>debug-level</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="163"><var>dbg_lvl</var></indexterm></cindex>
Run the operator with <samp>-r</samp> and then run the command with 
<samp>-D 5</samp> to increase the verbosity of the debugging output.
It is very important that your report contain the exact error messages 
and compile-time environment.
Include a copy of your sample input file, or place one on a 
publically accessible location, of the file(s).
Post the full bug report to the 
<uref><urefurl>http://sf.net/bugs/?group_id=3331</urefurl><urefdesc spaces=" ">NCO Project buglist</urefdesc></uref>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="164">installation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="165"><command>autoconf</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="166"><file>nco.configure.$&lbrace;GNU_TRP&rbrace;.foo</file></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="167"><file>nco.config.log.$&lbrace;GNU_TRP&rbrace;.foo</file></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="168"><file>nco.make.$&lbrace;GNU_TRP&rbrace;.foo</file></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="169"><file>config.guess</file></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="170"><file>configure.eg</file></indexterm></cindex>
<para>Build failures count as bugs.
Our limited machine access means we cannot fix all build failures.
The information we need to diagnose, and often fix, build failures
are the three files output by <acronym><acronymword>GNU</acronymword></acronym> build tools,  
<file>nco.config.log.$&lbrace;GNU_TRP&rbrace;.foo</file>,
<file>nco.configure.$&lbrace;GNU_TRP&rbrace;.foo</file>, 
and <file>nco.make.$&lbrace;GNU_TRP&rbrace;.foo</file>.
The file <file>configure.eg</file> shows how to produce these files.
Here <code>$&lbrace;GNU_TRP&rbrace;</code> is the &quot;<acronym><acronymword>GNU</acronymword></acronym> architecture triplet&quot;,
the <var>chip-vendor-OS</var> string returned by <file>config.guess</file>.
Please send us your improvements to the examples supplied in
<file>configure.eg</file>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="171">regressions archive</indexterm></cindex>
The regressions archive at <url><urefurl>http://dust.ess.uci.edu/nco/rgr</urefurl></url>
contains the build output from our standard test systems.
You may find you can solve the build problem yourself by examining the
differences between these files and your own.
</para>
<html endspaces=" ">
&lt;a name=&quot;str&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#str &textndash;&gt;
</html>
</section>
</chapter>
<node name="Strategies" spaces=" "><nodename>Strategies</nodename><nodenext spaces=" ">Common features</nodenext><nodeprev spaces=" ">Introduction</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<chapter spaces=" "><sectiontitle>Operator Strategies</sectiontitle>

<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Philosophy</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Climate Model Paradigm</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Temporary Output Files</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Appending Variables</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Simple Arithmetic and Interpolation</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Statistics vs. Concatenation</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Large Numbers of Files</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Large Datasets</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Memory Requirements</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Performance</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<html endspaces=" ">
&lt;a name=&quot;phl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#phl &textndash;&gt;
</html>
<node name="Philosophy" spaces=" "><nodename>Philosophy</nodename><nodenext spaces=" ">Climate Model Paradigm</nodenext><nodeprev spaces=" ">Strategies</nodeprev><nodeup spaces=" ">Strategies</nodeup></node>
<section spaces=" "><sectiontitle>Philosophy</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="172">philosophy</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="173">climate model</indexterm></cindex>

<para>The main design goal is command line operators which perform useful,
scriptable operations on netCDF files.  
Many scientists work with models and observations which produce too much
data to analyze in tabular format.
Thus, it is often natural to reduce and massage this raw or primary
level data into summary, or second level data, e.g., temporal or spatial
averages. 
These second level data may become the inputs to graphical and
statistical packages, and are often more suitable for archival and
dissemination to the scientific community.
<acronym><acronymword>NCO</acronymword></acronym> performs a suite of operations useful in manipulating data
from the primary to the second level state.
<cindex index="cp" spaces=" "><indexterm index="cp" number="174"><acronym><acronymword>IDL</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="175">Matlab</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="176"><acronym><acronymword>NCL</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="177">Perl</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="178">Yorick</indexterm></cindex>
Higher level interpretive languages (e.g., <acronym><acronymword>IDL</acronymword></acronym>, Yorick,
Matlab, <acronym><acronymword>NCL</acronymword></acronym>, Perl, Python),
and lower level compiled languages (e.g., C, Fortran) can always perform  
any task performed by <acronym><acronymword>NCO</acronymword></acronym>, but often with more overhead.
NCO, on the other hand, is limited to a much smaller set of arithmetic
and metadata operations than these full blown languages.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="179">command line switches</indexterm></cindex>
<para>Another goal has been to implement enough command line switches so that 
frequently used sequences of these operators can be executed from a
shell script or batch file.
Finally, <acronym><acronymword>NCO</acronymword></acronym> was written to consume the absolute minimum
amount of system memory required to perform a given job.
The arithmetic operators are extremely efficient; their exact memory
usage is detailed in <ref label="Memory-Requirements"><xrefnodename>Memory Requirements</xrefnodename></ref>.
</para>
<html endspaces=" ">
&lt;a name=&quot;clm&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#clm &textndash;&gt;
</html>
</section>
<node name="Climate-Model-Paradigm" spaces=" "><nodename>Climate Model Paradigm</nodename><nodenext spaces=" ">Temporary Output Files</nodenext><nodeprev spaces=" ">Philosophy</nodeprev><nodeup spaces=" ">Strategies</nodeup></node>
<section spaces=" "><sectiontitle>Climate Model Paradigm</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="180">climate model</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="181"><acronym><acronymword>NCAR</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="182"><acronym><acronymword>GCM</acronymword></acronym></indexterm></cindex>

<para><acronym><acronymword>NCO</acronymword></acronym> was developed at <acronym><acronymword>NCAR</acronymword></acronym> to aid analysis and
manipulation of datasets produced by General Circulation Models
(<acronym><acronymword>GCM</acronymword></acronym>s).  
<acronym><acronymword>GCM</acronymword></acronym> datasets share many features with other gridded scientific
datasets and so provide a useful paradigm for the explication of the
<acronym><acronymword>NCO</acronymword></acronym> operator set. 
Examples in this manual use a <acronym><acronymword>GCM</acronymword></acronym> paradigm because latitude,
longitude, time, temperature and other fields related to our natural
environment are as easy to visualize for the layman as the expert.
</para>
<html endspaces=" ">
&lt;a name=&quot;out&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#out &textndash;&gt;
</html>
</section>
<node name="Temporary-Output-Files" spaces=" "><nodename>Temporary Output Files</nodename><nodenext spaces=" ">Appending Variables</nodenext><nodeprev spaces=" ">Climate Model Paradigm</nodeprev><nodeup spaces=" ">Strategies</nodeup></node>
<section spaces=" "><sectiontitle>Temporary Output Files </sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="183">data safety</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="184">error tolerance</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="185">safeguards</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="186">temporary output files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="187">temporary files</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> operators are designed to be reasonably fault tolerant, so
that a system failure or user-abort of the operation (e.g., with
<kbd>C-c</kbd>) does not cause loss of data.
The user-specified <var>output-file</var> is only created upon successful
completion of the operation  
<footnote><para>The <command>ncrename</command> and <command>ncatted</command> operators are
exceptions to this rule.
<xref label="ncrename-netCDF-Renamer"><xrefnodename>ncrename netCDF Renamer</xrefnodename></xref>.</para></footnote>.
This is accomplished by performing all operations in a temporary copy
of <var>output-file</var>.
The name of the temporary output file is constructed by appending
<code>.pid<var>&lt;process ID&gt;</var>.<var>&lt;operator name&gt;</var>.tmp</code> to the
user-specified <var>output-file</var> name.  
When the operator completes its task with no fatal errors, the temporary
output file is moved to the user-specified <var>output-file</var>.
This imbues the process with fault-tolerance since fatal error
(e.g., disk space fills up) affect only the temporary output file,
leaving the final output file not created if it did not already exist. 
Note the construction of a temporary output file uses more disk space
than just overwriting existing files &textldquo;in place&textrdquo; (because there may be
two copies of the same file on disk until the <acronym><acronymword>NCO</acronymword></acronym> operation
successfully concludes and the temporary output file overwrites the
existing <var>output-file</var>).  
<cindex index="cp" spaces=" "><indexterm index="cp" number="188">performance</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="189">operator speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="190">speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="191">execution time</indexterm></cindex>
Also, note this feature increases the execution time of the operator
by approximately the time it takes to copy the <var>output-file</var>
<footnote><para>The OS-specific system move command is used.
This is <command>mv</command> for UNIX, and <command>move</command> for Windows.</para></footnote>.
Finally, note this fault-tolerant feature allows the <var>output-file</var>
to be the same as the <var>input-file</var> without any danger of
&textldquo;overlap&textrdquo;. 
</para>
<html endspaces=" ">
&lt;a name=&quot;tmp_fl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#tmp_fl &textndash;&gt;
&lt;a name=&quot;no_tmp_fl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#no_tmp_fl &textndash;&gt;
&lt;a name=&quot;wrt_tmp_fl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#wrt_tmp_fl &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="192"><code>--no_tmp_fl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="193"><code>--wrt_tmp_fl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="194"><code>--write_tmp_fl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="195"><code>--create_ram</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="196"><code>--open_ram</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="197"><acronym><acronymword>RAM</acronymword></acronym> disks</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="198"><acronym><acronymword>RAM</acronymword></acronym> files</indexterm></cindex>
<para>Over time many &textldquo;power users&textrdquo; have requested a way to turn-off the
fault-tolerance safety feature of automatically creating a temporary
file. 
Often these users build and execute production data analysis scripts
that are repeated frequently on large datasets.
Obviating an extra file write can then conserve significant disk space
and time.  
For this purpose <acronym><acronymword>NCO</acronymword></acronym> has, since version 4.2.1 in August, 2012, 
made configurable the controls over temporary file creation.
The <samp>--wrt_tmp_fl</samp> and equivalent <samp>--write_tmp_fl</samp> switches 
ensure <acronym><acronymword>NCO</acronymword></acronym> writes output to an intermediate temporary file.
This is and has always been the default behavior so there is currently
no need to specify these switches.
However, the default may change some day, especially since writing to
RAM disks (<pxref label="RAM-disks"><xrefnodename>RAM disks</xrefnodename></pxref>) may some day become the default.
The <samp>--no_tmp_fl</samp> switch causes <acronym><acronymword>NCO</acronymword></acronym> to write directly to
the final output file instead of to an intermediate temporary file. 
&textldquo;Power users&textrdquo; may wish to invoke this switch to increase performance
(i.e., reduce wallclock time) when manipulating large files. 
When eschewing temporary files, users may forsake the ability to have
the same name for both <var>output-file</var> and <var>input-file</var> since, as   
described above, the temporary file prevented overlap issues.
However, if the user creates the output file in <acronym><acronymword>RAM</acronymword></acronym> (<pxref label="RAM-disks"><xrefnodename>RAM disks</xrefnodename></pxref>)
then it is still possible to have the same name for both
<var>output-file</var> and <var>input-file</var>.
</para><example endspaces=" ">
<pre xml:space="preserve">ncks in.nc out.nc # Default: create out.pid.tmp.nc then move to out.nc
ncks --wrt_tmp_fl in.nc out.nc # Same as default
ncks --no_tmp_fl in.nc out.nc # Create out.nc directly on disk
ncks --no_tmp_fl in.nc in.nc # ERROR-prone! Overwrite in.nc with itself
ncks --create_ram --no_tmp_fl in.nc in.nc # Create in RAM, write to disk
ncks --open_ram --no_tmp_fl in.nc in.nc # Read into RAM, write to disk
</pre></example>
<noindent></noindent>
<para>There is no reason to expect the fourth example to work.
The behavior of overwriting a file while reading from the same file is
undefined, much as is the shell command <samp>cat foo &gt; foo</samp>.
Although it may &textldquo;work&textrdquo; in some cases, it is unreliable.
One way around this is to use <samp>--create_ram</samp> so that the
output file is not written to disk until the input file is closed,
<xref label="RAM-disks"><xrefnodename>RAM disks</xrefnodename></xref>.
However, as of 20130328, the behavior of the <samp>--create_ram</samp> and
<samp>--open_ram</samp> examples has not been thoroughly tested.
</para>
<para>The <acronym><acronymword>NCO</acronymword></acronym> authors have seen compelling use cases for utilizing
the <acronym><acronymword>RAM</acronymword></acronym> switches, though not (yet) for combining them with
<samp>--no_tmp_fl</samp>. 
<acronym><acronymword>NCO</acronymword></acronym> implements both options because they are largely
independent of eachother.
It is up to &textldquo;power users&textrdquo; to discover which best fit their needs.
We welcome accounts of your experiences posted to the forums.
</para>
<html endspaces=" ">
&lt;a name=&quot;-A&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-A &textndash;&gt;
&lt;a name=&quot;-O&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-O &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="199"><code>-A</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="200"><code>-O</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="201"><code>--apn</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="202"><code>--append</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="203"><code>--ovr</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="204"><code>--overwrite</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="205">overwriting files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="206">appending variables</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="207">appending to files</indexterm></cindex>
<para>Other safeguards exist to protect the user from inadvertently
overwriting data.
If the <var>output-file</var> specified for a command is a pre-existing file,
then the operator will prompt the user whether to overwrite (erase) the
existing <var>output-file</var>, attempt to append to it, or abort the
operation. 
However, in processing large amounts of data, too many interactive
questions slows productivity.
Therefore <acronym><acronymword>NCO</acronymword></acronym> also implements two ways to override its own
safety features, the <samp>-O</samp> and <samp>-A</samp> switches.
Specifying <samp>-O</samp> tells the operator to overwrite any existing
<var>output-file</var> without prompting the user interactively.
Specifying <samp>-A</samp> tells the operator to attempt to append to any
existing <var>output-file</var> without prompting the user interactively.
These switches are useful in batch environments because they suppress
interactive keyboard input.
</para>
<html endspaces=" ">
&lt;a name=&quot;apn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#apn &textndash;&gt;
&lt;a name=&quot;append&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#append &textndash;&gt;
</html>
</section>
<node name="Appending-Variables" spaces=" "><nodename>Appending Variables</nodename><nodenext spaces=" ">Simple Arithmetic and Interpolation</nodenext><nodeprev spaces=" ">Temporary Output Files</nodeprev><nodeup spaces=" ">Strategies</nodeup></node>
<section spaces=" "><sectiontitle>Appending Variables</sectiontitle>
<para>Adding variables from one file to another is often desirable.
<cindex index="cp" spaces=" "><indexterm index="cp" number="208">concatenation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="209">appending variables</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="210">merging files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="211">pasting variables</indexterm></cindex>
This is referred to as <dfn>appending</dfn>, although some prefer the
terminology <dfn>merging</dfn> <footnote><para>The terminology <dfn>merging</dfn> is
reserved for an (unwritten) operator which replaces hyperslabs of a
variable in one file with hyperslabs of the same variable from another 
file</para></footnote> or <dfn>pasting</dfn>. 
Appending is often confused with what <acronym><acronymword>NCO</acronymword></acronym> calls
<dfn>concatenation</dfn>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="212">record dimension</indexterm></cindex>
In <acronym><acronymword>NCO</acronymword></acronym>, concatenation refers to splicing a variable
along the record dimension.  
The length along the record dimension of the output is the sum of the
lengths of the input files. 
Appending, on the other hand, refers to copying a variable from one file
to another file which may or may not already contain the variable 
<footnote><para>Yes, the terminology is confusing. 
By all means mail me if you think of a better nomenclature.
Should <acronym><acronymword>NCO</acronymword></acronym> use <dfn>paste</dfn> instead of <dfn>append</dfn>?
</para></footnote>. 
<acronym><acronymword>NCO</acronymword></acronym> can append or concatenate just one variable, or all the
variables in a file at the same time.
</para>
<para>In this sense, <command>ncks</command> can append variables from one file to
another file. 
This capability is invoked by naming two files on the command line,
<var>input-file</var> and <var>output-file</var>. 
When <var>output-file</var> already exists, the user is prompted whether to
<dfn>overwrite</dfn>, <dfn>append/replace</dfn>, or <dfn>exit</dfn> from the command.
Selecting <dfn>overwrite</dfn> tells the operator to erase the existing
<var>output-file</var> and replace it with the results of the operation.
Selecting <dfn>exit</dfn> causes the operator to exit&textmdash;the <var>output-file</var>
will not be touched in this case.
Selecting <dfn>append/replace</dfn> causes the operator to attempt to place
the results of the operation in the existing <var>output-file</var>, 
<xref label="ncks-netCDF-Kitchen-Sink"><xrefnodename>ncks netCDF Kitchen Sink</xrefnodename></xref>.
</para>
<html endspaces=" ">
&lt;a name=&quot;unn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#unn &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="213">union of files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="214">disjoint files</indexterm></cindex>
<para>The simplest way to create the union of two files is
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -A fl_1.nc fl_2.nc
</pre></example>
<para>This puts the contents of <file>fl_1.nc</file> into <file>fl_2.nc</file>. 
The <samp>-A</samp> is optional. 
On output, <file>fl_2.nc</file> is the union of the input files,
regardless of whether they share dimensions and variables, 
or are completely disjoint.
The append fails if the input files have differently named record
dimensions (since netCDF supports only one), or have dimensions of the
same name but different sizes.
</para>
<html endspaces=" ">
&lt;a name=&quot;bnr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#bnr &textndash;&gt;
</html>
</section>
<node name="Simple-Arithmetic-and-Interpolation" spaces=" "><nodename>Simple Arithmetic and Interpolation</nodename><nodenext spaces=" ">Statistics vs. Concatenation</nodenext><nodeprev spaces=" ">Appending Variables</nodeprev><nodeup spaces=" ">Strategies</nodeup></node>
<section spaces=" "><sectiontitle>Simple Arithmetic and Interpolation</sectiontitle>

<para>Users comfortable with <acronym><acronymword>NCO</acronymword></acronym> semantics may find it easier to
perform some simple mathematical operations in <acronym><acronymword>NCO</acronymword></acronym> rather than  
higher level languages. 
<command>ncbo</command> (<pxref label="ncbo-netCDF-Binary-Operator"><xrefnodename>ncbo netCDF Binary Operator</xrefnodename></pxref>) does file
addition, subtraction, multiplication, division, and broadcasting. 
It even does group broadcasting.
<command>ncflint</command> (<pxref label="ncflint-netCDF-File-Interpolator"><xrefnodename>ncflint netCDF File Interpolator</xrefnodename></pxref>) does
file addition, subtraction, multiplication and interpolation. 
Sequences of these commands can accomplish simple yet powerful
operations from the command line.
</para>
<html endspaces=" ">
&lt;a name=&quot;statisticians&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#statisticians &textndash;&gt;
&lt;a name=&quot;averagers&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#averagers &textndash;&gt;
</html>
</section>
<node name="Statistics-vs_002e-Concatenation" spaces=" "><nodename>Statistics vs. Concatenation</nodename><nodenext spaces=" ">Large Numbers of Files</nodenext><nodeprev spaces=" ">Simple Arithmetic and Interpolation</nodeprev><nodeup spaces=" ">Strategies</nodeup></node>
<section spaces=" "><sectiontitle>Statistics vs.&noeos; Concatenation</sectiontitle>

<html endspaces=" ">
&lt;a name=&quot;sym_ncea&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#sym_ncea &textndash;&gt;
&lt;a name=&quot;sym_nces&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#sym_nces &textndash;&gt;
&lt;a name=&quot;sym_ncrcat&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#sym_ncrcat &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="215">symbolic links</indexterm></cindex>
<para>The most frequently used operators of <acronym><acronymword>NCO</acronymword></acronym> are probably the
<dfn>statisticians</dfn> (i.e., tools that do statistics) and concatenators.
Because there are so many types of statistics like averaging (e.g.,
across files, within a file, over the record dimension, over other
dimensions, with or without weights and masks) and of concatenating
(across files, along the record dimension, along other dimensions),
there are currently no fewer than five operators which tackle these two
purposes: <command>ncra</command>, <command>nces</command>, <command>ncwa</command>,
<command>ncrcat</command>, and <command>ncecat</command>.  
These operators do share many capabilities <footnote><para>Currently
<command>nces</command> and <command>ncrcat</command> are symbolically linked to the
<command>ncra</command> executable, which behaves slightly differently based on
its invocation name (i.e., <samp>argv[0]</samp>). 
These three operators share the same source code, and merely have
different inner loops.</para></footnote>, though each has its unique specialty.
Two of these operators, <command>ncrcat</command> and <command>ncecat</command>, 
concatenate hyperslabs across files. 
The other two operators, <command>ncra</command> and <command>nces</command>, compute
statistics across (and/or within) files 
<footnote><para>The third averaging operator, <command>ncwa</command>, is the most
sophisticated averager in <acronym><acronymword>NCO</acronymword></acronym>. 
However, <command>ncwa</command> is in a different class than <command>ncra</command> and
<command>nces</command> because it operates on a single file per invocation (as
opposed to multiple files).    
On that single file, however, <command>ncwa</command> provides a richer set of 
averaging options&textmdash;including weighting, masking, and broadcasting.</para></footnote>.  
First, let&textrsquo;s describe the concatenators, then the statistics tools.
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Concatenation</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Averaging</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Interpolating</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<html endspaces=" ">
&lt;a name=&quot;cnc&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnc &textndash;&gt;
</html>
<node name="Concatenation" spaces=" "><nodename>Concatenation</nodename><nodenext spaces=" ">Averaging</nodenext><nodeprev spaces=" ">Statistics vs. Concatenation</nodeprev><nodeup spaces=" ">Statistics vs. Concatenation</nodeup></node>
<subsection spaces=" "><sectiontitle>Concatenators <command>ncrcat</command> and <command>ncecat</command></sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="216"><command>ncecat</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="217"><command>ncrcat</command></indexterm></cindex>

<para>Joining together independent files along a common record dimension is
called <dfn>concatenation</dfn>.    
<command>ncrcat</command> is designed for concatenating record variables, while
<command>ncecat</command> is designed for concatenating fixed length variables.
Consider five files, <file>85.nc</file>, <file>86.nc</file>, 
<w>&dots; <file>89.nc</file></w> each containing a year&textrsquo;s worth of data.  
Say you wish to create from them a single file, <file>8589.nc</file>
containing all the data, i.e., spanning all five years.
If the annual files make use of the same record variable, then
<command>ncrcat</command> will do the job nicely with, e.g., 
<code>ncrcat 8?.nc 8589.nc</code>. 
The number of records in the input files is arbitrary and can vary from
file to file. 
<xref label="ncrcat-netCDF-Record-Concatenator"><xrefnodename>ncrcat netCDF Record Concatenator</xrefnodename></xref>, for a complete description of
<command>ncrcat</command>. 
</para>
<para>However, suppose the annual files have no record variable, and thus
their data are all fixed length. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="218">ensemble</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="219">climate model</indexterm></cindex>
For example, the files may not be conceptually sequential, but rather
members of the same group, or <dfn>ensemble</dfn>. 
Members of an ensemble may have no reason to contain a record dimension.
<command>ncecat</command> will create a new record dimension (named <var>record</var>
by default) with which to glue together the individual files into the
single ensemble file.
If <command>ncecat</command> is used on files which contain an existing record
dimension, that record dimension is converted to a fixed-length
dimension of the same name and a new record dimension (named
<code>record</code>) is created.  
Consider five realizations, <file>85a.nc</file>, <file>85b.nc</file>, 
<w>&dots; <file>85e.nc</file></w> of 1985 predictions from the same climate
model. 
Then <code>ncecat 85?.nc 85_ens.nc</code> glues together the individual
realizations into the single file, <file>85_ens.nc</file>. 
If an input variable was dimensioned [<code>lat</code>,<code>lon</code>], it will
have dimensions [<code>record</code>,<code>lat</code>,<code>lon</code>] in the output file.
<w>A restriction</w> of <command>ncecat</command> is that the hyperslabs of the
processed variables must be the same from file to file.
Normally this means all the input files are the same size, and contain 
data on different realizations of the same variables.
<xref label="ncecat-netCDF-Ensemble-Concatenator"><xrefnodename>ncecat netCDF Ensemble Concatenator</xrefnodename></xref>, for a complete description
of <command>ncecat</command>. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="220"><command>ncpdq</command></indexterm></cindex>
<html endspaces=" ">
&lt;a name=&quot;dmn_cat&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dmn_cat &textndash;&gt;
</html>
<para><command>ncpdq</command> makes it possible to concatenate files along any
dimension, not just the record dimension.
First, use <command>ncpdq</command> to convert the dimension to be concatenated
(i.e., extended with data from other files) into the record dimension. 
Second, use <command>ncrcat</command> to concatenate these files.
Finally, if desirable, use <command>ncpdq</command> to revert to the original
dimensionality.
As a concrete example, say that files <file>x_01.nc</file>, <file>x_02.nc</file>,
<w>&dots; <file>x_10.nc</file></w> contain time-evolving datasets from spatially
adjacent regions.
The time and spatial coordinates are <code>time</code> and <code>x</code>, respectively.
Initially the record dimension is <code>time</code>.
Our goal is to create a single file that contains joins all the
spatially adjacent regions into one single time-evolving dataset.
</para><example endspaces=" ">
<pre xml:space="preserve">for idx in 01 02 03 04 05 06 07 08 09 10; do # Bourne Shell
  ncpdq -a x,time x_$&lbrace;idx&rbrace;.nc foo_$&lbrace;idx&rbrace;.nc # Make x record dimension
done
ncrcat foo_??.nc out.nc       # Concatenate along x
ncpdq -a time,x out.nc out.nc # Revert to time as record dimension
</pre></example>

<para>Note that <command>ncrcat</command> will not concatenate fixed-length variables, 
whereas <command>ncecat</command> concatenates both fixed-length and record
variables along a new record variable.
To conserve system memory, use <command>ncrcat</command> where possible.
</para>
<html endspaces=" ">
&lt;a name=&quot;avg&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#avg &textndash;&gt;
</html>
</subsection>
<node name="Averaging" spaces=" "><nodename>Averaging</nodename><nodenext spaces=" ">Interpolating</nodenext><nodeprev spaces=" ">Concatenation</nodeprev><nodeup spaces=" ">Statistics vs. Concatenation</nodeup></node>
<subsection spaces=" "><sectiontitle>Averagers <command>nces</command>, <command>ncra</command>, and <command>ncwa</command> </sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="221"><command>nces</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="222"><command>ncra</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="223"><command>ncwa</command></indexterm></cindex>

<para>The differences between the averagers <command>ncra</command> and <command>nces</command>
are analogous to the differences between the concatenators.
<command>ncra</command> is designed for averaging record variables from at least
one file, while <command>nces</command> is designed for averaging fixed length
variables from multiple files.
<command>ncra</command> performs a simple arithmetic average over the record
dimension of all the input files, with each record having an equal
weight in the average. 
<command>nces</command> performs a simple arithmetic average of all the input
files, with each file having an equal weight in the average. 
Note that <command>ncra</command> cannot average fixed-length variables,
but <command>nces</command> can average both fixed-length and record variables.  
To conserve system memory, use <command>ncra</command> rather than
<command>nces</command> where possible (e.g., if each <var>input-file</var> is one
record long). 
The file output from <command>nces</command> will have the same dimensions
(meaning dimension names as well as sizes) as the input hyperslabs  
(<pxref label="nces-netCDF-Ensemble-Statistics"><xrefnodename>nces netCDF Ensemble Statistics</xrefnodename></pxref>, for a complete description of 
<command>nces</command>).  
The file output from <command>ncra</command> will have the same dimensions as
the input hyperslabs except for the record dimension, which will have a   
size <w>of 1</w> (<pxref label="ncra-netCDF-Record-Averager"><xrefnodename>ncra netCDF Record Averager</xrefnodename></pxref>, for a complete
description of <command>ncra</command>). 
</para>
<html endspaces=" ">
&lt;a name=&quot;ntp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ntp &textndash;&gt;
</html>
</subsection>
<node name="Interpolating" spaces=" "><nodename>Interpolating</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">Averaging</nodeprev><nodeup spaces=" ">Statistics vs. Concatenation</nodeup></node>
<subsection spaces=" "><sectiontitle>Interpolator <command>ncflint</command></sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="224"><command>ncflint</command></indexterm></cindex>

<para><command>ncflint</command> can interpolate data between or two files.
Since no other operators have this ability, the description of
interpolation is given fully on the <command>ncflint</command> reference page
(<pxref label="ncflint-netCDF-File-Interpolator"><xrefnodename>ncflint netCDF File Interpolator</xrefnodename></pxref>). 
Note that this capability also allows <command>ncflint</command> to linearly
rescale any data in a netCDF file, e.g., to convert between differing
units. 
</para>
<html endspaces=" ">
&lt;a name=&quot;lrg&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#lrg &textndash;&gt;
</html>
</subsection>
</section>
<node name="Large-Numbers-of-Files" spaces=" "><nodename>Large Numbers of Files</nodename><nodenext spaces=" ">Large Datasets</nodenext><nodeprev spaces=" ">Statistics vs. Concatenation</nodeprev><nodeup spaces=" ">Strategies</nodeup></node>
<section spaces=" "><sectiontitle>Large Numbers of Files</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="225">files, numerous input</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="226"><code>-n <var>loop</var></code></indexterm></cindex>

<para>Occasionally one desires to digest (i.e., concatenate or average)
hundreds or thousands of input files.
<cindex index="cp" spaces=" "><indexterm index="cp" number="227">automagic</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="228"><acronym><acronymword>NASA EOSDIS</acronymword></acronym></indexterm></cindex>
Unfortunately, data archives (e.g., <acronym><acronymword>NASA EOSDIS</acronymword></acronym>) may not
name netCDF files in a format understood by the <samp>-n <var>loop</var></samp>
switch (<pxref label="Specifying-Input-Files"><xrefnodename>Specifying Input Files</xrefnodename></pxref>) that automagically generates
arbitrary numbers of input filenames. 
The <samp>-n <var>loop</var></samp> switch has the virtue of being concise,
and of minimizing the command line.
This helps keeps output file small since the command line is stored
as metadata in the <code>history</code> attribute 
(<pxref label="History-Attribute"><xrefnodename>History Attribute</xrefnodename></pxref>). 
However, the <samp>-n <var>loop</var></samp> switch is useless when there is no
simple, arithmetic pattern to the input filenames (e.g.,
<file>h00001.nc</file>, <file>h00002.nc</file>, <w>&dots; <file>h90210.nc</file></w>).
Moreover, filename globbing does not work when the input files are too
numerous or their names are too lengthy (when strung together as a
single argument) to be passed by the calling shell to the <acronym><acronymword>NCO</acronymword></acronym>
operator
<footnote><para>The exact length which exceeds the operating system internal
limit for command line lengths varies from <acronym><acronymword>OS</acronymword></acronym> to <acronym><acronymword>OS</acronymword></acronym>
and from shell to shell.  
<acronym><acronymword>GNU</acronymword></acronym> <code>bash</code> may not have any arbitrary fixed limits to the
size of command line arguments. 
Many <acronym><acronymword>OS</acronymword></acronym>s cannot handle command line arguments (including
results of file globbing) exceeding 4096 characters.</para></footnote>.
When this occurs, the <acronym><acronymword>ANSI</acronymword></acronym> C-standard <code>argc</code>-<code>argv</code> 
method of passing arguments from the calling shell to a C-program (i.e.,
an <acronym><acronymword>NCO</acronymword></acronym> operator) breaks down. 
There are (at least) three alternative methods of specifying the input 
filenames to <acronym><acronymword>NCO</acronymword></acronym> in environment-limited situations.
</para>
<html endspaces=" ">
&lt;a name=&quot;stdin&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#stdin &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="229">standard input</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="230"><code>stdin</code></indexterm></cindex>
<para>The recommended method for sending very large numbers (hundreds or
more, typically) of input filenames to the multi-file operators is
to pass the filenames with the <acronym><acronymword>UNIX</acronymword></acronym> <dfn>standard input</dfn>
feature, aka <code>stdin</code>: 
</para><example endspaces=" ">
<pre xml:space="preserve"># Pipe large numbers of filenames to stdin
/bin/ls | grep $&lbrace;CASEID&rbrace;_'......'.nc | ncecat -o foo.nc
</pre></example>
<para>This method avoids all constraints on command line size imposed by
the operating system. 
A drawback to this method is that the <code>history</code> attribute
(<pxref label="History-Attribute"><xrefnodename>History Attribute</xrefnodename></pxref>) does not record the name of any input 
files since the names were not passed on the command line.
This makes determining the data provenance at a later date difficult.
<cindex index="cp" spaces=" "><indexterm index="cp" number="231"><code>nco_input_file_number</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="232"><code>nco_input_file_list</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="233">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="234">attributes, global</indexterm></cindex>
To remedy this situation, multi-file operators store the number of
input files in the <code>nco_input_file_number</code> global attribute and the
input file list itself in the <code>nco_input_file_list</code> global attribute
(<pxref label="File-List-Attributes"><xrefnodename>File List Attributes</xrefnodename></pxref>).
Although this does not preserve the exact command used to generate the
file, it does retains all the information required to reconstruct the
command and determine the data provenance.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="235">globbing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="236">shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="237">extended regular expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="238">regular expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="239">pattern matching</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="240"><command>xargs</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="241"><acronym><acronymword>UNIX</acronymword></acronym></indexterm></cindex>
<para>A second option is to use the <acronym><acronymword>UNIX</acronymword></acronym> <command>xargs</command> command.
This simple example selects as input to <command>xargs</command> all the
filenames in the current directory that match a given pattern.
For illustration, consider a user trying to average millions of 
files which each have a six character filename. 
If the shell buffer cannot hold the results of the corresponding
globbing operator, <file>??????.nc</file>, then the filename globbing
technique will fail. 
Instead we express the filename pattern as an extended regular 
expression, <file>......\.nc</file> (<pxref label="Subsetting-Files"><xrefnodename>Subsetting Files</xrefnodename></pxref>).
We use <command>grep</command> to filter the directory listing for this pattern
and to pipe the results to <command>xargs</command> which, in turn, passes the
matching filenames to an <acronym><acronymword>NCO</acronymword></acronym> multi-file operator, e.g.,
<command>ncecat</command>.
</para><example endspaces=" ">
<pre xml:space="preserve"># Use xargs to transfer filenames on the command line
/bin/ls | grep $&lbrace;CASEID&rbrace;_'......'.nc | xargs -x ncecat -o foo.nc
</pre></example>
<cindex index="cp" spaces=" "><indexterm index="cp" number="242">pipes</indexterm></cindex>
<para>The single quotes protect the only sensitive parts of the extended
regular expression (the <command>grep</command> argument), and allow shell
interpolation (the <code>$&lbrace;CASEID&rbrace;</code> variable substitution) to
proceed unhindered on the rest of the command.
<command>xargs</command> uses the <acronym><acronymword>UNIX</acronymword></acronym> pipe feature to append the
suitably filtered input file list to the end of the <command>ncecat</command>
command options.  
<cindex index="cp" spaces=" "><indexterm index="cp" number="243">output file</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="244">input files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="245"><code>-o <var>fl_out</var></code></indexterm></cindex>
The <code>-o foo.nc</code> switch ensures that the input files supplied by
<command>xargs</command> are not confused with the output file name. 
<command>xargs</command> does, unfortunately, have its own limit (usually about 
20,000 characters) on the size of command lines it can pass.
Give <command>xargs</command> the <samp>-x</samp> switch to ensure it dies if it
reaches this internal limit.
When this occurs, use either the <code>stdin</code> method above, or the
symbolic link presented next.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="246">symbolic links</indexterm></cindex>
<para>Even when its internal limits have not been reached, the
<command>xargs</command> technique may not be sophisticated enough to handle 
all situations. 
A full scripting language like Perl can handle any level of complexity
of filtering input filenames, and any number of filenames.
The technique of last resort is to write a script that creates symbolic 
links between the irregular input filenames and a set of regular,
arithmetic filenames that the <samp>-n <var>loop</var></samp> switch understands. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="247">Perl</indexterm></cindex>
For example, the following Perl script creates a monotonically
enumerated symbolic link to up to one million <file>.nc</file> files in a
directory. If there are 999,999 netCDF files present, the links are
named <file>000001.nc</file> to <file>999999.nc</file>: 
<cindex index="cp" spaces=" "><indexterm index="cp" number="248"><code>-n <var>loop</var></code></indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve"># Create enumerated symbolic links
/bin/ls | grep \.nc | perl -e \
'$idx=1;while(&lt;STDIN&gt;)&lbrace;chop;symlink $_,sprintf(&quot;%06d.nc&quot;,$idx++);&rbrace;'
ncecat -n 999999,6,1 000001.nc foo.nc
# Remove symbolic links when finished
/bin/rm ??????.nc
</pre></example>
<para>The <samp>-n <var>loop</var></samp> option tells the <acronym><acronymword>NCO</acronymword></acronym> operator to
automatically generate the filnames of the symbolic links.
This circumvents any <acronym><acronymword>OS</acronymword></acronym> and shell limits on command line size.
The symbolic links are easily removed once <acronym><acronymword>NCO</acronymword></acronym> is finished.
<cindex index="cp" spaces=" "><indexterm index="cp" number="249"><code>history</code></indexterm></cindex>
One drawback to this method is that the <code>history</code> attribute
(<pxref label="History-Attribute"><xrefnodename>History Attribute</xrefnodename></pxref>) retains the filename list of the symbolic
links, rather than the data files themselves. 
This makes it difficult to determine the data provenance at a later
date. 
</para>
</section>
<node name="Large-Datasets" spaces=" "><nodename>Large Datasets</nodename><nodenext spaces=" ">Memory Requirements</nodenext><nodeprev spaces=" ">Large Numbers of Files</nodeprev><nodeup spaces=" ">Strategies</nodeup></node>
<section spaces=" "><sectiontitle>Large Datasets</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="250">large datasets</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="251">LFS</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="252">Large File Support</indexterm></cindex>

<para><dfn>Large datasets</dfn> are those files that are comparable in size to the
amount of random access memory (<acronym><acronymword>RAM</acronymword></acronym>) in your computer.
Many users of <acronym><acronymword>NCO</acronymword></acronym> work with files larger than <w>100 MB</w>.
Files this large not only push the current edge of storage technology, 
they present special problems for programs which attempt to access the  
entire file at once, such as <command>nces</command> and <command>ncecat</command>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="253">swap space</indexterm></cindex>
If you work with a <w>300 MB</w> files on a machine with only <w>32 MB</w> of
memory then you will need large amounts of swap space (virtual memory on
disk) and <acronym><acronymword>NCO</acronymword></acronym> will work slowly, or even fail. 
There is no easy solution for this.
The best strategy is to work on a machine with sufficient amounts of
memory and swap space. 
Since about 2004, many users have begun to produce or analyze files
exceeding <w>2 GB</w> in size. 
These users should familiarize themselves with <acronym><acronymword>NCO</acronymword></acronym>&textrsquo;s Large
File Support (<acronym><acronymword>LFS</acronymword></acronym>) capabilities (<pxref label="Large-File-Support"><xrefnodename>Large File Support</xrefnodename></pxref>).
The next section will increase your familiarity with <acronym><acronymword>NCO</acronymword></acronym>&textrsquo;s
memory requirements.
With this knowledge you may re-design your data reduction approach to
divide the problem into pieces solvable in memory-limited situations.   
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="254">server</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="255"><acronym><acronymword>UNICOS</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="256">Cray</indexterm></cindex>
<para>If your local machine has problems working with large files, try running
<acronym><acronymword>NCO</acronymword></acronym> from a more powerful machine, such as a network server.  
Certain machine architectures, e.g., Cray <acronym><acronymword>UNICOS</acronymword></acronym>, have special 
commands which allow one to increase the amount of interactive memory.
<cindex index="cp" spaces=" "><indexterm index="cp" number="257"><code>ilimit</code></indexterm></cindex>
On Cray systems, try to increase the available memory with the
<code>ilimit</code> command.    
<cindex index="cp" spaces=" "><indexterm index="cp" number="258"><acronym><acronymword>GNU</acronymword></acronym>/Linux</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="259"><code>ulimit</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="260"><code>core dump</code></indexterm></cindex>
If you get a memory-related core dump 
(e.g., <samp>Error exit (core dumped)</samp>) on a <acronym><acronymword>GNU</acronymword></acronym>/Linux system,
try increasing the process-available memory with <code>ulimit</code>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="261">speed</indexterm></cindex>
<para>The speed of the <acronym><acronymword>NCO</acronymword></acronym> operators also depends on file size.
When processing large files the operators may appear to hang, or do
nothing, for large periods of time.
In order to see what the operator is actually doing, it is useful to
activate a more verbose output mode.
This is accomplished by supplying a number greater <w>than 0</w> to the
<samp>-D <var>debug-level</var></samp> (or <samp>--debug-level</samp>, or
<samp>--dbg_lvl</samp>) switch.
<cindex index="cp" spaces=" "><indexterm index="cp" number="262"><code>-D <var>debug-level</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="263"><code>--debug-level <var>debug-level</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="264"><code>--dbg_lvl <var>debug-level</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="265"><var>debug-level</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="266"><var>dbg_lvl</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="267">debugging</indexterm></cindex>
When the <var>debug-level</var> is nonzero, the operators report their
current status to the terminal through the <var>stderr</var> facility.
Using <samp>-D</samp> does not slow the operators down. 
Choose a <var>debug-level</var> <w>between 1</w> <w>and 3</w> for most situations,
e.g., <code>nces -D 2 85.nc 86.nc 8586.nc</code>.
<w>A full</w> description of how to estimate the actual amount of memory the
multi-file <acronym><acronymword>NCO</acronymword></acronym> operators consume is given in 
<ref label="Memory-Requirements"><xrefnodename>Memory Requirements</xrefnodename></ref>. 
</para>
<html endspaces=" ">
&lt;a name=&quot;mmr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mmr &textndash;&gt;
</html>
</section>
<node name="Memory-Requirements" spaces=" "><nodename>Memory Requirements</nodename><nodenext spaces=" ">Performance</nodenext><nodeprev spaces=" ">Large Datasets</nodeprev><nodeup spaces=" ">Strategies</nodeup></node>
<section spaces=" "><sectiontitle>Memory Requirements</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="268">memory requirements</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="269">memory available</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="270"><acronym><acronymword>RAM</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="271">swap space</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="272">peak memory usage</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="273"><code>--ram_all</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="274"><code>--open_ram</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="275"><code>--diskless_all</code></indexterm></cindex>

<para>Many people use <acronym><acronymword>NCO</acronymword></acronym> on gargantuan files which dwarf the
memory available (free <acronym><acronymword>RAM</acronymword></acronym> plus swap space) even on today&textrsquo;s powerful
machines. 
These users want <acronym><acronymword>NCO</acronymword></acronym> to consume the least memory possible
so that their scripts do not have to tediously cut files into smaller
pieces that fit into memory. 
We commend these greedy users for pushing <acronym><acronymword>NCO</acronymword></acronym> to its limits!
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="276">threads</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="277">OpenMP</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="278">shared memory machines</indexterm></cindex>
<para>This section describes the memory <acronym><acronymword>NCO</acronymword></acronym> requires during
operation.
The required memory is based on the underlying algorithms.
The description below is the memory usage per thread.
Users with shared memory machines may use the threaded <acronym><acronymword>NCO</acronymword></acronym>
operators (<pxref label="OpenMP-Threading"><xrefnodename>OpenMP Threading</xrefnodename></pxref>).
The peak and sustained memory usage will scale accordingly,
i.e., by the number of threads.
Memory consumption patterns of all operators are similar, with
the exception of <command>ncap2</command>.
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Single and Multi-file Operators</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Memory for ncap2</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<node name="Single-and-Multi_002dfile-Operators" spaces=" "><nodename>Single and Multi-file Operators</nodename><nodenext spaces=" ">Memory for ncap2</nodenext><nodeprev spaces=" ">Memory Requirements</nodeprev><nodeup spaces=" ">Memory Requirements</nodeup></node>
<subsection spaces=" "><sectiontitle>Single and Multi-file Operators</sectiontitle>

<cindex index="cp" spaces=" "><indexterm index="cp" number="279">multi-file operators</indexterm></cindex>
<para>The multi-file operators currently comprise the record operators,
<command>ncra</command> and <command>ncrcat</command>, and the ensemble operators,
<command>nces</command> and <command>ncecat</command>. 
The record operators require <emph>much less</emph> memory than the ensemble 
operators. 
This is because the record operators operate on one single record (i.e.,
time-slice) at a time, whereas the ensemble operators retrieve the
entire variable into memory. 
Let <math>MS</math> be the peak sustained memory demand of an operator,
<math>FT</math> be the memory required to store the entire contents of all the 
variables to be processed in an input file,
<math>FR</math> be the memory required to store the entire contents of a
single record of each of the variables to be processed in an input file, 
<math>VR</math> be the memory required to store a single record of the
largest record variable to be processed in an input file, 
<math>VT</math> be the memory required to store the largest variable 
to be processed in an input file,
<math>VI</math> be the memory required to store the largest variable 
which is not processed, but is copied from the initial file to the
output file. 
All operators require <math>MI = VI</math> during the initial copying of
variables from the first input file to the output file. 
This is the <emph>initial</emph> (and transient) memory demand.
The <emph>sustained</emph> memory demand is that memory required by the
operators during the processing (i.e., averaging, concatenation)
phase which lasts until all the input files have been processed.
The operators have the following memory requirements: 
<command>ncrcat</command> requires <math>MS &lt;= VR</math>. 
<command>ncecat</command> requires <math>MS &lt;= VT</math>. 
<command>ncra</command> requires <math>MS = 2FR + VR</math>. 
<command>nces</command> requires <math>MS = 2FT + VT</math>. 
<command>ncbo</command> requires <math>MS &lt;= 3VT</math> 
(both input variables and the output variable).
<command>ncflint</command> requires <math>MS &lt;= 3VT</math>
(both input variables and the output variable).
<command>ncpdq</command> requires <math>MS &lt;= 2VT</math>
(one input variable and the output variable).
<command>ncwa</command> requires <math>MS &lt;= 8VT</math> (see below).
Note that only variables that are processed, e.g., averaged,
concatenated, or differenced, contribute to <math>MS</math>. 
Variables which do not appear in the output file 
(<pxref label="Subsetting-Files"><xrefnodename>Subsetting Files</xrefnodename></pxref>) are never read and contribute nothing
to the memory requirements. 
</para>
<para>Further note that some operators perform internal type-promotion on some
variables prior to arithmetic (<pxref label="Type-Conversion"><xrefnodename>Type Conversion</xrefnodename></pxref>).
For example, <command>ncra</command> and <command>nces</command> both promote integer
types to double-precision floating point prior to arithmetic, then 
perform the arithmetic, then demote back to the original integer type
after arithmetic.
This preserves the on-disk storage type while obtaining the accuracy
advantages of floating point arithmetic. 
Since version 4.3.6 (released in September, 2013), <acronym><acronymword>NCO</acronymword></acronym> also
by default converts single-precision floating point to double-precision
prior to arithmetic, which incurs the same <acronym><acronymword>RAM</acronymword></acronym> penalty.
Hence, the sustained memory required for integer variables and
single-precision floats are two or four-times their on-disk,
uncompressed, unpacked sizes if they meet the rules for automatic
internal promotion. 
Put another way, disabling auto-promotion of single-precision variables
(with <samp>--flt</samp>) considerably reduces the <acronym><acronymword>RAM</acronymword></acronym> footprint 
of arithmetic operators.
</para>
<para>The <samp>--open_ram</samp> switch (and switches that invoke it like
<samp>--ram_all</samp> and <samp>--diskless_all</samp>) incurs a <acronym><acronymword>RAM</acronymword></acronym>
penalty.
These switches cause each input file to be copied to <acronym><acronymword>RAM</acronymword></acronym> upon
opening. 
Hence any operator invoking these switches utilizes an additional
<math>FT</math> of <acronym><acronymword>RAM</acronymword></acronym> (i.e., <math>MS += FT</math>).
See <ref label="RAM-disks"><xrefnodename>RAM disks</xrefnodename></ref> for further details. 
</para>
<html endspaces=" ">
&lt;a name=&quot;mmr_ncwa&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mmr_ncwa &textndash;&gt;
</html>
<para><command>ncwa</command> consumes between two and seven times the memory of a
variable in order to process it. 
Peak consumption occurs when storing simultaneously in memory 
one input variable, one tally array,
one input weight, one conformed/working weight, one weight tally, 
one input mask, one conformed/working mask, and
one output variable. 
When invoked, the weighting and masking features contribute up to
three-sevenths and two-sevenths of these requirements apiece.
If weights and masks are <emph>not</emph> specified 
(i.e., no <samp>-w</samp> or <samp>-a</samp> options)
then <command>ncwa</command> requirements drop to <math>MS &lt;= 3VT</math>
(one input variable, one tally array, and the output variable). 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="280">OpenMP</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="281">threads</indexterm></cindex>
<para>The above memory requirements must be multiplied by the number of
threads <var>thr_nbr</var> (<pxref label="OpenMP-Threading"><xrefnodename>OpenMP Threading</xrefnodename></pxref>).
<cindex index="cp" spaces=" "><indexterm index="cp" number="282"><code>-t <var>thr_nbr</var></code></indexterm></cindex>
If this causes problems then reduce (with <samp>-t <var>thr_nbr</var></samp>) the
number of threads.
</para>
</subsection>
<node name="Memory-for-ncap2" spaces=" "><nodename>Memory for ncap2</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">Single and Multi-file Operators</nodeprev><nodeup spaces=" ">Memory Requirements</nodeup></node>
<subsection spaces=" "><sectiontitle>Memory for <command>ncap2</command></sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="283"><command>ncap2</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="284">binary operations</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="285">unary operations</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="286">memory leaks</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="287">left hand casting</indexterm></cindex>
<para><command>ncap2</command> has unique memory requirements due its ability to process
arbitrarily long scripts of any complexity.
All scripts acceptable to <command>ncap2</command> are ultimately processed as a
sequence of binary or unary operations.
<command>ncap2</command> requires <math>MS &lt;= 2VT</math> under most conditions.
An exception to this is when left hand casting (<pxref label="Left-hand-casting"><xrefnodename>Left hand
casting</xrefnodename></pxref>) is used to stretch the size of derived variables beyond the
size of any input variables.
Let <math>VC</math> be the memory required to store the largest variable
defined by left hand casting.
In this case, <math>MS &lt;= 2VC</math>.
</para>
<findex index="fn" spaces=" "><indexterm index="fn" number="4" mergedindex="cp">malloc()</indexterm></findex>
<para><command>ncap2</command> scripts are complete dynamic and may be of arbitrary
length. 
A script that contains many thousands of operations, may uncover a
slow memory leak even though each single operation consumes little
additional memory. 
Memory leaks are usually identifiable by their memory usage signature.
Leaks cause peak memory usage to increase monotonically with time
regardless of script complexity. 
Slow leaks are very difficult to find.
Sometimes a <command>malloc()</command> (or <command>new[]</command>) failure is the
only noticeable clue to their existance.
If you have good reasons to believe that a memory allocation failure  
is ultimately due to an <acronym><acronymword>NCO</acronymword></acronym> memory leak (rather than
inadequate <acronym><acronymword>RAM</acronymword></acronym> on your system), then we would be very
interested in receiving a detailed bug report. 
</para>
</subsection>
</section>
<node name="Performance" spaces=" "><nodename>Performance</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">Memory Requirements</nodeprev><nodeup spaces=" ">Strategies</nodeup></node>
<section spaces=" "><sectiontitle>Performance</sectiontitle>

<cindex index="cp" spaces=" "><indexterm index="cp" number="288">papers</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="289">overview</indexterm></cindex>
<para>An overview of <acronym><acronymword>NCO</acronymword></acronym> capabilities as of about 2006 is in
Zender, C. S. (2008), 
&textldquo;Analysis of Self-describing Gridded Geoscience Data with netCDF Operators (NCO)&textrdquo;,
Environ. Modell. Softw., doi:10.1016/j.envsoft.2008.03.004.
This paper is also available at
<url><urefurl>http://dust.ess.uci.edu/ppr/ppr_Zen08.pdf</urefurl></url>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="290">scaling</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="291">performance</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> performance and scaling for arithmetic operations is
described in 
Zender, C. S., and H. J. Mangalam (2007), 
&textldquo;Scaling Properties of Common Statistical Operators for Gridded Datasets&textrdquo;, 
Int. <w>J. High</w> Perform. Comput. Appl., 21(4), 485-498,
doi:10.1177/1094342007083802. 
This paper is also available at
<url><urefurl>http://dust.ess.uci.edu/ppr/ppr_ZeM07.pdf</urefurl></url>.
</para>
<para>It is helpful to be aware of the aspects of <acronym><acronymword>NCO</acronymword></acronym> design 
that can limit its performance:
</para><enumerate first="1" endspaces=" ">
<listitem> 
<cindex index="cp" spaces=" "><indexterm index="cp" number="292">buffering</indexterm></cindex>
<para>No data buffering is performed during <command>nc_get_var</command> and
<command>nc_put_var</command> operations.  
<cindex index="cp" spaces=" "><indexterm index="cp" number="293">performance</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="294">operator speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="295">speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="296">execution time</indexterm></cindex>
Hyperslabs too large too hold in core memory will suffer substantial
performance penalties because of this. 
</para>
</listitem><listitem> 
<cindex index="cp" spaces=" "><indexterm index="cp" number="297">monotonic coordinates</indexterm></cindex>
<para>Since coordinate variables are assumed to be monotonic, the search for 
bracketing the user-specified limits should employ a quicker algorithm,
like bisection, than the two-sided incremental search currently
implemented.  
</para>
</listitem><listitem> 
<cindex index="cp" spaces=" "><indexterm index="cp" number="298"><var>C_format</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="299"><var>FORTRAN_format</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="300"><var>signedness</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="301"><var>scale_format</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="302"><var>add_offset</var></indexterm></cindex>
<para><var>C_format</var>, <var>FORTRAN_format</var>, <var>signedness</var>,
<var>scale_format</var> and <var>add_offset</var> attributes are ignored by
<command>ncks</command> when printing variables to screen. 
</para>
</listitem><listitem>
<cindex index="cp" spaces=" "><indexterm index="cp" number="303">Yorick</indexterm></cindex>
<para>In the late 1990s it was discovered that some random access operations
on large files on certain architectures (e.g., <acronym><acronymword>UNICOS</acronymword></acronym>) were
much slower with <acronym><acronymword>NCO</acronymword></acronym> than with similar operations performed
using languages that bypass the netCDF interface (e.g., Yorick).    
This may have been a penalty of unnecessary byte-swapping in the netCDF 
interface.  
It is unclear whether such problems exist in present day (2007)
netCDF/<acronym><acronymword>NCO</acronymword></acronym> environments, where unnecessary byte-swapping has
been reduced or eliminated.
</para></listitem></enumerate>

<html endspaces=" ">
&lt;a name=&quot;ftr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ftr &textndash;&gt;
</html>
</section>
</chapter>
<node name="Common-features" spaces=" "><nodename>Common features</nodename><nodenext spaces=" ">Operator Reference Manual</nodenext><nodeprev spaces=" ">Strategies</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<chapter spaces=" "><sectiontitle>NCO Features</sectiontitle>

<para>Many features have been implemented in more than one operator and are
described here for brevity. 
The description of each feature is preceded by a box listing the
operators for which the feature is implemented. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="304">command line switches</indexterm></cindex>
Command line switches for a given feature are consistent across all
operators wherever possible. 
If no &textldquo;key switches&textrdquo; are listed for a feature, then that particular
feature is automatic and cannot be controlled by the user. 
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Internationalization</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Metadata Optimization</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">OpenMP Threading</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Command Line Options</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Specifying Input Files</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Specifying Output Files</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Remote storage</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Retaining Retrieved Files</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">File Formats and Conversion</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Large File Support</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Subsetting Files</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Subsetting Coordinate Variables</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Group Path Editing</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">C and Fortran Index Conventions</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Hyperslabs</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Stride</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Record Appending</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Subcycle</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Multislabs</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Wrapped Coordinates</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Auxiliary Coordinates</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">UDUnits Support</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Rebasing Time Coordinate</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Multiple Record Dimensions</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Missing Values</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Chunking</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Deflation</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">MD5 digests</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Buffer sizes</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">RAM disks</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Packed data</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Operation Types</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Type Conversion</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Batch Mode</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">History Attribute</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">File List Attributes</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">CF Conventions</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ARM Conventions</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Operator Version</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<html endspaces=" ">
&lt;a name=&quot;i18n&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#i18n &textndash;&gt;
</html>
<node name="Internationalization" spaces=" "><nodename>Internationalization</nodename><nodenext spaces=" ">Metadata Optimization</nodenext><nodeprev spaces=" ">Common features</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Internationalization</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="305">Internationalization</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="306">I18N</indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
</para></cartouche>
<cindex index="cp" spaces=" "><indexterm index="cp" number="307">L10N</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> support for <dfn>internationalization</dfn> of textual input
and output (e.g., Warning messages) is nascent.
We hope to produce foreign language string catalogues in 2004.
<!-- c fxm: Work on this section -->
</para>
<html endspaces=" ">
&lt;a name=&quot;hdr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hdr &textndash;&gt;
&lt;a name=&quot;hdr_pad&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hdr_pad &textndash;&gt;
</html>
</section>
<node name="Metadata-Optimization" spaces=" "><nodename>Metadata Optimization</nodename><nodenext spaces=" ">OpenMP Threading</nodenext><nodeprev spaces=" ">Internationalization</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Metadata Optimization</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="308">metadata optimization</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="309">performance</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="310">operator speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="311">speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="312">execution time</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="313"><code>nc__enddef()</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="314"><code>--hdr_pad <var>hdr_pad</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="315"><code>--header_pad <var>hdr_pad</var></code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
Short options: None&linebreak;
Long options: <samp>--hdr_pad</samp>, <samp>--header_pad</samp>&linebreak; 
</para></cartouche>
<para><acronym><acronymword>NCO</acronymword></acronym> supports padding headers to improve the speed of future
metadata operations.
Use the <samp>--hdr_pad</samp> and <samp>--header_pad</samp> switches to request
that <var>hdr_pad</var> bytes be inserted into the metadata section of the
output file.
Future metadata expansions will not incur the netCDF3 performance
penalty of copying the entire output file unless the expansion exceeds
the amount of header padding exceeded.
This can be beneficial when it is known that some metadata will be added
at a future date.
</para>
<para>This optimization exploits the netCDF library <code>nc__enddef()</code>
function, which behaves differently with different versions of netCDF.
It will improve speed of future metadata expansion with <code>CLASSIC</code>
and <code>64bit</code> netCDF files, though not necessarily with <code>NETCDF4</code> 
files, i.e., those created by the netCDF interface to the <acronym><acronymword>HDF5</acronymword></acronym>
library (<pxref label="File-Formats-and-Conversion"><xrefnodename>File Formats and Conversion</xrefnodename></pxref>).
</para>
<html endspaces=" ">
&lt;a name=&quot;omp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#omp &textndash;&gt;
&lt;a name=&quot;openmp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#openmp &textndash;&gt;
</html>
</section>
<node name="OpenMP-Threading" spaces=" "><nodename>OpenMP Threading</nodename><nodenext spaces=" ">Command Line Options</nodenext><nodeprev spaces=" ">Metadata Optimization</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>OpenMP Threading</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="316">OpenMP</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="317">threads</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="318"><acronym><acronymword>SMP</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="319">shared memory parallelism</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="320">parallelism</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="321"><code>nco_openmp_thread_number</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="322"><code>--thr_nbr <var>thr_nbr</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="323"><code>--threads <var>thr_nbr</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="324"><code>--omp_num_threads <var>thr_nbr</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="325"><code>-t <var>thr_nbr</var></code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncap2</command>, <command>ncbo</command>, <command>nces</command>, <command>ncecat</command>,
<command>ncflint</command>, <command>ncpdq</command>, <command>ncra</command>, <command>ncrcat</command>,
<command>ncwa</command>&linebreak;
Short options: <samp>-t</samp>&linebreak;
Long options: <samp>--thr_nbr</samp>, <samp>--threads</samp>,
<samp>--omp_num_threads</samp>&linebreak; 
</para></cartouche>
<para><acronym><acronymword>NCO</acronymword></acronym> supports shared memory parallelism (<acronym><acronymword>SMP</acronymword></acronym>) when
compiled with an OpenMP-enabled compiler.
Threads requests and allocations occur in two stages.
First, users may request a specific number of threads <var>thr_nbr</var> with
the <samp>-t</samp> switch (or its long option equivalents, <samp>--thr_nbr</samp>,
<samp>--threads</samp>, and <samp>--omp_num_threads</samp>).
If not user-specified, OpenMP obtains <var>thr_nbr</var> from the
<code>OMP_NUM_THREADS</code> environment variable, if present, or from the
<acronym><acronymword>OS</acronymword></acronym>, if not.
</para>
<cartouche endspaces=" ">
<para>Caveat:
Unfortunately, threading does not improve <acronym><acronymword>NCO</acronymword></acronym> throughput (i.e.,
wallclock time) because nearly all <acronym><acronymword>NCO</acronymword></acronym> operations are
I/O-bound. 
This means that <acronym><acronymword>NCO</acronymword></acronym> spends negligible time doing anything
compared to reading and writing. 
We have seen some and can imagine other use cases where
<command>ncwa</command>, <command>ncpdq</command>, and <command>ncap2</command> (with long scripts) 
will complete faster due to threading. 
The main benefits of threading so far have been to isolate the serial
from parallel portions of code. 
This parallelism is now exploited by OpenMP but then runs into the I/O
bottleneck during output. 
The bottleneck could be ameliorated for large files by the use of
MPI-enabled calls in the netCDF4 library when the underlying filesystem
is parallel (e.g., <acronym><acronymword>PVFS</acronymword></acronym> or <acronym><acronymword>JFS</acronymword></acronym>).
Implementation of the parallel output calls in <acronym><acronymword>NCO</acronymword></acronym> is not a
goal of our current funding and would require new volunteers or funding.  
</para></cartouche>

<cindex index="cp" spaces=" "><indexterm index="cp" number="326"><var>thr_nbr</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="327"><code>OMP_NUM_THREADS</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="328"><command>ncrcat</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="329"><command>ncwa</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="330"><command>ncap2</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="331"><command>ncpdq</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="332">large datasets</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> may modify <var>thr_nbr</var> according to its own internal
settings before it requests any threads from the system.
Certain operators contain hard-code limits to the number of threads they
request.
We base these limits on our experience and common sense, and to reduce
potentially wasteful system usage by inexperienced users.
For example, <code>ncrcat</code> is extremely I/O-intensive so we restrict
<math><var>thr_nbr</var> &lt;= 2</math> for <code>ncrcat</code>.
This is based on the notion that the best performance that can be
expected from an operator which does no arithmetic is to have one thread
reading and one thread writing simultaneously.
In the future (perhaps with netCDF4), we hope to demonstrate significant
threading improvements with operators like <code>ncrcat</code> by performing
multiple simultaneous writes. 
</para>
<para>Compute-intensive operators (<code>ncap2</code>, <code>ncwa</code> and <code>ncpdq</code>) 
benefit most from threading.
The greatest increases in throughput due to threading occur on
large datasets where each thread performs millions, at least,
of floating point operations.
Otherwise, the system overhead of setting up threads probably outweighs 
the speed enhancements due to <acronym><acronymword>SMP</acronymword></acronym> parallelism.
However, we have not yet demonstrated that the <acronym><acronymword>SMP</acronymword></acronym> parallelism 
scales beyond four threads for these operators.
Hence we restrict <math><var>thr_nbr</var> &lt;= 4</math> for all operators.
We encourage users to play with these limits (edit file
<file>nco_omp.c</file>) and send us their feedback.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="333">debugging</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="334"><var>dbg_lvl</var></indexterm></cindex>
<para>Once the initial <var>thr_nbr</var> has been modified for any
operator-specific limits, <acronym><acronymword>NCO</acronymword></acronym> requests the system to allocate 
a team of <var>thr_nbr</var> threads for the body of the code.
The operating system then decides how many threads to allocate
based on this request.
Users may keep track of this information by running the operator with
<math><var>dbg_lvl</var> &gt; 0</math>.
</para>
<para>By default, threaded operators attach one global attribute,
<code>nco_openmp_thread_number</code>, to any file they create or modify. 
This attribute contains the number of threads the operator used to
process the input files. 
This information helps to verify that the answers with threaded and
non-threaded operators are equal to within machine precision.
<cindex index="cp" spaces=" "><indexterm index="cp" number="335">benchmarks</indexterm></cindex>
This information is also useful for benchmarking.
</para>
<html endspaces=" ">
&lt;a name=&quot;cmd_ln&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cmd_ln &textndash;&gt;
</html>
</section>
<node name="Command-Line-Options" spaces=" "><nodename>Command Line Options</nodename><nodenext spaces=" ">Specifying Input Files</nodenext><nodeprev spaces=" ">OpenMP Threading</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Command Line Options</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="336">command line options</indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
</para></cartouche>
<cindex index="cp" spaces=" "><indexterm index="cp" number="337"><acronym><acronymword>POSIX</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="338"><acronym><acronymword>UNIX</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="339"><acronym><acronymword>GNU</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="340">switches</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> achieves flexibility by using <dfn>command line options</dfn>.
These options are implemented in all traditional <acronym><acronymword>UNIX</acronymword></acronym> commands 
as single letter <dfn>switches</dfn>, e.g., <samp>ls -l</samp>.
For many years <acronym><acronymword>NCO</acronymword></acronym> used only single letter option names.
In late 2002, we implemented <acronym><acronymword>GNU</acronymword></acronym>/<acronym><acronymword>POSIX</acronymword></acronym> extended
or long option names for all options.
This was done in a backward compatible way such that the full
functionality of <acronym><acronymword>NCO</acronymword></acronym> is still available through the familiar 
single letter options.
In the future, however, some features of <acronym><acronymword>NCO</acronymword></acronym> may require the
use of long options, simply because we have nearly run out of single
letter options.
More importantly, mnemonics for single letter options are often
non-intuitive so that long options provide a more natural way of
expressing intent.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="341">long options</indexterm></cindex>
<para>Extended options, also called long options, are implemented using the
system-supplied <file>getopt.h</file> header file, if possible. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="342"><code>BSD</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="343"><code>getopt</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="344"><code>getopt_long</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="345"><file>getopt.h</file></indexterm></cindex>
This provides the <command>getopt_long</command> function to <acronym><acronymword>NCO</acronymword></acronym>
<footnote spaces="\n"><para>If a <command>getopt_long</command> function cannot be found on the system, 
<acronym><acronymword>NCO</acronymword></acronym> will use the <command>getopt_long</command> from the
<command>my_getopt</command> package by Benjamin Sittler
<email><emailaddress>bsittler&arobase;iname.com</emailaddress></email>.
This is <acronym><acronymword>BSD</acronymword></acronym>-licensed software available from  
<uref><urefurl>http://www.geocities.com/ResearchTriangle/Node/9405/#my_getopt</urefurl></uref>.</para></footnote>. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="346"><code>-D <var>debug-level</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="347"><code>--dbg_lvl <var>debug-level</var></code></indexterm></cindex>
<para>The syntax of <dfn>short options</dfn> (single letter options) is
<kbd>-<var>key</var> <var>value</var></kbd> (dash-key-space-value).
Here, <var>key</var> is the single letter option name, e.g., 
<samp>-D 2</samp>.
</para>
<para>The syntax of <dfn>long options</dfn> (multi-letter options) is 
<kbd>--<var>long_name</var> <var>value</var></kbd>
(dash-dash-key-space-value), e.g., <samp>--dbg_lvl 2</samp> or
<kbd>--<var>long_name</var>=<var>value</var></kbd>
(dash-dash-key-equal-value), e.g., <samp>--dbg_lvl=2</samp>.
Thus the following are all valid for the <samp>-D</samp> (short version)
or <samp>--dbg_lvl</samp> (long version) command line option.
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -D 3 in.nc        # Short option
ncks --dbg_lvl=3 in.nc # Long option, preferred form
ncks --dbg_lvl 3 in.nc # Long option, alternate form
</pre></example>
<noindent></noindent>
<para>The last example is preferred for two reasons.
First, <samp>--dbg_lvl</samp> is more specific and less ambiguous than
<samp>-D</samp>.
The long option form makes scripts more self documenting and less error
prone.  
Often long options are named after the source code variable whose value 
they carry.
Second, the equals sign <kbd>=</kbd> joins the key (i.e., <var>long_name</var>) to   
the value in an uninterruptible text block. 
Experience shows that users are less likely to mis-parse commands when
restricted to this form.
</para>
<para><acronym><acronymword>GNU</acronymword></acronym> implements a superset of the <acronym><acronymword>POSIX</acronymword></acronym> standard 
which allows any unambiguous truncation of a valid option to be used.
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -D 3 in.nc        # Short option
ncks --dbg_lvl=3 in.nc # Long option, full form
ncks --dbg=3 in.nc     # Long option, unambiguous truncation
ncks --db=3 in.nc      # Long option, unambiguous truncation
ncks --d=3 in.nc       # Long option, ambiguous truncation
</pre></example>
<noindent></noindent>
<para>The first four examples are equivalent and will work as expected.
The final example will exit with an error since <command>ncks</command> cannot
disambiguate whether <samp>--d</samp> is intended as a truncation of
<samp>--dbg_lvl</samp>, of <samp>--dimension</samp>, or of some other long option. 
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> provides many long options for common switches.
For example, the debugging level may be set in all operators with any
of the switches <samp>-D</samp>, <samp>--debug-level</samp>, or <samp>--dbg_lvl</samp>.
This flexibility allows users to choose their favorite mnemonic.
For some, it will be <samp>--debug</samp> (an unambiguous truncation of
<samp>--debug-level</samp>, and other will prefer <samp>--dbg</samp>.
Interactive users usually prefer the minimal amount of typing, i.e.,
<samp>-D</samp>.
We recommend that scripts which are re-usable employ some form of 
the long options for future maintainability.
</para>
<para>This manual generally uses the short option syntax in examples.
This is for historical reasons and to conserve space in printed output.
Users are expected to pick the unambiguous truncation of each option
name that most suits their taste.
</para>
<html endspaces=" ">
&lt;a name=&quot;fl_in&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#fl_in &textndash;&gt;
&lt;a name=&quot;in&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#in &textndash;&gt;
&lt;a name=&quot;input&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#input &textndash;&gt;
</html>
</section>
<node name="Specifying-Input-Files" spaces=" "><nodename>Specifying Input Files</nodename><nodenext spaces=" ">Specifying Output Files</nodenext><nodeprev spaces=" ">Command Line Options</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Specifying Input Files</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="348">input files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="349">globbing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="350">regular expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="351">wildcards</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="352"><code>NINTAP</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="353">Processor, <acronym><acronymword>CCM</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="354"><acronym><acronymword>CCM</acronymword></acronym> Processor</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="355"><code>-n <var>loop</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="356"><code>--nintap <var>loop</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="357"><code>-p <var>input-path</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="358"><code>--pth <var>input-path</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="359"><code>--path <var>input-path</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="360"><var>input-path</var></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability (<code>-n</code>): <command>nces</command>, <command>ncecat</command>, <command>ncra</command>, <command>ncrcat</command>&linebreak;
Availability (<code>-p</code>): All operators&linebreak;
Short options: <samp>-n</samp>, <samp>-p</samp>&linebreak;
Long options: <samp>--nintap</samp>, <samp>--pth</samp>, <samp>--path</samp>&linebreak;
</para></cartouche>
<para>It is important that users be able to specify multiple input files
without typing every filename in full, often a tedious task even
by graduate student standards.
<cindex index="cp" spaces=" "><indexterm index="cp" number="361"><acronym><acronymword>UNIX</acronymword></acronym></indexterm></cindex>
There are four different ways of specifying input files to <acronym><acronymword>NCO</acronymword></acronym>:
explicitly typing each, using <acronym><acronymword>UNIX</acronymword></acronym> shell wildcards, and using
the <acronym><acronymword>NCO</acronymword></acronym> <samp>-n</samp> and <samp>-p</samp> switches (or their long option
equivalents, <samp>--nintap</samp> or <samp>--pth</samp> and <samp>--path</samp>,
respectively). 
Techniques to augment these methods to specify arbitrary numbers (e.g.,
thousands) and patterns of filenames are discussed separately 
(<pxref label="Large-Numbers-of-Files"><xrefnodename>Large Numbers of Files</xrefnodename></pxref>).
</para>
<para>To illustrate these methods, consider the simple problem of using
<command>ncra</command> to average five input files, <file>85.nc</file>, <file>86.nc</file>,
<w>&dots; <file>89.nc</file></w>, and store the results in <file>8589.nc</file>.
Here are the four methods in order.
They produce identical answers.
</para><example endspaces=" ">
<pre xml:space="preserve">ncra 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra 8[56789].nc 8589.nc
ncra -p <var>input-path</var> 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra -n 5,2,1 85.nc 8589.nc
</pre></example>
<para>The first method (explicitly specifying all filenames) works by brute 
force. 
The second method relies on the operating system shell to <dfn>glob</dfn>
(expand) the <dfn>regular expression</dfn> <code>8[56789].nc</code>.
The shell passes valid filenames which match the expansion to
<command>ncra</command>.
The third method uses the <samp>-p <var>input-path</var></samp> argument to specify 
the directory where all the input files reside.
<acronym><acronymword>NCO</acronymword></acronym> prepends <var>input-path</var> (e.g.,
<file>/data/usrname/model</file>) to all <var>input-files</var> (though not to
<var>output-file</var>).  
Thus, using <samp>-p</samp>, the path to any number of input files need only
be specified once.
Note <var>input-path</var> need not end with <samp>/</samp>; the <samp>/</samp> is
automatically generated if necessary. 
</para>
<para>The last method passes (with <samp>-n</samp>) syntax concisely describing 
the entire set of filenames
<footnote><para>The <samp>-n</samp> option is a backward compatible superset of the
<code>NINTAP</code> option from the <acronym><acronymword>NCAR</acronymword></acronym> <acronym><acronymword>CCM</acronymword></acronym> Processor.</para></footnote>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="362">multi-file operators</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="363">files, multiple</indexterm></cindex>
This option is only available with the <dfn>multi-file operators</dfn>:
<command>ncra</command>, <command>ncrcat</command>, <command>nces</command>, and <command>ncecat</command>.
By definition, multi-file operators are able to process an arbitrary
number of <var>input-files</var>.
This option is very useful for abbreviating lists of filenames
representable as
<var>alphanumeric_prefix</var>+<var>numeric_suffix</var>+<file>.</file>+<var>filetype</var>
where <var>alphanumeric_prefix</var> is a string of arbitrary length and
composition, <var>numeric_suffix</var> is a fixed width field of digits, and
<var>filetype</var> is a standard filetype indicator. 
For example, in the file <file>ccm3_h0001.nc</file>, we have
<var>alphanumeric_prefix</var> = <file>ccm3_h</file>, <var>numeric_suffix</var> =
<file>0001</file>, and <var>filetype</var> = <file>nc</file>.
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> is able to decode lists of such filenames encoded using the
<samp>-n</samp> option. 
The simpler (3-argument) <samp>-n</samp> usage takes the form 
<code>-n <var>file_number</var>,<var>digit_number</var>,<var>numeric_increment</var></code>
where <var>file_number</var> is the number of files, <var>digit_number</var> is
the fixed number of numeric digits comprising the <var>numeric_suffix</var>,
and <var>numeric_increment</var> is the constant, integer-valued difference
between the <var>numeric_suffix</var> of any two consecutive files.
The value of <var>alphanumeric_prefix</var> is taken from the input file,
which serves as a template for decoding the filenames.
In the example above, the encoding <code>-n 5,2,1</code> along with the input
file name <file>85.nc</file> tells <acronym><acronymword>NCO</acronymword></acronym> to
construct five (5) filenames identical to the template <file>85.nc</file>
except that the final two (2) digits are a numeric suffix to be
incremented by one (1) for each successive file.
Currently <var>filetype</var> may be either be empty, <file>nc</file>,
<file>cdf</file>, <file>hdf</file>, or <file>hd5</file>. 
If present, these <var>filetype</var> suffixes (and the preceding <file>.</file>)
are ignored by <acronym><acronymword>NCO</acronymword></acronym> as it uses the <samp>-n</samp> arguments to
locate, evaluate, and compute the <var>numeric_suffix</var> component of
filenames. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="364">wrapped filenames</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="365">climate model</indexterm></cindex>
<para>Recently the <samp>-n</samp> option has been extended to allow convenient
specification of filenames with &textldquo;circular&textrdquo; characteristics.
This means it is now possible for <acronym><acronymword>NCO</acronymword></acronym> to automatically
generate filenames which increment regularly until a specified maximum
value, and then wrap back to begin again at a specified minimum value. 
The corresponding <samp>-n</samp> usage becomes more complex, taking one or
two additional arguments for a total of four or five, respectively: 
<code>-n
<var>file_number</var>,<var>digit_number</var>,<var>numeric_increment</var>[,<var>numeric_max</var>[,<var>numeric_min</var>]]</code>
where <var>numeric_max</var>, if present, is the maximum integer-value of 
<var>numeric_suffix</var> and <var>numeric_min</var>, if present, is the minimum
integer-value of <var>numeric_suffix</var>.
Consider, for example, the problem of specifying non-consecutive input
files where the filename suffixes end with the month index.  
In climate modeling it is common to create summertime and wintertime
averages which contain the averages of the months June&textndash;July&textndash;August,
and December&textndash;January&textndash;February, respectively:
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -n 3,2,1 85_06.nc 85_0608.nc
ncra -n 3,2,1,12 85_12.nc 85_1202.nc
ncra -n 3,2,1,12,1 85_12.nc 85_1202.nc
</pre></example>
<para>The first example shows that three arguments to the <samp>-n</samp> option
suffice to specify consecutive months (<code>06, 07, 08</code>) which do not
&textldquo;wrap&textrdquo; back to a minimum value.
The second example shows how to use the optional fourth and fifth
elements of the <samp>-n</samp> option to specify a wrap value to <acronym><acronymword>NCO</acronymword></acronym>.
The fourth argument to <samp>-n</samp>, if present, specifies the maximum
integer value of <var>numeric_suffix</var>.
In this case the maximum value <w>is 12,</w> and will be formatted as
<file>12</file> in the filename string. 
The fifth argument to <samp>-n</samp>, if present, specifies the minimum
integer value of <var>numeric_suffix</var>.
The default minimum filename suffix <w>is 1,</w> which is formatted as
<file>01</file> in this case.   
Thus the second and third examples have the same effect, that is, they
automatically generate, in order, the filenames <file>85_12.nc</file>,
<file>85_01.nc</file>, and <file>85_02.nc</file> as input to <acronym><acronymword>NCO</acronymword></acronym>.
</para>
<html endspaces=" ">
&lt;a name=&quot;fl_out&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#fl_out &textndash;&gt;
&lt;a name=&quot;out&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#out &textndash;&gt;
&lt;a name=&quot;output&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#output &textndash;&gt;
</html>
</section>
<node name="Specifying-Output-Files" spaces=" "><nodename>Specifying Output Files</nodename><nodenext spaces=" ">Remote storage</nodenext><nodeprev spaces=" ">Specifying Input Files</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Specifying Output Files</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="366">output file</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="367">input files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="368">positional arguments</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="369">command line switches</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="370"><code>-o <var>fl_out</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="371"><code>--output <var>fl_out</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="372"><code>--fl_out <var>fl_out</var></code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
Short options: <samp>-o</samp>&linebreak;
Long options: <samp>--fl_out</samp>, <samp>--output</samp>&linebreak;
</para></cartouche>
<para><acronym><acronymword>NCO</acronymword></acronym> commands produce no more than one output file, <var>fl_out</var>. 
Traditionally, users specify <var>fl_out</var> as the final argument to the
operator, following all input file names. 
This is the <dfn>positional argument</dfn> method of specifying input and
ouput file names.
The positional argument method works well in most applications.
<acronym><acronymword>NCO</acronymword></acronym> also supports specifying <var>fl_out</var> using the command
line switch argument method, <samp>-o <var>fl_out</var></samp>.
</para>
<para>Specifying <var>fl_out</var> with a switch, rather than as a positional
argument, allows <var>fl_out</var> to precede input files in the argument
list. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="373">multi-file operators</indexterm></cindex>
This is particularly useful with multi-file operators for three reasons.
Multi-file operators may be invoked with hundreds (or more) filenames.
Visual or automatic location of <var>fl_out</var> in such a list is
difficult when the only syntactic distinction between input and output
files is their position.
<cindex index="cp" spaces=" "><indexterm index="cp" number="374"><command>xargs</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="375">input files</indexterm></cindex>
Second, specification of a long list of input files may be difficult
(<pxref label="Large-Numbers-of-Files"><xrefnodename>Large Numbers of Files</xrefnodename></pxref>).
Making the input file list the final argument to an operator facilitates 
using <command>xargs</command> for this purpose.
Some alternatives to <command>xargs</command> are very ugly and undesirable.
Finally, many users are more comfortable specifying output files 
with <samp>-o <var>fl_out</var></samp> near the beginning of an argument list.
<cindex index="cp" spaces=" "><indexterm index="cp" number="376">compilers</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="377">linkers</indexterm></cindex>
Compilers and linkers are usually invoked this way.
</para>
<para>Users should specify <var>fl_out</var> using either (not both) method.
If <var>fl_out</var> is specified twice (once with the switch and once as
the last positional argument), then the positional argument takes
precedence. 
</para>
<html endspaces=" ">
&lt;a name=&quot;rmt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rmt &textndash;&gt;
</html>
</section>
<node name="Remote-storage" spaces=" "><nodename>Remote storage</nodename><nodenext spaces=" ">Retaining Retrieved Files</nodenext><nodeprev spaces=" ">Specifying Output Files</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Accessing Remote Files</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="378"><code>rcp</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="379"><code>scp</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="380"><file>.rhosts</file></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="381"><acronym><acronymword>NCAR MSS</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="382"><acronym><acronymword>MSS</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="383">Mass Store System</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="384"><acronym><acronymword>URL</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="385"><code>ftp</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="386"><code>sftp</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="387"><code>wget</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="388">remote files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="389">synchronous file access</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="390">asynchronous file access</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="391"><code>--pth <var>input-path</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="392"><code>--path <var>input-path</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="393"><code>--lcl <var>output-path</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="394"><code>--local <var>output-path</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="395"><code>-l <var>output-path</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="396"><file>.netrc</file></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="397"><code>history</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
Short options: <samp>-p</samp>, <samp>-l</samp>&linebreak;
Long options: <samp>--pth</samp>, <samp>--path</samp>, <samp>--lcl</samp>, <samp>--local</samp>&linebreak;
</para></cartouche>
<para>All <acronym><acronymword>NCO</acronymword></acronym> operators can retrieve files from remote sites as well 
as from the local file system.
<w>A remote</w> site can be an anonymous <acronym><acronymword>FTP</acronymword></acronym> server, a machine on
which the user has <command>rcp</command>, <command>scp</command>, or <command>sftp</command>
privileges, <acronym><acronymword>NCAR</acronymword></acronym>&textrsquo;s Mass Storage System (<acronym><acronymword>MSS</acronymword></acronym>), or
an <acronym><acronymword>OPeNDAP</acronymword></acronym> server.
Examples of each are given below, following a brief description of the 
particular access protocol.
</para>
<html endspaces=" ">
&lt;a name=&quot;ftp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ftp &textndash;&gt;
</html>
<para>To access a file via an anonymous <acronym><acronymword>FTP</acronymword></acronym> server, supply the
remote file&textrsquo;s <acronym><acronymword>URL</acronymword></acronym>.
<acronym><acronymword>FTP</acronymword></acronym> is an intrinsically insecure protocol because it transfers
passwords in plain text format. 
Users should access sites using anonymous <acronym><acronymword>FTP</acronymword></acronym>, or better yet,
secure <acronym><acronymword>FTP</acronymword></acronym> when possible. 
Some <acronym><acronymword>FTP</acronymword></acronym> servers require a login/password combination for a
valid user account.
<acronym><acronymword>NCO</acronymword></acronym> allows these transactions so long as the required
information is stored in the <file>.netrc</file> file. 
Usually this information is the remote machine name, login, and
password, in plain text, separated by those very keywords, e.g.,
</para><example endspaces=" ">
<pre xml:space="preserve">machine dust.ess.uci.edu login zender password bushlied
</pre></example>
<para>Eschew using valuable passwords for <acronym><acronymword>FTP</acronymword></acronym> transactions, since
<file>.netrc</file> passwords are potentially exposed to eavesdropping
software
<footnote><para><acronym><acronymword>NCO</acronymword></acronym> does not implement command line options to
specify <acronym><acronymword>FTP</acronymword></acronym> logins and passwords because copying those data
into the <code>history</code> global attribute in the output file (done by
default) poses an unacceptable security risk. 
</para></footnote>. 
</para>
<html endspaces=" ">
&lt;a name=&quot;sftp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#sftp &textndash;&gt;
</html>
<para><acronym><acronymword>SFTP</acronymword></acronym>, i.e., secure <acronym><acronymword>FTP</acronymword></acronym>, uses <acronym><acronymword>SSH</acronymword></acronym>-based 
security protocols that solve the security issues associated with
plain <acronym><acronymword>FTP</acronymword></acronym>.  
<acronym><acronymword>NCO</acronymword></acronym> supports <acronym><acronymword>SFTP</acronymword></acronym> protocol access to files
specified with a homebrew syntax of the form
</para><example endspaces=" ">
<pre xml:space="preserve">sftp://machine.domain.tld:/path/to/filename
</pre></example>
<para>Note the second colon following the top-level-domain, <code>tld</code>.
This syntax is a hybrid between an <acronym><acronymword>FTP URL</acronymword></acronym> and a standard
remote file syntax.
</para>
<html endspaces=" ">
&lt;a name=&quot;rcp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rcp &textndash;&gt;
&lt;a name=&quot;scp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#scp &textndash;&gt;
</html>
<para>To access a file using <command>rcp</command> or <command>scp</command>, specify the
Internet address of the remote file.
Of course in this case you must have <command>rcp</command> or <command>scp</command>
privileges which allow transparent (no password entry required) access
to the remote machine. 
This means that <file>~/.rhosts</file> or <file>~/ssh/authorized_keys</file> must
be set accordingly on both local and remote machines.   
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="398"><acronym><acronymword>HPSS</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="399"><command>hsi</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="400"><command>msrcp</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="401"><command>msread</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="402"><command>nrnet</command></indexterm></cindex>
<html endspaces=" ">
&lt;a name=&quot;hpss&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hpss &textndash;&gt;
&lt;a name=&quot;HPSS&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#HPSS &textndash;&gt;
&lt;a name=&quot;hsi&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hsi &textndash;&gt;
&lt;a name=&quot;HSI&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#HSI &textndash;&gt;
&lt;a name=&quot;msrcp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#msrcp &textndash;&gt;
&lt;a name=&quot;msread&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#msread &textndash;&gt;
&lt;a name=&quot;nrnet&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#nrnet &textndash;&gt;
</html>
<para>To access a file on a High Performance Storage System (<acronym><acronymword>HPSS</acronymword></acronym>) 
(such as that at <acronym><acronymword>NCAR</acronymword></acronym>, <acronym><acronymword>ECMWF</acronymword></acronym>, <acronym><acronymword>LANL</acronymword></acronym>,
<acronym><acronymword>DKRZ</acronymword></acronym>, <acronym><acronymword>LLNL</acronymword></acronym>) specify the full <acronym><acronymword>HPSS</acronymword></acronym> pathname
of the remote file.  
<acronym><acronymword>NCO</acronymword></acronym> will attempt to detect whether the local machine has direct
(synchronous) <acronym><acronymword>HPSS</acronymword></acronym> access. 
In this case, <acronym><acronymword>NCO</acronymword></acronym> attempts to use the Hierarchical Storage
Interface (<acronym><acronymword>HSI</acronymword></acronym>) command <command>hsi get</command>
<footnote><para>The <command>hsi</command> command must be in the user&textrsquo;s path in one of
the following directories: <code>/usr/local/bin</code>, <code>/opt/hpss/bin</code>,
or <code>/ncar/opt/hpss/hsi</code>.
Tell us if the <acronym><acronymword>HPSS</acronymword></acronym> installation at your site places the
<command>hsi</command> command in a different location, and we will add that
location to the list of acceptable paths to search for <command>hsi</command>.
</para></footnote>.
</para>
<para>The following examples show how one might analyze files stored on  
remote systems.
<!-- c HPSS syntax at http://www2.cisl.ucar.edu/docs/hpss/hsi -->
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -l . ftp://dust.ess.uci.edu/pub/zender/nco/in.nc
ncks -l . sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco/in.nc
ncks -l . dust.ess.uci.edu:/home/zender/nco/data/in.nc
ncks -l . /ZENDER/nco/in.nc
ncks -l . /home/zender/nco/in.nc
ncks -l . http://thredds-test.ucar.edu/thredds/dodsC/testdods/in.nc 
</pre></example>
<noindent></noindent>
<para>The first example works verbatim if your system is connected to the
Internet and is not behind a firewall. 
The second example works if you have <command>sftp</command> access to the
machine <code>dust.ess.uci.edu</code>.
The third example works if you have <command>rcp</command> or <command>scp</command>
access to the machine <code>dust.ess.uci.edu</code>. 
The fourth and fifth examples work on <acronym><acronymword>NCAR</acronymword></acronym> computers with
local access to the <acronym><acronymword>HPSS</acronymword></acronym> <command>hsi get</command> command
<footnote><para><acronym><acronymword>NCO</acronymword></acronym> supported the old <acronym><acronymword>NCAR</acronymword></acronym> Mass Storage
System (<acronym><acronymword>MSS</acronymword></acronym>) until version 4.0.7 in April, 2011.
<acronym><acronymword>NCO</acronymword></acronym> supported <acronym><acronymword>MSS</acronymword></acronym>-retrievals via a variety of
mechanisms including the <command>msread</command>, <command>msrcp</command>, and
<command>nrnet</command> commands invoked either automatically or with sentinels
like <command>ncks -p mss:/ZENDER/nco -l . in.nc</command>.
Once the <acronym><acronymword>MSS</acronymword></acronym> was decommissioned in March, 2011, support for
these retrieval mechanisms was replaced by support for <acronym><acronymword>HPSS</acronymword></acronym>
in <acronym><acronymword>NCO</acronymword></acronym>.
</para></footnote>.
The sixth command works if your local version of <acronym><acronymword>NCO</acronymword></acronym> is
<acronym><acronymword>OPeNDAP</acronymword></acronym>-enabled (this is fully described in <ref label="OPeNDAP"><xrefnodename>OPeNDAP</xrefnodename></ref>),
or if the remote file is accessible via <command>wget</command>.
The above commands can be rewritten using the <samp>-p <var>input-path</var></samp> 
option as follows: 
<cindex index="cp" spaces=" "><indexterm index="cp" number="403"><code>-p <var>input-path</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="404"><var>input-path</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="405"><code>-l <var>output-path</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="406"><var>output-path</var></indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -p ftp://dust.ess.uci.edu/pub/zender/nco -l . in.nc
ncks -p sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco -l . in.nc
ncks -p dust.ess.uci.edu:/home/zender/nco -l . in.nc
ncks -p /ZENDER/nco -l . in.nc
ncks -p /home/zender/nco -l . in.nc # HPSS
ncks -p http://thredds-test.ucar.edu/thredds/dodsC/testdods \ 
     -l . in.nc
</pre></example>
<noindent></noindent>
<para>Using <samp>-p</samp> is recommended because it clearly separates the
<var>input-path</var> from the filename itself, sometimes called the
<dfn>stub</dfn>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="407">stub</indexterm></cindex>
When <var>input-path</var> is not explicitly specified using <samp>-p</samp>,
<acronym><acronymword>NCO</acronymword></acronym> internally generates an <var>input-path</var> from the first
input filename.  
The automatically generated <var>input-path</var> is constructed by stripping 
the input filename of everything following the final <samp>/</samp> character
(i.e., removing the stub).
The <samp>-l <var>output-path</var></samp> option tells <acronym><acronymword>NCO</acronymword></acronym> where to
store the remotely retrieved file.
It has no effect on locally-retrieved files, or on the output file.
Often the path to a remotely retrieved file is quite different than the
path on the local machine where you would like to store the file.
If <samp>-l</samp> is not specified then <acronym><acronymword>NCO</acronymword></acronym> internally generates an
<var>output-path</var> by simply setting <var>output-path</var> equal to
<var>input-path</var> stripped of any machine names.
If <samp>-l</samp> is not specified and the remote file resides on the
<acronym><acronymword>NCAR</acronymword></acronym> <acronym><acronymword>HPSS</acronymword></acronym> system, then the leading character of
<var>input-path</var>, <samp>/</samp>, is also stripped from <var>output-path</var>.
Specifying <var>output-path</var> as <samp>-l ./</samp> tells <acronym><acronymword>NCO</acronymword></acronym> to
store the remotely retrieved file and the output file in the current
directory. 
Note that <samp>-l .</samp> is equivalent to <samp>-l ./</samp> though the latter is
syntactically more clear.
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">OPeNDAP</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<html endspaces=" ">
&lt;a name=&quot;dap&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dap &textndash;&gt;
&lt;a name=&quot;DAP&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#DAP &textndash;&gt;
&lt;a name=&quot;DODS&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#DODS &textndash;&gt;
&lt;a name=&quot;OPeNDAP&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#OPeNDAP &textndash;&gt;
&lt;a name=&quot;dods&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dods &textndash;&gt;
&lt;a name=&quot;opendap&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#opendap &textndash;&gt;
</html>
<node name="OPeNDAP" spaces=" "><nodename>OPeNDAP</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">Remote storage</nodeprev><nodeup spaces=" ">Remote storage</nodeup></node>
<subsection spaces=" "><sectiontitle><acronym><acronymword>OPeNDAP</acronymword></acronym></sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="408"><acronym><acronymword>DAP</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="409"><acronym><acronymword>DODS</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="410"><acronym><acronymword>HTTP</acronymword></acronym> protocol</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="411"><env>DODS_ROOT</env></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="412">Distributed Oceanographic Data System</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="413">oceanography</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="414">data access protocol</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="415">Open-source Project for a Network Data Access Protocol</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="416"><acronym><acronymword>OPeNDAP</acronymword></acronym>.</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="417">server</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="418">client-server</indexterm></cindex>
<para>The Distributed Oceanographic Data System (<acronym><acronymword>DODS</acronymword></acronym>) provides
useful replacements for common data interface libraries like netCDF.
The <acronym><acronymword>DODS</acronymword></acronym> versions of these libraries implement network
transparent access to data via a client-server data access protocol
that uses the <acronym><acronymword>HTTP</acronymword></acronym> protocol for communication.
Although <acronym><acronymword>DODS</acronymword></acronym>-technology originated with oceanography data,
it applyies to virtually all scientific data.
In recognition of this, the data access protocol underlying
<acronym><acronymword>DODS</acronymword></acronym> (which is what <acronym><acronymword>NCO</acronymword></acronym> cares about) has been 
renamed the Open-source Project for a Network Data Access Protocol, 
<acronym><acronymword>OPeNDAP</acronymword></acronym>.
We use the terms <acronym><acronymword>DODS</acronymword></acronym> and <acronym><acronymword>OPeNDAP</acronymword></acronym> interchangeably,
and often write <acronym><acronymword>OPeNDAP</acronymword></acronym>/<acronym><acronymword>DODS</acronymword></acronym> for now. 
In the future we will deprecate <acronym><acronymword>DODS</acronymword></acronym> in favor of
<acronym><acronymword>DAP</acronymword></acronym> or <acronym><acronymword>OPeNDAP</acronymword></acronym>, as appropriate
<footnote spaces="\n"><cindex index="cp" spaces=" "><indexterm index="cp" number="419"><acronym><acronymword>NVODS</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="420">National Virtual Ocean Data System</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="421">open source</indexterm></cindex>
<para><acronym><acronymword>DODS</acronymword></acronym> is being deprecated because it is ambiguous, referring
both to a protocol and to a collection of (oceanography) data.
It is superceded by two terms.
<acronym><acronymword>DAP</acronymword></acronym> is the discipline-neutral Data Access Protocol at the
heart of <acronym><acronymword>DODS</acronymword></acronym>.
The National Virtual Ocean Data System (<acronym><acronymword>NVODS</acronymword></acronym>) refers to the
collection of oceanography data and oceanographic extensions to
<acronym><acronymword>DAP</acronymword></acronym>. 
In other words, <acronym><acronymword>NVODS</acronymword></acronym> is implemented with <acronym><acronymword>OPeNDAP</acronymword></acronym>.
<acronym><acronymword>OPeNDAP</acronymword></acronym> is <emph>also</emph> the open source project which
maintains, develops, and promulgates the <acronym><acronymword>DAP</acronymword></acronym> standard. 
<acronym><acronymword>OPeNDAP</acronymword></acronym> and <acronym><acronymword>DAP</acronymword></acronym> really are interchangeable.
Got it yet?</para></footnote>.
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> may be <acronym><acronymword>DAP</acronymword></acronym>-enabled by linking
<acronym><acronymword>NCO</acronymword></acronym> to the <acronym><acronymword>OPeNDAP</acronymword></acronym> libraries. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="422"><file>Makefile</file></indexterm></cindex>
This is described in the <acronym><acronymword>OPeNDAP</acronymword></acronym> documentation and
automagically implemented in <acronym><acronymword>NCO</acronymword></acronym> build mechanisms
<footnote spaces="\n"><para>Automagic support for <acronym><acronymword>DODS</acronymword></acronym> version 3.2.x was deprecated in 
December, 2003 after <acronym><acronymword>NCO</acronymword></acronym> version 2.8.4.
<acronym><acronymword>NCO</acronymword></acronym> support for <acronym><acronymword>OPeNDAP</acronymword></acronym> versions 3.4.x commenced in
December, 2003, with <acronym><acronymword>NCO</acronymword></acronym> version 2.8.5.
<acronym><acronymword>NCO</acronymword></acronym> support for <acronym><acronymword>OPeNDAP</acronymword></acronym> versions 3.5.x commenced in
June, 2005, with <acronym><acronymword>NCO</acronymword></acronym> version 3.0.1.
<acronym><acronymword>NCO</acronymword></acronym> support for <acronym><acronymword>OPeNDAP</acronymword></acronym> versions 3.6.x commenced in
June, 2006, with <acronym><acronymword>NCO</acronymword></acronym> version 3.1.3.
<acronym><acronymword>NCO</acronymword></acronym> support for <acronym><acronymword>OPeNDAP</acronymword></acronym> versions 3.7.x commenced in
January, 2007, with <acronym><acronymword>NCO</acronymword></acronym> version 3.1.9.</para></footnote>.
The <file>./configure</file> mechanism automatically enables <acronym><acronymword>NCO</acronymword></acronym> as
<acronym><acronymword>OPeNDAP</acronymword></acronym> clients if it can find the required
<acronym><acronymword>OPeNDAP</acronymword></acronym> libraries
<footnote spaces="\n"><para>The minimal set of libraries required to build <acronym><acronymword>NCO</acronymword></acronym> as
<acronym><acronymword>OPeNDAP</acronymword></acronym> clients, where <acronym><acronymword>OPeNDAP</acronymword></acronym> is supplied as a
separate library apart from <file>libnetcdf.a</file>, are, in link order,
<file>libnc-dap.a</file>, <file>libdap.a</file>, and 
<file>libxml2</file> and <file>libcurl.a</file>.</para></footnote>.
 in the usual locations.
The <env>$DODS_ROOT</env> environment variable may be used to override the 
default <acronym><acronymword>OPeNDAP</acronymword></acronym> library location at <acronym><acronymword>NCO</acronymword></acronym>
compile-time.  
Building <acronym><acronymword>NCO</acronymword></acronym> with <file>bld/Makefile</file> and the command
<code>make DODS=Y</code> adds the (non-intuitive) commands to link to the
<acronym><acronymword>OPeNDAP</acronymword></acronym> libraries installed in the <env>$DODS_ROOT</env>
directory.  
The file <file>doc/opendap.sh</file> contains a generic script intended to help
users install <acronym><acronymword>OPeNDAP</acronymword></acronym> before building <acronym><acronymword>NCO</acronymword></acronym>.
The documentation at the 
<uref><urefurl>http://www.opendap.org</urefurl><urefdesc spaces=" ">OPeNDAP Homepage</urefdesc></uref>
is voluminous.
Check there and on the
<uref><urefurl>http://www.unidata.ucar.edu/packages/dods/home/mailLists/</urefurl><urefdesc spaces=" ">DODS mail lists</urefdesc></uref>.
to learn more about the extensive capabilities of <acronym><acronymword>OPeNDAP</acronymword></acronym>
<footnote spaces="\n"><para>We are most familiar with the <acronym><acronymword>OPeNDAP</acronymword></acronym> ability to enable 
network-transparent data access.
<cindex index="cp" spaces=" "><indexterm index="cp" number="423">constraint expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="424">server-side processing</indexterm></cindex>
<acronym><acronymword>OPeNDAP</acronymword></acronym> has many other features, including sophisticated
hyperslabbing and server-side processing via <dfn>constraint expressions</dfn>.
If you know more about this, please consider writing a section
on &quot;<acronym><acronymword>OPeNDAP</acronymword></acronym> Capabilities of Interest to <acronym><acronymword>NCO</acronymword></acronym> Users&quot;
for incorporation in the <cite>NCO User Guide</cite>.</para></footnote>.
</para>
<para>Once <acronym><acronymword>NCO</acronymword></acronym> is <acronym><acronymword>DAP</acronymword></acronym>-enabled the operators are
<acronym><acronymword>OPeNDAP</acronymword></acronym> clients.  
All <acronym><acronymword>OPeNDAP</acronymword></acronym> clients have network transparent access to
any files controlled by a <acronym><acronymword>OPeNDAP</acronymword></acronym> server. 
Simply specify the input file path(s) in <acronym><acronymword>URL</acronymword></acronym> notation and all 
<acronym><acronymword>NCO</acronymword></acronym> operations may be performed on remote files made
accessible by a <acronym><acronymword>OPeNDAP</acronymword></acronym> server. 
This command tests the basic functionality of <acronym><acronymword>OPeNDAP</acronymword></acronym>-enabled  
<acronym><acronymword>NCO</acronymword></acronym> clients: 
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -O -o ~/foo.nc -C -H -v one -l /tmp \
  -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
% ncks -H -v one ~/foo.nc
one = 1
</pre></example>
<para>The <code>one = 1</code> outputs confirm (first) that <command>ncks</command> correctly
retrieved data via the  <acronym><acronymword>OPeNDAP</acronymword></acronym> protocol and (second) that 
<command>ncks</command> created a valid local copy of the subsetted remote file.
With minor changes to the above command, netCDF4 can be used as both the
input and output file format:
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -4 -O -o ~/foo.nc -C -H -v one -l /tmp \
  -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in_4.nc
% ncks -H -v one ~/foo.nc
one = 1
</pre></example>
<para>And, of course, <acronym><acronymword>OPeNDAP</acronymword></acronym>-enabled <acronym><acronymword>NCO</acronymword></acronym> clients continue
to support other, orthogonal features such as UDUnits 
(<pxref label="UDUnits-Support"><xrefnodename>UDUnits Support</xrefnodename></pxref>):
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -u -C -H -v wvl -d wvl,'0.4 micron','0.7 micron' \
  -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in_4.nc
% wvl[0]=5e-07 meter
</pre></example>

<para>The next command is a more advanced example which demonstrates the real
power of <acronym><acronymword>OPeNDAP</acronymword></acronym>-enabled <acronym><acronymword>NCO</acronymword></acronym> clients.
The <command>ncwa</command> client requests an equatorial hyperslab from remotely
stored <acronym><acronymword>NCEP reanalyses data</acronymword></acronym> of the <w>year 1969</w>.
The <acronym><acronymword>NOAA</acronymword></acronym> <acronym><acronymword>OPeNDAP</acronymword></acronym> server (hopefully!) serves these data. 
The local <command>ncwa</command> client then computes and stores (locally) the
regional mean surface pressure <w>(in Pa)</w>. 
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -C -a lat,lon,time -d lon,-10.,10. -d lat,-10.,10. -l /tmp -p \
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.dailyavgs/surface \
  pres.sfc.1969.nc ~/foo.nc
</pre></example>
<noindent></noindent>
<cindex index="cp" spaces=" "><indexterm index="cp" number="425">packing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="426">unpacking</indexterm></cindex>
<para>All with one command!
The data in this particular input file also happen to be packed
(<pxref label="Methods-and-functions"><xrefnodename>Methods and functions</xrefnodename></pxref>), although this is completely transparent
to the user since <acronym><acronymword>NCO</acronymword></acronym> automatically unpacks data before
attempting arithmetic. 
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> obtains remote files from the <acronym><acronymword>OPeNDAP</acronymword></acronym> server
(e.g., <file>www.cdc.noaa.gov</file>) rather than the local machine. 
Input files are first copied to the local machine, then processed.
The <acronym><acronymword>OPeNDAP</acronymword></acronym> server performs data access, hyperslabbing,
and transfer to the local machine.
<cindex index="cp" spaces=" "><indexterm index="cp" number="427">I/O</indexterm></cindex>
This allows the I/O to appear to <acronym><acronymword>NCO</acronymword></acronym> as if the input files
were local.  
The local machine performs all arithmetic operations.
Only the hyperslabbed output data are transferred over the network (to
the local machine) for the number-crunching to begin.
The advantages of this are obvious if you are examining small parts of
large files stored at remote locations.
</para>
<html endspaces=" ">
&lt;a name=&quot;rtn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rtn &textndash;&gt;
</html>
</subsection>
</section>
<node name="Retaining-Retrieved-Files" spaces=" "><nodename>Retaining Retrieved Files</nodename><nodenext spaces=" ">File Formats and Conversion</nodenext><nodeprev spaces=" ">Remote storage</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Retaining Retrieved Files</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="428">file deletion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="429">file removal</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="430">file retention</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="431"><code>-R</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="432"><code>--rtn</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="433"><code>--retain</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
Short options: <samp>-R</samp>&linebreak;
Long options: <samp>--rtn</samp>, <samp>--retain</samp>&linebreak;
</para></cartouche>
<para>In order to conserve local file system space, files retrieved from
remote locations are automatically deleted from the local file system 
once they have been processed.
Many <acronym><acronymword>NCO</acronymword></acronym> operators were constructed to work with numerous
large (e.g., <w>200 MB</w>) files. 
Retrieval of multiple files from remote locations is done serially. 
Each file is retrieved, processed, then deleted before the cycle
repeats.  
In cases where it is useful to keep the remotely-retrieved files on the
local file system after processing, the automatic removal feature may be 
disabled by specifying <samp>-R</samp> on the command line.
</para>
<para>Invoking <code>-R</code> disables the default printing behavior of
<command>ncks</command>.
This allows <command>ncks</command> to retrieve remote files without
automatically trying to print them.
See <ref label="ncks-netCDF-Kitchen-Sink"><xrefnodename>ncks netCDF Kitchen Sink</xrefnodename></ref>, for more details.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="434"><acronym><acronymword>FTP</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="435"><acronym><acronymword>SSH</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="436"><acronym><acronymword>msrcp</acronymword></acronym></indexterm></cindex>
<para>Note that the remote retrieval features of <acronym><acronymword>NCO</acronymword></acronym> can always be
used to retrieve <emph>any</emph> file, including non-netCDF files, via
<command>SSH</command>, anonymous <acronym><acronymword>FTP</acronymword></acronym>, or <command>msrcp</command>.
Often this method is quicker than using a browser, or running an
<acronym><acronymword>FTP</acronymword></acronym> session from a shell window yourself.
<cindex index="cp" spaces=" "><indexterm index="cp" number="437">server</indexterm></cindex>
For example, say you want to obtain a <acronym><acronymword>JPEG</acronymword></acronym> file from a weather
server. 
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -R -p ftp://weather.edu/pub/pix/jpeg -l . storm.jpg
</pre></example>
<noindent></noindent>
<para>In this example, <command>ncks</command> automatically performs an anonymous
<acronym><acronymword>FTP</acronymword></acronym> login to the remote machine and retrieves the specified
file. 
When <command>ncks</command> attempts to read the local copy of <file>storm.jpg</file>
as a netCDF file, it fails and exits, leaving  <file>storm.jpg</file> in
the current directory.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="438"><acronym><acronymword>DODS</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="439">server</indexterm></cindex>
<para>If your <acronym><acronymword>NCO</acronymword></acronym> is <acronym><acronymword>DAP</acronymword></acronym>-enabled (<pxref label="OPeNDAP"><xrefnodename>OPeNDAP</xrefnodename></pxref>),
then you may use <acronym><acronymword>NCO</acronymword></acronym> to retrieve any files (including netCDF,
<acronym><acronymword>HDF</acronymword></acronym>, etc.) served by an <acronym><acronymword>OPeNDAP</acronymword></acronym> server to your local 
machine. 
For example, 
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -R -l . -p \
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.dailyavgs/surface \
  pres.sfc.1969.nc
</pre></example>
<para>It may occasionally be useful to use <acronym><acronymword>NCO</acronymword></acronym> to transfer files
when your other preferred methods are not available locally.
</para>
<html endspaces=" ">
&lt;a name=&quot;fl_fmt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#fl_fmt &textndash;&gt;
&lt;a name=&quot;hdf&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hdf &textndash;&gt;
&lt;a name=&quot;64bit&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#64bit &textndash;&gt;
&lt;a name=&quot;netcdf4&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#netcdf4 &textndash;&gt;
</html>
</section>
<node name="File-Formats-and-Conversion" spaces=" "><nodename>File Formats and Conversion</nodename><nodenext spaces=" ">Large File Support</nodenext><nodeprev spaces=" ">Retaining Retrieved Files</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>File Formats and Conversion</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="440"><acronym><acronymword>HDF</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="441">netCDF2</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="442">netCDF3</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="443">netCDF4</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="444"><code>NETCDF4_CLASSIC</code> files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="445"><code>NETCDF4</code> files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="446"><code>CLASSIC</code> files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="447"><code>64BIT</code> files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="448"><code>--3</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="449"><code>-3</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="450"><code>-4</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="451"><code>-6</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="452"><code>-7</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="453"><code>--4</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="454"><code>--7</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="455"><code>--netcdf4</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="456"><code>--fl_fmt</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="457"><code>--file_format</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="458"><code>--64bit</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncap2</command>, <command>ncbo</command>, <command>nces</command>,
<command>ncecat</command>, <command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>,
<command>ncra</command>, <command>ncrcat</command>, <command>ncwa</command>&linebreak;
Short options: <samp>-3</samp>, <samp>-4</samp>, <samp>-6</samp>, <samp>-7</samp>&linebreak;
Long options: <samp>--3</samp>, <samp>--4</samp>, <samp>--64bit</samp>, <samp>--7</samp>, <samp>--fl_fmt</samp>,
<samp>--netcdf4</samp>&linebreak;  
</para></cartouche>
<para>All <acronym><acronymword>NCO</acronymword></acronym> operators support (read and write) all three (or four, 
depending on how one counts) file formats supported by netCDF4.
The default output file format for all operators is the input file
format. 
The operators listed under &textldquo;Availability&textrdquo; above allow the user to
specify the output file format independent of the input file format. 
These operators allow the user to convert between the various file
formats. 
(The operators <command>ncatted</command> and <command>ncrename</command> do not support
these switches so they always write the output netCDF file in the same
format as the input netCDF file.) 
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">File Formats</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Determining File Format</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">File Conversion</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Autoconversion</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<node name="File-Formats" spaces=" "><nodename>File Formats</nodename><nodenext spaces=" ">Determining File Format</nodenext><nodeprev spaces=" ">File Formats and Conversion</nodeprev><nodeup spaces=" ">File Formats and Conversion</nodeup></node>
<subsection spaces=" "><sectiontitle>File Formats</sectiontitle>
<para>netCDF supports four types of files: <code>CLASSIC</code>, <code>64BIT</code>,
<code>NETCDF4</code>, and <code>NETCDF4_CLASSIC</code>, 
The <code>CLASSIC</code> format is the traditional 32-bit offset written by
netCDF2 and netCDF3.
As of 2005, nearly all netCDF datasets were in <code>CLASSIC</code> format. 
The <code>64BIT</code> format was added in Fall, 2004.
As of 2010, many netCDF datasets were in <code>64BIT</code> format. 
As of 2013, many netCDF datasets are in <code>NETCDF4_CLASSIC</code> format. 
</para>
<para>The <code>NETCDF4</code> format uses <acronym><acronymword>HDF5</acronymword></acronym> as the file storage layer. 
The files are (usually) created, accessed, and manipulated using the 
traditional netCDF3 <acronym><acronymword>API</acronymword></acronym> (with numerous extensions).
The <code>NETCDF4_CLASSIC</code> format refers to netCDF4 files created with
the <code>NC_CLASSIC_MODEL</code> mask.
Such files use <acronym><acronymword>HDF5</acronymword></acronym> as the back-end storage format (unlike
netCDF3), though they incorporate only netCDF3 features.
Hence <code>NETCDF4_CLASSIC</code> files are entirely readable by applications 
that use only the netCDF3 <acronym><acronymword>API</acronymword></acronym> (though the applications must be
linked with the netCDF4 library).
<acronym><acronymword>NCO</acronymword></acronym> must be built with <w>netCDF4</w> to write files in the new
<code>NETCDF4</code> and <code>NETCDF4_CLASSIC</code> formats, and to read files in
these formats. 
Datasets in the default <code>CLASSIC</code> or the newer <code>64BIT</code> formats 
have maximum backwards-compatibility with older applications. 
<acronym><acronymword>NCO</acronymword></acronym> has deep support for <code>NETCDF4</code> formats.
If performance or disk-space as important as backwards compatibility,
then use <code>NETCDF4_CLASSIC</code> instead of <code>CLASSIC</code> format files. 
As of 2014, <acronym><acronymword>NCO</acronymword></acronym> support for the <code>NETCDF4</code> format is
nearly complete and the most powerful and disk/<acronym><acronymword>RAM</acronymword></acronym> efficient 
workflows will utilize this format.
</para>
<para>As mentioned above, all operators write use the input file format for
output files unless told otherwise.
Toggling the short option <samp>-6</samp> or the long option <samp>--64bit</samp>
switch (or its <var>key</var>-<var>value</var> equivalent <samp>--fl_fmt=64bit</samp>)
produces the netCDF3 64-bit offset format named <code>64BIT</code>. 
<acronym><acronymword>NCO</acronymword></acronym> must be built with <w>netCDF 3.6</w> or higher to produce
a <code>64BIT</code> file.
Using the <samp>-4</samp> switch (or its long option equivalents
<samp>--4</samp> or <samp>--netcdf4</samp>), or setting its <var>key</var>-<var>value</var>
equivalent <samp>--fl_fmt=netcdf4</samp> produces a <code>NETCDF4</code> file
(i.e., with all supported <acronym><acronymword>HDF5</acronymword></acronym> features).
Using the <samp>-7</samp> switch (or its long option equivalent
<samp>--7</samp>
<footnote spaces="\n"><para>The reason (and mnemonic) for <samp>-7</samp> is that <code>NETCDF4_CLASSIC</code>
files include great features of both netCDF3 (compatibility) and
netCDF4 (compression, chunking) and, well, <math>3+4=7</math>.</para></footnote>, or 
setting its <var>key</var>-<var>value</var> equivalent
<samp>--fl_fmt=netcdf4_classic</samp> produces a <code>NETCDF4_CLASSIC</code>  
file (i.e., with all supported <acronym><acronymword>HDF5</acronymword></acronym> features like compression
and chunking but without groups or new atomic types).
Operators given the <samp>-3</samp> (or <samp>--3</samp>) switch without arguments
will (attempt to) produce netCDF3 <code>CLASSIC</code> output, even from
netCDF4 input files.  
</para>
<para>Note that <code>NETCDF4</code> and <code>NETCDF4_CLASSIC</code> are the same
binary format. 
The latter simply causes a writing application to fail if it attempts to 
write a <code>NETCDF4</code> file that cannot be completely read by the
netCDF3 library. 
Conversely, <code>NETCDF4_CLASSIC</code> indicates to a reading application
that all of the file contents are readable with the netCDF3 library. 
<acronym><acronymword>NCO</acronymword></acronym> has supported reading/writing basic <code>NETCDF4</code> and
<code>NETCDF4_CLASSIC</code> files since October, 2005.
</para>
<html endspaces=" ">
&lt;a name=&quot;fmt_inq&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#fmt_inq &textndash;&gt;
</html>
</subsection>
<node name="Determining-File-Format" spaces=" "><nodename>Determining File Format</nodename><nodenext spaces=" ">File Conversion</nodenext><nodeprev spaces=" ">File Formats</nodeprev><nodeup spaces=" ">File Formats and Conversion</nodeup></node>
<subsection spaces=" "><sectiontitle>Determining File Format</sectiontitle>
<para>Input files often end with the generic <code>.nc</code> suffix that leaves
(perhaps by intention) the internal file format ambiguous.
There are at least three ways to discover the internal format of a
netCDF-supported file.
These methods determine whether it is a classic (32-bit offset) or newer
64-bit offset netCDF3 format, or is a netCDF4 format. 
Each method returns the information using slightly different terminology 
that becomes easier to understand with practice.
</para>
<para>First, examine the first line of global metadata output by <samp>ncks -M</samp>:  
<cindex index="cp" spaces=" "><indexterm index="cp" number="459">netCDF3 classic file format</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="460">netCDF4 classic file format</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="461">netCDF4 file format</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="462">32-bit offset file format</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="463">64-bit offset file format</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="464"><command>ncks</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="465"><code>-M</code></indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -M foo_3c.nc
Summary of foo_3c.nc: filetype = NC_FORMAT_CLASSIC, 0 groups ...
% ncks -M foo_364.nc
Summary of foo_364.nc: filetype = NC_FORMAT_64BIT, 0 groups ...
% ncks -M foo_4c.nc
Summary of foo_4c.nc: filetype = NC_FORMAT_NETCDF4_CLASSIC, 0 groups ...
% ncks -M foo_4.nc
Summary of foo_4.nc: filetype = NC_FORMAT_NETCDF4, 0 groups ...
</pre></example>
<para>This method requires a netCDF4-enabled <acronym><acronymword>NCO</acronymword></acronym> version 3.9.0+
(i.e., from 2007 or later).
<cindex index="cp" spaces=" "><indexterm index="cp" number="466">extended file format</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="467">underlying file format</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="468"><code>NC_FORMAT_NC3</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="469"><code>NC_FORMAT_NC_HDF5</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="470"><code>NC_FORMAT_NC_HDF4</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="471"><code>NC_FORMAT_PNETCDF</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="472"><code>NC_FORMAT_DAP2</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="473"><code>NC_FORMAT_DAP4</code></indexterm></cindex>
As of <acronym><acronymword>NCO</acronymword></acronym> version 4.4.0 (January, 2014), <command>ncks</command> will
also print the extended or underlying format of the input file.
The extended filetype will be one of the six underlying formats that
are accessible through the netCDF <acronym><acronymword>API</acronymword></acronym>.
These formats are
<code>NC_FORMAT_NC3</code> (classic and 64-bit versions of netCDF3 formats),
<code>NC_FORMAT_NC_HDF5</code> (classic and extended versions of netCDF4, and
&textldquo;pure&textrdquo; HDF5 format),
<code>NC_FORMAT_NC_HDF4</code> (HDF4 format),
<code>NC_FORMAT_PNETCDF</code> (PnetCDF format),
<code>NC_FORMAT_DAP2</code> (accessed via DAP2 protocol), and
<code>NC_FORMAT_DAP4</code> (accessed via DAP2 protocol).
For example,
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -D 2 -M hdf.hdf
Summary of hdf.hdf: filetype = NC_FORMAT_NETCDF4 (representation of \
  extended/underlying filetype NC_FORMAT_HDF4), 0 groups ...
% ncks -D 2 -M http://thredds-test.ucar.edu/thredds/dodsC/testdods/in.nc
Summary of http://thredds-test.ucar.edu/thredds/dodsC/testdods/in.nc: \
  filetype = NC_FORMAT_CLASSIC (representation of extended/underlying \
  filetype NC_FORMAT_DAP2), 0 groups  
% ncks -D 2 -M foo_4.nc
Summary of foo_4.nc: filetype = NC_FORMAT_NETCDF4 (representation of \
  extended/underlying filetype NC_FORMAT_HDF5), 0 groups  
</pre></example>
<para>The extended filetype determines some of the capabilities that netCDF
has to alter the file.
</para>
<para>Second, query the file with <samp>ncdump -k</samp>:
<cindex index="cp" spaces=" "><indexterm index="cp" number="474"><command>ncdump</command></indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">% ncdump -k foo_3c.nc
classic
% ncdump -k foo_364.nc
64-bit-offset
% ncdump -k foo_4c.nc
netCDF-4 classic model
% ncdump -k foo_4.nc
netCDF-4
</pre></example>
<para>This method requires a netCDF4-enabled <acronym><acronymword>netCDF</acronymword></acronym> 3.6.2+
(i.e., from 2007 or later).
</para>
<para>The third option uses the POSIX-standard <command>od</command> (octal dump)
command:   
<cindex index="cp" spaces=" "><indexterm index="cp" number="475"><command>od</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="476">octal dump</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">% od -An -c -N4 foo_3c.nc
   C   D   F 001
% od -An -c -N4 foo_364.nc
   C   D   F 002
% od -An -c -N4 foo_4c.nc
 211   H   D   F
% od -An -c -N4 foo_4.nc
 211   H   D   F
</pre></example>
<para>This option works without <acronym><acronymword>NCO</acronymword></acronym> and <command>ncdump</command>.
Values of <samp>C D F 001</samp> and <samp>C D F 002</samp> indicate 32-bit
(classic) and 64-bit netCDF3 formats, respectively, while values of
<samp>211 H D F</samp> indicate either of the newer netCDF4 file formats.
</para>
<html endspaces=" ">
&lt;a name=&quot;hdf2nc&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hdf2nc &textndash;&gt;
&lt;a name=&quot;3to4&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#3to4 &textndash;&gt;
&lt;a name=&quot;4to3&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#4to3 &textndash;&gt;
</html>
</subsection>
<node name="File-Conversion" spaces=" "><nodename>File Conversion</nodename><nodenext spaces=" ">Autoconversion</nodenext><nodeprev spaces=" ">Determining File Format</nodeprev><nodeup spaces=" ">File Formats and Conversion</nodeup></node>
<subsection spaces=" "><sectiontitle>File Conversion</sectiontitle>
<para>Let us demonstrate converting a file from any netCDF-supported
input format into any netCDF output format (subject to limits of the
output format).  
Here the input file <file>in.nc</file> may be in any of these formats:
netCDF3 (classic and 64bit), netCDF4 (classic and extended), HDF4, HDF5,
HDF-EOS (version 2 or 5), and DAP. 
The switch determines the output format written in the comment:
</para><example endspaces=" ">
<pre xml:space="preserve">ncks --fl_fmt=classic in.nc foo_3c.nc # netCDF3 classic
ncks --fl_fmt=64bit in.nc foo_364.nc # netCDF3 64bit
ncks --fl_fmt=netcdf4_classic in.nc foo_4c.nc # netCDF4 classic
ncks --fl_fmt=netcdf4 in.nc foo_4.nc # netCDF4 
ncks -3 in.nc foo_3c.nc # netCDF3 classic
ncks --3 in.nc foo_3c.nc # netCDF3 classic
ncks -6 in.nc foo_364.nc # netCDF3 64bit
ncks --64 in.nc foo_364.nc # netCDF3 64bit
ncks -4 in.nc foo_4.nc # netCDF4 
ncks --4 in.nc foo_4.nc # netCDF4 
ncks -7 in.nc foo_4c.nc # netCDF4 classic
ncks --7 in.nc foo_4c.nc # netCDF4 classic
</pre></example>
<para>Of course since most operators support these switches, the
&textldquo;conversions&textrdquo; can be done at the output stage of arithmetic
or metadata processing rather than requiring a separate step.
Producing (netCDF3) <code>CLASSIC</code> or <code>64BIT</code> files from
<code>NETCDF4_CLASSIC</code> files will always work. 
</para>
<html endspaces=" ">
&lt;a name=&quot;autoconversion&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#autoconversion &textndash;&gt;
&lt;a name=&quot;autocnv&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#autocnv &textndash;&gt;
</html>
</subsection>
<node name="Autoconversion" spaces=" "><nodename>Autoconversion</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">File Conversion</nodeprev><nodeup spaces=" ">File Formats and Conversion</nodeup></node>
<subsection spaces=" "><sectiontitle>Autoconversion</sectiontitle>
<para>Because of the dearth of support for netCDF4 amongst tools and user
communities (including the <acronym><acronymword>CF</acronymword></acronym> conventions), it is often useful
to convert netCDF4 to netCDF3 for certain applications.
Until <acronym><acronymword>NCO</acronymword></acronym> version 4.4.0 (January, 2014), producing netCDF3
files from netCDF4 files only worked if the input files contained no 
netCDF4-specific features (e.g., atomic types, multiple record
dimensions, or groups). 
As of <acronym><acronymword>NCO</acronymword></acronym> version 4.4.0, <command>ncks</command> supports
<dfn>autoconversion</dfn> of many netCDF4 features to their closest
netCDF3-compatible representations.
Since converting netCDF4 to netCDF3 results in loss of features, 
&textldquo;automatic down-conversion&textrdquo; may be a more precise description of what  
we term autoconversion.
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> employs three algorithms to downconvert netCDF4 to netCDF3:
</para><enumerate first="1" endspaces=" ">
<listitem> 
<cindex index="cp" spaces=" "><indexterm index="cp" number="477">autoconversion</indexterm></cindex>
<para>Autoconversion of atomic types:
Autoconversion automatically promotes <code>NC_UBYTE</code> to <code>NC_SHORT</code>, 
and <code>NC_USHORT</code> to <code>NC_INT</code>.
It automatically demotes the three types <code>NC_UINT</code>,
<code>NC_UINT64</code>, and <code>NC_INT64</code> to <code>NC_INT</code>. 
And it converts <code>NC_STRING</code> to <code>NC_CHAR</code>.
All numeric conversions work for attributes and variables of any rank.
Two numeric types (<code>NC_UBYTE</code> and <code>NC_USHORT</code>) are
<emph>promoted</emph> to types with greater range (and greater storage). 
This extra range is often not used so promotion perhaps conveys
the wrong impression.
However, promotion never truncates values or loses data (this perhaps
justifies the extra storage).
Three numeric types (<code>NC_UINT</code>, <code>NC_UINT64</code> and
<code>NC_INT64</code>) are <emph>demoted</emph>.
Since the input range is larger than the output range, demotion can
result in numeric truncation and thus loss of data. 
In such cases, it would possible to convert the data to floating point
values instead. 
If this feature interests you, please be the squeaky wheel and let us
know.
</para>
<para>String conversions (to <code>NC_CHAR</code>) work for all attributes, but
not for variables.
This is because attributes are at most one-dimensional and may be of any
size whereas variables require gridded dimensions that usually do not
fit the ragged sizes of text strings.
Hence scalar <code>NC_STRING</code> attributes are correctly converted to and
stored as <code>NC_CHAR</code> attributes in the netCDF3 output file, but
<code>NC_STRING</code> variables are not correctly converted. 
If this limitation annoys or enrages you, please let us know by being
the squeaky wheel.
</para>
</listitem><listitem>
<cindex index="cp" spaces=" "><indexterm index="cp" number="478"><code>--fix_rec_dmn all</code></indexterm></cindex>
<para>Convert multiple record dimensions to fixed-size dimensions.
Many netCDF4 and <acronym><acronymword>HDF5</acronymword></acronym> datasets have multiple unlimited
dimensions.
Since a netCDF3 file may have at most one unlimited dimension, all but
possibly one unlimited dimension from the input file must be converted
to fixed-length dimensions prior to storing netCDF4 input as netCDF3
output.
By invoking <code>--fix_rec_dmn all</code> the user ensures the output file
will adhere to netCDF3 conventions and the user need not know the names
of the specific record dimensions to fix.
See <ref label="ncks-netCDF-Kitchen-Sink"><xrefnodename>ncks netCDF Kitchen Sink</xrefnodename></ref> for a description of the
<samp>--fix_rec_dmn</samp> option. 
</para>
</listitem><listitem>
<cindex index="cp" spaces=" "><indexterm index="cp" number="479">flattening</indexterm></cindex>
<para>Flattening (removal) of groups.
Many netCDF4 and <acronym><acronymword>HDF5</acronymword></acronym> datasets have group hierarchies.
Since a netCDF3 file may not have any groups, groups in the input file
must be removed.
This is also called &textldquo;flattening&textrdquo; the hierarchical file.
See <ref label="Group-Path-Editing"><xrefnodename>Group Path Editing</xrefnodename></ref> for a description of the <acronym><acronymword>GPE</acronymword></acronym>
option <samp>-G :</samp> to flatten files.
</para></listitem></enumerate>

<para>Putting the three algorithms together, one sees that the recipe to 
convert netCDF4 to netCDF4 becomes increasingly complex as the netCDF4
features in the input file become more elaborate:
</para><example endspaces=" ">
<pre xml:space="preserve"># Convert file with netCDF4 atomic types
ncks -3 in.nc4 out.nc3
# Convert file with multiple record dimensions + netCDF4 atomic types
ncks -3 --fix_rec_dmn=all in.nc4 out.nc3
# Convert file with groups, multiple record dimensions + netCDF4 atomic types
ncks -3 -G : --fix_rec_dmn=all in.nc4 out.nc3
</pre></example>
<para>Future versions of <acronym><acronymword>NCO</acronymword></acronym> may automatically invoke the record
dimension fixation and group flattening when converting to netCDF3
(rather than requiring it be specified manually).
If this feature would interest you, please let us know.
</para>
<html endspaces=" ">
&lt;a name=&quot;lfs&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#lfs &textndash;&gt;
</html>
</subsection>
</section>
<node name="Large-File-Support" spaces=" "><nodename>Large File Support</nodename><nodenext spaces=" ">Subsetting Files</nodenext><nodeprev spaces=" ">File Formats and Conversion</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Large File Support</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="480">LFS</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="481">Large File Support</indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
Short options: none&linebreak;
Long options: none&linebreak;
</para></cartouche>
<para><acronym><acronymword>NCO</acronymword></acronym> has Large File Support (<acronym><acronymword>LFS</acronymword></acronym>), meaning that 
<acronym><acronymword>NCO</acronymword></acronym> can write files larger than <w>2 GB</w> on some 32-bit
operating systems with netCDF libraries earlier than <w>version 3.6</w>. 
If desired, LFS support must be configured when both netCDF and
<acronym><acronymword>NCO</acronymword></acronym> are installed.
netCDF <w>versions 3.6</w> and higher support 64-bit file addresses as part
of the netCDF standard.
We recommend that users ignore LFS support which is difficult to
configure and is implemented in <acronym><acronymword>NCO</acronymword></acronym> only to support netCDF
versions prior <w>to 3.6</w>.  
This obviates the need for configuring explicit LFS support in
applications (such as <acronym><acronymword>NCO</acronymword></acronym>) that now support 64-bit files   
directly through the netCDF interface.
See <ref label="File-Formats-and-Conversion"><xrefnodename>File Formats and Conversion</xrefnodename></ref> for instructions on accessing 
the different file formats, including 64-bit files, supported by the
modern netCDF interface. 
</para>
<para>If you are still interested in explicit LFS support for netCDF versions
prior <w>to 3.6</w>, know that LFS support depends on a complex,
interlocking set of operating system  
<footnote spaces="\n"><para>Linux and <acronym><acronymword>AIX</acronymword></acronym> do support <acronym><acronymword>LFS</acronymword></acronym>.</para></footnote>
and netCDF support issues.
The netCDF LFS 
<uref><urefurl>http://my.unidata.ucar.edu/content/software/netcdf/faq-lfs.html</urefurl><urefdesc>FAQ</urefdesc></uref>
describes the various file size limitations imposed by different
versions of the netCDF standard.
<acronym><acronymword>NCO</acronymword></acronym> and netCDF automatically attempt to configure LFS at
build time. 
</para>
<html endspaces=" ">
&lt;a name=&quot;grp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#grp &textndash;&gt;
&lt;a name=&quot;var&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#var &textndash;&gt;
&lt;a name=&quot;xcl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xcl &textndash;&gt;
&lt;a name=&quot;sbs&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#sbs &textndash;&gt;
&lt;a name=&quot;subset&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#subset &textndash;&gt;
</html>
</section>
<node name="Subsetting-Files" spaces=" "><nodename>Subsetting Files</nodename><nodenext spaces=" ">Subsetting Coordinate Variables</nodenext><nodeprev spaces=" ">Large File Support</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Subsetting Files</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="482">subsetting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="483">union</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="484">intersection</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="485">exclusion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="486">extraction</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="487"><code>-v <var>var</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="488"><code>--variable <var>var</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="489"><code>-g <var>grp</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="490"><code>--grp <var>grp</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="491"><code>--group <var>grp</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="492"><code>-x</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="493"><code>--exclude</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="494"><code>--xcl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="495"><code>--unn</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="496"><code>--union</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Options <code>--unn</code>&linebreak;
Availability: <command>ncbo</command>, <command>nces</command>,
<command>ncecat</command>, <command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>,
<command>ncra</command>, <command>ncrcat</command>, <command>ncwa</command>&linebreak; 
Short options: &linebreak;
Long options: <samp>--unn</samp> and <samp>--union</samp>&linebreak;
Options <code>-g <var>grp</var></code>&linebreak;
Availability: <command>ncbo</command>, <command>nces</command>,
<command>ncecat</command>, <command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>,
<command>ncra</command>, <command>ncrcat</command>, <command>ncwa</command>&linebreak; 
Short options: <samp>-g</samp>&linebreak;
Long options: <samp>--grp</samp> and <samp>--group</samp>&linebreak;
Options <code>-v <var>var</var></code> and <code>-x</code>&linebreak;
Availability: (<command>ncap2</command>), <command>ncbo</command>, <command>nces</command>,
<command>ncecat</command>, <command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>,
<command>ncra</command>, <command>ncrcat</command>, <command>ncwa</command>&linebreak;
Short options: <samp>-v</samp>, <samp>-x</samp>&linebreak;
Long options: <samp>--variable</samp>, <samp>--exclude</samp> or <samp>--xcl</samp>&linebreak;
</para></cartouche>
<para>Subsetting variables refers to explicitly specifying variables and
groups to be included or excluded from operator actions.
Subsetting is controlled by the <samp>-v <var>var</var>[,&dots;]</samp> and
<samp>-x</samp> options for directly specifying variables.  
Specifying groups, whether in addition to or instead of variables,
is quite similar and is controlled by the <samp>-g <var>grp</var>[,&dots;]</samp>
and <samp>-x</samp> options.
<w>A list</w> of variables or groups to extract is specified following the
<samp>-v</samp> and <samp>-g</samp> options, e.g., <samp>-v time,lat,lon</samp> or
<samp>-g grp1,grp2</samp>.
Both options may be specified simultaneously and <acronym><acronymword>NCO</acronymword></acronym> will
extract the intersection of the lists, i.e., only variables of the
specified names found in groups of the specified names.
The <samp>--unn</samp> option causes <acronym><acronymword>NCO</acronymword></acronym> to extract the
union, rather than the intersection, of the specified groups and
variables. 
Not using the <samp>-v</samp> or <samp>-g</samp> option is equivalent to specifying
all variables or groupp, respectively.  
The <samp>-x</samp> option causes the list of variables specified with
<samp>-v</samp> to be <emph>excluded</emph> rather than <emph>extracted</emph>.
Thus <samp>-x</samp> saves typing when you only want to extract fewer than
half of the variables in a file.
</para>
<para>Variables or groups explicitly specified for extraction with
<samp>-v <var>var</var>[,&dots;]</samp> or <samp>-g <var>grp</var>[,&dots;]</samp>
<emph>must</emph> be present in the input file or an error will result.
Variables explicitly specified for <emph>exclusion</emph> with
<samp>-x -v <var>var</var>[,&dots;]</samp> need not be present in the input
file.
To accord with the sophistication of the underlying hierarchy,
group subsetting is controlled by a few powerful yet subtle syntactical
distinctions.
When learning this syntax it is helpful to keep in mind the similarity
between group hierarchies and directory structures. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="497"><command>mv</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="498"><command>cp</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="499">recursion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="500">recursive</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="501">anchor</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="502">anchoring</indexterm></cindex>
<html endspaces=" ">
&lt;a name=&quot;rcr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rcr &textndash;&gt;
&lt;a name=&quot;recursion&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#recursion &textndash;&gt;
&lt;a name=&quot;recursive&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#recursive &textndash;&gt;
&lt;a name=&quot;ncr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncr &textndash;&gt;
&lt;a name=&quot;anchor&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#anchor &textndash;&gt;
&lt;a name=&quot;anchoring&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#anchoring &textndash;&gt;
</html>
<para>Two properties of subsetting, recursion and anchoring, are best
illustrated by reminding the user of their <acronym><acronymword>UNIX</acronymword></acronym> equivalents.
The <acronym><acronymword>UNIX</acronymword></acronym> command <command>mv src dst</command> moves <file>src</file>
<emph>and all its subdirectories</emph> (and all their subdirectories etc.)
to <file>dst</file>.
In other words <command>mv</command> is, by default, <emph>recursive</emph>.
In contrast, the <acronym><acronymword>UNIX</acronymword></acronym> command <command>cp src dst</command> moves
<file>src</file>, and only <file>src</file>, to <file>dst</file>,
If <file>src</file> is a directory, not a file, then that command fails.
One must explicitly request to copy directories recursively, i.e.,
with <command>cp -r src dst</command>.
In <acronym><acronymword>NCO</acronymword></acronym> recursive extraction (and copying) of groups is the
default (like with <command>mv</command>, not with <command>cp</command>).
Recursion is turned off by appending a trailing slash to the path.
</para>
<para>These <acronym><acronymword>UNIX</acronymword></acronym> commands also illustrate a property we call
<emph>anchoring</emph>. 
The command <command>mv src dst</command> moves (recursively) the source
directory <file>src</file> to the destination directory <file>dst</file>. 
If <file>src</file> begins with the slash character then the specified path is
relative to the root directory, otherwise the path is relative to the
current working directory. 
In other words, an initial slash character anchors the subsequent path
to the root directory.
In <acronym><acronymword>NCO</acronymword></acronym> an initial slash anchors the path at the root group.
Paths that begin and end with slash characters (e.g., <file>//</file>,
<file>/g1/</file>, and <file>/g1/g2/</file>) are both anchored and non-recursive. 
</para>
<para>Consider the following commands, all of which may be assumed to end with
<samp>in.nc out.nc</samp>:
</para><example endspaces=" ">       
<pre xml:space="preserve">ncks -g  g1  # Extract, recursively, all groups with a g1 component
ncks -g  g1/ # Extract, non-recursively, all groups terminating in g1
ncks -g /g1  # Extract, recursively, root group g1
ncks -g /g1/ # Extract, non-recursively root group g1
ncks -g //   # Extract, non-recursively the root group
</pre></example>
<para>The first command is probably the most useful and common.
It would extract these groups, if present, and all their direct
ancestors and children:
<file>/g1</file>, <file>/g2/g1</file>, and <file>/g3/g1/g2</file>.
In other words, the simplest form of <samp>-g grp</samp> grabs all groups that 
(and their direct ancestors and children, recursively) that have
<file>grp</file> as a complete component of their path.
A simple string match is insufficient, <var>grp</var> must be a complete 
component (i.e., group name) in the path.
The option <samp>-g g1</samp> would not extract these groups because <file>g1</file> 
is not a complete component of the path: <file>/g12</file>, <file>/fg1</file>, and
<file>/g1g1</file>.
The second command above shows how a terminating slash character
<kbd>/</kbd> cancels the recursive copying of groups.
An argument to <samp>-g</samp> which terminates with a slash character
extracts the group and its direct ancestors, but none of its children. 
The third command above shows how an initial slash character <kbd>/</kbd>
anchors the argument to the root group.
The third command would not extract the group <file>/g2/g1</file> because 
the <file>g1</file> group is not at the root level, but it would extract,
any group <file>/g1</file> at the root level and all its children,
recursively.  
The fourth command is the non-recursive version of the third command.
The fifth command is a special case of the fourth command.
</para>
<html endspaces=" ">
&lt;a name=&quot;unn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#unn &textndash;&gt;
&lt;a name=&quot;nsx&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#nsx &textndash;&gt;
&lt;a name=&quot;union&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#union &textndash;&gt;
&lt;a name=&quot;intersection&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#intersection &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="503">union</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="504">intersection</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="505"><code>--unn</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="506"><code>--union</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="507"><code>--nsx</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="508"><code>--intersection</code></indexterm></cindex>
<para>As mentioned above, both <samp>-v</samp> and <samp>-g</samp> options may be
specified simultaneously and <acronym><acronymword>NCO</acronymword></acronym> will, by default, extract the
intersection of the lists, i.e., the specified variables found in the
specified groups
<footnote spaces="\n"><para>Intersection-mode can also be explicitly invoked with the <samp>--nsx</samp>
or <samp>--intersection</samp> switches.
These switches are supplied for clarity and consistency and do
absolutely nothing since intersection-mode is the default.</para></footnote>.
The <samp>--unn</samp> option causes <acronym><acronymword>NCO</acronymword></acronym> to extract the
union, rather than the intersection, of the specified groups and
variables. 
Consider the following commands (which may be assumed to end with
<samp>in.nc out.nc</samp>):
</para><example endspaces=" ">       
<pre xml:space="preserve"># Intersection-mode subsetting (default)
ncks -g  g1  -v v1 # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1
ncks -g /g1  -v v1 # Yes: /g1/v1, /g1/g2/v1. No: /v1, /g2/v1, /g2/g1/v1
ncks -g  g1/ -v v1 # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1, /g1/g2/v1
ncks -v  g1/v1     # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1, /g1/g2/v1
ncks -g /g1/ -v v1 # Yes: /g1/v1. No: /g2/g1/v1, /v1, /g2/v1 ...
ncks -v /g1/v1     # Yes: /g1/v1. No: /g2/g1/v1, /v1, /g2/v1 ...

# Union-mode subsetting (invoke with --unn or --union)
ncks -g  g1  -v v1 --unn # All variables in  g1 or progeny, or named v1
ncks -g /g1  -v v1 --unn # All variables in /g1 or progeny, or named v1
ncks -g  g1/ -v v1 --unn # All variables in  g1 or named v1
ncks -g /g1/ -v v1 --unn # All variables in /g1 or named v1
</pre></example>
<para>The first command (<samp>-g g1 -v v1</samp>) extracts the variable <file>v1</file>
from any group named <file>g1</file> or descendent <file>g1</file>.
The second command extracts <file>v1</file> from any root group
named <file>g1</file> and any descendent groups as well.
The third and fourth commands are equivalent ways of extracting
<file>v1</file> only from the root group named <file>g1</file> (not its
descendents). 
The fifth and sixth commands are equivalent ways of extracting the 
variable <file>v1</file> only from the root group named <file>g1</file>.
Subsetting in union-mode (with <samp>--unn</samp>) causes all variables to be
extracted which meet either one or both of the specifications of the 
variable and group specifications.
Union-mode subsetting is simply the logical &textldquo;OR&textrdquo; of intersection-mode
subsetting. 
As discussed below, the group and variable specifications may be comma
separated lists of regular expressions for added control over
subsetting. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="509">memory requirements</indexterm></cindex>
<para>Remember, if averaging or concatenating large files stresses your
systems memory or disk resources, then the easiest solution is often to
subset (with <samp>-g</samp> and/or <samp>-v</samp>) to retain only the most
important variables (<pxref label="Memory-Requirements"><xrefnodename>Memory Requirements</xrefnodename></pxref>).
</para><example endspaces=" ">       
<pre xml:space="preserve">ncks          in.nc out.nc # Extract all groups and variables
ncks -v scl   # Extract variable scl from all groups
ncks -g g1    # Extract group g1 and descendents
ncks -x -g g1 # Extract all groups except g1 and descendents
ncks -g g2,g3 -v scl # Extract scl from groups g2 and g3
</pre></example>
<para>Overwriting and appending work as expected:
</para><example endspaces=" ">
<pre xml:space="preserve"># Replace scl in group g2 in out.nc with scl from group g2 from in.nc
ncks -A -g g2 -v scl in.nc out.nc
</pre></example>

<para>Due to its special capabilities, <command>ncap2</command> interprets the
<samp>-v</samp> switch differently 
(<pxref label="ncap2-netCDF-Arithmetic-Processor"><xrefnodename>ncap2 netCDF Arithmetic Processor</xrefnodename></pxref>). 
For <command>ncap2</command>, the <samp>-v</samp> switch takes no arguments and
indicates that <emph>only</emph> user-defined variables should be output. 
<command>ncap2</command> neither accepts nor understands the <var>-x</var> and
<var>-g</var> switches.
</para>
<html endspaces=" ">
&lt;a name=&quot;rx&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rx &textndash;&gt;
&lt;a name=&quot;wildcarding&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#wildcarding &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="510">extended regular expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="511">regular expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="512">pattern matching</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="513">wildcards</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="514"><command>grep -E</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="515"><command>egrep</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="516"><command>ncatted</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="517"><acronym><acronymword>GNU</acronymword></acronym></indexterm></cindex>
<para>Regular expressions the syntax that <acronym><acronymword>NCO</acronymword></acronym> use pattern-match
object names in netCDF file against user requests.
The user can select all variables beginning with the string <samp>DST</samp>
from an input file by supplying the regular expression <samp>^DST</samp> to
the <samp>-v</samp> switch, i.e., <samp>-v '^DST'</samp>. 
The meta-characters used to express pattern matching operations are
<samp>^$+?.*[]&lbrace;&rbrace;|</samp>. 
If the regular expression pattern matches <emph>any</emph> part of a variable 
name then that variable is selected.
This capability is also called <dfn>wildcarding</dfn>, and is very useful for 
sub-setting large data files.
</para>
<para>Extended regular expressions are defined by the <acronym><acronymword>POSIX</acronymword></acronym>
<command>grep -E</command> (aka <command>egrep</command>) command.  
As of <acronym><acronymword>NCO</acronymword></acronym> 2.8.1 (August, 2003), variable name arguments
to the <samp>-v</samp> switch may contain <dfn>extended regular expressions</dfn>.
As of <acronym><acronymword>NCO</acronymword></acronym> 3.9.6 (January, 2009), variable names arguments 
to <command>ncatted</command> may contain <dfn>extended regular expressions</dfn>. 
As of <acronym><acronymword>NCO</acronymword></acronym> 4.2.4 (November, 2012), group name arguments 
to the <samp>-g</samp> switch may contain <dfn>extended regular expressions</dfn>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="518"><acronym><acronymword>POSIX</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="519"><code>regex</code></indexterm></cindex>
<para>Because of its wide availability, <acronym><acronymword>NCO</acronymword></acronym> uses the <acronym><acronymword>POSIX</acronymword></acronym>  
regular expression library <code>regex</code>.  
Regular expressions of arbitary complexity may be used.
Since netCDF variable names are relatively simple constructs, only a 
few varieties of variable wildcards are likely to be useful.
For convenience, we define the most useful pattern matching operators
here: 
<cindex index="cp" spaces=" "><indexterm index="cp" number="520"><code>.</code> (wildcard character)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="521"><code>$</code> (wildcard character)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="522"><code>^</code> (wildcard character)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="523"><code>?</code> (filename expansion)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="524"><code>*</code> (filename expansion)</indexterm></cindex>
</para><table commandarg="samp" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="samp">^</itemformat></item>
</tableterm><tableitem><para>Matches the beginning of a string
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">$</itemformat></item>
</tableterm><tableitem><para>Matches the end of a string
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">.</itemformat></item>
</tableterm><tableitem><para>Matches any single character
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>The most useful repetition and combination operators are
<cindex index="cp" spaces=" "><indexterm index="cp" number="525"><code>?</code> (wildcard character)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="526"><code>*</code> (wildcard character)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="527"><code>+</code> (wildcard character)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="528"><code>|</code> (wildcard character)</indexterm></cindex>
</para><table commandarg="samp" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="samp">?</itemformat></item>
</tableterm><tableitem><para>The preceding regular expression is optional and matched at most once
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">*</itemformat></item>
</tableterm><tableitem><para>The preceding regular expression will be matched zero or more times
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">+</itemformat></item>
</tableterm><tableitem><para>The preceding regular expression will be matched one or more times
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">|</itemformat></item>
</tableterm><tableitem><para>The preceding regular expression will be joined to the following regular
expression.
The resulting regular expression matches any string matching either
subexpression. 
</para></tableitem></tableentry></table>
<noindent></noindent>

<para>To illustrate the use of these operators in extracting variables
and groups, consider file <file>in_grp.nc</file> with groups
<code>g0</code>&textndash;<code>g9</code>, and subgroups <code>s0</code>&textndash;<code>s9</code>, in each of
those groups, and file <file>in.nc</file> with variables <code>Q</code>,
<code>Q01</code>&textndash;<code>Q99</code>, <code>Q100</code>, <code>QAA</code>&textndash;<code>QZZ</code>,
<code>Q_H2O</code>, <code>X_H2O</code>, <code>Q_CO2</code>, <code>X_CO2</code>.  
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -v '.+' in.nc               # All variables (default)
ncks -v 'Q.?' in.nc              # Variables that contain Q
ncks -v '^Q.?' in.nc             # Variables that start with Q
ncks -v '^Q+.?.' in.nc           # Q, Q0--Q9, Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q..' in.nc             # Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q[0-9][0-9]' in.nc     # Q01--Q99, Q100
ncks -v '^Q[[:digit:]]&lbrace;2&rbrace;' in.nc # Q01--Q99
ncks -v 'H2O$' in.nc             # Q_H2O, X_H2O 
ncks -v 'H2O$|CO2$' in.nc        # Q_H2O, X_H2O, Q_CO2, X_CO2 
ncks -v '^Q[0-9][0-9]$' in.nc    # Q01--Q99
ncks -v '^Q[0-6][0-9]|7[0-3]' in.nc # Q01--Q73, Q100
ncks -v '(Q[0-6][0-9]|7[0-3])$' in.nc # Q01--Q73
ncks -v '^[a-z]_[a-z]&lbrace;3&rbrace;$' in.nc # Q_H2O, X_H2O, Q_CO2, X_CO2
ncks -g 'g.' in_grp.nc           # 10 Groups g0-g9
ncks -g 's.' in_grp.nc       # 100 sub-groups g0/s0, g0/s1, ... g9/s9
ncks -g 'g.' -v 'v.' in_grp.nc   # All variables 'v.' in groups 'g.'
</pre></example>
<para>Beware&textmdash;two of the most frequently used repetition pattern matching
operators, <samp>*</samp> and <samp>?</samp>, are also valid pattern matching
operators for filename expansion (globbing) at the shell-level.
Confusingly, their meanings in extended regular expressions and in
shell-level filename expansion are significantly different.
In an extended regular expression, <samp>*</samp> matches zero or more
occurences of the preceding regular expression. 
Thus <samp>Q*</samp> selects all variables, and <samp>Q+.*</samp> selects all
variables containing <samp>Q</samp> (the <samp>+</samp> ensures the preceding item 
matches at least once).
To match zero or one occurence of the preceding regular expression,   
use <samp>?</samp>.
Documentation for the <acronym><acronymword>UNIX</acronymword></acronym> <command>egrep</command> command details the
extended regular expressions which <acronym><acronymword>NCO</acronymword></acronym> supports.
</para>
<html endspaces=" ">
&lt;a name=&quot;globbing&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#globbing &textndash;&gt;
&lt;a name=&quot;glb&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#glb &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="529">globbing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="530">shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="531"><command>bash</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="532"><command>csh</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="533">quotes</indexterm></cindex>
<para>One must be careful to protect any special characters in the regular
expression specification from being interpreted (globbed) by the shell.
This is accomplish by enclosing special characters within single or
double quotes
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -v Q?? in.nc out.nc   # Error: Shell attempts to glob wildcards
ncra -v '^Q+..' in.nc out.nc # Correct: NCO interprets wildcards
ncra -v '^Q+..' in*.nc out.nc # Correct: NCO interprets, Shell globs 
</pre></example>
<para>The final example shows that commands may use a combination of variable
wildcarding and shell filename expansion (globbing).
For globbing, <samp>*</samp> and <samp>?</samp> <emph>have nothing to do</emph> with the 
preceding regular expression!
In shell-level filename expansion, <samp>*</samp> matches any string,
including the null string and <samp>?</samp> matches any single character. 
Documentation for <command>bash</command> and <command>csh</command> describe the rules of
filename expansion (globbing).
</para>
<html endspaces=" ">
&lt;a name=&quot;crd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#crd &textndash;&gt;
&lt;a name=&quot;-C&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-C &textndash;&gt;
&lt;a name=&quot;-c&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-c &textndash;&gt;
</html>
</section>
<node name="Subsetting-Coordinate-Variables" spaces=" "><nodename>Subsetting Coordinate Variables</nodename><nodenext spaces=" ">Group Path Editing</nodenext><nodeprev spaces=" ">Subsetting Files</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Subsetting Coordinate Variables</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="534">subsetting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="535"><code>-C</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="536"><code>-c</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="537"><code>--no-coords</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="538"><code>--no-crd</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="539"><code>--coords</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="540"><code>--crd</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncap2</command>, <command>ncbo</command>, <command>nces</command>,
<command>ncecat</command>, <command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>,
<command>ncra</command>, <command>ncrcat</command>, <command>ncwa</command>&linebreak; 
Short options: <samp>-C</samp>, <samp>-c</samp>&linebreak;
Long options: <samp>--no-coords</samp>, <samp>--no-crd</samp>, <samp>--crd</samp>, <samp>--coords</samp>&linebreak;
</para></cartouche>
<para>By default, coordinates variables associated with any variable appearing
in the <var>input-file</var> will be placed in the <var>output-file</var>, even
if they are not explicitly specified, e.g., with the <samp>-v</samp> switch.
Thus variables with a latitude coordinate <code>lat</code> always carry the
values of <code>lat</code> with them into the <var>output-file</var>.
This feature can be disabled with <samp>-C</samp>, which causes <acronym><acronymword>NCO</acronymword></acronym>
to not automatically add coordinates to the variables appearing in the
<var>output-file</var>. 
However, using <samp>-C</samp> does not preclude the user from including some
coordinates in the output files simply by explicitly selecting the
coordinates with the <var>-v</var> option.
The <samp>-c</samp> option, on the other hand, is a shorthand way of
automatically specifying that <emph>all</emph> coordinate variables in the
<var>input-files</var> should appear in the <var>output-file</var>.
Thus <samp>-c</samp> allows the user to select all the coordinate variables
without having to know their names.
<cindex index="cp" spaces=" "><indexterm index="cp" number="541"><acronym><acronymword>CF</acronymword></acronym> conventions</indexterm></cindex>
As of <acronym><acronymword>NCO</acronymword></acronym> version 3.9.6 (January, 2009) 
both <samp>-c</samp> and <samp>-C</samp> honor the <acronym><acronymword>CF</acronymword></acronym> <code>coordinates</code>
convention described in <ref label="CF-Conventions"><xrefnodename>CF Conventions</xrefnodename></ref>.
As of <acronym><acronymword>NCO</acronymword></acronym> version 4.0.8 (April, 2011) 
both <samp>-c</samp> and <samp>-C</samp> honor the <acronym><acronymword>CF</acronymword></acronym> <code>bounds</code>
convention described in <ref label="CF-Conventions"><xrefnodename>CF Conventions</xrefnodename></ref>. 
</para>
<html endspaces=" ">
&lt;a name=&quot;gpe&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#gpe &textndash;&gt;
</html>
</section>
<node name="Group-Path-Editing" spaces=" "><nodename>Group Path Editing</nodename><nodenext spaces=" ">C and Fortran Index Conventions</nodenext><nodeprev spaces=" ">Subsetting Coordinate Variables</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Group Path Editing</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="542"><code>-G <var>gpe_dsc</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="543"><code>--gpe <var>gpe_dsc</var></code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Options <code>-G <var>gpe_dsc</var></code>&linebreak;
Availability: <command>ncbo</command>, <command>ncecat</command>, <command>nces</command>,
<command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>, <command>ncra</command>,
<command>ncrcat</command>, <command>ncwa</command>&linebreak;
Short options: <samp>-G</samp>&linebreak;
Long options: <samp>--gpe</samp>&linebreak;
</para></cartouche>

<para><dfn>Group Path Editing</dfn>, or <acronym><acronymword>GPE</acronymword></acronym>, allows the user to
restructure (i.e., add, remove, and rename groups) in the output file 
relative to the input file based on the instructions they provide.
As of  <acronym><acronymword>NCO</acronymword></acronym> 4.2.3 (November, 2012), all operators that accept  
netCDF4 files with groups accept the <samp>-G</samp> switch, or its
long-option equivalent <samp>--gpe</samp>.
To master <acronym><acronymword>GPE</acronymword></acronym> one must understand the meaning of the
required <var>gpe_dsc</var> structure/argument that specifies the
transformation of input-to-output group paths.
</para>
<para>Each <var>gpe_dsc</var> contains up to three elements (two are optional) in
the following order:&linebreak;   
<var>gpe_dsc</var> = <var>grp_pth</var>:<var>lvl_nbr</var> or <var>grp_pth</var>&arobase;<var>lvl_nbr</var>
</para>
<table commandarg="var" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="var">grp_pth</itemformat></item>
</tableterm><tableitem><para>Group Path.
<cindex index="cp" spaces=" "><indexterm index="cp" number="544">group path</indexterm></cindex>
This (optional) component specifies the output group path that should be 
appended after any editing (i.e., deletion or truncation) of the input
path is performed.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="var">lvl_nbr</itemformat></item>
</tableterm><tableitem><para>The number of levels to delete (from the head) or truncate (from the
tail) of the input path.
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>If both components of the argument are present, then a single character,
either the colon or at-sign (<code>:</code> or <code>&arobase;</code>), must separate them.
If only <var>grp_pth</var> is specifed, the separator character may be
omitted, e.g., <samp>-G g1</samp>.
If only <var>lvl_nbr</var> is specifed, the separator character is still
required to indicate it is a <var>lvl_nbr</var> arugment and not a
<var>grp_pth</var>, e.g., <samp>-G :-1</samp> or <samp>-G &arobase;1</samp>.
</para>
<para>If the at-sign separator character <code>&arobase;</code> is used instead of the colon
separator character <code>:</code>, then the following <var>lvl_nbr</var> arugment 
must be positive and it will be assumed to refer to Truncation-Mode.
Hence, <samp>-G :-1</samp> is the same as <samp>-G &arobase;1</samp>. 
This is simply a way of making the <var>lvl_nbr</var> argument
positive-definite. 
</para>
<html endspaces=" ">
&lt;a name=&quot;flatten&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#flatten &textndash;&gt;
&lt;a name=&quot;delete&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#delete &textndash;&gt;
&lt;a name=&quot;truncate&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#truncate &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="545"><code>&arobase;</code> (separator character)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="546"><code>:</code> (separator character)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="547">delete (groups)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="548">truncate (groups)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="549">flatten (groups)</indexterm></cindex>
<para><acronym><acronymword>GPE</acronymword></acronym> has three editing modes: Delete, Truncate, and
Flatten.
Select one of <acronym><acronymword>GPE</acronymword></acronym>&textrsquo;s three editing modes by supplying a
<var>lvl_nbr</var> that is positive, negative, or zero for Delete-, 
Truncate- and Flatten-mode, respectively. 
</para>
<para>In Delete-mode, <var>lvl_nbr</var> is a positive integer which specifies
the maximum number of group path components (i.e., groups) that
<acronym><acronymword>GPE</acronymword></acronym> will try to delete from the head of <var>grp_pth</var>. 
For example <math><var>lvl_nbr</var> = 3</math> changes the input path
<file>/g1/g2/g3/g4/g5</file> to the output path <file>/g4/g5</file>.
Input paths with <var>lvl_nbr</var> or fewer components (groups)
are completely erased and the output path commences from the root  
level. 
</para>
<para>In other words, <acronym><acronymword>GPE</acronymword></acronym> is tolerant of specifying too many group
components to delete. 
It deletes as many as possible, without complaint, and then begins to
flatten the file (which will fail if namespace conflicts arise).
</para>
<para>In Truncate-mode, <var>lvl_nbr</var> is a negative integer which specifies
the maximum number of group path components (i.e., groups) that
<acronym><acronymword>GPE</acronymword></acronym> will try to truncate from the tail of <var>grp_pth</var>. 
For example <math><var>lvl_nbr</var> = -3</math> changes the input path
<file>/g1/g2/g3/g4/g5</file> to the output path <file>/g1/g2</file>.
Input paths with <var>lvl_nbr</var> or fewer components (groups)
are completely erased and the output path commences from the root  
level. 
</para>
<para>In Flatten-mode, indicated by the separator character alone
or with <math><var>lvl_nbr</var> = 0</math>, <acronym><acronymword>GPE</acronymword></acronym> removes the entire group
path from the input file and constructs the output path beginning at the
root level.  
For example <code>-G :0</code> and <code>-G :</code> are identical and change the
input path <file>/g1/g2/g3/g4/g5</file> to the output path <file>/</file> whereas
<code>-G g1:0</code> and <code>-G g1:</code> are identical and result in the output 
path <file>/g1</file> for all variables.
</para>
<para>Subsequent to the alteration of the input path by the specified
editing mode, if any, <acronym><acronymword>GPE</acronymword></acronym> prepends (in Delete Mode)
or Appends (in Truncate-mode) any specifed <var>grp_pth</var> to the output
path. 
For example <code>-G g2</code> changes the input paths <file>/</file> and <file>/g1</file>
to <file>/g2</file> and <file>/g1/g2</file>, respectively.
Likewise, <code>-G g2/g3</code> changes the input paths <file>/</file> and <file>/g1</file>
to <file>/g2/g3</file> and <file>/g1/g2/g3</file>, respectively.
When <var>grp_pth</var> and <var>lvl_nbr</var> are both specified, the editing
actions are taken in sequence so that, e.g., <code>-G g1/g2:2</code> 
changes the input paths <file>/</file> and <file>/h1/h2/h3/h4</file>
to <file>/g1/g2</file> and <file>/g1/g2/h3/h4</file>, respectively.
Likewise, <code>-G g1/g2:-2</code> changes the input paths <file>/</file> and
<file>/h1/h2/h3/h4</file> to <file>/g1/g2</file> and <file>/h1/h2/g1/g2</file>,
respectively. 
</para>
<para>Combining <acronym><acronymword>GPE</acronymword></acronym> with subsetting (<pxref label="Subsetting-Files"><xrefnodename>Subsetting Files</xrefnodename></pxref>) 
yields powerful control over the extracted (or excluded) variables and
groups and their placement in the output file as shown by the following
commands. 
All commands below may be assumed to end with <samp>in.nc out.nc</samp>.
</para><example endspaces=" ">       
<pre xml:space="preserve"># Prepending paths without editing:
ncks                   # /g?/v? -&gt; /g?/v?
ncks             -v v1 # /g?/v1 -&gt; /g?/v1
ncks       -g g1       # /g1/v? -&gt; /g1/v?
ncks -G o1             # /g?/v? -&gt; /o1/g?/v?
ncks -G o1 -g g1       # /g1/v? -&gt; /o1/g1/v?
ncks       -g g1 -v v1 # /g1/v1 -&gt; /g1/v1
ncks -G o1       -v v1 # /g?/v1 -&gt; /o1/g?/v1
ncks -G o1 -g g1 -v v1 # /g1/v1 -&gt; /o1/g1/v1
ncks -G g1 -g /  -v v1 # /v1    -&gt; /g1/v1
ncks -G g1/g2    -v v1 # /g?/v1 -&gt; /g1/g2/g?/v1
# Delete-mode: Delete from and Prepend to path head
# Syntax: -G [ppn]:lvl_nbr = # of levels to delete
ncks -G :1    -g g1    -v v1 # /g1/v1    -&gt; /v1
ncks -G :1    -g g1/g1 -v v1 # /g1/g1/v1 -&gt; /g1/v1
ncks -G :2    -g g1/g1 -v v1 # /g1/g1/v1 -&gt; /v1
ncks -G :2    -g g1    -v v1 # /g1/v1    -&gt; /v1
ncks -G g2:1  -g g1    -v v1 # /g1/v1    -&gt; /g2/v1
ncks -G g2:2  -g g1/g1 -v v1 # /g1/g1/v1 -&gt; /g2/v1
ncks -G g2:1  -g /     -v v1 # /v1       -&gt; /g2/v1
ncks -G g2:1           -v v1 # /v1       -&gt; /g2/v1
ncks -G g2:1  -g g1/g1 -v v1 # /g1/g1/v1 -&gt; /g2/g1/v1
# Flatten-mode: Remove all input path components
# Syntax: -G [apn]: colon without numerical argument
ncks -G :            -v v1 # /g?/v1    -&gt; /v1
ncks -G :   -g g1    -v v1 # /g1/v1    -&gt; /v1
ncks -G :   -g g1/g1 -v v1 # /g1/g1/v1 -&gt; /v1
ncks -G g2:          -v v1 # /g?/v1    -&gt; /g2/v1
ncks -G g2:                # /g?/v?    -&gt; /g2/v?
ncks -G g2: -g g1/g1 -v v1 # /g1/g1/v1 -&gt; /g2/v1
# Truncate-mode: Truncate from and Append to path tail
# Syntax: -G [apn]:-lvl_nbr = # of levels to truncate
# NB: -G [apn]:-lvl_nbr is equivalent to -G [apn]&arobase;lvl_nbr
ncks -G :-1   -g g1    -v v1 # /g1/v1    -&gt; /v1
ncks -G :-1   -g g1/g2 -v v1 # /g1/g2/v1 -&gt; /g1/v1
ncks -G :-2   -g g1/g2 -v v1 # /g1/g2/v1 -&gt; /v1
ncks -G :-2   -g g1    -v v1 # /g1/v1    -&gt; /v1
ncks -G g2:-1          -v v1 # /g?/v1    -&gt; /g2/v1
ncks -G g2:-1 -g g1    -v v1 # /g1/v1    -&gt; /g2/v1
ncks -G g1:-1 -g g1/g2 -v v1 # /g1/g2/v1 -&gt; /g1/g1/v1
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;mv&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mv &textndash;&gt;
&lt;a name=&quot;move&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#move &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="550">move groups</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="551">groups, moving</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="552">rename groups</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="553">groups, renaming</indexterm></cindex>
<para>Until fall 2013 (netCDF version 4.3.1-pre1), netCDF contained no library
function for renaming groups, and therefore <command>ncrename</command> cannot
rename groups.
However, <acronym><acronymword>NCO</acronymword></acronym> built on earlier versions of netCDF than 4.3.1
can use a <acronym><acronymword>GPE</acronymword></acronym>-based workaround mechanism to &textldquo;rename&textrdquo;
groups. 
The <acronym><acronymword>GPE</acronymword></acronym> mechanism actually <emph>moves</emph> (i.e., copies to a new
location) groups, a more arduous procedure than simply renaming them.
<acronym><acronymword>GPE</acronymword></acronym> applies to all selected groups, so, in the general case,
one must move only the desired group to a new file, and then merge that
new file with the original to obtain a file where the desired group has
been &textldquo;renamed&textrdquo; and all else is unchanged.
Here is how to &textldquo;rename&textrdquo; group <file>/g4</file> to group <file>/f4</file> with
<acronym><acronymword>GPE</acronymword></acronym> instead of <command>ncrename</command>
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -O -G f4:1 -g g4 ~/nco/data/in_grp.nc ~/tmp.nc # Move /g4 to /f4
ncks -O -x -g g4 ~/nco/data/in_grp.nc ~/out.nc # Excise /g4
ncks -A ~/tmp.nc ~/out.nc # Add /f4 to new file
</pre></example>
<para>If the original group <file>g4</file> is not excised from <file>out.nc</file> (step
two above), then the final output file would contain both <file>g4</file> and
a copy named <file>f4</file>.
Thus GPE can be used to both &textldquo;rename&textrdquo; and copy groups.
The recommended way to rename groups when when netCDF version 4.3.1 is
availale is to use <command>ncrename</command> (<pxref label="ncrename-netCDF-Renamer"><xrefnodename>ncrename netCDF Renamer</xrefnodename></pxref>).
</para>
<html endspaces=" ">
&lt;a name=&quot;xmp_flt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_flt &textndash;&gt;
</html>
<para>One may wish to flatten hierarchical group files for many reasons.
These include <w>1. Obtaining</w> flat netCDF3 files for use with tools
that do not work with netCDF4 files, <w>2. Splitting</w> apart
hierarchies to re-assemble into different hierarchies, and
<w>3. Providing</w> a subset of a hierarchical file with the simplest
possible storage structure.
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -O -G : -g cesm -3 ~/nco/data/cmip5.nc ~/cesm.nc # Extract /cesm to /
</pre></example>
<para>The <option>-3</option> switch
<footnote><para>Note that the <option>-3</option> switch should appear <emph>after</emph> the
<option>-G</option> and <option>-g</option> switches. 
This is due to an artifact of the <acronym><acronymword>GPE</acronymword></acronym> implementation which we
wish to remove in the future.</para></footnote>
specifies the output dataset should be in netCDF3
format, the <option>-G :</option> option flattens all extracted groups, and the
<option>-g cesm</option> option extracts only the <code>cesm</code> group and leaves
all other groups (e.g., <code>ecmwf</code>, <code>giss</code>).
</para>
<html endspaces=" ">
&lt;a name=&quot;dismember&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dismember &textndash;&gt;
&lt;a name=&quot;disaggregate&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#disaggregate &textndash;&gt;
&lt;a name=&quot;ncdismember&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncdismember &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="554">disaggregate</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="555">dismember</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="5" mergedindex="cp">ncdismember</indexterm></findex>
<para>Let us show how to completely disaggregate (or, more memorably)
<emph>dismember</emph> a hierarchical dataset.
For now we take this to mean: store each group as a standalone flat
dataset in netCDF3 format.
This can be accomplished by looping the previous example over all
groups. 
This script <file>ncdismember</file> dismembers the input file <var>fl_in</var>
specified in the first argument and places the resulting files in the
directory <var>drc_out</var> specified by the second argument:
</para><example endspaces=" ">
<verbatim xml:space="preserve">
cat &gt; ~/ncdismember &lt;&lt; 'EOF'
# Purpose: Dismember netCDF4/HDF5 hierarchical files. CF-check them.
# Place each input file group in separate netCDF3 output file
# Described in NCO User Guide at http://nco.sf.net/nco.html#dismember
# Requirements: NCO 4.3.x+, UNIX shell utilities awk, grep, sed
# Optional: CFchecker command https://bitbucket.org/mde_/cfchecker

# Usage:
# ncdismember &lt;fl_in&gt; &lt;drc_out&gt; [flg_cf] [cf_vrs] [opt]
# where fl_in is input file/URL to dismember, drc_out is output directory,
# CF-compliance check is performed when optional third argument is 'cf',
# Optional fourth argument cf_vrs is CF version to check
# Optional fifth argument opt passes straight through to ncks
# chmod a+x ~/sh/ncdismember
# ncdismember ~/nco/data/mdl.nc /data/zender/nco/tmp
# ncdismember ~/nco/data/mdl.nc /data/zender/nco/tmp
# ncdismember http://dust.ess.uci.edu/nco/mdl.nc /data/zender/nco/tmp
# ncdismember http://thredds-test.ucar.edu/thredds/dodsC/testdods/foo.nc /data/zender/nco/tmp
# ncdismember ~/nco/data/mdl.nc /data/zender/nco/tmp cf
# ncdismember ~/nco/data/mdl.nc /data/zender/nco/tmp cf 1.3
# ncdismember ~/nco/data/mdl.nc /data/zender/nco/tmp cf 1.5 --fix_rec_dmn=all

# Command line argument defaults
fl_in=&quot;${HOME}/nco/data/mdl.nc&quot; # [sng] Input file to dismember/check
drc_out=&quot;${DATA}/nco/tmp&quot; # [sng] Output directory
flg_cf='0' # [flg] Perform CF-compliance check
cf_vrs='1.5' # [sng] Compliance-check this CF version (e.g., '1.5')
opt='' # [flg] Additional ncks options (e.g., '--fix_rec_dmn=all')

# Command line argument option parsing
if [ -n &quot;${1}&quot; ]; then fl_in=${1}; fi
if [ -n &quot;${2}&quot; ]; then drc_out=${2}; fi
if [ -n &quot;${3}&quot; ]; then flg_cf=${3}; fi
if [ -n &quot;${4}&quot; ]; then cf_vrs=${4}; fi
if [ -n &quot;${5}&quot; ]; then opt=${5}; fi

# Prepare output directory
echo &quot;NCO dismembering file ${fl_in}&quot;
fl_stb=$(basename ${fl_in})
drc_out=${drc_out}/${fl_stb}
mkdir -p ${drc_out}
cd ${drc_out}
# Obtain group list
grp_lst=`ncks --cdl -m ${fl_in} | grep '// group' | awk '{$1=$2=$3=&quot;&quot;;sub(/^  */,&quot;&quot;,$0);print}'`
IFS=$'\n' # Change Internal-Field-Separator from &lt;Space&gt;&lt;Tab&gt;&lt;Newline&gt; to &lt;Newline&gt;
for grp_in in ${grp_lst} ; do
    # Replace slashes by dots for output group filenames
    grp_out=`echo ${grp_in} | sed 's/\///' | sed 's/\//./g'`
    if [ &quot;${grp_out}&quot; = '' ]; then grp_out='root' ; fi
    # Tell older NCO/netCDF if HDF4 with --hdf4 switch (signified by .hdf/.HDF suffix)
    hdf4=`echo ${fl_in} | awk '{if(match(tolower($1),&quot;.hdf$&quot;)) hdf4=&quot;--hdf4&quot;; print hdf4}'`
    # Flatten to netCDF3, anchor, no history, no temporary file, padding, HDF4 flag, options
    ncks -O -3 -G : -g ${grp_in}/ -h --no_tmp_fl --hdr_pad=40 ${hdf4} ${opt} ${fl_in} ${drc_out}/${grp_out}.nc
    if [ ${flg_cf} = 'cf' ]; then
       # cfchecker needs Conventions &lt;= 1.5
       ncatted -h -a Conventions,global,o,c,&quot;CF-${cf_vrs}&quot; ${drc_out}/${grp_out}.nc
    else # !flg_cf
       echo ${drc_out}/${grp_out}.nc
    fi # !flg_cf
done
if [ ${flg_cf} = 'cf' ]; then
    echo &quot;CFchecker reports CF-compliance of each group in flat netCDF3 format&quot;
    cfchecker -c ${cf_vrs} *.nc
fi # !flg_cf
EOF
chmod 755 ~/ncdismember # Make command executable
/bin/mv -f ~/ncdismember ~/sh # Store in location on $PATH, e.g., /usr/local/bin

zender@roulee:~$ ncdismember ~/nco/data/mdl.nc ${DATA}/nco/tmp
NCO dismembering file /home/zender/nco/data/mdl.nc
/data/zender/nco/tmp/mdl.nc/cesm.cesm_01.nc
/data/zender/nco/tmp/mdl.nc/cesm.cesm_02.nc
/data/zender/nco/tmp/mdl.nc/cesm.nc
/data/zender/nco/tmp/mdl.nc/ecmwf.ecmwf_01.nc
/data/zender/nco/tmp/mdl.nc/ecmwf.ecmwf_02.nc
/data/zender/nco/tmp/mdl.nc/ecmwf.nc
/data/zender/nco/tmp/mdl.nc/root.nc
</verbatim>
</example>
<para>A (potentially more portable) binary executable could be written to
dismember all groups with a single invocation, yet dismembering without
loss of information is possible now with this simple script on all 
platforms with UNIXy utilities.
Note that all dimensions inherited by groups in the input file are
correctly placed by <command>ncdismember</command> into the flat files.
Moreover, each output file preserves the group metadata of all ancestor
groups, including the global metadata from the input file.
As written, the script could fail on groups that contain advanced
netCDF4 features because the user requests (with the <samp>-3</samp> switch)
that output be netCDF3 classic format.  
However, <command>ncks</command> detects many format incompatibilities in advance
and works around them.
For example, <command>ncks</command> autoconverts netCDF4-only atomic-types (such
as <code>NC_STRING</code> and <code>NC_UBYTE</code>) to corresponding netCDF3
atomic types (<code>NC_CHAR</code> and <code>NC_SHORT</code>) when the output format
is netCDF3. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="556"><acronym><acronymword>CF</acronymword></acronym> compliance checker</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="6" mergedindex="cp">cfchecker</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="7" mergedindex="cp">ncdismember</indexterm></findex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="557">compliance checker</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="558">Martin Schultz</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="559">Michael Decker</indexterm></cindex>
<para>One application of dismembering is to check the <acronym><acronymword>CF</acronymword></acronym>-compliance of each 
group in a file. 
When invoked with the optional third argumnt <samp>cf</samp>,
<command>ncdismember</command> passes each file it generates to the freely
available
<footnote><para>CFchecker is developed by Michael Decker and Martin Schultz at
Forschungszentrum J<accent type="uml" bracketed="off">u</accent>lich and distributed at
<uref><urefurl>https://bitbucket.org/mde_/cfchecker</urefurl></uref>.</para></footnote> 
<command>cfchecker</command> command.
</para><example endspaces=" ">
<verbatim xml:space="preserve">
zender@roulee:~$ ncdismember ~/nco/data/mdl.nc /data/zender/nco/tmp cf
NCO dismembering file /home/zender/nco/data/mdl.nc
CFchecker reports CF-compliance of each group in flat netCDF3 format
WARNING: Using the default (non-CF) Udunits database
cesm.cesm_01.nc: 
INFO: INIT:     running CFchecker version 1.5.15
INFO: INIT:     checking compliance with convention CF-1.5
INFO: INIT:     using standard name table version: 25, last modified: 2013-07-05T05:40:30Z
INFO: INIT:     using area type table version: 2, date: 10 July 2013
INFO: 2.4:      no axis information found in dimension variables, not checking dimension order
WARNING: 3:     variable &quot;tas1&quot; contains neither long_name nor standard_name attribute
WARNING: 3:     variable &quot;tas2&quot; contains neither long_name nor standard_name attribute
INFO: 3.1:      variable &quot;tas1&quot; does not contain units attribute
INFO: 3.1:      variable &quot;tas2&quot; does not contain units attribute
--------------------------------------------------
cesm.cesm_02.nc: 
...
</verbatim>
</example>
<para>By default the <acronym><acronymword>CF</acronymword></acronym> version checked is determined automatically by
<command>cfchecker</command>. 
The user can override this default by supplying a supported <acronym><acronymword>CF</acronymword></acronym> version,
e.g., <samp>1.3</samp>, as an optional fourth argument to
<command>ncdismember</command>. 
Current valid <acronym><acronymword>CF</acronymword></acronym> options are <samp>1.0</samp>, <samp>1.1</samp>,
<samp>1.2</samp>, <samp>1.3</samp>, <samp>1.4</samp>, and <samp>1.5</samp>. 
</para>
<html endspaces=" ">
&lt;a name=&quot;diwg&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#diwg &textndash;&gt;
</html>
<para>Our development and testing of <command>ncdismember</command> is funded by our
involvement in <acronym><acronymword>NASA</acronymword></acronym>&textrsquo;s Dataset Interoperability Working Group 
(<uref><urefurl>https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group</urefurl><urefdesc spaces="\n">DIWG</urefdesc></uref>), though our interest extends beyond <acronym><acronymword>NASA</acronymword></acronym> datasets.
Taken together, <acronym><acronymword>NCO</acronymword></acronym>&textrsquo;s features (autoconversion to netCDF3  
atomic types, fixing multiple record dimensions, autosensing
<acronym><acronymword>HDF4</acronymword></acronym> input, scoping rules for CF conventions) make 
<command>ncdismember</command> reliable and friendly for both dismembering 
hierarchical files and for <acronym><acronymword>CF</acronymword></acronym>-compliance checks. 
Most <acronym><acronymword>HDF4</acronymword></acronym> and <acronym><acronymword>HDF5</acronymword></acronym> datasets can be checked for 
<acronym><acronymword>CF</acronymword></acronym>-compliance with a one-line command. 
Example compliance checks of common <acronym><acronymword>NASA</acronymword></acronym> datasets are at
<uref><urefurl>http://dust.ess.uci.edu/diwg</urefurl></uref>.
Our long-term goal is to enrich the hierarchical data model with the 
expressivity and syntactic power of <acronym><acronymword>CF</acronymword></acronym> conventions.
</para>
<para><acronym><acronymword>NASA</acronymword></acronym> asked the <acronym><acronymword>DIWG</acronymword></acronym> to prepare a one-page summary
of the procedure necessary to check <acronym><acronymword>HDF</acronymword></acronym> files for
<acronym><acronymword>CF</acronymword></acronym>-compliance: 
</para><example endspaces=" ">
<verbatim xml:space="preserve">
cat &gt; ~/ncdismember.txt &lt;&lt; 'EOF'
    Preparing an RPM-based OS to Test Hierarchical Files for CF-Compliance

By Charlie Zender, UCI &amp; NASA Dataset Interoperability Working Group (DIWG)

Installation Summary:
1. HDF4 [with netCDF support _disabled_]
2. HDF5
3. netCDF version 4.3.1 (or later) [with HDF4 support _enabled_]
4. NCO version 4.4.0 (or later)
5. numpy
6. netcdf4-python
7. python-lxml
8. CFunits-python
9. CFChecker
10. ncdismember

All 10 packages can use default installs _except_ HDF4 and netCDF.
Following instructions for Fedora Core 20 (FC20), an RPM-based Linux OS
Feedback and changes for other Linux-based OS's welcome to zender at uci.edu
${H4DIR}, ${H5DIR}, ${NETCDFDIR}, ${NCODIR}, may all be different
For simplicity CZ sets them all to /usr/local

# 1. HDF4. Build in non-default manner. Turn-off its own netCDF support.
# Per http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html
# HDF4 support not necessary though it makes ncdismember more comprehensive
wget -c http://www.hdfgroup.org/ftp/HDF/HDF_Current/src/hdf-4.2.9.tar.gz
tar xvzf hdf-4.2.9.tar.gz
cd hdf-4.2.9
./configure --enable-shared --disable-netcdf --disable-fortran --prefix=${H4DIR}
make &amp;&amp; make check &amp;&amp; make install

# 2. HDF5. Build normally. RPM may work too. Please let me know if so.
# HDF5 is a necessary pre-requisite for netCDF4
wget -c ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4/hdf5-1.8.11.tar.gz
tar xvzf hdf5-1.8.11.tar.gz
cd hdf5-1.8.11
./configure --enable-shared --prefix=${H5DIR}
make &amp;&amp; make check &amp;&amp; make install

# 3. netCDF version 4.3.1. Build in non-default manner with HDF4. No RPM.
# Per http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html
# Earlier versions of netCDF may fail checking some HDF4 files
wget -c ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.3.1.tar.gz
tar xvzf netcdf-4.3.1.tar.gz
cd netcdf-4.3.1
CPPFLAGS=&quot;-I${H5DIR}/include -I${H4DIR}/include&quot; \
LDFLAGS=&quot;-L${H5DIR}/lib -L${H4DIR}/lib&quot; \
./configure --enable-hdf4 --enable-hdf4-file-tests
make &amp;&amp; make check &amp;&amp; make installas

# 4. NCO version 4.4.2. No RPM for this version. Must install by hand.
# Earlier versions of NCO are relatively useless for ncdismember
cd ${DATA}/tmp
wget http://nco.sourceforge.net/src/nco-4.4.2.tar.gz .
tar xvzf nco-4.4.2.tar.gz
cd nco-4.4.2
./configure --prefix=${NCODIR}
make &amp;&amp; make install

# 5. numpy
sudo yum install numpy -y

# 6. netcdf4-python
sudo yum install netcdf4-python -y

# 7. python-lxml
sudo yum install python-lxml -y

# 8. CFunits-python. No RPM available. Must install by hand.
# http://code.google.com/p/cfunits-python/
cd ${DATA}/tmp
wget http://cfunits-python.googlecode.com/files/cfunits-0.9.6.tar.gz .
cd cfunits-0.9.6
sudo python setup.py install

# 9. CFChecker. No RPM available. Must install by hand.
# https://bitbucket.org/mde_/cfchecker
cd ${DATA}/tmp
wget https://bitbucket.org/mde_/cfchecker/downloads/CFchecker-1.5.15.tar.bz2 . 
tar xvjf CFchecker-1.5.15.tar.bz2 
cd CFchecker
sudo python setup.py install

# 10. ncdismember. Copy script from http://nco.sf.net/nco.html#ncdismember
# Store dismembered files somewhere, e.g., ${DATA}/nco/tmp/hdf
mkdir -p ${DATA}/nco/tmp/hdf
# Many datasets work with a simpler command...
ncdismember ~/nco/data/in.nc ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ~/nco/data/mdl.nc ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ${DATA}/hdf/AMSR_E_L2_Rain_V10_200905312326_A.hdf \
            ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ${DATA}/hdf/BUV-Nimbus04_L3zm_v01-00-2012m0203t144121.h5 \
            ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ${DATA}/hdf/HIRDLS-Aura_L3ZAD_v06-00-00-c02_2005d022-2008d077.he5 ${DATA}/nco/tmp/hdf cf 1.5
# Some datasets, typically .h5, require the --fix_rec_dmn=all argument
ncdismember_${DATA}/hdf/GATMO_npp_d20100906_t1935191_e1935505_b00012_c20110707155932065809_noaa_ops.h5 ${DATA}/nco/tmp/hdf cf 1.5 --fix_rec_dmn=all
ncdismember ${DATA}/hdf/mabel_l2_20130927t201800_008_1.h5 \
            ${DATA}/nco/tmp/hdf cf 1.5 --fix_rec_dmn=all
EOF
</verbatim>
</example>
<!-- c scp ~/ncdismember.pdf dust.ess.uci.edu:/var/www/html/diwg -->
<para>A <acronym><acronymword>PDF</acronymword></acronym> version of these instructions is available
<uref><urefurl>http://dust.ess.uci.edu/diwg/ncdismember.pdf</urefurl><urefdesc spaces=" ">here</urefdesc></uref>.
</para>
<html endspaces=" ">
&lt;a name=&quot;ftn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ftn &textndash;&gt;
&lt;a name=&quot;-F&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-F &textndash;&gt;
</html>
</section>
<node name="C-and-Fortran-Index-Conventions" spaces=" "><nodename>C and Fortran Index Conventions</nodename><nodenext spaces=" ">Hyperslabs</nodenext><nodeprev spaces=" ">Group Path Editing</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>C and Fortran Index conventions</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="560">index convention</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="561">Fortran index convention</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="562">C index convention</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="563"><code>-F</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="564"><code>--fortran</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncbo</command>, <command>nces</command>, <command>ncecat</command>,
<command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>, <command>ncra</command>,
<command>ncrcat</command>, <command>ncwa</command>&linebreak; 
Short options: <samp>-F</samp>&linebreak;
Long options: <samp>--fortran</samp>&linebreak;
</para></cartouche>
<cindex index="cp" spaces=" "><indexterm index="cp" number="565">I/O</indexterm></cindex>
<para>The <samp>-F</samp> switch changes <acronym><acronymword>NCO</acronymword></acronym> to read and write with
the Fortran index convention. 
By default, <acronym><acronymword>NCO</acronymword></acronym> uses C-style (0-based) indices for all I/O. 
<w>In C</w>, indices count <w>from 0</w> (rather <w>than 1</w>), and
dimensions are ordered from slowest (inner-most) to fastest
(outer-most) varying.
In Fortran, indices count <w>from 1</w> (rather <w>than 0</w>), and
dimensions are ordered from fastest (inner-most) to slowest 
(outer-most) varying.  
<cindex index="cp" spaces=" "><indexterm index="cp" number="566">transpose</indexterm></cindex>
Hence <w>C and</w> Fortran data storage conventions represent mathematical
transposes of eachother.
<cindex index="cp" spaces=" "><indexterm index="cp" number="567">record variable</indexterm></cindex>
Note that record variables contain the record dimension as the most
slowly varying dimension.  
See <ref label="ncpdq-netCDF-Permute-Dimensions-Quickly"><xrefnodename>ncpdq netCDF Permute Dimensions Quickly</xrefnodename></ref> for techniques
to re-order (including transpose) dimensions and to reverse data
storage order.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="568">record dimension</indexterm></cindex>
<para>Consider a file <file>85.nc</file> containing <w>12 months</w> of data in the
record dimension <code>time</code>.
The following hyperslab operations produce identical results, a
June-July-August average of the data:
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -d time,5,7 85.nc 85_JJA.nc
ncra -F -d time,6,8 85.nc 85_JJA.nc
</pre></example>

<para>Printing variable <var>three_dmn_var</var> in file <file>in.nc</file> first with
the <w>C indexing</w> convention, then with Fortran indexing convention
results in the following output formats: 
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -v three_dmn_var in.nc
lat[0]=-90 lev[0]=1000 lon[0]=-180 three_dmn_var[0]=0 
...
% ncks -F -v three_dmn_var in.nc
lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0 
...
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;-d&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-d &textndash;&gt;
&lt;a name=&quot;dmn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dmn &textndash;&gt;
&lt;a name=&quot;hyp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hyp &textndash;&gt;
&lt;a name=&quot;hyperslab&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hyperslab &textndash;&gt;
</html>
</section>
<node name="Hyperslabs" spaces=" "><nodename>Hyperslabs</nodename><nodenext spaces=" ">Stride</nodenext><nodeprev spaces=" ">C and Fortran Index Conventions</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Hyperslabs </sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="569">hyperslab</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="570">dimension limits</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="571">coordinate limits</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="572"><code>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="573"><code>--dimension <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="574"><code>--dmn <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncbo</command>, <command>nces</command>, <command>ncecat</command>,
<command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>, <command>ncra</command>,
<command>ncrcat</command>, <command>ncwa</command>&linebreak; 
Short options: <samp>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>&linebreak;
Long options: 
<samp>--dimension <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>,&linebreak; 
<samp>--dmn <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>&linebreak;
</para></cartouche>
<para><w>A <dfn>hyperslab</dfn></w> is a subset of a variable&textrsquo;s data.
The coordinates of a hyperslab are specified with the 
<code>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code> short
option (or with the same arguments to the <samp>--dimension</samp> or
<samp>--dmn</samp> long options).   
At least one hyperslab argument (<var>min</var>, <var>max</var>, or <var>stride</var>)
must be present. 
The bounds of the hyperslab to be extracted are specified by the
associated <var>min</var> and <var>max</var> values. 
<w>A half</w>-open range is specified by omitting either the <var>min</var> or
<var>max</var> parameter.
The separating comma must be present to indicate the omission of one of
these arguments.
The unspecified limit is interpreted as the maximum or minimum value in 
the unspecified direction.  
<w>A cross</w>-section at a specific coordinate is extracted by specifying only
the <var>min</var> limit and omitting a trailing comma. 
Dimensions not mentioned are passed with no reduction in range.
The dimensionality of variables is not reduced (in the case of a
cross-section, the size of the constant dimension will be one). 
</para><example endspaces=" ">
<pre xml:space="preserve"># First and second indices of lon dimension
ncks -F -d lon,1,2 in.nc out.nc
# Second and third indices of lon dimension
ncks -d lon,1,2 in.nc out.nc
</pre></example>

<cindex index="cp" spaces=" "><indexterm index="cp" number="575">stride</indexterm></cindex>
<para>Coordinate values should be specified using real notation with a decimal 
point required in the value, whereas dimension indices are specified
using integer notation without a decimal point. 
This convention serves only to differentiate coordinate values from
dimension indices.
It is independent of the type of any netCDF coordinate variables.
For a given dimension, the specified limits must both be coordinate
values (with decimal points) or dimension indices (no decimal points).
</para>
<para>If values of a coordinate-variable are used to specify a range or
cross-section, then the coordinate variable must be monotonic (values
either increasing or decreasing). 
In this case, command-line values need not exactly match coordinate
values for the specified dimension. 
Ranges are determined by seeking the first coordinate value to occur in
the closed range [<var>min</var>,<var>max</var>] and including all subsequent
values until one falls outside the range. 
The coordinate value for a cross-section is the coordinate-variable
value closest to the specified value and must lie within the range or
coordinate-variable values. 
The <var>stride</var> argument, if any, must be a dimension index, not a
coordinate value.
<xref label="Stride"><xrefnodename>Stride</xrefnodename></xref>, for more information on the <var>stride</var> option.
</para><example endspaces=" ">
<pre xml:space="preserve"># All longitude values between 1 and 2 degrees
ncks -d lon,1.0,2.0 in.nc out.nc
# All longitude values between 1 and 2 degrees
ncks -F -d lon,1.0,2.0 in.nc out.nc
# Every other longitude value between 0 and 90 degrees
ncks -F -d lon,0.0,90.0,2 in.nc out.nc
</pre></example>

<para>As of version 4.2.1 (August, 2012), <acronym><acronymword>NCO</acronymword></acronym> allows one to extract
the last <var>N</var> elements of a hyperslab.
Negative integers as <var>min</var> or <var>max</var> elements of a hyperslab
specification indicate offsets from the end (Python also uses this
convention). 
Previously, for example, <samp>-d time,-2,-1</samp> caused a domain error. 
Now it means select the second-to-last and penultimate timesteps.
Negative integers work for <var>min</var> and <var>max</var> indices, though not
for <var>stride</var>. 
</para><example endspaces=" ">
<pre xml:space="preserve"># Last two indices of lon dimension
ncks -F -d lon,1,-2 in.nc out.nc
# First to penultimate indices of lon dimension
ncks -F -d lon,1,-2 in.nc out.nc
# Third-to-last to last index of lon dimension
ncks -F -d lon,-3,-1 in.nc out.nc
# Third-to-last to last index of lon dimension
ncks -F -d lon,-3, in.nc out.nc
</pre></example>
<noindent></noindent>
<para>As shown, we recommend using a full floating point suffix of <code>.0</code>
instead of simply <code>.</code> in order to make obvious the selection of
hyperslab elements based on coordinate value rather than index.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="576"><code>NC_CHAR</code></indexterm></cindex>
<para>User-specified coordinate limits are promoted to double-precision values 
while searching for the indices which bracket the range. 
Thus, hyperslabs on coordinates of type <code>NC_CHAR</code> are computed
numerically rather than lexically, so the results are unpredictable. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="577">wrapped coordinates</indexterm></cindex>
<para>The relative magnitude of <var>min</var> and <var>max</var> indicate to the
operator whether to expect a <dfn>wrapped coordinate</dfn>
(<pxref label="Wrapped-Coordinates"><xrefnodename>Wrapped Coordinates</xrefnodename></pxref>), such as longitude.
If <math><var>min</var> &gt; <var>max</var></math>, the <acronym><acronymword>NCO</acronymword></acronym> expects the
coordinate to be wrapped, and a warning message will be printed.
When this occurs, <acronym><acronymword>NCO</acronymword></acronym> selects all values outside the domain
[<math><var>max</var> &lt; <var>min</var></math>], i.e., all the values exclusive of the
values which would have been selected if <var>min</var> and <var>max</var> were
swapped. 
If this seems confusing, test your command on just the coordinate
variables with <command>ncks</command>, and then examine the output to ensure
<acronym><acronymword>NCO</acronymword></acronym> selected the hyperslab you expected (coordinate wrapping
is currently only supported by <command>ncks</command>). 
</para>
<para>Because of the way wrapped coordinates are interpreted, it is very
important to make sure you always specify hyperslabs in the
monotonically increasing sense, i.e., <math><var>min</var> &lt; <var>max</var></math>
(even if the underlying coordinate variable is monotonically
decreasing). 
The only exception to this is when you are indeed specifying a wrapped
coordinate.  
The distinction is crucial to understand because the points selected by, 
e.g., <code>-d longitude,50.,340.</code>, are exactly the complement of the
points selected by <code>-d longitude,340.,50.</code>.
</para>
<para>Not specifying any hyperslab option is equivalent to specifying full
ranges of all dimensions. 
This option may be specified more than once in a single command 
(each hyperslabbed dimension requires its own <code>-d</code> option).
</para>
<html endspaces=" ">
&lt;a name=&quot;srd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#srd &textndash;&gt;
&lt;a name=&quot;stride&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#stride &textndash;&gt;
</html>
</section>
<node name="Stride" spaces=" "><nodename>Stride</nodename><nodenext spaces=" ">Record Appending</nodenext><nodeprev spaces=" ">Hyperslabs</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Stride </sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="578">stride</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="579"><code>-d <var>dim</var>,[<var>min</var>],[<var>max</var>],<var>stride</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="580"><code>--dimension <var>dim</var>,[<var>min</var>],[<var>max</var>],<var>stride</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="581"><code>--dmn <var>dim</var>,[<var>min</var>],[<var>max</var>],<var>stride</var></code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncbo</command>, <command>nces</command>, <command>ncecat</command>,
<command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>, <command>ncra</command>,
<command>ncrcat</command>, <command>ncwa</command>&linebreak; 
Short options: <samp>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>&linebreak;
Long options: 
<samp>--dimension <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>,&linebreak; 
<samp>--dmn <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>&linebreak;
</para></cartouche>
<para>All data operators support specifying a <dfn>stride</dfn> for any and all
dimensions at the same time.
The <var>stride</var> is the spacing between consecutive points in a
hyperslab. 
<w>A <var>stride</var></w> <w>of 1</w> picks all the elements of the hyperslab, and
a <var>stride</var> <w>of 2</w> skips every other element, etc.&eosperiod;
<command>ncks</command> multislabs support strides, and are more powerful than
the regular hyperslabs supported by the other operators
(<pxref label="Multislabs"><xrefnodename>Multislabs</xrefnodename></pxref>).
Using the <var>stride</var> option for the record dimension with
<command>ncra</command> and <command>ncrcat</command> makes it possible, for instance, to
average or concatenate regular intervals across multi-file input data sets.
</para>
<para>The <var>stride</var> is specified as the optional fourth argument to the
<samp>-d</samp> hyperslab specification:  
<code>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code>.
Specify <var>stride</var> as an integer (i.e., no decimal point) following
the third comma in the <samp>-d</samp> argument.  
There is no default value for <var>stride</var>. 
Thus using <samp>-d time,,,2</samp> is valid but <samp>-d time,,,2.0</samp> and
<samp>-d time,,,</samp> are not.
When <var>stride</var> is specified but <var>min</var> is not, there is an
ambiguity as to whether the extracted hyperslab should begin with (using
C-style, 0-based indexes) <w>element 0</w> or element <samp>stride-1</samp>.
<acronym><acronymword>NCO</acronymword></acronym> must resolve this ambiguity and it chooses <w>element 0</w>
as the first element of the hyperslab when <var>min</var> is not specified.
Thus <samp>-d time,,,<var>stride</var></samp> is syntactically equivalent to
<samp>-d time,0,,<var>stride</var></samp>.
This means, for example, that specifying the operation 
<samp>-d time,,,2</samp> on the array <samp>1,2,3,4,5</samp> selects the hyperslab
<samp>1,3,5</samp>. 
To obtain the hyperslab <samp>2,4</samp> instead, simply explicitly specify
the starting index <w>as 1,</w> i.e., <samp>-d time,1,,2</samp>. 
</para>
<para>For example, consider a file <file>8501_8912.nc</file> which contains 60
consecutive months of data. 
Say you wish to obtain just the March data from this file.
Using 0-based subscripts (<pxref label="C-and-Fortran-Index-Conventions"><xrefnodename>C and Fortran Index Conventions</xrefnodename></pxref>) these 
data are stored in records <w>2, 14, &dots; 50</w> so the desired
<var>stride</var> <w>is 12.</w>
Without the <var>stride</var> option, the procedure is very awkward.
One could use <command>ncks</command> five times and then use <command>ncrcat</command> to  
concatenate the resulting files together:
<cindex index="cp" spaces=" "><indexterm index="cp" number="582">Bourne Shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="583">C Shell</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">for idx in 02 14 26 38 50; do # Bourne Shell
  ncks -d time,$&lbrace;idx&rbrace; 8501_8912.nc foo.$&lbrace;idx&rbrace;
done
foreach idx (02 14 26 38 50) # C Shell
  ncks -d time,$&lbrace;idx&rbrace; 8501_8912.nc foo.$&lbrace;idx&rbrace;
end
ncrcat foo.?? 8589_03.nc
rm foo.??
</pre></example>
<para>With the <var>stride</var> option, <command>ncks</command> performs this hyperslab
extraction in one operation:
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -d time,2,,12 8501_8912.nc 8589_03.nc
</pre></example>
<para><xref label="ncks-netCDF-Kitchen-Sink"><xrefnodename>ncks netCDF Kitchen Sink</xrefnodename></xref>, for more information on <command>ncks</command>.
</para>
<para>Applying the <var>stride</var> option to the record dimension in
<command>ncra</command> and <command>ncrcat</command> makes it possible, for instance, to
average or concatenate regular intervals across multi-file input data
sets. 
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8589_03.nc
ncrcat -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8503_8903.nc
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;rec_apn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rec_apn &textndash;&gt;
&lt;a name=&quot;record_append&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#record_append &textndash;&gt;
</html>
</section>
<node name="Record-Appending" spaces=" "><nodename>Record Appending</nodename><nodenext spaces=" ">Subcycle</nodenext><nodeprev spaces=" ">Stride</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Record Appending</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="584">record append</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="585"><code>--rec_apn</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="586"><code>--record_append</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncra</command>, <command>ncrcat</command>&linebreak; 
Short options: None&linebreak;
Long options: 
<samp>--rec_apn</samp>, <samp>--record_append</samp>&linebreak;
</para></cartouche>
<para>As of version 4.2.6 (March, 2013), <acronym><acronymword>NCO</acronymword></acronym> allows both
Multi-File, Multi-Record operators (<command>ncra</command> and <command>ncrcat</command>)
to append their output directly to the end of an existing file.
This feature may be used to augment a target file, rather than construct
it from scratch. 
This helps, for example, when a timeseries is concatenated from input
data that becomes available in stages rather than all at once.
In such cases this switch significantly speeds writing.
</para>
<para>Consider the use case where one wishes to preserve the contents of
<file>fl_1.nc</file>, and add to them new records contained in
<file>fl_2.nc</file>. 
Previously the output had to be placed in a third file, <file>fl_3.nc</file>
(which could also safely be named <file>fl_2.nc</file>), via
</para><example endspaces=" ">
<pre xml:space="preserve">ncrcat -O fl_1.nc fl_2.nc fl_3.nc
</pre></example>
<para>Under the hood this operation copies all information in
<file>fl_1.nc</file> and <file>fl_2.nc</file> not once but twice.
The first copy is performed through the netCDF interface, as all
data from <file>fl_1.nc</file> and <file>fl_2.nc</file> are extracted and placed in
the output file.
The second copy occurs (usually much) more quickly as the (by default)
temporary output file is copied (sometimes a quick re-link suffices) to
the final output file (<pxref label="Temporary-Output-Files"><xrefnodename>Temporary Output Files</xrefnodename></pxref>).
All this copying is expensive for large files. 
</para>
<para>The new <samp>--record_append</samp> switch causes all records in
<file>fl_2.nc</file> to be appended to the end of the corresponding records in
<file>fl_1.nc</file>: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncrcat --rec_apn fl_2.nc fl_1.nc
</pre></example>
<para>The ordering of the filename arguments may seem non-intuitive.
If the record variable represents time in these files, then the
values in <file>fl_1.nc</file> precede those in <file>fl_2.nc</file>, so why
do the files appear in the reverse order on the command line?
<file>fl_1.nc</file> is the last file named because it is the pre-existing
output file to which we are appending all of the other input files
(in this case only <file>fl_2.nc</file>).
The contents of <file>fl_1.nc</file> are completely preserved, and only
values in <file>fl_2.nc</file> (and any other input files) are copied. 
This switch avoids the necessity of copying all of <file>fl_1.nc</file> 
through the netCDF interface to a new output file.
The <samp>--rec_apn</samp> switch automatically puts <acronym><acronymword>NCO</acronymword></acronym> into 
append mode (<pxref label="Appending-Variables"><xrefnodename>Appending Variables</xrefnodename></pxref>), so specifying <samp>-A</samp> is
redundant, and simultaneously specifying overwrite mode with <samp>-O</samp>
causes an error.  
By default, NCO works in an intermediate temporary file.
Power users may combine <samp>--rec_apn</samp> with the <samp>--no_tmp_fl</samp>
switch (<pxref label="Temporary-Output-Files"><xrefnodename>Temporary Output Files</xrefnodename></pxref>):
</para><example endspaces=" ">
<pre xml:space="preserve">ncrcat --rec_apn --no_tmp_fl fl_2.nc fl_1.nc
</pre></example>
<para>This avoids creating an intermediate file, and copies only the
minimal amount of data (i.e., all of <file>fl_2.nc</file>). 
Hence, it is fast.
We recommend users try to understand the safety trade-offs involved. 
</para>
<html endspaces=" ">
&lt;a name=&quot;subcycle&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#subcycle &textndash;&gt;
&lt;a name=&quot;ssc&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ssc &textndash;&gt;
&lt;a name=&quot;duration&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#duration &textndash;&gt;
&lt;a name=&quot;drn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#drn &textndash;&gt;
&lt;a name=&quot;mro&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mro &textndash;&gt;
</html>
</section>
<node name="Subcycle" spaces=" "><nodename>Subcycle</nodename><nodenext spaces=" ">Multislabs</nodenext><nodeprev spaces=" ">Record Appending</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Subcycle</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="587">duration</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="588">sub-cycle</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="589">subcycle</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="590">MRO</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="591">Multi-Record Operator</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="592"><code>--mro</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="593"><code>-d <var>dim</var>,[<var>min</var>],[<var>max</var>],[<var>stride</var>],[<var>subcycle</var>]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="594"><code>--dimension <var>dim</var>,[<var>min</var>],[<var>max</var>],[<var>stride</var>],[<var>subcycle</var>]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="595"><code>--dmn <var>dim</var>,[<var>min</var>],[<var>max</var>],[<var>stride</var>],<var>subcycle</var>]</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncra</command>, <command>ncrcat</command>&linebreak; 
Short options: <samp>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>][,[<var>subcycle</var>]]]]</samp>&linebreak;
Long options: 
<samp>--mro</samp>
<samp>--dimension <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>][,[<var>subcycle</var>]]]]</samp>&linebreak;
<samp>--dmn <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>][,[<var>subcycle</var>]]]]</samp>&linebreak;
</para></cartouche>
<para>As of version 4.2.1 (August, 2012), <acronym><acronymword>NCO</acronymword></acronym> allows both Multi-File,
Multi-Record operators, <command>ncra</command> and <command>ncrcat</command>, to extract
and operate on multiple groups of records. 
These groups may be connected to physical <emph>sub-cycles</emph> of a
periodic nature, e.g., months of a year, or hours of a day. 
Or they may be thought of as groups of a specifed duration.
The feature and the terminology to describe it are new.
For now, we call this the <dfn>subcycle feature</dfn>, sometimes abbreviated
<acronym><acronymword>SSC</acronymword></acronym>
<footnote><para>When originally released in 2012 this was called the
<dfn>duration feature</dfn>, and was abbreviated <acronym><acronymword>DRN</acronymword></acronym>.</para></footnote>.
</para>
<para>The subcycle feature allows processing of groups of records
separated by regular intervals of records.
It is perhaps best illustrated by an extended example which describes
how to solve the same problem both with and without the <acronym><acronymword>SSC</acronymword></acronym> feature.
</para>
<para>The first task in climate data processing is often creating seasonal
cycles. 
Suppose a 150-year climate simulation produces 150 output files, each
comprising 12 records, each record a monthly mean:
<file>1850.nc</file>, <file>1851.nc</file>, ... <file>1999.nc</file>.
Our goal is to create a single file containing the summertime (June,
July, and August, aka JJA) mean.
Traditionally, we would first compute the climatological monthly
mean for each month of summer. 
Each of these is a 150-year mean, i.e., 
</para><example endspaces=" ">
<pre xml:space="preserve"># Step 1: Create climatological monthly files clm06.nc..clm08.nc
for mth in &lbrace;6..8&rbrace;; do
  mm=`printf &quot;%02d&quot; $mth`
  ncra -O -F -d time,$&lbrace;mm&rbrace;,,12 -n 150,4,1 1850.nc clm$&lbrace;mm&rbrace;.nc
done
# Step 2: Average climatological monthly files into summertime mean
ncra -O clm06 clm07.nc clm08.nc clm_JJA.nc
</pre></example>
<noindent></noindent>
<para>So far, nothing is unusual and this task can be performed by any
<acronym><acronymword>NCO</acronymword></acronym> version. 
The <acronym><acronymword>SSC</acronymword></acronym> feature makes obsolete the need for the shell loop
used in <w>Step 1</w> above. 
</para>
<para>The new <acronym><acronymword>SSC</acronymword></acronym> option aggregates more than one input record at a time
before performing arithmetic operations, and, with an additional
switch, allows us to archive those results in multiple record output
(MRO) files.  
This reduces the task of producing the climatological summertime
mean to one step:
</para><example endspaces=" ">
<pre xml:space="preserve"># Step 1: Compute climatological summertime mean
ncra -O -F -d time,6,,12,3 -n 150,4,1 1850.nc clm_JJA.nc
</pre></example>
<noindent></noindent>
<para>The <acronym><acronymword>SSC</acronymword></acronym> option instructs <command>ncra</command> (or <command>ncrcat</command>)
to process  files in groups of three records. 
To better understand the meaning of each argument to the <samp>-d</samp>
hyperslab option, read it this way: &textldquo;for the time dimension start with
the sixth record, continue without end, repeat the process every twelfth
record, and define a sub-cycle as three consecutive records&textrdquo;. 
</para>
<para>A separate option, <samp>--mro</samp>, instructs <command>ncra</command> to output 
its results from each sub-group, and to produce a <dfn>Multi-Record Output</dfn>
(MRO) file rather than a <dfn>Single-Record Output</dfn> (SRO) file.
Unless <samp>--mro</samp> is specified, <command>ncra</command> collects together all
the sub-groups, operates on their ensemble, and produces a single
output record. 
The addition of <samp>--mro</samp> to the above example causes <command>ncra</command>
to archive all (150) annual summertime means to one file:
</para><example endspaces=" ">
<pre xml:space="preserve"># Step 1: Archive all 150 summertime means in one file
ncra --mro -O -F -d time,6,,12,3 -n 150,4,1 1850.nc 1850_2009_JJA.nc
# ...or all (150) annual means...
ncra --mro -O -d time,,,12,12 -n 150,4,1 1850.nc 1850_2009.nc
</pre></example>
<noindent></noindent>
<para>These operations generate and require no intermediate files.
This contrasts to previous <acronym><acronymword>NCO</acronymword></acronym> methods, which require
generating, averaging, then catenating 150 files.  
The <samp>--mro</samp> option has no effect on, or rather is redundant for, 
<command>ncrcat</command> since <command>ncrcat</command> always outputs all selected
records.
</para>
<html endspaces=" ">
&lt;a name=&quot;msa&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#msa &textndash;&gt;
&lt;a name=&quot;mlt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mlt &textndash;&gt;
</html>
</section>
<node name="Multislabs" spaces=" "><nodename>Multislabs</nodename><nodenext spaces=" ">Wrapped Coordinates</nodenext><nodeprev spaces=" ">Subcycle</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Multislabs </sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="596">multislab</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="597">multi-hyperslab</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="598"><acronym><acronymword>MSA</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="599"><code>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="600"><code>--dimension <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="601"><code>--dmn <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="602"><code>--msa</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="603"><code>--msa_usr_rdr</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="604"><code>--msa_user_order</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncbo</command>, <command>nces</command>, <command>ncecat</command>,
<command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>, <command>ncra</command>,
<command>ncrcat</command>&linebreak; 
Short options: <samp>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>&linebreak;
Long options: 
<samp>--dimension <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>,&linebreak; 
<samp>--dmn <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>&linebreak;
<samp>--msa_usr_rdr</samp>, <samp>--msa_user_order</samp>&linebreak;
</para></cartouche>
<para>A <w>multislab</w> is a union of one or more hyperslabs.
One defines multislabs by chaining together hyperslab commands, i.e., 
<kbd>-d</kbd> options (<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>).
Support for specifying a <dfn>multi-hyperslab</dfn> or <dfn>multislab</dfn> for
any variable was first added to <command>ncks</command> in late 2002.
The other operators received these capabilities in April 2008.
Multi-slabbing is often referred to by the acronym <acronym><acronymword>MSA</acronymword></acronym>,
which stands for &textldquo;Multi-Slabbing Algorithm&textrdquo;.
As explained below, the user may additionally request that the
multislabs be returned in the user-specified order, rather than the
on-disk storage order. 
Although <acronym><acronymword>MSA</acronymword></acronym> user-ordering has been available in all operators
since 2008, most users were unaware of it since the documentation
(below, and in the man pages) was not written until July 2013.
</para>
<para>Multislabs overcome many restraints that limit simple hyperslabs.
<w>A single</w> <kbd>-d</kbd> option can only specify a contiguous and/or
a regularly spaced multi-dimensional data array.
Multislabs are constructed from multiple <kbd>-d</kbd> options and may
therefore have non-regularly spaced arrays.
For example, suppose it is desired to operate on all longitudes
from 10.0 to 20.0 and from 80.0 to <w>90.0 degrees</w>.
The combined range of longitudes is not selectable in a single 
hyperslab specfication of the form 
<samp>-d <var>dimension</var>,<var>min</var>,<var>max</var></samp> or  
<samp>-d <var>dimension</var>,<var>min</var>,<var>max</var>,<var>stride</var></samp> because its
elements are irregularly spaced in coordinate space (and presumably 
in index space too). 
The multislab specification for obtaining these values is simply
the union of the hyperslabs specifications that comprise the multislab,
i.e., 
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -d lon,10.,20. -d lon,80.,90. in.nc out.nc
ncks -d lon,10.,15. -d lon,15.,20. -d lon,80.,90. in.nc out.nc
</pre></example>
<noindent></noindent>
<para>Any number of hyperslabs specifications may be chained together
to specify the multislab.
<acronym><acronymword>MSA</acronymword></acronym> creates an output dimension equal in size to the sum of
the sizes of the multislabs.
This can be used to extend and or pad coordinate grids.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="605">stride</indexterm></cindex>
<para>Users may specify redundant ranges of indices in a multislab, e.g., 
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -d lon,0,4 -d lon,2,9,2 in.nc out.nc
</pre></example>
<noindent></noindent>
<para>This command retrieves the first five longitudes, and then every other
longitude value up to the tenth.
Elements 0, 2, <w>and 4</w> are specified by both hyperslab arguments (hence
this is redundant) but will count only once if an arithmetic operation
is being performed.  
This example uses index-based (not coordinate-based) multislabs because
the <var>stride</var> option only supports index-based hyper-slabbing. 
<xref label="Stride"><xrefnodename>Stride</xrefnodename></xref>, for more information on the <var>stride</var> option.
</para>
<para>Multislabs are more efficient than the alternative of sequentially
performing hyperslab operations and concatenating the results.
<cindex index="cp" spaces=" "><indexterm index="cp" number="606">I/O</indexterm></cindex>
This is because <acronym><acronymword>NCO</acronymword></acronym> employs a novel multislab algorithm to
minimize the number of I/O operations when retrieving irregularly spaced
data from disk.
The <acronym><acronymword>NCO</acronymword></acronym> multislab algorithm retrieves each element from disk
once and only once.
Thus users may take some shortcuts in specifying multislabs and the
algorithm will obtain the intended values.
Specifying redundant ranges is not encouraged, but may be useful on
occasion and will not result in unintended consequences.
</para>
<para>Suppose the <var>Q</var> variable contains three dimensional arrays of
distinct chemical constituents in no particular order.
We are interested in the NOy species in a certain geographic range. 
Say that NO, NO2, and N2O5 are <w>elements 0</w>, 1, <w>and 5</w> of the
<var>species</var> dimension of <var>Q</var>.
The multislab specification might look something like
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -d species,0,1 -d species,5 -d lon,0,4 -d lon,2,9,2 in.nc out.nc
</pre></example>
<noindent></noindent>
<para>Multislabs are powerful because they may be specified for every
dimension at the same time.
Thus multislabs obsolete the need to execute multiple <command>ncks</command>
commands to gather the desired range of data.
</para>
<html endspaces=" ">
&lt;a name=&quot;msa_usr_rdr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#msa_usr_rdr &textndash;&gt;
</html>
<para>The <acronym><acronymword>MSA</acronymword></acronym> user-order switch <samp>--msa_usr_rdr</samp> (or
<samp>--msa_user_order</samp>, both of which shorten to <samp>--msa</samp>) 
requests that the multislabs be output in the user-specified
order from the command-line, rather than in the input-file on-disk
storage order.  
This allows the user to perform complex data re-ordering in one
operation that would otherwise require cumbersome steps of
hyperslabbing, concatenating, and permuting. 
Consider the recent example of a user who needed to convert datasets
stored with the longitude coordinate <code>Lon</code> ranging from
[&minus;180,180) to datasets that follow the [0,360) convention.
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -H -v Lon in.nc
Lon[0]=-180
Lon[1]=-90
Lon[2]=0
Lon[3]=90
</pre></example>
<noindent></noindent>
<para>Although simple in theory, this task requires both mathematics to
change the numerical value of the longitude coordinate, data
hyperslabbing to split the input on-disk arrays at Greenwich, and data
re-ordering within to stitch the western hemisphere onto the eastern
hemisphere at the date-line.
The <samp>--msa</samp> user-order switch overrides the default that data are
output in the same order in which they are stored on-disk in the input
file, and instead stores them in the same order as the multi-slabs are
given to the command line.
This default is intuitive and is not important in most uses.
However, the <acronym><acronymword>MSA</acronymword></acronym> user-order switch allows users to meet
their output order needs by specifying multi-slabs in a certain order.
Compare the results of default ordering to user-ordering for longitude:
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -O -H       -v Lon -d Lon,0.,180. -d Lon,-180.,-1.0 in.nc
Lon[0]=-180 
Lon[1]=-90 
Lon[2]=0 
Lon[3]=90 
% ncks -O -H --msa -v Lon -d Lon,0.,180. -d Lon,-180.,-1.0 in.nc
Lon[0]=0 
Lon[1]=90 
Lon[2]=-180 
Lon[3]=-90 
</pre></example>
<noindent></noindent>
<para>The two multi-slabs are the same but they can be presented to screen,
or to an output file, in either order. 
The second example shows how to place the western hemisphere after the
eastern hemisphere, although they are stored in the opposite order in
the input file. 
</para>
<para>With this background, one sees that the following commands suffice to
rotate the input file by 180 degrees longitude:
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -O -v LatLon --msa -d Lon,0.,180. -d Lon,-180.,-1.0 in.nc out.nc
% ncap2 -O -s 'where(Lon &lt; 0) Lon=Lon+360' out.nc out.nc
% ncks -C -H -v LatLon ~/nco/data/in.nc
Lat[0]=-45 Lon[0]=-180 LatLon[0]=0 
Lat[0]=-45 Lon[1]=-90 LatLon[1]=1 
Lat[0]=-45 Lon[2]=0 LatLon[2]=2 
Lat[0]=-45 Lon[3]=90 LatLon[3]=3 
Lat[1]=45 Lon[0]=-180 LatLon[4]=4 
Lat[1]=45 Lon[1]=-90 LatLon[5]=5 
Lat[1]=45 Lon[2]=0 LatLon[6]=6 
Lat[1]=45 Lon[3]=90 LatLon[7]=7 
% ncks -C -H -v LatLon ~/out.nc
Lat[0]=-45 Lon[0]=0 LatLon[0]=2 
Lat[0]=-45 Lon[1]=90 LatLon[1]=3 
Lat[0]=-45 Lon[2]=180 LatLon[2]=0 
Lat[0]=-45 Lon[3]=270 LatLon[3]=1 
Lat[1]=45 Lon[0]=0 LatLon[4]=6 
Lat[1]=45 Lon[1]=90 LatLon[5]=7 
Lat[1]=45 Lon[2]=180 LatLon[6]=4 
Lat[1]=45 Lon[3]=270 LatLon[7]=5 
</pre></example>
<noindent></noindent>
<para>There are other workable, valid methods to accomplish this rotation, yet
none are simpler nor more efficient than utilizing <acronym><acronymword>MSA</acronymword></acronym>
user-ordering. 
Some final comments on applying this algorithm:
Be careful to specify hemispheres that do not overlap, e.g., by
inadvertently specifying coordinate ranges that both include Greenwich. 
Some users will find using index-based rather than coordinate-based
hyperslabs makes this clearer.
</para>
<html endspaces=" ">
&lt;a name=&quot;wrp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#wrp &textndash;&gt;
</html>
</section>
<node name="Wrapped-Coordinates" spaces=" "><nodename>Wrapped Coordinates</nodename><nodenext spaces=" ">Auxiliary Coordinates</nodenext><nodeprev spaces=" ">Multislabs</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Wrapped Coordinates</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="607">wrapped coordinates</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="608">longitude</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="609"><code>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="610"><code>--dimension <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="611"><code>--dmn <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncks</command>&linebreak;
Short options: <samp>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>&linebreak;
Long options: 
<samp>--dimension <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>,&linebreak; 
<samp>--dmn <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>&linebreak;
</para></cartouche>
<para><w>A <dfn>wrapped coordinate</dfn></w> is a coordinate whose values increase or
decrease monotonically (nothing unusual so far), but which represents a
dimension that ends where it begins (i.e., wraps around on itself).
Longitude (i.e., degrees on a circle) is a familiar example of a wrapped
coordinate.
Longitude increases to the East of Greenwich, England, where it is
defined to be zero.
Halfway around the globe, the longitude is <w>180 degrees</w> East (or West). 
Continuing eastward, longitude increases to <w>360 degrees</w> East at
Greenwich. 
The longitude values of most geophysical data are either in the range
[0,360), or [&minus;180,180).
In either case, the Westernmost and Easternmost longitudes are
numerically separated by <w>360 degrees</w>, but represent contiguous
regions on the globe.
For example, the Saharan desert stretches from roughly 340 to 
<w>50 degrees</w> East.
Extracting the hyperslab of data representing the Sahara from a global
dataset presents special problems when the global dataset is stored
consecutively in longitude from 0 to <w>360 degrees</w>.
This is because the data for the Sahara will not be contiguous in the
<var>input-file</var> but is expected by the user to be contiguous in the
<var>output-file</var>. 
In this case, <command>ncks</command> must invoke special software routines to
assemble the desired output hyperslab from multiple reads of the
<var>input-file</var>. 
</para>
<para>Assume the domain of the monotonically increasing longitude coordinate
<code>lon</code> is <math>0 &lt; <var>lon</var> &lt; 360</math>. 
<command>ncks</command> will extract a hyperslab which crosses the Greenwich
meridian simply by specifying the westernmost longitude as <var>min</var> and
the easternmost longitude as <var>max</var>.
The following commands extract a hyperslab containing the Saharan desert:
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -d lon,340.,50. in.nc out.nc
ncks -d lon,340.,50. -d lat,10.,35. in.nc out.nc
</pre></example>
<noindent></noindent>
<para>The first example selects data in the same longitude range as the Sahara. 
The second example further constrains the data to having the same
latitude as the Sahara.
The coordinate <code>lon</code> in the <var>output-file</var>, <file>out.nc</file>, will
no longer be monotonic! 
The values of <code>lon</code> will be, e.g., <samp>340, 350, 0, 10, 20, 30,
40, 50</samp>. 
This can have serious implications should you run <file>out.nc</file> through
another operation which expects the <code>lon</code> coordinate to be
monotonically increasing.
Fortunately, the chances of this happening are slim, since <code>lon</code>
has already been hyperslabbed, there should be no reason to hyperslab
<code>lon</code> again.
Should you need to hyperslab <code>lon</code> again, be sure to give
dimensional indices as the hyperslab arguments, rather than coordinate
values (<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>).
</para>
<html endspaces=" ">
&lt;a name=&quot;aux&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#aux &textndash;&gt;
&lt;a name=&quot;auxiliary&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#auxiliary &textndash;&gt;
&lt;a name=&quot;-X&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-X &textndash;&gt;
&lt;a name=&quot;std_nm&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#std_nm &textndash;&gt;
&lt;a name=&quot;standard_name&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#standard_name &textndash;&gt;
</html>
</section>
<node name="Auxiliary-Coordinates" spaces=" "><nodename>Auxiliary Coordinates</nodename><nodenext spaces=" ">UDUnits Support</nodenext><nodeprev spaces=" ">Wrapped Coordinates</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Auxiliary Coordinates</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="612"><code>-X</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="613"><code>--auxiliary</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="614"><code>standard_name</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="615"><code>coordinates</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="616"><acronym><acronymword>CF</acronymword></acronym> conventions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="617"><code>-X <var>lon_min</var>,<var>lon_max</var>,<var>lat_min</var>,<var>lat_max</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="618"><code>--auxiliary <var>lon_min</var>,<var>lon_max</var>,<var>lat_min</var>,<var>lat_max</var></code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncbo</command>, <command>nces</command>, <command>ncecat</command>,
<command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>, <command>ncra</command>,
<command>ncrcat</command>&linebreak; 
Short options: <samp>-X <var>lon_min</var>,<var>lon_max</var>,<var>lat_min</var>,<var>lat_max</var></samp>&linebreak;
Long options: 
<samp>--auxiliary <var>lon_min</var>,<var>lon_max</var>,<var>lat_min</var>,<var>lat_max</var></samp>&linebreak;
</para></cartouche>
<para>Utilize auxiliary coordinates specified in values of the coordinate
variable&textrsquo;s <code>standard_name</code> attributes, if any, when interpreting 
hyperslab and multi-slab options. 
Also <samp>--auxiliary</samp>.
This switch supports hyperslabbing cell-based grids over coordinate
ranges. 
This works on datasets that associate coordinate variables to
grid-mappings using the <acronym><acronymword>CF</acronymword></acronym>-convention (<pxref label="CF-Conventions"><xrefnodename>CF Conventions</xrefnodename></pxref>)   
<code>coordinates</code> and <code>standard_name</code> attributes described 
<uref><urefurl>http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#coordinate-system</urefurl><urefdesc spaces=" ">here</urefdesc></uref>. 
Currently, <acronym><acronymword>NCO</acronymword></acronym> understands auxiliary coordinate variables 
pointed to by the <code>standard_name</code> attributes for <var>latitude</var> and 
<var>longitude</var>.   
Cells that contain a value within the user-specified range 
[<var>lon_min</var>,<var>lon_max</var>,<var>lat_min</var>,<var>lat_max</var>] are
included in the output hyperslab.  
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="619">cell-based grids</indexterm></cindex>
<para>A cell-based grid collapses the horizontal spatial information 
(latitude and longitude) and stores it along a one-dimensional
coordinate that has a one-to-one mapping to both latitude and longitude
coordinates. 
Rectangular (in longitude and latitude) horizontal hyperslabs cannot
be selected using the typical procedure (<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>) of
separately specifying <samp>-d</samp> arguments for longitude and latitude.
Instead, when the <samp>-X</samp> is used, <acronym><acronymword>NCO</acronymword></acronym> learns the names of
the latitude and longitude coordinates by searching the
<code>standard_name</code> attribute of all variables until it finds
the two variables whose <code>standard_name</code>&textrsquo;s are &textldquo;latitude&textrdquo; and 
&textldquo;longitude&textrdquo;, respectively. 
This <code>standard_name</code> attribute for latitude and longitude
coordinates follows the <acronym><acronymword>CF</acronymword></acronym>-convention  
(<pxref label="CF-Conventions"><xrefnodename>CF Conventions</xrefnodename></pxref>). 
</para>
<para>Putting it all together, consider a variable <var>gds_3dvar</var> output from 
simulations on a cell-based geodesic grid. 
Although the variable contains three dimensions of data (time, latitude,
and longitude), it is stored in the netCDF file with only two dimensions,
<code>time</code> and <code>gds_crd</code>.  
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -m -C -v gds_3dvar ~/nco/data/in.nc
gds_3dvar: type NC_FLOAT, 2 dimensions, 4 attributes, chunked? no, \
 compressed? no, packed? no, ID = 41
gds_3dvar RAM size is 10*8*sizeof(NC_FLOAT) = 80*4 = 320 bytes
gds_3dvar dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 20 \ 
 (CRD)(REC)
gds_3dvar dimension 1: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
gds_3dvar attribute 0: long_name, size = 17 NC_CHAR, value = \ 
 Geodesic variable
gds_3dvar attribute 1: units, size = 5 NC_CHAR, value = meter
gds_3dvar attribute 2: coordinates, size = 15 NC_CHAR, value = \
 lat_gds lon_gds
gds_3dvar attribute 3: purpose, size = 64 NC_CHAR, value = \ 
 Test auxiliary coordinates like those that define geodesic grids
</pre></example>
<para>The <code>coordinates</code> attribute lists the names of the latitude and
longitude coordinates, <code>lat_gds</code> and <code>lon_gds</code>, respectively. 
The <code>coordinates</code> attribute is recommended though optional.
With it, the user can immediately identify which variables contain
the latitude and longitude coordinates.
Without a <code>coordinates</code> attribute it would be unclear at first
glance whether a variable resides on a cell-based grid.
In this example, <code>time</code> is a normal record dimension and
<code>gds_crd</code> is the cell-based dimension.
</para>
<para>The cell-based grid file must contain two variables whose
<code>standard_name</code> attributes are &textldquo;latitude&textrdquo;, and &textldquo;longitude&textrdquo;:
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -m -C -v lat_gds,lon_gds ~/nco/data/in.nc
lat_gds: type NC_DOUBLE, 1 dimensions, 4 attributes, \
 chunked? no, compressed? no, packed? no, ID = 37
lat_gds RAM size is 8*sizeof(NC_DOUBLE) = 8*8 = 64 bytes
lat_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
lat_gds attribute 0: long_name, size = 8 NC_CHAR, value = Latitude
lat_gds attribute 1: standard_name, size = 8 NC_CHAR, value = latitude
lat_gds attribute 2: units, size = 6 NC_CHAR, value = degree
lat_gds attribute 3: purpose, size = 62 NC_CHAR, value = \ 
 1-D latitude coordinate referred to by geodesic grid variables

lon_gds: type NC_DOUBLE, 1 dimensions, 4 attributes, \
 chunked? no, compressed? no, packed? no, ID = 38
lon_gds RAM size is 8*sizeof(NC_DOUBLE) = 8*8 = 64 bytes
lon_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
lon_gds attribute 0: long_name, size = 9 NC_CHAR, value = Longitude
lon_gds attribute 1: standard_name, size = 9 NC_CHAR, value = longitude
lon_gds attribute 2: units, size = 6 NC_CHAR, value = degree
lon_gds attribute 3: purpose, size = 63 NC_CHAR, value = \
 1-D longitude coordinate referred to by geodesic grid variables
</pre></example>
<para>In this example <code>lat_gds</code> and <code>lon_gds</code> represent the 
latitude or longitude, respectively, of cell-based variables.
These coordinates (must) have the same single dimension (<code>gds_crd</code>,
in this case) as the cell-based variables.
And the coordinates must be one-dimensional&textmdash;multidimensional
coordinates will not work.
</para>
<para>This infrastructure allows <acronym><acronymword>NCO</acronymword></acronym> to identify, interpret, and
process (e.g., hyperslab) the variables on cell-based grids as easily
as it works with regular grids.
To time-average all the values between zero and <w>180 degrees</w>
longitude and between plus and minus <w>30 degress</w> latitude, we use
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -O -X 0.,180.,-30.,30. -v gds_3dvar in.nc out.nc
</pre></example>
<para><acronym><acronymword>NCO</acronymword></acronym> accepts multiple <samp>-X</samp> arguments for cell-based grid
multi-slabs, just as it accepts multiple <samp>-d</samp> arguments for 
multi-slabs of regular coordinates.
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -O -X 0.,180.,-30.,30. -X 270.,315.,45.,90. in.nc out.nc
</pre></example>
<para>The arguments to <samp>-X</samp> are always interpreted as floating point
numbers, i.e., as coordinate values rather than dimension indices
so that these two commands produce identical results
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -X 0.,180.,-30.,30. in.nc out.nc
ncra -X 0,180,-30,30 in.nc out.nc
</pre></example>
<para>In contrast, arguments to <samp>-d</samp> require decimal places to be
recognized as coordinates not indices (<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>).  
We recommend always using decimal points with <samp>-X</samp> arguments
to avoid confusion.
</para>
<html endspaces=" ">
&lt;a name=&quot;UDUnits&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#UDUnits &textndash;&gt;
&lt;a name=&quot;UDUnits2&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#UDUnits2 &textndash;&gt;
</html>
</section>
<node name="UDUnits-Support" spaces=" "><nodename>UDUnits Support</nodename><nodenext spaces=" ">Rebasing Time Coordinate</nodenext><nodeprev spaces=" ">Auxiliary Coordinates</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>UDUnits Support </sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="620">UDUnits</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="621">Unidata</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="622"><code>units</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="623">attribute, <code>units</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="624"><code>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="625"><code>--dimension <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="626"><code>--dmn <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncbo</command>, <command>nces</command>, <command>ncecat</command>,
<command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>, <command>ncra</command>,
<command>ncrcat</command>, <command>ncwa</command>&linebreak; 
Short options: <samp>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>&linebreak;
Long options: 
<samp>--dimension <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>,&linebreak; 
<samp>--dmn <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</samp>&linebreak;
</para></cartouche>
<para>There is more than one way to hyperskin a cat.
The <uref><urefurl>http://www.unidata.ucar.edu/packages/udunits</urefurl><urefdesc spaces=" ">UDUnits</urefdesc></uref> package 
provides a library which, if present, <acronym><acronymword>NCO</acronymword></acronym> uses to translate
user-specified physical dimensions into the physical dimensions of data
stored in netCDF files.
Unidata provides UDUnits under the same terms as netCDF, so sites should
install both.
Compiling <acronym><acronymword>NCO</acronymword></acronym> with UDUnits support is currently optional but
may become required in a future version of <acronym><acronymword>NCO</acronymword></acronym>.
</para>
<para>Two examples suffice to demonstrate the power and convenience of UDUnits  
support. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="627">MKS units</indexterm></cindex>
First, consider extraction of a variable containing non-record
coordinates with physical dimensions stored in MKS units.
In the following example, the user extracts all wavelengths
in the visible portion of the spectrum in terms of the units
very frequently used in visible spectroscopy, microns:
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -C -H -v wvl -d wvl,&quot;0.4 micron&quot;,&quot;0.7 micron&quot; in.nc
wvl[0]=5e-07 meter
</pre></example>
<noindent></noindent>
<cindex index="cp" spaces=" "><indexterm index="cp" number="628"><code>units</code></indexterm></cindex>
<para>The hyperslab returns the correct values because the <var>wvl</var> variable
is stored on disk with a length dimension that UDUnits recognizes in the 
<code>units</code> attribute.
The automagical algorithm that implements this functionality is worth
describing since understanding it helps one avoid some potential
pitfalls. 
First, the user includes the physical units of the hyperslab dimensions 
she supplies, separated by a simple space from the numerical values of
the hyperslab limits.
She encloses each coordinate specifications in quotes so that the shell
does not break the <emph>value-space-unit</emph> string into separate
arguments before passing them to <acronym><acronymword>NCO</acronymword></acronym>. 
Double quotes (<kbd>&quot;foo&quot;</kbd>) or single quotes (<kbd>'foo'</kbd>) are equally
valid for this purpose. 
Second, <acronym><acronymword>NCO</acronymword></acronym> recognizes that units translation is requested
because each hyperslab argument contains text characters and non-initial
spaces.  
Third, <acronym><acronymword>NCO</acronymword></acronym> determines whether the <var>wvl</var> is dimensioned
with a coordinate variable that has a <code>units</code> attribute. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="629">coordinate variable</indexterm></cindex>
In this case, <var>wvl</var> itself is a coordinate variable.
The value of its <code>units</code> attribute is <code>meter</code>. 
Thus <var>wvl</var> passes this test so UDUnits conversion is attempted. 
If the coordinate associated with the variable does not contain a 
<code>units</code> attribute, then <acronym><acronymword>NCO</acronymword></acronym> aborts.
Fourth, <acronym><acronymword>NCO</acronymword></acronym> passes the specified and desired dimension strings  
(microns are specified by the user, meters are required by
<acronym><acronymword>NCO</acronymword></acronym>) to the UDUnits library.
Fifth, the UDUnits library that these dimension are commensurate
and it returns the appropriate linear scaling factors to convert from 
microns to meters to <acronym><acronymword>NCO</acronymword></acronym>.
If the units are incommensurate (i.e., not expressible in the same
fundamental MKS units), or are not listed in the UDUnits database, then 
NCO aborts since it cannot determine the user&textrsquo;s intent.
Finally, <acronym><acronymword>NCO</acronymword></acronym> uses the scaling information to convert the
user-specified hyperslab limits into the same physical dimensions as
those of the corresponding cooridinate variable on disk.
At this point, <acronym><acronymword>NCO</acronymword></acronym> can perform a coordinate hyperslab using
the same algorithm as if the user had specified the hyperslab without
requesting units conversion.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="630"><code>units</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="631"><code>time</code></indexterm></cindex>
<para>The translation and dimensional innterpretation of time coordinates
shows a more powerful, and probably more common, UDUnits application.
In this example, the user prints all data between <w>4 PM</w> and <w>7 PM</w>
on <w>December 8</w>, 1999, from a variable whose time dimension is hours 
since the year 1900:
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -u -H -C -v time_udunits -d time_udunits,&quot;1999-12-08 \
  16:00:0.0&quot;,&quot;1999-12-08 19:00:0.0&quot; in.nc
time_udunits[1]=876018 hours since 1900-01-01 00:00:0.0
</pre></example>
<noindent></noindent>
<cindex index="cp" spaces=" "><indexterm index="cp" number="632">stride</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="633">whitespace</indexterm></cindex>
<para>Here, the user invokes the stride (<pxref label="Stride"><xrefnodename>Stride</xrefnodename></pxref>) capability to obtain 
every other timeslice.
This is possible because the UDUnits feature is additive, not
exclusive&textmdash;it works in conjunction with all other hyperslabbing
(<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>) options and in all operators which support
hyperslabbing.
The following example shows how one might average data in a 
time period spread across multiple input files
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -d time,&quot;1939-09-09 12:00:0.0&quot;,&quot;1945-05-08 00:00:0.0&quot; \
  in1.nc in2.nc in3.nc out.nc
</pre></example>
<noindent></noindent>
<para>Note that there is no excess whitespace before or after the individual
elements of the <samp>-d</samp> argument.
<cindex index="cp" spaces=" "><indexterm index="cp" number="634">shell</indexterm></cindex>
This is important since, as far as the shell knows, <samp>-d</samp> takes
only <emph>one</emph> command-line argument.
Parsing this argument into its component
<code><var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]</code> elements 
(<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>) is the job of <acronym><acronymword>NCO</acronymword></acronym>.
When unquoted whitespace is present between these elements, the shell
passes <acronym><acronymword>NCO</acronymword></acronym> arugment fragments which will not parse as
intended. 
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> implemented support for the UDUnits2 library with version   
3.9.2 (August, 2007).
The
<uref><urefurl>http://www.unidata.ucar.edu/software/udunits/udunits-2/udunits2.html</urefurl><urefdesc spaces="\n">UDUnits2</urefdesc></uref> package supports non-ASCII characters and logarithmic units. 
We are interested in user-feedback on these features.
</para>
<para>One aspect that deserves mention is that UDUnits, and thus
<acronym><acronymword>NCO</acronymword></acronym>, supports run-time definition of the location of the
relevant UDUnits databases. 
With UDUnits <w>version 1</w>, users may specify the directory which
contains the UDUnits database, <file>udunits.dat</file>, via the
<code>UDUNITS_PATH</code> environment variable.
With UDUnits <w>version 2</w>, users may specify the UDUnits database file 
itself, <file>udunits2.xml</file>, via the <code>UDUNITS2_XML_PATH</code> 
environment variable.
</para><example endspaces=" ">
<pre xml:space="preserve"># UDUnits1
export UDUNITS_PATH='/unusual/location/share/udunits'
# UDUnits2
export UDUNITS2_XML_PATH='/unusual/location/share/udunits/udunits2.xml'
</pre></example>
<para>This run-time flexibility can enable the full functionality of
pre-built binaries on machines with libraries in different locations.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="635">Climate and Forecast Metadata Convention</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="636"><acronym><acronymword>CF</acronymword></acronym> conventions</indexterm></cindex>
<para>The <uref><urefurl>http://www.unidata.ucar.edu/packages/udunits</urefurl><urefdesc spaces=" ">UDUnits</urefdesc></uref>
package documentation describes the supported formats of time
dimensions. 
Among the metadata conventions that adhere to these formats are the  
<uref><urefurl>http://cf-pcmdi.llnl.gov</urefurl><urefdesc spaces=" \n">Climate and Forecast (CF) Conventions</urefdesc></uref> and the 
<uref><urefurl>http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html</urefurl><urefdesc spaces="\n">Cooperative Ocean/Atmosphere Research Data Service (COARDS) Conventions</urefdesc></uref>.
The following <samp>-d arguments</samp> extract the same data using 
commonly encountered time dimension formats: 
<!-- c fxm add more formats here -->
</para><example endspaces=" ">
<pre xml:space="preserve">-d time,'1918-11-11 00:00:0.0','1939-09-09 00:00:0.0'
-d time,'1918-11-11 00:00:0.0','1939-09-09 00:00:0.0'
-d time,'1918-11-11T00:00:0.0Z','1939-09-09T00:00:0.0Z'
-d time,'1918-11-11','1939-09-09'
-d time,'1918-11-11','1939-9-9'
</pre></example>
<noindent></noindent>
<para>All of these formats include at least one dash <kbd>-</kbd> in a
non-leading character position (a dash in a leading character position 
is a negative sign). 
<acronym><acronymword>NCO</acronymword></acronym> assumes that a space, colon, or non-leading dash in a
limit string indicates that a UDUnits units conversion is requested.
Some date formats like YYYYMMDD that are valid in UDUnits are ambiguous
to <acronym><acronymword>NCO</acronymword></acronym> because it cannot distinguish a purely numerical date
(i.e., no dashes or text characters in it) from a coordinate or index
value: 
</para><example endspaces=" ">
<pre xml:space="preserve">-d time,1918-11-11 # Interpreted as the date November 11, 1918
-d time,19181111   # Interpreted as time-dimension index 19181111
-d time,19181111.  # Interpreted as time-coordinate value 19181111.0
</pre></example>
<para>Hence, use the YYYY-MM-DD format rather than YYYYMMDD for dates.
</para>
<noindent></noindent>
<para>As of version 4.0.0 (January, 2010), <acronym><acronymword>NCO</acronymword></acronym> supports some
calendar attributes specified by the <acronym><acronymword>CF</acronymword></acronym> conventions. 
</para><table commandarg="asis" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="asis"><strong>Supported types:</strong> </itemformat></item>
</tableterm><tableitem><para>&quot;365_day&quot;/&quot;noleap&quot;, &quot;360_day&quot;, &quot;gregorian&quot;, &quot;standard&quot; 
</para></tableitem></tableentry><tableentry><tableterm><item spaces="  "><itemformat command="asis"><strong>Unsupported types:</strong> </itemformat></item>
</tableterm><tableitem><para>&quot;366_day&quot;/&quot;all_leap&quot;,&quot;proleptic_gregorian&quot;,&quot;julian&quot;,&quot;none&quot; 
</para></tableitem></tableentry></table>
<para>Unsupported types default to mixed Gregorian/Julian as defined by 
UDUnits. 
</para>
<noindent></noindent> <para>An Example: Consider the following netCDF variable
</para>
<example endspaces=" ">
<pre xml:space="preserve">variables:
  double lon_cal(lon_cal) ;
    lon_cal:long_name = &quot;lon_cal&quot; ;
    lon_cal:units = &quot;days since 1964-2-28 0:0:0&quot; ;
    lon_cal:calendar = &quot;365_day&quot; ;
data:
  lon_cal = 1,2,3,4,5,6,7,8,9,10;
</pre></example> 
<para><samp>ncks -v lon_cal -d lon_cal,'1964-3-1 0:00:0.0','1964-3-4 00:00:0.0'</samp>
results in <code>lon_cal=1,2,3,4</code>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="637">MKS units</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="638">God</indexterm></cindex>
<para>netCDF variables should always be stored with MKS (i.e., God&textrsquo;s) units,
so that application programs may assume MKS dimensions apply to all
input variables. 
The UDUnits feature is intended to alleviate some of the <acronym><acronymword>NCO</acronymword></acronym>
user&textrsquo;s pain when handling MKS units.
It connects users who think in human-friendly units (e.g.,
miles, millibars, days) to extract data which are always stored in God&textrsquo;s
units, MKS (e.g., meters, Pascals, seconds). 
The feature is not intended to encourage writers to store data in 
esoteric units (e.g., furlongs, pounds per square inch, fortnights). 
</para>
<html endspaces=" ">
&lt;a name=&quot;time_rebase&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#time_rebase &textndash;&gt;
&lt;a name=&quot;rbs&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rbs &textndash;&gt;
</html>
</section>
<node name="Rebasing-Time-Coordinate" spaces=" "><nodename>Rebasing Time Coordinate</nodename><nodenext spaces=" ">Multiple Record Dimensions</nodenext><nodeprev spaces=" ">UDUnits Support</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Rebasing Time Coordinate</sectiontitle>
<cartouche endspaces=" ">
<para>Availability: 
<command>ncra</command>, <command>ncrcat</command> 
Short options: None&linebreak;
</para></cartouche>

<para>Time rebasing is invoked when numerous files share a common record
coordinate, and the record coordinate units change among input files.
The rebasing is performed automatically if and only if UDUnits is
installed. 
Usually rebasing occurs when the recoordinate is a time-based variable, 
and times are recorded in units of a time-since-basetime, and the
basetime changes from file to file.
Since the output file can have only one unit (i.e., one basetime) for
the record coordinate, <acronym><acronymword>NCO</acronymword></acronym>, in such cases, chooses the units
of the first input file to be the units of the output file.
It is necessary to &textldquo;rebase&textrdquo; all the input record variables to this
output time unit in order for the output file to have the correct
values.  
</para>
<para>For example suppose the time coordinate is in hours and each day in
January is stored in its own daily file.
Each daily file records the temperature variable <code>tpt(time)</code> 
with an (unadjusted) <code>time</code> coordinate value between 0&textndash;23 hours,
and uses the <code>units</code> attribute to advance the base time:
</para><example endspaces=" ">
<pre xml:space="preserve">file01.nc time:units=&quot;hours since 1990-1-1&quot;   
file02.nc time:units=&quot;hours since 1990-1-2&quot;   
...
file31.nc time:units=&quot;hours since 1990-1-31&quot;   
</pre></example>

<example endspaces=" ">
<pre xml:space="preserve">// Mean noontime temperature in January
ncra -v tpt -d time,&quot;1990-1-1 12:00:00&quot;,&quot;1990-1-31 23:59:59&quot;,24 \
      file??.nc noon.nc    

// Concatenate day2 noon through day3 noon records
ncrcat -v tpt -d time,&quot;1990-1-2 12:00:00&quot;,&quot;1990-1-3 11:59:59&quot; \ 
      file01.nc file02.nc file03.nc noon.nc    

// Results: time is &quot;re-based&quot; to the time units in &quot;file01.nc&quot;
time=36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, \
     51, 52, 53, 54, 55, 56, 57, 58, 59 ;
  
// If we repeat the above command but with only two input files...
ncrcat -v tpt -d time,&quot;1990-1-2 12:00:00&quot;,&quot;1990-1-3 11:59:59&quot; \
      file02.nc file03 noon.nc    

// ...then output time coordinate is based on time units in &quot;file02.nc&quot;
time = 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, \ 
     26, 27, 28, 29, 30, 31, 32, 33, 34, 35 ;
</pre></example>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.2.1 (August, 2012), <acronym><acronymword>NCO</acronymword></acronym>
automatically rebases not only the record coordinate (<code>time</code>, here) 
but also any bounds associated with the record coordinate (e.g.,
<code>time_bnds</code>) (<pxref label="CF-Conventions"><xrefnodename>CF Conventions</xrefnodename></pxref>).
</para>
<html endspaces=" ">
&lt;a name=&quot;mrd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mrd &textndash;&gt;
</html>
</section>
<node name="Multiple-Record-Dimensions" spaces=" "><nodename>Multiple Record Dimensions</nodename><nodenext spaces=" ">Missing Values</nodenext><nodeprev spaces=" ">Rebasing Time Coordinate</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Multiple Record Dimensions</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="639">netCDF4</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="640"><code>--mrd</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="641"><code>--multiple_record_dimensions</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: 
<command>ncecat</command>, <command>ncpdq</command> 
Short options: None&linebreak;
Long options: <samp>--mrd</samp>&linebreak;
</para></cartouche>
<para>The netCDF3 file format allows only one record dimension, and that
dimension must be the first dimension (i.e., the least rapidly varying 
dimension) of any variable in which it appears.
This imposes certain rules on how operators must perform operations
that alter the ordering of dimensions or the number of record variables.
The netCDF4 file format has no such restrictions.
Files and variables may have any number of record dimensions in any
order.
This additional flexibility of netCDF4 can only be realized by
selectively abandoning the constraints that would make operations
behave completely consistently between netCDF3 and netCDF4 files.
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> chooses, by default, to impose netCDF3-based constraints
on netCDF4 files. 
This reduces the number of unanticipated consequences and keeps the
operators functioning in a familiar way.
Put another way, <acronym><acronymword>NCO</acronymword></acronym> limits production of additional record
dimensions so processing netCDF4 files leads to the same results as
processing netCDF4 files.
Users can override this default with the <samp>--mrd</samp> (or 
<samp>--multiple_record_dimension</samp>) switch, which enables netCDF4
variables to accumulate additional record dimensions.
</para>
<para>How can additional record dimensions be produced?
Most commonly <command>ncecat</command> (in record-aggregate mode) defines a new
leading record dimension.
In netCDF4 files this becomes an additional record dimension unless the
original record dimension is changed to a fixed dimension (as must be
done in netCDF3 files). 
Also when <command>ncpdq</command> reorders dimensions it can preserve the
&textldquo;record&textrdquo; property of record variables.
<command>ncpdq</command> tries to define as a record dimension whichever
dimension ends up first in a record variable, and, in netCDF4 files,
this becomes an additional record dimension unless the original record
dimension is changed to a fixed dimension (as must be done in netCDF3
files). 
It it easier if <command>ncpdq</command> and <command>ncecat</command> do not increase
the number of record dimensions in a variable so that is the default.
Use <samp>--mrd</samp> to override this.
</para>
<html endspaces=" ">
&lt;a name=&quot;missing_value&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#missing_value &textndash;&gt;
&lt;a name=&quot;_FillValue&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#_FillValue &textndash;&gt;
&lt;a name=&quot;fll_val&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#fll_val &textndash;&gt;
&lt;a name=&quot;mss_val&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mss_val &textndash;&gt;
&lt;a name=&quot;mss&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mss &textndash;&gt;
</html>
</section>
<node name="Missing-Values" spaces=" "><nodename>Missing Values</nodename><nodenext spaces=" ">Chunking</nodenext><nodeprev spaces=" ">Multiple Record Dimensions</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Missing values</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="642">missing values</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="643">data, missing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="644">averaging data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="645"><code>missing_value</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="646"><code>_FillValue</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncap2</command>, <command>ncbo</command>, <command>nces</command>,
<command>ncflint</command>, <command>ncpdq</command>, <command>ncra</command>, <command>ncwa</command>&linebreak; 
Short options: None&linebreak;
</para></cartouche>

<para>The phrase <dfn>missing data</dfn> refers to data points that are missing,
invalid, or for any reason not intended to be arithmetically processed
in the same fashion as valid data.  
<cindex index="cp" spaces=" "><indexterm index="cp" number="647">arithmetic operators</indexterm></cindex>
The <acronym><acronymword>NCO</acronymword></acronym> arithmetic operators attempt to handle missing data in
an intelligent fashion. 
There are four steps in the <acronym><acronymword>NCO</acronymword></acronym> treatment of missing data:
</para><enumerate first="1" endspaces=" ">
<listitem> 
<para>Identifying variables that may contain missing data. 
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> follows the convention that missing data should be stored
with the <var>_FillValue</var> specified in the variable&textrsquo;s <code>_FillValue</code> 
attributes. 
The <emph>only</emph> way <acronym><acronymword>NCO</acronymword></acronym> recognizes that a variable <emph>may</emph>
contain missing data is if the variable has a <code>_FillValue</code>
attribute. 
In this case, any elements of the variable which are numerically equal
to the <var>_FillValue</var> are treated as missing data.
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> adopted the behavior that the default attribute name, if 
any, assumed to specify the value of data to ignore is <code>_FillValue</code> 
with version 3.9.2 (August, 2007).
Prior to that, the <code>missing_value</code> attribute, if any, was assumed to  
specify the value of data to ignore.
Supporting both of these attributes simultaneously is not practical.
Hence the behavior <acronym><acronymword>NCO</acronymword></acronym> once applied to <var>missing_value</var> it
now applies to any <var>_FillValue</var>. 
<acronym><acronymword>NCO</acronymword></acronym> now treats any <var>missing_value</var> as normal data 
<footnote spaces="\n"><para>The old functionality, i.e., where the ignored values are indicated by
<code>missing_value</code> not <code>_FillValue</code>, may still be selected 
<emph>at <acronym><acronymword>NCO</acronymword></acronym> build time</emph> by compiling <acronym><acronymword>NCO</acronymword></acronym> 
with the token definition 
<!-- c @kbd{CPPFLAGS='-DNCO_MSS_VAL_SNG=missing_value'}. -->
<kbd>CPPFLAGS='-UNCO_USE_FILL_VALUE'</kbd>.
</para></footnote>.
</para>
<findex index="fn" spaces=" "><indexterm index="fn" number="8" mergedindex="cp">ncrename</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="9" mergedindex="cp">ncatted</indexterm></findex>
<para>It has been and remains most advisable to create both <code>_FillValue</code> 
and <code>missing_value</code> attributes with identical values in datasets.
Many legacy datasets contain only <code>missing_value</code> attributes.
<acronym><acronymword>NCO</acronymword></acronym> can help migrating datasets between these conventions.
One may use <command>ncrename</command> (<pxref label="ncrename-netCDF-Renamer"><xrefnodename>ncrename netCDF Renamer</xrefnodename></pxref>) to
rename all <code>missing_value</code> attributes to <code>_FillValue</code>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncrename -a .missing_value,_FillValue inout.nc
</pre></example>
<para>Alternatively, one may use
<command>ncatted</command> (<pxref label="ncatted-netCDF-Attribute-Editor"><xrefnodename>ncatted netCDF Attribute Editor</xrefnodename></pxref>) to
add a <code>_FillValue</code> attribute to all variables
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -O -a _FillValue,,o,f,1.0e36 inout.nc
</pre></example>

</listitem><listitem> 
<para>Converting the <var>_FillValue</var> to the type of the variable, if
neccessary. 
</para>
<para>Consider a variable <var>var</var> of type <var>var_type</var> with a
<code>_FillValue</code> attribute of type <var>att_type</var> containing the
value <var>_FillValue</var>.  
As a guideline, the type of the <code>_FillValue</code> attribute should be
the same as the type of the variable it is attached to.
If <var>var_type</var> equals <var>att_type</var> then <acronym><acronymword>NCO</acronymword></acronym>
straightforwardly compares each value of <var>var</var> to
<var>_FillValue</var> to determine which elements of <var>var</var> are to be
treated as missing data. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="648">C language</indexterm></cindex>
If not, then <acronym><acronymword>NCO</acronymword></acronym> converts <var>_FillValue</var> from
<var>att_type</var> to <var>var_type</var> by using the implicit conversion rules
<w>of C</w>, or, if <var>att_type</var> is <code>NC_CHAR</code>
<footnote><para>For example, the <acronym><acronymword>DOE</acronymword></acronym> <acronym><acronymword>ARM</acronymword></acronym> program often
uses <var>att_type</var> = <code>NC_CHAR</code> and <var>_FillValue</var> =
<samp>-99999.</samp>. 
</para></footnote>, by typecasting the results of the <w>C function</w>
<code>strtod(<var>_FillValue</var>)</code>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="649"><command>ncatted</command></indexterm></cindex>
You may use the <acronym><acronymword>NCO</acronymword></acronym> operator <command>ncatted</command> to change the
<code>_FillValue</code> attribute and all data whose data is
<var>_FillValue</var> to a new value
(<pxref label="ncatted-netCDF-Attribute-Editor"><xrefnodename>ncatted netCDF Attribute Editor</xrefnodename></pxref>).
</para>
</listitem><listitem> 
<para>Identifying missing data during arithmetic operations.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="650">performance</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="651">operator speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="652">speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="653">execution time</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="654">arithmetic operators</indexterm></cindex>
<para>When an <acronym><acronymword>NCO</acronymword></acronym> arithmetic operator processes a variable <var>var</var>
with a <code>_FillValue</code> attribute, it compares each value of
<var>var</var> to <var>_FillValue</var> before performing an operation.
Note the <var>_FillValue</var> comparison imposes a performance penalty
on the operator.
Arithmetic processing of variables which contain the
<code>_FillValue</code> attribute always incurs this penalty, even when
none of the data are missing.
Conversely, arithmetic processing of variables which do not contain the
<code>_FillValue</code> attribute never incurs this penalty.
In other words, do not attach a <code>_FillValue</code> attribute to a
variable which does not contain missing data.
This exhortation can usually be obeyed for model generated data, but it
may be harder to know in advance whether all observational data will be
valid or not.
</para>
</listitem><listitem> 
<para>Treatment of any data identified as missing in arithmetic operators.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="655"><command>nces</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="656"><command>ncra</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="657"><command>ncwa</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="658"><command>ncbo</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="659"><command>ncflint</command></indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> averagers (<command>ncra</command>, <command>nces</command>, <command>ncwa</command>)
do not count any element with the value <var>_FillValue</var> towards the
average. 
<command>ncbo</command> and <command>ncflint</command> define a <var>_FillValue</var> result  
when either of the input values is a <var>_FillValue</var>.
Sometimes the <var>_FillValue</var> may change from file to file in a
multi-file operator, e.g., <command>ncra</command>.
<acronym><acronymword>NCO</acronymword></acronym> is written to account for this (it always compares a
variable to the <var>_FillValue</var> assigned to that variable in the
current file). 
Suffice it to say that, in all known cases, <acronym><acronymword>NCO</acronymword></acronym> does &textldquo;the
right thing&textrdquo;. 
</para>
<para>It is impossible to determine and store the correct result of a binary  
operation in a single variable.
One such corner case occurs when both operands have differing
<var>_FillValue</var> attributes, i.e., attributes with different
numerical values.
Since the output (result) of the operation can only have one
<var>_FillValue</var>, some information may be lost.
In this case, <acronym><acronymword>NCO</acronymword></acronym> always defines the output variable to have
the same <var>_FillValue</var> as the first input variable.
Prior to performing the arithmetic operation, all values of the second
operand equal to the second <var>_FillValue</var> are replaced with the
first <var>_FillValue</var>.
Then the arithmetic operation proceeds as normal, comparing each element 
of each operand to a single <var>_FillValue</var>.
Comparing each element to two distinct <var>_FillValue</var>&textrsquo;s would be
much slower and would be no likelier to yield a more satisfactory
answer. 
In practice, judicious choice of <var>_FillValue</var> values prevents any
important information from being lost.
</para></listitem></enumerate>

<html endspaces=" ">
&lt;a name=&quot;chunking&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#chunking &textndash;&gt;
&lt;a name=&quot;cnk&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnk &textndash;&gt;
&lt;a name=&quot;cnk_sz&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnk_sz &textndash;&gt;
&lt;a name=&quot;chunk_size&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#chunk_size &textndash;&gt;
</html>
</section>
<node name="Chunking" spaces=" "><nodename>Chunking</nodename><nodenext spaces=" ">Deflation</nodenext><nodeprev spaces=" ">Missing Values</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Chunking</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="660"><code>--cnk_byt</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="661"><code>--cnk_dmn</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="662"><code>--cnk_map</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="663"><code>--cnk_plc</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="664"><code>--cnk_scl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="665"><code>--chunk_byte</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="666"><code>--chunk_dimension</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="667"><code>--chunk_map</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="668"><code>--chunk_policy</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="669"><code>--chunk_scalar</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="670">chunking</indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncap2</command>, <command>ncbo</command>, <command>nces</command>,
<command>ncecat</command>, <command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>,
<command>ncra</command>, <command>ncrcat</command>, <command>ncwa</command>&linebreak;
Short options: none&linebreak;
Long options: 
<samp>--cnk_byt <var>cnk_sz</var></samp>, <samp>--chunk_byte <var>cnk_sz</var></samp>&linebreak;
<samp>--cnk_dmn <var>dmn_nm</var>,<var>cnk_sz</var></samp>,
<samp>--chunk_dimension <var>dmn_nm</var>,<var>cnk_sz</var></samp>&linebreak;,
<samp>--cnk_map <var>cnk_map</var></samp>, <samp>--chunk_map <var>cnk_map</var></samp>,&linebreak;
<samp>--cnk_plc <var>cnk_plc</var></samp>, <samp>--chunk_policy <var>cnk_plc</var></samp>,&linebreak;
<samp>--cnk_scl <var>cnk_sz</var></samp>, <samp>--chunk_scalar <var>cnk_sz</var></samp>&linebreak;
</para></cartouche>

<para>All netCDF4-enabled <acronym><acronymword>NCO</acronymword></acronym> operators that define variables 
support a plethora of chunksize options.
Chunking can significantly accelerate or degrade read/write access
to large datasets.
Dataset chunking issues are described in detail
<uref><urefurl>http://www.hdfgroup.org/HDF5/doc/H5.user/Chunking.html</urefurl><urefdesc>here</urefdesc></uref>,
<uref><urefurl>http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_why_it_matters</urefurl><urefdesc>here</urefdesc></uref>,
and
<uref><urefurl>http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes</urefurl><urefdesc>here</urefdesc></uref>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="671">chunking policy</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="672">chunking map</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="673">chunksize</indexterm></cindex>
<para>The <acronym><acronymword>NCO</acronymword></acronym> chunking implementation is designed to be flexible. 
Users control three aspects of the chunking implementation.
These are known as the <dfn>chunking policy</dfn>, <dfn>chunking map</dfn>,
and <dfn>chunksize</dfn>.
The first two are high-level mechanisms that apply to an entire file
and all variables and dimensions, while the third allows per-dimension
specification of parameters. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="674">hyperslab</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="10" mergedindex="cp">ncpdq</indexterm></findex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="675">packing</indexterm></cindex>
The implementation is a hybrid of the <command>ncpdq</command> packing policies  
(<pxref label="ncpdq-netCDF-Permute-Dimensions-Quickly"><xrefnodename>ncpdq netCDF Permute Dimensions Quickly</xrefnodename></pxref>), and the hyperslab
specifications (<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>).
Each aspect is intended to have a sensible default, so that most users
will only need to set one switch to obtain sensible chunking.
Power users can tune the three switches in tandem to obtain optimal
performance. 
</para>
<para>The user specifies the desired chunking policy with the <samp>-P</samp> switch 
(or its long option equivalents, <samp>--cnk_plc</samp> and
<samp>--chunk_policy</samp>) and its <var>cnk_plc</var> argument.
Five chunking policies are currently implemented:&linebreak;   
<cindex index="cp" spaces=" "><indexterm index="cp" number="676"><samp>all</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="677"><samp>g2d</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="678"><samp>g3d</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="679"><samp>xpl</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="680"><samp>xst</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="681"><samp>cnk_all</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="682"><samp>cnk_g2d</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="683"><samp>cnk_g3d</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="684"><samp>cnk_xpl</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="685"><samp>cnk_xst</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="686"><samp>plc_all</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="687"><samp>plc_g2d</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="688"><samp>plc_g3d</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="689"><samp>plc_xpl</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="690"><samp>plc_xst</samp></indexterm></cindex>
</para><table commandarg="dfn" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunk All Variables [<emph>default</emph>]</itemformat></item>
</tableterm><tableitem><para>Definition: Chunk all variables possible.
For obvious reasons, scalar variables cannot be chunked.&linebreak;
Alternate invocation: <code>ncchunk</code>&linebreak;
<var>cnk_plc</var> key values: <samp>all</samp>, <samp>cnk_all</samp>, <samp>plc_all</samp>&linebreak;
Mnemonic: All&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunk Variables with at least Two Dimensions</itemformat></item>
</tableterm><tableitem><para>Definition: Chunk all variables possible with at least two dimensions&linebreak;
Alternate invocation: none&linebreak;
<var>cnk_plc</var> key values: <samp>g2d</samp>, <samp>cnk_g2d</samp>, <samp>plc_g2d</samp>&linebreak;
Mnemonic: <emph>G</emph>reater than or equal to <emph>2</emph> <emph>D</emph>imensions&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunk Variables with at least Three Dimensions</itemformat></item>
</tableterm><tableitem><para>Definition: Chunk all variables possible with at least three dimensions&linebreak;
Alternate invocation: none&linebreak;
<var>cnk_plc</var> key values: <samp>g3d</samp>, <samp>cnk_g3d</samp>, <samp>plc_g3d</samp>&linebreak;
Mnemonic: <emph>G</emph>reater than or equal to <emph>3</emph> <emph>D</emph>imensions&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunk Variables Containing Explicitly Chunked Dimensions</itemformat></item>
</tableterm><tableitem><para>Definition: Chunk all variables possible that contain at least one
dimension whose chunksize was explicitly set with the <samp>--cnk_dmn</samp> option.
Alternate invocation: none&linebreak;
<var>cnk_plc</var> key values: <samp>xpl</samp>, <samp>cnk_xpl</samp>, <samp>plc_xpl</samp>&linebreak;
Mnemonic: E<emph>XPL</emph>icitly specified dimensions&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunk Variables with Existing Chunk Sizes</itemformat></item>
</tableterm><tableitem><para>Definition: Chunk all variables possible that are already chunked in the
input file.
When used in conjunction with <samp>cnk_map=xst</samp> this option preserves
and copies the chunking parameters from the input to the output file.
Alternate invocation: none&linebreak;
<var>cnk_plc</var> key values: <samp>xst</samp>, <samp>cnk_xst</samp>, <samp>plc_xst</samp>&linebreak;
Mnemonic: E<emph>X</emph>i<emph>ST</emph>ing chunk sizes&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Unchunking</itemformat></item>
</tableterm><tableitem><para>Definition: Unchunk all variables possible. 
The <acronym><acronymword>HDF5</acronymword></acronym> storge layer requires that record variables (i.e.,
variables that contain at least one record dimension) must be chunked.
Also variables that are compressed or use checksums must be chunked.
Such variables cannot be unchunked.&linebreak;  
Alternate invocation: <code>ncunchunk</code>&linebreak;
<var>cnk_plc</var> key values: <samp>uck</samp>, <samp>cnk_uck</samp>, <samp>plc_uck</samp>, <samp>unchunk</samp>&linebreak;
Mnemonic: <emph>U</emph>n<emph>C</emph>hun<emph>K</emph>&linebreak;
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>Equivalent key values are fully interchangeable.
Multiple equivalent options are provided to satisfy disparate needs
and tastes of <acronym><acronymword>NCO</acronymword></acronym> users working with scripts and from the
command line.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="691">chunking map</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="692">degenerate dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="693"><var>cnk_map</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="694"><code>-M <var>cnk_map</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="695"><code>--cnk_map <var>cnk_map</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="696"><code>--map <var>cnk_map</var></code></indexterm></cindex>
<para>The chunking algorithms must know the chunksizes of each dimension of
each variable to be chunked.
The correspondence between the input variable shape and the chunksizes
is called the <dfn>chunking map</dfn>. 
The user specifies the desired chunking map with the <samp>-M</samp> switch
(or its long option equivalents, <samp>--cnk_map</samp> and
<samp>--chunk_map</samp>) and its <var>cnk_map</var> argument.
Four chunking maps are currently implemented:&linebreak;
<cindex index="cp" spaces=" "><indexterm index="cp" number="697"><samp>dmn</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="698"><samp>scl</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="699"><samp>prd</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="700"><samp>lfp</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="701"><samp>rd1</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="702"><samp>xst</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="703"><samp>rew</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="704"><samp>cnk_dmn</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="705"><samp>cnk_scl</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="706"><samp>cnk_prd</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="707"><samp>cnk_lfp</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="708"><samp>cnk_rd1</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="709"><samp>cnk_xst</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="710"><samp>map_dmn</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="711"><samp>map_scl</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="712"><samp>map_prd</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="713"><samp>map_lfp</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="714"><samp>map_rd1</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="715"><samp>map_xst</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="716">Chris Barker</indexterm></cindex>
</para><table commandarg="dfn" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunksize Equals Dimension Size [<emph>default</emph>]</itemformat></item>
</tableterm><tableitem><para>Definition: Chunksize defaults to dimension size. 
Explicitly specify chunksizes for particular dimensions with
<samp>--cnk_dmn</samp> option.&linebreak;
<var>cnk_map</var> key values: <samp>dmn</samp>, <samp>cnk_dmn</samp>, <samp>map_dmn</samp>&linebreak;
Mnemonic: <emph>D</emph>i<emph>M</emph>e<emph>N</emph>sion&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunksize Equals Dimension Size except Record Dimension</itemformat></item>
</tableterm><tableitem><para>Definition: Chunksize equals dimension size except record dimension has size one.
Explicitly specify chunksizes for particular dimensions with
<samp>--cnk_dmn</samp> option.&linebreak;
<var>cnk_map</var> key values: <samp>rd1</samp>, <samp>cnk_rd1</samp>, <samp>map_rd1</samp>&linebreak;
Mnemonic: <emph>R</emph>ecord <emph>D</emph>imension size <emph>1</emph>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunksize Equals Scalar Size Specified</itemformat></item>
</tableterm><tableitem><para>Definition: Chunksize for all dimensions is set with the
<samp>--cnk_scl</samp> option.&linebreak; 
<var>cnk_map</var> key values: <samp>xpl</samp>, <samp>cnk_xpl</samp>, <samp>map_xpl</samp>&linebreak;
Mnemonic: E<emph>XPL</emph>icitly specified dimensions&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunksize Product Matches Scalar Size Specified</itemformat></item>
</tableterm><tableitem><para>Definition: The product of the chunksizes for each variable
matches (approximately equals) the size specified with the
<samp>--cnk_scl</samp> option.
A dimension of size one is said to be <emph>degenerate</emph>.
For a variable of rank <var>R</var> (i.e., with <var>R</var> non-degenerate
dimensions), the chunksize in each non-degenerate dimension is the
<var>R</var>th root of <var>cnk_scl</var>.&linebreak;
<var>cnk_map</var> key values: <samp>prd</samp>, <samp>cnk_prd</samp>, <samp>map_prd</samp>&linebreak;
Mnemonic: <emph>PR</emph>o<emph>D</emph>uct&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunksize Lefter Product Matches Scalar Size Specified</itemformat></item>
</tableterm><tableitem><para>Definition: The product of the chunksizes for each variable
(approximately) equals the size specified with the <samp>--cnk_scl</samp> 
option.
This is accomplished by using dimension sizes as chunksizes for the
rightmost (most rapidly varying) dimensions, and then &textldquo;flexing&textrdquo; the
chunksize of the leftmost (least rapidly varying) dimensions such that 
the product of all chunksizes matches the specified size.
All dimensions to the left of and including the first record dimension
define the left-hand side.
This map was first proposed by Chris Barker.&linebreak;
<var>cnk_map</var> key values: <samp>lfp</samp>, <samp>cnk_lfp</samp>, <samp>map_lfp</samp>&linebreak;
Mnemonic: <emph>L</emph>e<emph>F</emph>ter <emph>P</emph>roduct&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunksize Equals Existing Chunksize</itemformat></item>
</tableterm><tableitem><para>Definition: Chunksizes are copied from the input to the output
file for every variable that is chunked in the input file.
Variables not chunked in the input file will be chunked with 
default mappings.&linebreak;
<var>cnk_map</var> key values: <samp>xst</samp>, <samp>cnk_xst</samp>, <samp>map_xst</samp>&linebreak;
Mnemonic: E<emph>X</emph>i<emph>ST</emph>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Chunksize Balances 1D and (N-1)-D Access to N-D Variable</itemformat></item>
</tableterm><tableitem><para>Definition: Chunksizes are chosen so that 1-D and (<var>(N-1)</var>)-D
hyperslabs of <var>3</var>-D variables (e.g., point-timeseries orn
latitude/longitude surfaces of 3-D fields) both require approximately
the number of chunks. 
Hence their access time should be balanced.
Russ Rew explains the motivation and derivation for this strategy 
<uref><urefurl>http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes</urefurl><urefdesc spaces="\n">here</urefdesc></uref>.&linebreak;
<var>cnk_map</var> key values: <samp>rew</samp>, <samp>cnk_rew</samp>, <samp>map_rew</samp>&linebreak;
Mnemonic: Russ <emph>REW</emph>&linebreak;
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>It is possible to combine the above chunking map algorithms with
user-specified per-dimension (though not per-variable) chunksizes that 
override specific chunksizes determined by the maps above. 
The user specifies the per-dimension chunksizes with the (equivalent) 
long options <samp>--cnk_dmn</samp> or <samp>--chunk_dimension</samp>).
The option takes two comma-separated arguments,
<var>dmn_nm</var>,<var>cnk_sz</var>, which are the dimension name and its
chunksize, respectively. 
The <samp>--cnk_dmn</samp> option may be used as many times as necessary.
</para>
<html endspaces=" ">
&lt;a name=&quot;xmp_cnk&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_cnk &textndash;&gt;
&lt;a name=&quot;xmp_chunk&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_chunk &textndash;&gt;
</html>
<example endspaces=" ">
<pre xml:space="preserve"># Simple chunking and unchunking
ncks -O -4 --cnk_plc=all     in.nc out.nc # Chunk in.nc
ncks -O -4 --cnk_plc=unchunk in.nc out.nc # Unchunk in.nc

# Chunk data then unchunk it, printing informative metadata
ncks -O -4 -D 4 --cnk_plc=all ~/nco/data/in.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_plc=uck ~/foo.nc ~/foo.nc

# Set total chunksize to 8192 B
ncks -O -4 -D 4 --cnk_plc=all --cnk_byt=8192 ~/nco/data/in.nc ~/foo.nc

# More complex chunking procedures, with informative metadata
ncks -O -4 -D 4 --cnk_scl=8 ~/nco/data/in.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_scl=8 dstmch90_clm.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_dmn lat,64 --cnk_dmn lon,128 dstmch90_clm.nc \ 
 ~/foo.nc 
ncks -O -4 -D 4 --cnk_plc=uck ~/foo.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_plc=g2d --cnk_map=rd1 --cnk_dmn lat,32 \
 --cnk_dmn lon,128 dstmch90_clm_0112.nc ~/foo.nc

# Chunking works with all operators...
ncap2 -O -4 -D 4 --cnk_scl=8 -S ~/nco/data/ncap2_tst.nco \ 
 ~/nco/data/in.nc ~/foo.nc
ncbo -O -4 -D 4 --cnk_scl=8 -p ~/nco/data in.nc in.nc ~/foo.nc
ncecat -O -4 -D 4 -n 12,2,1 --cnk_dmn lat,32 \ 
 -p /data/zender/dstmch90 dstmch90_clm01.nc ~/foo.nc
ncflint -O -4 -D 4 --cnk_scl=8 ~/nco/data/in.nc ~/foo.nc
ncpdq -O -4 -D 4 -P all_new --cnk_scl=8 -L 5 ~/nco/data/in.nc ~/foo.nc
ncrcat -O -4 -D 4 -n 12,2,1 --cnk_dmn lat,32 \ 
 -p /data/zender/dstmch90 dstmch90_clm01.nc ~/foo.nc
ncwa -O -4 -D 4 -a time --cnk_plc=g2d --cnk_map=rd1 --cnk_dmn lat,32 \ 
 --cnk_dmn lon,128 dstmch90_clm_0112.nc ~/foo.nc
</pre></example>

<cindex index="cp" spaces=" "><indexterm index="cp" number="717">record dimension</indexterm></cindex>
<para>It is appropriate to conclude by informing users about an aspect of
chunking that may not be expected.
Three types of variables are <emph>always</emph> chunked: Record variables,
Deflated (compressed) variables, and Checksummed variables.
Hence all variables that contain a record dimension are also chunked
(since data must be chunked in all dimensions, not just one).
Unless otherwise specified by the user, the other (fixed, non-record) 
dimensions of record variables are assigned default chunk sizes. 
The <acronym><acronymword>HDF5</acronymword></acronym> layer does all this automatically to optimize the
on-disk variable/file storage geometry of record variables.
Do not be surprised to learn that files created without any explicit
instructions to activate chunking nevertheless contain chunked
variables. 
</para>
<html endspaces=" ">
&lt;a name=&quot;dfl_lvl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dfl_lvl &textndash;&gt;
&lt;a name=&quot;dfl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dfl &textndash;&gt;
&lt;a name=&quot;lz&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#lz &textndash;&gt;
&lt;a name=&quot;lz77&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#lz77 &textndash;&gt;
&lt;a name=&quot;deflate&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#deflate &textndash;&gt;
&lt;a name=&quot;deflation&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#deflation &textndash;&gt;
</html>
</section>
<node name="Deflation" spaces=" "><nodename>Deflation</nodename><nodenext spaces=" ">MD5 digests</nodenext><nodeprev spaces=" ">Chunking</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Deflation</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="718"><code>-L</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="719"><code>--deflate</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="720"><code>--dfl_lvl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="721">Lempel-Ziv deflation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="722">compression</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="723">deflation</indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncap2</command>, <command>ncbo</command>, <command>nces</command>,
<command>ncecat</command>, <command>ncflint</command>, <command>ncks</command>, <command>ncpdq</command>,
<command>ncra</command>, <command>ncrcat</command>, <command>ncwa</command>&linebreak;
Short options: <samp>-L</samp>&linebreak;
Long options: <samp>--dfl_lvl</samp>, <samp>--deflate</samp>&linebreak;  
</para></cartouche>

<para>All <acronym><acronymword>NCO</acronymword></acronym> operators that define variables support 
the netCDF4 feature of storing variables compressed with Lempel-Ziv
deflation. 
The Lempel-Ziv algorithm is a lossless data compression technique.
Activate this deflation with the <code>-L <var>dfl_lvl</var></code> short option
(or with the same argument to the <samp>--dfl_lvl</samp> or <samp>--deflate</samp> 
long options).
Specify the deflation level <var>dfl_lvl</var> on a scale from 
no deflation (<var>dfl_lvl = 0</var>) to maximum deflation 
(<var>dfl_lvl = 9</var>).
Minimal deflation (<var>dfl_lvl = 1</var>) achieves considerable storage
compression with little time penalty.
Higher deflation levels require more time for compression.
File sizes resulting from minimal (<var>dfl_lvl = 1</var>) and maximal   
(<var>dfl_lvl = 9</var>) deflation levels typically differ by a few
percent in size. 
</para>
<para>To compress an entire file using deflation, use
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -4 -L 0 in.nc out.nc # No deflation (fast, no time penalty)
ncks -4 -L 1 in.nc out.nc # Minimal deflation (little time penalty)
ncks -4 -L 9 in.nc out.nc # Maximal deflation (much slower)
</pre></example>

<para>Unscientific testing shows that deflation compresses typical climate
datasets by 30-60%.  
Packing, a lossy compression technique available for all netCDF files 
(see <ref label="Packed-data"><xrefnodename>Packed data</xrefnodename></ref>), can easily compress files by 50%.
Packed data may be deflated to squeeze datasets by about 80%:
</para><example endspaces=" ">
<pre xml:space="preserve">ncks  -4 -L 1 in.nc out.nc # Minimal deflation (~30-60% compression)
ncks  -4 -L 9 in.nc out.nc # Maximal deflation (~31-63% compression)
ncpdq         in.nc out.nc # Standard packing  (~50% compression)
ncpdq -4 -L 9 in.nc out.nc # Deflated packing  (~80% compression)
</pre></example>
<findex index="fn" spaces=" "><indexterm index="fn" number="11" mergedindex="cp">ncks</indexterm></findex>
<para><command>ncks</command> prints deflation parameters, if any, to screen
(<pxref label="ncks-netCDF-Kitchen-Sink"><xrefnodename>ncks netCDF Kitchen Sink</xrefnodename></pxref>).
</para>
<html endspaces=" ">
&lt;a name=&quot;md5&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#md5 &textndash;&gt;
&lt;a name=&quot;digest&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#digest &textndash;&gt;
&lt;a name=&quot;hash&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hash &textndash;&gt;
&lt;a name=&quot;integrity&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#integrity &textndash;&gt;
&lt;a name=&quot;security&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#security &textndash;&gt;
</html>
</section>
<node name="MD5-digests" spaces=" "><nodename>MD5 digests</nodename><nodenext spaces=" ">Buffer sizes</nodenext><nodeprev spaces=" ">Deflation</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>MD5 digests</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="724"><code>--md5_digest</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="725"><code>--md5_dgs</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="726"><code>--md5_wrt_att</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="727"><code>--md5_write_attribute</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="728">integrity</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="729">security</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="730">digest</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="731">hash</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="732">MD5 digest</indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: 
<command>ncecat</command>, <command>ncks</command>, <command>ncrcat</command>&linebreak;
Short options: &linebreak;
Long options: <samp>--md5_dgs</samp>, <samp>--md5_digest</samp>, <samp>--md5_wrt_att</samp>, <samp>--md5_write_attribute</samp>&linebreak;
</para></cartouche>

<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.1.0 (April, 2012), <acronym><acronymword>NCO</acronymword></acronym> 
supports data integrity verification using the <acronym><acronymword>MD5</acronymword></acronym> digest
algorithm. 
This support is currently implemented in <command>ncks</command> and in the
multifile concantenators <command>ncecat</command> and <command>ncrcat</command>.
Activate it with the <samp>--md5_dgs</samp> or <samp>--md5_digest</samp> long
options. 
As of <acronym><acronymword>NCO</acronymword></acronym> version 4.3.3 (July, 2013), <acronym><acronymword>NCO</acronymword></acronym> 
will write the <acronym><acronymword>MD5</acronymword></acronym> digest of each variable as an
<code>NC_CHAR</code> attribute named <code>MD5</code>.
This support is currently implemented in <command>ncks</command> and in the
multifile concantenators <command>ncecat</command> and <command>ncrcat</command>.
Activate it with the <samp>--md5_wrt_att</samp> or
<samp>--md5_write_attribute</samp> long options.
</para>
<para>The behavior and verbosity of the <acronym><acronymword>MD5</acronymword></acronym> digest is
operator-dependent.
When activating <acronym><acronymword>MD5</acronymword></acronym> digests with <command>ncks</command> it is assumed
that the user simply wishes to see the digest of every variable and this
is done when the debugging level exceeds one.
This incurs only the minor overhead of performing the hash algorithm for
each variable read.
<acronym><acronymword>MD5</acronymword></acronym> digests may be activated in both the one- and two-filename
argument forms of <command>ncks</command>, which are used for printing and for
sub-setting, respectively.
The <acronym><acronymword>MD5</acronymword></acronym> digests are shown as a 32-character hexadecimal string 
in which each two characters represent one byte of the 16-byte digest:
</para><example endspaces=" ">
<pre xml:space="preserve">&gt; ncks -O -D 2 -C --md5 -v md5_a,md5_abc ~/nco/data/in.nc
...
ncks: INFO MD5(md5_a) = 0cc175b9c0f1b6a831c399e269772661
md5_a = 'a' 
ncks: INFO MD5(md5_abc) = 900150983cd24fb0d6963f7d28e17f72
lev[0]=100 md5_abc[0--2]='abc' 
&gt; ncks -O -D 2 -C -d lev,0 --md5 -v md5_a,md5_abc ~/nco/data/in.nc
...
ncks: INFO MD5(md5_a) = 0cc175b9c0f1b6a831c399e269772661
md5_a = 'a' 
ncks: INFO MD5(md5_abc) = 0cc175b9c0f1b6a831c399e269772661
lev[0]=100 md5_abc[0--0]='a' 
</pre></example>
<para>In fact these examples demonstrate the validity of the hash algorithm
since the <acronym><acronymword>MD5</acronymword></acronym> hashes of the strings &textldquo;a&textrdquo; and &textldquo;abc&textrdquo; are
widely known.
The second example shows that the hyperslab of variable <code>md5_abc</code>
(= &textldquo;abc&textrdquo;) consisting of only its first letter (= &textldquo;a&textrdquo;) has the same
hash as the variable <code>md5_a</code> (&textldquo;a&textrdquo;).
This illustrates that <acronym><acronymword>MD5</acronymword></acronym> digests act only on variable data,
not on metadata. 
</para>
<para>When activating <acronym><acronymword>MD5</acronymword></acronym> digests with <command>ncecat</command> or
<command>ncrcat</command> it is assumed that the user wishes to verify
that every variable written to disk has the same <acronym><acronymword>MD5</acronymword></acronym> digest 
as when it is subsequently read from disk.
This incurs the major additional overhead of reading in each variable
after it is written and performing the hash algorithm again on that to
compare to the original hash.
Moreover, it is assumed that such operations are generally done
&textldquo;production mode&textrdquo; where the user is not interested in actually
examining the digests herself.
The digests proceed silently unless the debugging level exceeds three:
</para><example endspaces=" ">
<pre xml:space="preserve">&gt; ncecat -O -D 4 --md5 -p ~/nco/data in.nc in.nc ~/foo.nc | grep MD5
...
ncecat: INFO MD5(wnd_spd) = bec190dd944f2ce2794a7a4abf224b28
ncecat: INFO MD5 digests of RAM and disk contents for wnd_spd agree
&gt; ncrcat -O -D 4 --md5 -p ~/nco/data in.nc in.nc ~/foo.nc | grep MD5
...
ncrcat: INFO MD5(wnd_spd) = 74699bb0a72b7f16456badb2c995f1a1
ncrcat: INFO MD5 digests of RAM and disk contents for wnd_spd agree
</pre></example>
<para>Regardless of the debugging level, an error is returned when the digests
of the variable read from the source file and from the output file
disagree.  
</para>
<para>These rules are evolving and as <acronym><acronymword>NCO</acronymword></acronym> pays more attention to
data integrity. 
We welcome feedback and suggestions from users.
</para>
<html endspaces=" ">
&lt;a name=&quot;bfr_sz_hnt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#bfr_sz_hnt &textndash;&gt;
&lt;a name=&quot;bfr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#bfr &textndash;&gt;
&lt;a name=&quot;buffer&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#buffer &textndash;&gt;
</html>
</section>
<node name="Buffer-sizes" spaces=" "><nodename>Buffer sizes</nodename><nodenext spaces=" ">RAM disks</nodenext><nodeprev spaces=" ">MD5 digests</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Buffer sizes</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="733"><code>--bfr_sz_hnt</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="734">Buffer sizes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="735">File buffers</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="736"><command>stat() system call</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="737">I/O block size</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="738">System calls</indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
Short options: &linebreak;
Long options: <samp>--bfr_sz_hnt</samp>, <samp>--buffer_size_hint</samp>&linebreak;  
</para></cartouche>

<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.2.0 (May, 2012), <acronym><acronymword>NCO</acronymword></acronym> 
allows the user to request specific buffer sizes to allocate for reading 
and writing files.
This buffer size determines how many system calls the netCDF layer must
invoke to read and write files.
By default, netCDF uses the preferred I/O block size returned as the
<samp>st_blksize</samp> member of the <samp>stat</samp> structure returned by the
<command>stat()</command> system call
<footnote spaces="\n"><para>On modern Linux systems the block size defaults to <w>8192 B</w>.
The GLADE filesystem at NCAR has a block size of <w>512 kB</w>.</para></footnote>.
Otherwise, netCDF uses twice the system pagesize.
Larger sizes can increase access speed by reducing the number of 
system calls netCDF makes to read/write data from/to disk.
Because netCDF cannot guarantee the buffer size request will be met, the 
actual buffer size granted by the system is printed as an INFO
statement. 
</para><example endspaces=" ">
<pre xml:space="preserve"># Request 2 MB file buffer instead of default 8 kB buffer
&gt; ncks -O -D 3 --bfr_sz=2097152 ~/nco/data/in.nc ~/foo.nc
...
ncks: INFO nc__open() will request file buffer size = 2097152 bytes
ncks: INFO nc__open() opened file with buffer size = 2097152 bytes
...
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ram_all&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ram_all &textndash;&gt;
&lt;a name=&quot;ram&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ram &textndash;&gt;
&lt;a name=&quot;diskless&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#diskless &textndash;&gt;
</html>
</section>
<node name="RAM-disks" spaces=" "><nodename>RAM disks</nodename><nodenext spaces=" ">Packed data</nodenext><nodeprev spaces=" ">Buffer sizes</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>RAM disks</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="739"><code>--ram_all</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="740"><code>--create_ram</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="741"><code>--open_ram</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="742"><code>--diskless_all</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="743"><acronym><acronymword>RAM</acronymword></acronym> disks</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="744"><acronym><acronymword>RAM</acronymword></acronym> files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="745"><code>NC_DISKLESS</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="746">diskless files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="747">memory requirements</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="748">memory available</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="749"><acronym><acronymword>RAM</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="750">swap space</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="751">peak memory usage</indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
Short options: &linebreak;
Long options: <samp>--ram_all</samp>, <samp>--create_ram</samp>, <samp>--open_ram</samp>,
<samp>--diskless_all</samp>&linebreak;   
</para></cartouche>

<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.2.1 (August, 2012), <acronym><acronymword>NCO</acronymword></acronym> supports
the use of diskless files, aka <acronym><acronymword>RAM</acronymword></acronym> disks, for file access and creation. 
Two independent switches, <samp>--open_ram</samp> and <samp>--create_ram</samp>,
control this feature. 
Before describing the specifics of these switches, we describe why many 
<acronym><acronymword>NCO</acronymword></acronym> operations will not benefit from them.
Essentially, reading/writing from/to <acronym><acronymword>RAM</acronymword></acronym> rather than disk only hastens 
the task when reads/writes to disk are avoided.
Most <acronym><acronymword>NCO</acronymword></acronym> operations are simple enough that they require a
single read-from/write-to disk for every block of input/output. 
Diskless access does not change this, but it does add an extra
read-from/write-to RAM. 
However this extra <acronym><acronymword>RAM</acronymword></acronym> write/read does avoid contention for limited
system resources like disk-head access.
Operators which may benefit from <acronym><acronymword>RAM</acronymword></acronym> disks include <command>ncwa</command>, which
may need to read weighting variables multiple times, the multi-file 
operators <command>ncra</command>, <command>ncrcat</command>, and <command>ncecat</command>,
which may try to write output at least once per input file, and
<command>ncap2</command> scripts which may be arbitrarily long and convoluted. 
</para>
<para>The <samp>--open_ram</samp> switch causes input files to copied to <acronym><acronymword>RAM</acronymword></acronym> when
opened. 
All further metadata and data access occurs in <acronym><acronymword>RAM</acronymword></acronym> and thus avoids
access time delays caused by disk-head movement.
Usually input data is read at most once so it is unlikely that
requesting input files be stored in <acronym><acronymword>RAM</acronymword></acronym> will save much time.
The likeliest exceptions are files that are accessed numerous times,
such as those analyzed extensively analyzed by <command>ncap2</command>. 
</para>
<para>Invoking <samp>--open_ram</samp>, <samp>--ram_all</samp>, or <samp>--diskless_all</samp>
uses much more system memory.
To copy the input file to <acronym><acronymword>RAM</acronymword></acronym> increases the sustained
memory use by exactly the on-disk filesize of the input file, i.e.,
<math>MS += FT</math>.
For large input files this can be a huge memory burden that starves
the rest of the <acronym><acronymword>NCO</acronymword></acronym> analysis of sufficient <acronym><acronymword>RAM</acronymword></acronym>.
To be safe, use <samp>--open_ram</samp>, <samp>--ram_all</samp>, or
<samp>--diskless_all</samp> only on files that are much (say at least a factor
of four) smaller than your available system <acronym><acronymword>RAM</acronymword></acronym>.
See <ref label="Memory-Requirements"><xrefnodename>Memory Requirements</xrefnodename></ref> for further details. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="752"><acronym><acronymword>RAM</acronymword></acronym> variables</indexterm></cindex>
<para>The <samp>--create_ram</samp> switch causes output files to be created in RAM,
rather than on disk. 
These files are copied to disk only when closed, i.e., when the
operator completes.
Creating files in <acronym><acronymword>RAM</acronymword></acronym> may save time, especially with <command>ncap2</command>
computations that are iterative, e.g., loops, and for multi-file
operators that write output every record (timestep) or file.
RAM files provide many of the same benefits as <acronym><acronymword>RAM</acronymword></acronym> variables in such
cases (<pxref label="RAM-variables"><xrefnodename>RAM variables</xrefnodename></pxref>). 
</para>
<para>Two switches, <samp>--ram_all</samp> and <samp>--diskless_all</samp>, are convenient
shortcuts for specifying both <samp>--create_ram</samp> and
<samp>--diskless_ram</samp>. 
Thus
</para><example endspaces=" ">
<pre xml:space="preserve">ncks in.nc out.nc # Default: Open in.nc on disk, write out.nc to disk
ncks --open_ram in.nc out.nc # Open in.nc in RAM, write out.nc to disk
ncks --create_ram in.nc out.nc # Create out.nc in RAM, write to disk
# Open in.nc in RAM, create out.nc in RAM, then write out.nc to disk
ncks --open_ram --create_ram in.nc out.nc
ncks --ram_all in.nc out.nc # Same as above
ncks --diskless_all in.nc out.nc # Same as above
</pre></example>

<para>It is straightforward to demonstrate the efficacy of <acronym><acronymword>RAM</acronymword></acronym> disks.
For <acronym><acronymword>NASA</acronymword></acronym> we constructed a test that employs <command>ncecat</command>
an arbitrary number (set to one hundred thousand) of files are all
symbolically linked to the same file. 
Everything is on the local filesystem (not <acronym><acronymword>DAP</acronymword></acronym>).
</para><example endspaces=" ">
<pre xml:space="preserve"># Create symbolic links for benchmark
cd $&lbrace;DATA&rbrace;/nco # Do all work here
for idx in &lbrace;1..99999&rbrace;; do
  idx_fmt=`printf &quot;%05d&quot; $&lbrace;idx&rbrace;`
  /bin/ln -s $&lbrace;DATA&rbrace;/nco/LPRM-AMSR_E_L3_D_SOILM3_V002-20120512T111931Z_20020619.nc \
             $&lbrace;DATA&rbrace;/nco/$&lbrace;idx_fmt&rbrace;.nc
done
# Benchmark time to ncecat one hundred thousand files
time ncecat --create_ram -O -u time -v ts -d Latitude,40.0 \ 
 -d Longitude,-105.0 -p $&lbrace;DATA&rbrace;/nco -n 99999,5,1 00001.nc ~/foo.nc
</pre></example>
<para>Run normally on a laptop in 201303, this completes in <w>21 seconds</w>.
The <samp>--create_ram</samp> reduces the elapsed time to <w>9 seconds</w>.
Some of this speed may be due to using symlinks and caching.
However, the efficacy of <samp>--create_ram</samp> is clear.
Placing the output file in <acronym><acronymword>RAM</acronymword></acronym> avoids thousands of disk writes.
It is not unreasonable to for <acronym><acronymword>NCO</acronymword></acronym> to process a million files
like this in a few minutes. 
However, there is no substitute for benchmarking with real files.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="753">temporary output files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="754">temporary files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="755"><code>--no_tmp_fl</code></indexterm></cindex>
<para>A completely independent way to reduce time spent writing files is 
to refrain from writing temporary output files.
This is accomplished with the <samp>--no_tmp_fl</samp> switch 
(<pxref label="Temporary-Output-Files"><xrefnodename>Temporary Output Files</xrefnodename></pxref>).
</para>
<html endspaces=" ">
&lt;a name=&quot;pck&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#pck &textndash;&gt;
&lt;a name=&quot;pack&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#pack &textndash;&gt;
</html>
</section>
<node name="Packed-data" spaces=" "><nodename>Packed data</nodename><nodenext spaces=" ">Operation Types</nodenext><nodeprev spaces=" ">RAM disks</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Packed data</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="756">packing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="757">unpacking</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="758"><code>add_offset</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="759"><code>scale_factor</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="760"><code>missing_value</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="761"><code>_FillValue</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="762"><command>pack(x)</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="763"><command>unpack(x)</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="764"><code>--hdf_upk</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="765"><code>--hdf_unpack</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncap2</command>, <command>ncbo</command>, <command>nces</command>,
<command>ncflint</command>, <command>ncpdq</command>, <command>ncra</command>, <command>ncwa</command>&linebreak; 
Short options: None&linebreak;
Long options: <samp>--hdf_upk</samp>, <samp>--hdf_unpack</samp>&linebreak;
</para></cartouche>

<para>The phrase <dfn>packed data</dfn> refers to data which are stored in the
standard netCDF3 packing format which employs a lossy algorithm.
See <ref label="ncks-netCDF-Kitchen-Sink"><xrefnodename>ncks netCDF Kitchen Sink</xrefnodename></ref> for a description of deflation, a 
lossless compression technique available with netCDF4 only.
Packed data may be deflated to save additional space.
</para>
<unnumberedsubsec spaces=" "><sectiontitle>Packing Algorithm</sectiontitle>
<para><dfn>Packing</dfn>
The standard netCDF packing algorithm (described
<uref><urefurl>http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Attribute-Conventions.html</urefurl><urefdesc spaces=" ">here</urefdesc></uref>)
produces data with
the same dynamic range as the original but which requires no more than
half the space to store.
Like all packing algorithms, it is <emph>lossy</emph>.
The packed variable is stored (usually) as type <code>NC_SHORT</code>
with the two attributes required to unpack the variable,
<code>scale_factor</code> and <code>add_offset</code>, stored at the original
(unpacked) precision of the variable
<footnote><para>Although not a part of the standard, <acronym><acronymword>NCO</acronymword></acronym> enforces
the policy that the <code>_FillValue</code> attribute, if any, of a packed
variable is also stored at the original precision.</para></footnote>.
Let <var>min</var> and <var>max</var> be the minimum and maximum values 
<w>of <var>x</var>.</w> 
<tex endspaces=" ">
$$
\rm
\eqalign{\hbox{scale\_factor} &amp;= (\hbox{max}-\hbox{min})/\hbox{ndrv}\cr
\hbox{add\_offset} &amp;= (\hbox{min}+\hbox{max})/2\cr
\hbox{pck} &amp;= (\hbox{upk}-\hbox{add\_offset})/\hbox{scale\_factor}\cr
 &amp;= {\hbox{ndrv}\times[\hbox{upk}-(\hbox{min}+\hbox{max})/2]\over\hbox{max}-\hbox{min}}\cr}
$$
</tex>
</para><sp value="1" line=" 1"></sp>
<para><var>scale_factor</var> = (<var>max</var>-<var>min</var>)/<var>ndrv</var>&linebreak;
<var>add_offset</var> = 0.5*(<var>min</var>+<var>max</var>)&linebreak;
<var>pck</var> = (<var>upk</var>-<var>add_offset</var>)/<var>scale_factor</var> = (<var>upk</var>-0.5*(<var>min</var>+<var>max</var>))*<var>ndrv</var>/(<var>max</var>-<var>min</var>)&linebreak; 
</para><sp value="1" line=" 1"></sp>
<para>where <var>ndrv</var> is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for <var>ndrv</var> is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type <code>NC_SHORT</code>, a two-byte
datatype, then there are at most <math>2^{16} = 65536</math> distinct values
representable.
In practice, the number of discretely representible values is taken
to be two less than the theoretical maximum.
This leaves space for a missing value and solves potential problems with
rounding that may occur during the unpacking of the variable.
Thus for <code>NC_SHORT</code>, <math>ndrv = 65536 - 2 = 65534</math>.
Less often, the variable may be packed into type <code>NC_CHAR</code>, 
where <math>ndrv = 2^{8} - 2 = 256 - 2 = 254</math>, or type <code>NC_INT</code> where
where <math>ndrv = 2^{32} - 2 = 4294967295 - 2 = 4294967293</math>.
One useful feature of (lossy) netCDF packing algorithm is that
additional, loss-less packing algorithms perform well on top of it. 
</para>
<html endspaces=" ">
&lt;a name=&quot;upk&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#upk &textndash;&gt;
&lt;a name=&quot;unpack&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#unpack &textndash;&gt;
</html>
</unnumberedsubsec>
<unnumberedsubsec spaces=" "><sectiontitle>Unpacking Algorithm</sectiontitle>
<para><dfn>Unpacking</dfn>
The unpacking algorithm depends on the presence of two attributes,
<code>scale_factor</code> and <code>add_offset</code>.
If <code>scale_factor</code> is present for a variable, the data are
multiplied by the value <var>scale_factor</var> after the data are read.
If <code>add_offset</code> is present for a variable, then the
<var>add_offset</var> value is added to the data after the data are read.
If both <code>scale_factor</code> and <code>add_offset</code> attributes are
present, the data are first scaled by <var>scale_factor</var> before the
offset <var>add_offset</var> is added.   
<tex endspaces=" ">
$$
\rm
\eqalign{\hbox{upk} &amp;= \hbox{scale\_factor}\times\hbox{pck} + \hbox{add\_offset}\cr 
&amp;= {\hbox{pck}\times(\hbox{max}-\hbox{min})\over\hbox{ndrv}} + {\hbox{min}+\hbox{max}\over2}\cr} 
$$
</tex>
</para><sp value="1" line=" 1"></sp>
<para><var>upk</var> = <var>scale_factor</var>*<var>pck</var> + <var>add_offset</var> = (<var>max</var>-<var>min</var>)*<var>pck</var>/<var>ndrv</var> + 0.5*(<var>min</var>+<var>max</var>)&linebreak;
</para><sp value="1" line=" 1"></sp>
<para>When <code>scale_factor</code> and <code>add_offset</code> are used for packing, the
associated variable (containing the packed data) is typically of type
<code>byte</code> or <code>short</code>, whereas the unpacked values are intended to
be of type <code>int</code>, <code>float</code>, or <code>double</code>. 
An attribute&textrsquo;s <code>scale_factor</code> and <code>add_offset</code> and
<code>_FillValue</code>, if any, should all be of the type intended for the
unpacked data, i.e., <code>int</code>, <code>float</code> or <code>double</code>. 
</para>
</unnumberedsubsec>
<unnumberedsubsec spaces=" "><sectiontitle>Default Handling of Packed Data</sectiontitle>
<html endspaces=" ">
&lt;a name=&quot;hdf_upk&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hdf_upk &textndash;&gt;
&lt;a name=&quot;hdf_unpack&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hdf_unpack &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="766">interoperability</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="767"><acronym><acronymword>HDF</acronymword></acronym> unpacking</indexterm></cindex>
<para>Most files originally written in <acronym><acronymword>HDF</acronymword></acronym> format use the
<acronym><acronymword>HDF</acronymword></acronym> packing/unpacking algorithm.
This algorithm is incompatible with the netCDF packing algorithm 
described above.   
The unpacking component of the <acronym><acronymword>HDF</acronymword></acronym> algorithm (described
<uref><urefurl>http://www.hdfgroup.org/HDF5/doc/UG/UG_frame10Datasets.html</urefurl><urefdesc spaces=" ">here</urefdesc></uref>) is
<tex endspaces=" ">
$$
\rm
\hbox{upk} = \hbox{scale\_factor}\times(\hbox{pck} - \hbox{add\_offset})
$$
</tex>
</para><sp value="1" line=" 1"></sp>
<para><var>upk</var> = <var>scale_factor</var>*(<var>pck</var> - <var>add_offset</var>)&linebreak;
</para><sp value="1" line=" 1"></sp>
<ignore>
NB: HDF packing documentation is hard to follow
MODIS HDF4 datasets definitely use the ``HDF algorithm''
However, must discriminate between HDF4 and HDF5 packing.
HDF4, or at least some HDF4 datasets implement the ``HDF algorithm'',
aka linear scale_factor and add_offset in other order from netCDF.
HDF5 only implements D-scaling, aka, decimal-scaling bit-packing.
D-Scaling uses add_offset (i.e., minimum data value) and scale_factor
(integer power of 10 by which the add_offset corrected data are divided
before storage) to control D-scaling.

@cindex scale-offset compression
@cindex scale-offset filter
@acronym{HDF} also calls this @dfn{scale-offset compression} and the
@dfn{scale-offset filter}.
</ignore>
<para>Confusingly, the (incompatible) netCDF and <acronym><acronymword>HDF</acronymword></acronym> algorithms both 
store their parameters in attributes with the same names
(<code>scale_factor</code> and <code>add_offset</code>).
Data packed with one algorithm should never be unpacked with the other;
doing so will result in incorrect answers.
Unfortunately, few users are aware that their datasets may be packed,
and fewer know the details of the packing algorithm employed.
This is what we in the &textldquo;bizness&textrdquo; call an <dfn>interoperability</dfn> issue
because it hampers data analysis performed on heterogeneous systems.
</para>
<para>As described below, <acronym><acronymword>NCO</acronymword></acronym> automatically unpacks data before
performing arithmetic.
This automatic unpacking occurs silently since there is usually no
reason to bother users with these details. 
There is as yet no generic way for <acronym><acronymword>NCO</acronymword></acronym> to know which
packing convention was used, so <acronym><acronymword>NCO</acronymword></acronym> <emph>assumes</emph> the netCDF 
convention was used. 
<acronym><acronymword>NCO</acronymword></acronym> uses the same convention for unpacking unless explicitly
told otherwise with the <samp>--hdf_upk</samp> (also <samp>--hdf_unpack</samp>)
switch. 
Until and unless a method of automatically detecting the packing method 
is devised, it must remain the user&textrsquo;s responsibility to tell
<acronym><acronymword>NCO</acronymword></acronym> when to use the <acronym><acronymword>HDF</acronymword></acronym> convention instead of the
netCDF convention to unpack. 
</para>
<para>If your data originally came from an <acronym><acronymword>HDF</acronymword></acronym> file (e.g.,
<acronym><acronymword>NASA</acronymword></acronym> <acronym><acronymword>EOS</acronymword></acronym>) then it was likely packed with the
<acronym><acronymword>HDF</acronymword></acronym> convention and must be unpacked with the same convention.
Our recommendation is to only request <acronym><acronymword>HDF</acronymword></acronym> unpacking when you 
are certain. 
Most packed datasets encountered by <acronym><acronymword>NCO</acronymword></acronym> will have used the
netCDF convention.
Those that were not will hopefully produce noticeably weird values when
unpacked by the wrong algorithm.
Before or after panicking, treat this as a clue to re-try your commands
with the <samp>--hdf_upk</samp> switch.
See <ref label="ncpdq-netCDF-Permute-Dimensions-Quickly"><xrefnodename>ncpdq netCDF Permute Dimensions Quickly</xrefnodename></ref> for an easy technique
to unpack data packed with the <acronym><acronymword>HDF</acronymword></acronym> convention, and then
re-pack it with the netCDF convention.
</para>
</unnumberedsubsec>
<unnumberedsubsec spaces=" "><sectiontitle>Default Handling of Packed Data</sectiontitle>
<para>All <acronym><acronymword>NCO</acronymword></acronym> arithmetic operators understand packed data.
The operators automatically unpack any packed variable in the input 
file which will be arithmetically processed.
For example, <command>ncra</command> unpacks all record variables, 
and <command>ncwa</command> unpacks all variable which contain a dimension to 
be averaged.
These variables are stored unpacked in the output file.
</para>
<para>On the other hand, arithmetic operators do not unpack non-processed
variables. 
For example, <command>ncra</command> leaves all non-record variables packed, 
and <command>ncwa</command> leaves packed all variables lacking an averaged
dimension.  
These variables (called fixed variables) are passed unaltered from the
input to the output file. 
Hence fixed variables which are packed in input files remain packed in
output files.
Completely packing and unpacking files is easily accomplished with
<command>ncpdq</command> (<pxref label="ncpdq-netCDF-Permute-Dimensions-Quickly"><xrefnodename>ncpdq netCDF Permute Dimensions Quickly</xrefnodename></pxref>).
Pack and unpack individual variables with <command>ncpdq</command> and the
<command>ncap2</command> <command>pack()</command> and <command>unpack()</command> functions
(<pxref label="Methods-and-functions"><xrefnodename>Methods and functions</xrefnodename></pxref>).
</para>
<html endspaces=" ">
&lt;a name=&quot;op_typ&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#op_typ &textndash;&gt;
</html>
</unnumberedsubsec>
</section>
<node name="Operation-Types" spaces=" "><nodename>Operation Types</nodename><nodenext spaces=" ">Type Conversion</nodenext><nodeprev spaces=" ">Packed data</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Operation Types</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="768">operation types</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="769"><code>avg</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="770"><code>sqravg</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="771"><code>avgsqr</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="772"><code>min</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="773"><code>max</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="774"><code>rmssdn</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="775"><code>rms</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="776"><code>ttl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="777"><code>sqrt</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="778">average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="779">mean</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="780">total</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="781">minimum</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="782">maximum</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="783">root-mean-square</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="784">standard deviation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="785">variance</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="786"><code>-y <var>op_typ</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="787"><code>--operation <var>op_typ</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="788"><code>--op_typ <var>op_typ</var></code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncap2</command>, <command>ncra</command>, <command>nces</command>, <command>ncwa</command>&linebreak;
Short options: <samp>-y</samp>&linebreak;
Long options: <samp>--operation</samp>, <samp>--op_typ</samp>&linebreak;
</para></cartouche>
<noindent></noindent>
<para>The <samp>-y <var>op_typ</var></samp> switch allows specification of many different
types of operations 
Set <var>op_typ</var> to the abbreviated key for the corresponding operation:
</para><table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">avg</itemformat></item>
</tableterm><tableitem><para>Mean value
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">sqravg</itemformat></item>
</tableterm><tableitem><para>Square of the mean
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">avgsqr</itemformat></item>
</tableterm><tableitem><para>Mean of sum of squares
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">max</itemformat></item>
</tableterm><tableitem><para>Maximium value
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">min</itemformat></item>
</tableterm><tableitem><para>Minimium value
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">rms</itemformat></item>
</tableterm><tableitem><para>Root-mean-square (normalized by <var>N</var>)
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">rmssdn</itemformat></item>
</tableterm><tableitem><para>Root-mean square (normalized by <var>N-1</var>)
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">sqrt</itemformat></item>
</tableterm><tableitem><para>Square root of the mean
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ttl</itemformat></item>
</tableterm><tableitem><para>Sum of values
</para></tableitem></tableentry></table>
<noindent></noindent>
<cindex index="cp" spaces=" "><indexterm index="cp" number="789">coordinate variable</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> assumes coordinate variables represent grid axes, e.g.,
longitude. 
The only rank-reduction which makes sense for coordinate variables
is averaging.
Hence <acronym><acronymword>NCO</acronymword></acronym> implements the operation type requested with
<samp>-y</samp> on all non-coordinate variables, not on coordinate variables.  
When an operation requires a coordinate variable to be reduced in
rank, i.e., from one dimension to a scalar or from one dimension to
a degenerate (single value) array, then <acronym><acronymword>NCO</acronymword></acronym> 
<emph>always averages</emph> the coordinate variable regardless of the
arithmetic operation type performed on the non-coordinate variables.
</para>
<para>The mathematical definition of each arithmetic operation is given below. 
<xref label="ncwa-netCDF-Weighted-Averager"><xrefnodename>ncwa netCDF Weighted Averager</xrefnodename></xref>, for additional information on
masks and normalization.
If an operation type is not specified with <samp>-y</samp> then the operator
performs an arithmetic average by default.
Averaging is described first so the terminology for the other operations
is familiar. 
</para>
<para><emph>Note for Info users</emph>: 
The definition of mathematical operations involving rank reduction
(e.g., averaging) relies heavily on mathematical expressions which
cannot be easily represented in Info.
<emph>See the <uref><urefurl>./nco.pdf</urefurl><urefdesc spaces=" ">printed manual</urefdesc></uref> for much more detailed and
complete documentation of this subject.</emph>
</para>
<tex endspaces=" ">
The masked, weighted average of a variable $\xxx$ can be generally
represented as
$$
\bar \xxx_{\jjj} = {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}  
$$
where $\bar \xxx_{\jjj}$ is the $\jjj$&textrsquo;th element of the output
hyperslab, $\xxx_{\idx}$ is the $\idx$&textrsquo;th element of the input
hyperslab, $\mssflg_{\idx}$ is~1 unless $\xxx_{\idx}$ equals the missing  
value, $\mskflg_{\idx}$ is~1 unless $\xxx_{\idx}$ is masked, and
$\wgt_{\idx}$ is the weight.  
This formiddable looking formula represents a simple weighted average
whose bells and whistles are all explained below. 
It is not too early to note, however, that when 
$\mssflg_{\idx} = \mskflg_{\idx} = \wgt_{\idx} = 1$, the 
generic averaging expression above reduces to a simple arithmetic
average.  
Furthermore, $\mskflg_{\idx} = \wgt_{\idx} = 1$ for all operators
except <command>ncwa</command>.
These variables are included in the discussion below for completeness,
and for possible future use in other operators. 

The size $\outnbr$ of the output hyperslab for a given variable is the
product of all the dimensions of the input variable which are not
averaged over. 
The size $\lmnnbr$ of the input hyperslab contributing to each 
$\bar \xxx_{\jjj}$ is simply the product of the sizes of all dimensions
which are averaged over (i.e., dimensions specified with <samp>-a</samp>).  
Thus $\lmnnbr$ is the number of input elements which <emph>potentially</emph> 
contribute to each output element.
An input element $\xxx_{\idx}$ contributes to the output element
$\xxx_{\jjj}$ except in two conditions:  
<cindex index="cp" spaces=" "><indexterm index="cp">missing values</indexterm></cindex>
<enumerate first="1" endspaces=" ">
<listitem spaces=" ">$\xxx_{\idx}$ equals the <var>missing value</var> 
(<pxref label="Missing-Values"><xrefnodename>Missing Values</xrefnodename></pxref>) for the variable. 
</listitem><listitem spaces=" ">$\xxx_{\idx}$ is located at a point where the mask condition 
(<pxref label="Mask-condition"><xrefnodename>Mask condition</xrefnodename></pxref>) is false.
</listitem></enumerate>
Points $\xxx_{\idx}$ in either of these two categories do not contribute 
to $\xxx_{\jjj}$&textmdash;they are ignored.
We now define these criteria more rigorously.

Each~$\xxx_{\idx}$ has an associated Boolean weight~$\mssflg_{\idx}$
whose value is~0 or~1 (false or true). 
The value of~$\mssflg_{\idx}$ is~1 (true) unless $\xxx_{\idx}$ equals
the <var>missing value</var> (<pxref label="Missing-Values"><xrefnodename>Missing Values</xrefnodename></pxref>) for the variable. 
Thus, for a variable with no <code>_FillValue</code> attribute,
$\mssflg_{\idx}$~is always~1.
All <acronym><acronymword>NCO</acronymword></acronym> arithmetic operators (<command>ncbo</command>,
<command>ncra</command>, <command>nces</command>, <command>ncflint</command>, <command>ncwa</command>) treat  
missing values analogously. 

Besides (weighted) averaging, <command>ncwa</command>, <command>ncra</command>, and
<command>nces</command> also compute some common non-linear operations which may
be specified with the <samp>-y</samp> switch (<pxref label="Operation-Types"><xrefnodename>Operation Types</xrefnodename></pxref>).
The other rank-reducing operations are simple variations of the generic
weighted mean described above.
The total value of~$\xxx$ (<code>-y ttl</code>) is 
$$
\bar \xxx_{\jjj} = \sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx_{\idx}  
$$
<cindex index="cp" spaces=" "><indexterm index="cp"><code>-N</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp"><code>numerator</code></indexterm></cindex>
Note that the total is the same as the numerator of the mean
of~$\xxx$, and may also be obtained in <command>ncwa</command> by using the
<samp>-N</samp> switch (<pxref label="ncwa-netCDF-Weighted-Averager"><xrefnodename>ncwa netCDF Weighted Averager</xrefnodename></pxref>).

The minimum value of~$\xxx$ (<code>-y min</code>) is 
$$
\bar \xxx_{\jjj} = \min [ \mssflg_{1} \mskflg_{1} \wgt_{1} \xxx_{1},
\mssflg_{2} \mskflg_{2} \wgt_{2} \xxx_{2}, \ldots, \mssflg_{\lmnnbr} 
\mskflg_{\lmnnbr} \wgt_{\lmnnbr} \xxx_{\lmnnbr} ] 
$$
Analogously, the maximum value of~$\xxx$ (<code>-y max</code>) is 
$$
\bar \xxx_{\jjj} = \max [ \mssflg_{1} \mskflg_{1} \wgt_{1} \xxx_{1},
\mssflg_{2} \mskflg_{2} \wgt_{2} \xxx_{2}, \ldots, \mssflg_{\lmnnbr} 
\mskflg_{\lmnnbr} \wgt_{\lmnnbr} \xxx_{\lmnnbr} ] 
$$
Thus the minima and maxima are determined after any weights are applied.

The square of the mean value of~$\xxx$ (<code>-y sqravg</code>) is 
$$
\bar \xxx_{\jjj} = \left( {\sum_{\idx=1}^{\idx=\lmnnbr}
\mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx} \xxx_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}} 
\right)^{2}    
$$
The mean of the sum of squares of~$\xxx$ (<code>-y avgsqr</code>) is 
$$
\bar \xxx_{\jjj} = {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx^{2}_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}} 
$$
If $\xxx$ represents a deviation from the mean of another variable,
$\xxx_{\idx} = \yyy_{\idx} - \bar{\yyy}$ (possibly created by
<command>ncbo</command> in a previous step), then applying <code>avgsqr</code> to
$\xxx$ computes the approximate variance of $\yyy$. 
Computing the true variance of~$\yyy$ requires subtracting~1 from the
denominator, discussed below.
For a large sample size however, the two results will be nearly
indistinguishable. 

The root mean square of~$\xxx$ (<code>-y rms</code>) is 
$$
\bar \xxx_{\jjj} = \sqrt{ {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} x^{2}_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}} 
$$
Thus <code>rms</code> simply computes the squareroot of the quantity computed
by <code>avgsqr</code>.

The root mean square of~$\xxx$ with standard-deviation-like
normalization (<code>-y rmssdn</code>) is implemented as follows.
When weights are not specified, this function is the same as the root
mean square of~$\xxx$ except one is subtracted from the sum in the
denominator 
$$
\bar \xxx_{\jjj} = \sqrt{ {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} 
\mskflg_{\idx} x^{2}_{\idx} \over -1 + \sum_{\idx=1}^{\idx=\lmnnbr}
\mssflg_{\idx} \mskflg_{\idx}} } 
$$
If $\xxx$ represents the deviation from the mean of another variable, 
$\xxx_{\idx} = \yyy_{\idx} - \bar{\yyy}$, then applying <code>rmssdn</code> to
$\xxx$ computes the standard deviation of~$\yyy$.
In this case the $-1$ in the denominator compensates for the degree of
freedom already used in computing $\bar{\yyy}$ in the numerator.
Consult a statistics book for more details.

When weights are specified it is unclear how to compensate for this
extra degree of freedom.
Weighting the numerator and denominator of the above by $\wgt_{\idx}$
and subtracting one from the denominator is only appropriate when all
the weights are~1.0.
When the weights are arbitrary (e.g., Gaussian weights), subtracting one 
from the sum in the denominator does not necessarily remove one degree
of freedom. 
Therefore when <code>-y rmssdn</code> is requested and weights are specified,
<command>ncwa</command> actually implements the <code>rms</code> procedure.
<command>nces</command> and <command>ncra</command>, which do not allow weights to be
specified, always implement the <code>rmssdn</code> procedure when asked.

<!-- c 20130827: Fedora Core 19 (FC19) broke here with "./nco.texi:6394: Missing dollarsign inserted." -->
<!-- c Ubuntu always built nco.texi fine -->
<!-- c Adding a dollarsign character right here breaks Ubuntu builds too -->
<!-- c Hence I must carefully spell-out the word dollarsign instead -->
<!-- c 20130829: Making many smaller TeX environments does not solve problem -->
<!-- c 20130910: Using latest texinfo.tex from GNU does not solve problem -->
<!-- c 20130910: Karl Berry solved problem by fixing bug in texinfo.tex -->
<!-- c Bug was triggered in Fedora by apostrophe in "User's Guide" (manual title) -->
<!-- c Bug not present in texinfo.tex version 2008-04-18.10 (used by Ubuntu 13.04) -->
<!-- c Bug     present in texinfo.tex version 2013-02-01.11 (used by FC19) -->
<!-- c Bug just fixed  in texinfo.tex version 2013-09-11 (committed by Karl) -->
<!-- c nco/autobld/texinfo.tex now contains fixed version -->
<!-- c Breakage always occurs near here -->
The square root of the mean of~$\xxx$ (<code>-y sqrt</code>) is 
$$
\bar \xxx_{\jjj} = \sqrt{ {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}}  
$$
</tex>
<para>The definitions of some of these operations are not universally useful.
Mostly they were chosen to facilitate standard statistical
computations within the <acronym><acronymword>NCO</acronymword></acronym> framework.
We are open to redefining and or adding to the above. 
If you are interested in having other statistical quantities
defined in <acronym><acronymword>NCO</acronymword></acronym> please contact the <acronym><acronymword>NCO</acronymword></acronym> project
(<pxref label="Help-Requests-and-Bug-Reports"><xrefnodename>Help Requests and Bug Reports</xrefnodename></pxref>).  
</para>
<noindent></noindent>
<para>EXAMPLES
</para>
<html endspaces=" ">
&lt;a name=&quot;min&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#min &textndash;&gt;
&lt;a name=&quot;max&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#max &textndash;&gt;
&lt;a name=&quot;rms&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rms &textndash;&gt;
</html>
<noindent></noindent>
<para>Suppose you wish to examine the variable <code>prs_sfc(time,lat,lon)</code> 
which contains a time series of the surface pressure as a function of
latitude and longitude.
Find the minimium value of <code>prs_sfc</code> over all dimensions:
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -y min -v prs_sfc in.nc foo.nc 
</pre></example>
<noindent></noindent>
<para>Find the maximum value of <code>prs_sfc</code> at each time interval for each
latitude: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -y max -v prs_sfc -a lon in.nc foo.nc
</pre></example>
<noindent></noindent>
<para>Find the root-mean-square value of the time-series of <code>prs_sfc</code> at
every gridpoint:
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -y rms -v prs_sfc in.nc foo.nc
ncwa -y rms -v prs_sfc -a time in.nc foo.nc
</pre></example>
<noindent></noindent>
<para>The previous two commands give the same answer but <command>ncra</command> is
preferred because it has a smaller memory footprint.
<cindex index="cp" spaces=" "><indexterm index="cp" number="790">degenerate dimension</indexterm></cindex>
A dimension of size one is said to be <dfn>degenerate</dfn>.
By default, <command>ncra</command> leaves the (degenerate) <code>time</code>
dimension in the output file (which is usually useful) whereas
<command>ncwa</command> removes the <code>time</code> dimension (unless <samp>-b</samp> is
given).
</para>
<noindent></noindent>
<para>These operations work as expected in multi-file operators.
Suppose that <code>prs_sfc</code> is stored in multiple timesteps per file
across multiple files, say <file>jan.nc</file>, <file>feb.nc</file>,
<file>march.nc</file>.  
We can now find the three month maximium surface pressure at every point.
</para><example endspaces=" ">
<pre xml:space="preserve">nces -y max -v prs_sfc jan.nc feb.nc march.nc out.nc
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;standard_deviation&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#standard_deviation &textndash;&gt;
&lt;a name=&quot;stddvn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#stddvn &textndash;&gt;
&lt;a name=&quot;sdn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#sdn &textndash;&gt;
&lt;a name=&quot;sdv&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#sdv &textndash;&gt;
&lt;a name=&quot;xmp_sdn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_sdn &textndash;&gt;
</html>
<noindent></noindent>
<cindex index="cp" spaces=" "><indexterm index="cp" number="791">standard deviation</indexterm></cindex>
<para>It is possible to use a combination of these operations to compute
the variance and standard deviation of a field stored in a single file
or across multiple files.
The procedure to compute the temporal standard deviation of the surface
pressure at all points in a single file <file>in.nc</file> involves three
steps. 
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -O -v prs_sfc -a time in.nc out.nc
ncbo -O -v prs_sfc in.nc out.nc out.nc 
ncra -O -y rmssdn out.nc out.nc
</pre></example>
<para>First construct the temporal mean of <code>prs_sfc</code> in the file
<file>out.nc</file>.
Next overwrite <file>out.nc</file> with the anomaly (deviation from the mean).
Finally overwrite <file>out.nc</file> with the root-mean-square of itself. 
Note the use of <samp>-y rmssdn</samp> (rather than <samp>-y rms</samp>) in the
final step. 
This ensures the standard deviation is correctly normalized by one fewer
than the number of time samples.
The procedure to compute the variance is identical except for the use of 
<samp>-y var</samp> instead of <samp>-y rmssdn</samp> in the final step.
</para>
<para><command>ncap2</command> can also compute statistics like standard deviations.
Brute-force implementation of formulae is one option, e.g.,
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'prs_sfc_sdn=sqrt((prs_sfc-prs_sfc.avg($time)^2).total($time)/($time.size-1))'
      in.nc out.nc
</pre></example>
<para>The operation may, of course, be broken into multiple steps in order  
to archive intermediate quantities, such as the time-anomalies
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'prs_sfc_anm=prs_sfc-prs_sfc.avg($time)' \
      -s 'prs_sfc_sdn=sqrt((prs_sfc_anm^2).total($time)/($time.size-1))' \
      in.nc out.nc
</pre></example>

<para><command>ncap2</command> supports intrinsic standard deviation functions
(<pxref label="Operation-Types"><xrefnodename>Operation Types</xrefnodename></pxref>) which simplify the above expression to
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'prs_sfc_sdn=(prs_sfc-prs_sfc.avg($time)).rmssdn($time)' in.nc out.nc
</pre></example>
<para>These instrinsic functions compute the answer quickly and concisely.
</para>
<para>The procedure to compute the spatial standard deviation of a field
in a single file <file>in.nc</file> involves three steps.
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -O -v prs_sfc,gw -a lat,lon -w gw in.nc out.nc
ncbo -O -v prs_sfc,gw in.nc out.nc out.nc
ncwa -O -y rmssdn -v prs_sfc -a lat,lon -w gw out.nc out.nc
</pre></example>
<para>First the appropriately weighted (with <samp>-w gw</samp>) spatial mean values
are written to the output file.  
This example includes the use of a weighted variable specified with
<samp>-w gw</samp>. 
When using weights to compute standard deviations one must remember to
include the weights in the initial output files so that they may be used
again in the final step. 
The initial output file is then overwritten with the gridpoint
deviations from the spatial mean.
Finally the root-mean-square of the appropriately weighted spatial
deviations is taken.  
</para>
<para>The <command>ncap2</command> solution to the spatially-weighted standard
deviation problem is 
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'prs_sfc_sdn=(prs_sfc*gw-prs_sfc*gw.avg($lat,$lon)).rmssdn($lat,$lon)' \
      in.nc out.nc
</pre></example>
<para>Be sure to multiply the variable by the weight prior to computing the
the anomalies and the standard deviation.
</para>
<para>The procedure to compute the standard deviation of a time-series across
multiple files involves one extra step since all the input must first be
collected into one file. 
</para><example endspaces=" ">
<pre xml:space="preserve">ncrcat -O -v tpt in.nc in.nc foo1.nc
ncwa -O -a time foo1.nc foo2.nc
ncbo -O -v tpt foo1.nc foo2.nc foo3.nc
ncra -O -y rmssdn foo3.nc out.nc
</pre></example>
<para>The first step assembles all the data into a single file.
Though this may consume a lot of temporary disk space, it is more or
less required by the <command>ncbo</command> operation in the third step.
</para>
<html endspaces=" ">
&lt;a name=&quot;typ_cnv&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#typ_cnv &textndash;&gt;
</html>
</section>
<node name="Type-Conversion" spaces=" "><nodename>Type Conversion</nodename><nodenext spaces=" ">Batch Mode</nodenext><nodeprev spaces=" ">Operation Types</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Type Conversion</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="792">type conversion</indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability (automatic type conversion): <command>ncap2</command>, <command>ncbo</command>, <command>nces</command>,
<command>ncflint</command>, <command>ncra</command>, <command>ncwa</command>&linebreak; 
Short options: None (it&textrsquo;s <emph>automatic</emph>)&linebreak;
Availability (manual type conversion): <command>nces</command>, <command>ncra</command>, <command>ncwa</command>&linebreak; 
Short options: None&linebreak;
Long options: <samp>--dbl</samp>, <samp>--flt</samp>, <samp>--rth_dbl</samp>, <samp>--rth_flt</samp>&linebreak; 
</para></cartouche>
<cindex index="cp" spaces=" "><indexterm index="cp" number="793">promotion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="794">demotion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="795">automatic type conversion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="796">manual type conversion</indexterm></cindex>
<para>Type conversion refers to the casting or coercion of one fundamental or
atomic data type to another, e.g., converting <code>NC_SHORT</code> (two
bytes) to <code>NC_DOUBLE</code> (eight bytes).  
Type conversion always <dfn>promotes</dfn> or <dfn>demotes</dfn> the range and/or 
precision of the values a variable can hold.
Type conversion is automatic when the language carries out this
promotion according to an internal set of rules without explicit user 
intervention. 
In contrast, manual type conversion refers to explicit user commands to
change the type of a variable or attribute.
Most type conversion happens automatically, yet there are situations in
which manual type conversion is advantageous.
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Automatic type conversion</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Promoting Single-precision to Double</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Manual type conversion</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<node name="Automatic-type-conversion" spaces=" "><nodename>Automatic type conversion</nodename><nodenext spaces=" ">Promoting Single-precision to Double</nodenext><nodeprev spaces=" ">Type Conversion</nodeprev><nodeup spaces=" ">Type Conversion</nodeup></node>
<subsection spaces=" "><sectiontitle>Automatic type conversion</sectiontitle>

<para>There are at least two reasons to avoid type conversions.
First, type conversions are expensive since they require creating
(temporary) buffers and casting each element of a variable from its 
storage type to some other type and then, often, converting it back.
Second, a dataset&textrsquo;s creator perhaps had a good reason for storing 
data as, say, <code>NC_FLOAT</code> rather than <code>NC_DOUBLE</code>.  
In a scientific framework there is no reason to store data with more
precision than the observations merit.
Normally this is single-precision, which guarantees 6&textndash;9 digits of
precision. 
Reasons to engage in type conversion include avoiding rounding 
errors and out-of-range limitations of less-precise types.
This is the case with most integers.
Thus <acronym><acronymword>NCO</acronymword></acronym> defaults to automatically promote integer types to
floating point when performing lengthy arithmetic, yet <acronym><acronymword>NCO</acronymword></acronym>
defaults to not promoting single to double-precision floats.
</para>
<para>Before discussing the more subtle floating point issues, we first
examine integer promotion. 
We will show how following parsimonious conversion rules dogmatically
can cause problems, and what <acronym><acronymword>NCO</acronymword></acronym> does about that. 
That said, there are situations in which implicit conversion of
single- to double-precision is also warranted.
Understanding the narrowness of these situations takes time, and we
hope the reader appreciates the following detailed discussion.
</para>
<para>Consider the average of the two <code>NC_SHORT</code>s <code>17000s</code> and
<code>17000s</code>.
A straightforward average without promotion results in garbage since the
intermediate value which holds their sum is also of type <code>NC_SHORT</code>
and thus overflows on (i.e., cannot represent) values greater than
32,767 
<footnote spaces="\n"><set name="flg" line=" flg"></set>
<tex endspaces=" ">
$32767 = 2^{15}-1$
<clear name="flg" line=" flg"></clear>
</tex>
<para><math>32767 = 2^15-1</math>
<clear name="flg" line=" flg"></clear>
</para></footnote>.
There are valid reasons for expecting this operation to succeed and 
the <acronym><acronymword>NCO</acronymword></acronym> philosophy is to make operators do what you want, not
what is purest.
Thus, unlike C and Fortran, but like many other higher level interpreted
languages, <acronym><acronymword>NCO</acronymword></acronym> arithmetic operators will perform automatic type
conversion on integers when all the following conditions are met
<footnote><para>Operators began performing automatic type conversions before
arithmetic in <acronym><acronymword>NCO</acronymword></acronym> <w>version 1.2</w>, August, 2000. 
Previous versions never performed unnecessary type conversion for
arithmetic.</para></footnote>: 
</para><enumerate first="1" endspaces=" ">
<listitem spaces=" "><para>The requested operation is arithmetic.
This is why type conversion is limited to the operators <command>ncap2</command>, 
<command>ncbo</command>, <command>nces</command>, <command>ncflint</command>, <command>ncra</command>, and
<command>ncwa</command>.   
</para></listitem><listitem spaces=" "><para>The arithmetic operation could benefit from type conversion.
Operations that could benefit include averaging, summation, or any
&quot;hard&quot; arithmetic that could overflow or underflow.  
Larger representable sums help avoid overflow, and more precision
helps to avoid underflow.
Type conversion does not benefit searching for minima and maxima
(<samp>-y min</samp>, or <samp>-y max</samp>).
</para></listitem><listitem spaces=" "><para>The variable on disk is of type <code>NC_BYTE</code>, <code>NC_CHAR</code>,
<code>NC_SHORT</code>, or <code>NC_INT</code>.
Type <code>NC_DOUBLE</code> is not promoted because there is no type of
higher precision.
Conversion of type <code>NC_FLOAT</code> is discussed in detail below. 
When it occurs, it follows the same procedure (promotion then
arithmetic then demotion) as conversion of integer types.
</para></listitem></enumerate>

<para>When these criteria are all met, the operator promotes the variable in
question to type <code>NC_DOUBLE</code>, performs all the arithmetic
operations, casts the <code>NC_DOUBLE</code> type back to the original type,
and finally writes the result to disk. 
The result written to disk may not be what you expect, because of
incommensurate ranges represented by different types, and because of
(lack of) rounding.
First, continuing the above example, the average (e.g., <samp>-y avg</samp>)
of <code>17000s</code> and <code>17000s</code> is written to disk as <code>17000s</code>. 
The type conversion feature of <acronym><acronymword>NCO</acronymword></acronym> makes this possible since
the arithmetic and intermediate values are stored as <code>NC_DOUBLE</code>s,
i.e., <code>34000.0d</code> and only the final result must be represented
as an <code>NC_SHORT</code>.
Without the type conversion feature of <acronym><acronymword>NCO</acronymword></acronym>, the average would
have been garbage (albeit predictable garbage near <code>-15768s</code>). 
Similarly, the total (e.g., <samp>-y ttl</samp>) of <code>17000s</code> and
<code>17000s</code> written to disk is garbage (actually <code>-31536s</code>) since 
the final result (the true total) of <math>34000</math> is outside the range
of type <code>NC_SHORT</code>.  
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="797"><code>trunc()</code></indexterm></cindex>
<para>After arithmetic is computed in double-precision for promoted variables,
the intermediate double-precision values must be demoted to the
variables&textrsquo; original storage type (e.g., from <code>NC_DOUBLE</code> to
<code>NC_SHORT</code>). 
<acronym><acronymword>NCO</acronymword></acronym> has handled this demotion in three ways in its history.
Prior to October, 2011 (version 4.0.8), <acronym><acronymword>NCO</acronymword></acronym> employed the
<w>C library</w> truncate function, <code>trunc()</code>
<footnote spaces="\n"><cindex index="cp" spaces=" "><indexterm index="cp" number="798">C language</indexterm></cindex>
<para>The actual type conversions with trunction were handled by intrinsic
type conversion, so the <code>trunc()</code> function was never explicitly
called, although the results would be the same if it were.</para></footnote>.
Truncation rounds <var>x</var> to the nearest integer not larger in absolute 
value.
For example, truncation rounds <code>1.0d</code>, <code>1.5d</code>, and
<code>1.8d</code> to the same value, <code>1s</code>. 
Clearly, truncation does not round floating point numbers to the nearest
integer! 
Yet truncation is how the <w>C language</w> performs implicit conversion of 
real numbers to integers.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="799">Neil Davis</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> stopped using truncation for demotion when an alert user
(Neil Davis) informed us that this caused a small bias in the packing
algorithm employed by <command>ncpdq</command>.
This led to <acronym><acronymword>NCO</acronymword></acronym> adopting rounding functions for demotion.
Rounding functions eliminated the small bias in the packing algorithm.
</para>
<findex index="fn" spaces=" "><indexterm index="fn" number="12" mergedindex="cp"><code>lround()</code>.</indexterm></findex>
<para>From February, 2012 through March, 2013 (versions 4.0.9&textndash;4.2.6),
<acronym><acronymword>NCO</acronymword></acronym> employed the <w>C library</w> family of rounding functions,
<code>lround()</code>. 
These functions round <var>x</var> to the nearest integer, halfway cases away
from zero.
The problem with <code>lround()</code> is that it always rounds real values
ending in <code>.5</code> away from zero.
This rounds, for example, <code>1.5d</code> and <code>2.5d</code> to <code>1s</code>
and <code>2s</code>, respectively.
</para>
<findex index="fn" spaces=" "><indexterm index="fn" number="13" mergedindex="cp"><code>lrint()</code>.</indexterm></findex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="800"><acronym><acronymword>IEEE</acronymword></acronym></indexterm></cindex>
<para>Since April, 2013 (version 4.3.0), <acronym><acronymword>NCO</acronymword></acronym> has employed the 
other <w>C library</w> family of rounding functions, <code>lrint()</code>. 
This algorithm rounds <var>x</var> to the nearest integer, using the current 
rounding direction.
Halfway cases are rounded to the nearest even integer.
This rounds, for example, both <code>1.5d</code> and <code>2.5d</code> to the same
value, <code>2s</code>, as recommended by the <acronym><acronymword>IEEE</acronymword></acronym>.
This rounding is symmetric: up half the time, down half the time.
This is the current and hopefully final demotion algorithm employed by  
<acronym><acronymword>NCO</acronymword></acronym>.
</para>
<para>Hence because of automatic conversion, <acronym><acronymword>NCO</acronymword></acronym> will compute the
average of <code>2s</code> and <code>3s</code> in double-precision arithmetic as 
<math>(<code>2.0d</code> + <code>3.0d</code>)/<code>2.0d</code>) = <code>2.5d</code></math>.
It then demotes this intermediate result back to <code>NC_SHORT</code> and
stores it on disk as  
<math><code>trunc(2.5d)</code> = <code>2s</code></math> (versions up to 4.0.8), 
<math><code>lround(2.5d)</code> = <code>3s</code></math> (versions 4.0.9&textndash;4.2.6), and
<math><code>lrint(2.5d)</code> = <code>2s</code></math> (versions 4.3.0 and later).
</para>
<html endspaces=" ">
&lt;a name=&quot;sp_dp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#sp_dp &textndash;&gt;
&lt;a name=&quot;dbl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dbl &textndash;&gt;
</html>
</subsection>
<node name="Promoting-Single_002dprecision-to-Double" spaces=" "><nodename>Promoting Single-precision to Double</nodename><nodenext spaces=" ">Manual type conversion</nodenext><nodeprev spaces=" ">Automatic type conversion</nodeprev><nodeup spaces=" ">Type Conversion</nodeup></node>
<subsection spaces=" "><sectiontitle>Promoting Single-precision to Double</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="801">promotion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="802">implicit conversion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="803"><code>--dbl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="804"><code>--rth_dbl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="805"><code>--flt</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="806"><code>--rth_flt</code></indexterm></cindex>
<para>Promotion of real numbers from single- to double-precision is
fundamental to scientific computing.
When it should occur depends on the precision of the inputs and the
number of operations.
Single-precision (four-byte) numbers contain about seven significant
figures, while double-precision contain about sixteen.
More, err, precisely, the <acronym><acronymword>IEEE</acronymword></acronym> single-precision representation
gives from <w>6 to 9</w> significant decimal digits precision
<footnote><para>According to Wikipedia&textrsquo;s summary of <acronym><acronymword>IEEE</acronymword></acronym> standard
754, &textldquo;If a decimal string with at most 6 significant digits is
converted to IEEE 754 single-precision and then converted back to the
same number of significant decimal, then the final string should match
the original; and if an IEEE 754 single-precision is converted to a
decimal string with at least 9 significant decimal and then converted
back to single, then the final number must match the original&textrdquo;.</para></footnote>.
And the <acronym><acronymword>IEEE</acronymword></acronym> double-precision representation
gives from <w>15 to 17</w> significant decimal digits precision
<footnote><para>According to Wikipedia&textrsquo;s summary of <acronym><acronymword>IEEE</acronymword></acronym> standard
754, &textldquo;If a decimal string with at most 15 significant digits is
converted to IEEE 754 double-precision representation and then converted
back to a string with the same number of significant digits, then the
final string should match the original; and if an IEEE 754 double
precision is converted to a decimal string with at least 17 significant
digits and then converted back to double, then the final number must
match the original&textrdquo;.</para></footnote>. 
Hence double-precision numbers represent about nine digits more
precision than single-precision numbers.
</para>
<para>Given these properties, there are at least two possible arithmetic
conventions for the treatment of real numbers:
<cindex index="cp" spaces=" "><indexterm index="cp" number="807">C language</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="808">Fortran</indexterm></cindex>
</para><enumerate first="1" endspaces=" ">
<listitem spaces=" "><para>Conservative, aka Fortran Convention
Automatic type conversion during arithmetic in the Fortran language is,
by default, performed only when necessary. 
All operands in an operation are converted to the most precise type
involved the operation before the arithmetic operation.
Expressions which involve only single-precision numbers are computed
entirely in single-precision.
Expressions involving mixed precision types are computed in the type
of higher precision.
<acronym><acronymword>NCO</acronymword></acronym> by default employs the Fortan Convention for promotion.
</para></listitem><listitem spaces=" "><para>Aggressive, aka C Convention
The <w>C language</w> is by default much more aggressive (and thus
wasteful) than Fortran, and will always implicitly convert single- 
to double-precision numbers for no good reason.
All real-number standard <w>C library</w> functions are double-precision, 
and <w>C programmers</w> must take extra steps to only utilize single
precision arithmetic.
The high level interpreted data analysis languages <acronym><acronymword>IDL</acronymword></acronym>,
Matlab, and <acronym><acronymword>NCL</acronymword></acronym> all adopt the <w>C Convention</w>.
</para></listitem></enumerate>

<para><acronym><acronymword>NCO</acronymword></acronym> does not automatically promote <code>NC_FLOAT</code> because, in
our judgement, the performance penalty of always doing so would outweigh
the potential benefits. 
The now-classic text &textldquo;Numerical Recipes <w>in C</w>&textrdquo; discusses this point
under the section &textldquo;Implicit Conversion of Float to Double&textrdquo;
<footnote><para>See <w>page 21</w> in Section 1.2 of the First edition for this
gem:
</para><quotation endspaces=" ">
<para>One does not need much experience in scientific computing to recognize
that the implicit conversion rules are, in fact, sheer madness!
In effect, they make it impossible to write efficient numerical
programs. 
</para></quotation>
</footnote>.
That said, such promotion is warranted in some circumstances.
</para>
<para>For example, rounding errors can accumulate to worrisome levels during
arithmetic performed on large arrays of single-precision floats. 
This use-case occurs often in geoscientific studies of climate where
thousands-to-millions of gridpoints may contribute to a single average.
If the inputs are all single-precision, then so should be the output.
However the intermediate results where running sums are accumulated may  
suffer from too much rounding or from underflow unless computed in
double-precision. 
</para>
<para>The order of operations matters to floating point math even when the
analytic expressions are equal. 
Cautious users feel disquieted when results from equally valid analyses
differ in the final bits instead of agreeing bit-for-bit.
For example, averaging arrays in multiple stages produces different
answers than averaging them in one step. 
This is easily seen in the computation of ensemble averages by two
different methods.
The <acronym><acronymword>NCO</acronymword></acronym> test file <file>in.nc</file> contains single- and
double-precision representations of the same temperature timeseries as 
<code>tpt_flt</code> and <code>tpt_dbl</code>.  
Pretend each datapoint in this timeseries represents a monthly-mean
temperature. 
We will mimic the derivation of a fifteen-year ensemble-mean January
temperature by concatenating the input file five times, and then
averaging the datapoints representing January two different ways.
In <w>Method 1</w> we derive the 15-year ensemble January average in two 
steps, as the average of three five-year averages.
This method is naturally used when each input file contains multiple
years and multiple input files are needed
<footnote><para>For example, the <acronym><acronymword>CMIP5</acronymword></acronym> archive tends to distribute
monthly average timeseries in 50-year chunks.</para></footnote>.
In <w>Method 2</w> we obtain 15-year ensemble January average in a single
step, by averaging all 15 Januaries at one time:
</para><example endspaces=" ">
<pre xml:space="preserve"># tpt_flt and tpt_dbl are identical except for precision
ncks --cdl -C -v tpt_flt,tpt_dbl ~/nco/data/in.nc
# tpt_dbl = 273.1, 273.2, 273.3, 273.4, 273.5, 273.6, 273.7, 273.8, 273.9, 274
# tpt_flt = 273.1, 273.2, 273.3, 273.4, 273.5, 273.6, 273.7, 273.8, 273.9, 274
# Create file with five &quot;ten-month years&quot; (i.e., 50 timesteps) of temperature data
ncrcat -O -v tpt_flt,tpt_dbl -p ~/nco/data in.nc in.nc in.nc in.nc in.nc ~/foo.nc
# Average 1st five &quot;Januaries&quot; (elements 1, 11, 21, 31, 41)
ncra --flt -O -F -d time,1,,10 ~/foo.nc ~/foo_avg1.nc
# Average 2nd five &quot;Januaries&quot; (elements 2, 12, 22, 32, 42)
ncra --flt -O -F -d time,2,,10 ~/foo.nc ~/foo_avg2.nc
# Average 3rd five &quot;Januaries&quot; (elements 3, 13, 23, 33, 43)
ncra --flt -O -F -d time,3,,10 ~/foo.nc ~/foo_avg3.nc
# Method 1: Obtain ensemble January average by averaging the averages
ncra --flt -O ~/foo_avg1.nc ~/foo_avg2.nc ~/foo_avg3.nc ~/foo_avg_mth1.nc
# Method 2: Obtain ensemble January average by averaging the raw data
# Employ ncra's &quot;subcycle&quot; feature (http://nco.sf.net/nco.html#ssc)
ncra --flt -O -F -d time,1,,10,3 ~/foo.nc ~/foo_avg_mth2.nc
# Difference the two methods
ncbo -O ~/foo_avg_mth1.nc ~/foo_avg_mth2.nc ~/foo_avg_dff.nc
ncks --cdl ~/foo_avg_dff.nc
# tpt_dbl = 5.6843418860808e-14 ;
# tpt_flt = -3.051758e-05 ;
</pre></example>
<para>Although the two methods are arithmetically equivalent, they produce
slightly different answers due to the different order of operations.
Moreover, it appears at first glance that the single-precision
answers suffer from greater error than the double-precision answers.
In fact both precisions suffer from non-zero rounding errors.
The answers differ negligibly to machine precision, which is about 
seven significant figures for single precision floats (<code>tpt_flt</code>),
and sixteen significant figures for double precision (<code>tpt_dbl</code>).
The input precision determines the answer precision.
</para>
<para>IEEE arithmetic guarantees that two methods will produce bit-for-bit
identical answers only if they compute the same operations in the same  
order.  
Bit-for-bit identical answers may also occur by happenstance when 
rounding errors exactly compensate one another.
This is demonstrated by repeating the example above with the
<samp>--dbl</samp> (or <samp>--rth_dbl</samp> for clarity) option which forces
conversion of single-precision numbers to double-precision prior to
arithmetic. 
Now <command>ncra</command> will treat the first value of <code>tpt_flt</code>,
<code>273.1000f</code>, as <code>273.1000000000000d</code>. 
Arithmetic on <code>tpt_flt</code> then proceeds in double-precision until the
final answer, which is converted back to single-precision for final
storage. 
</para><example endspaces=" ">
<pre xml:space="preserve"># Average 1st five &quot;Januaries&quot; (elements 1, 11, 21, 31, 41)
ncra --dbl -O -F -d time,1,,10 ~/foo.nc ~/foo_avg1.nc
# Average 2nd five &quot;Januaries&quot; (elements 2, 12, 22, 32, 42)
ncra --dbl -O -F -d time,2,,10 ~/foo.nc ~/foo_avg2.nc
# Average 3rd five &quot;Januaries&quot; (elements 3, 13, 23, 33, 43)
ncra --dbl -O -F -d time,3,,10 ~/foo.nc ~/foo_avg3.nc
# Method 1: Obtain ensemble January average by averaging the averages
ncra --dbl -O ~/foo_avg1.nc ~/foo_avg2.nc ~/foo_avg3.nc ~/foo_avg_mth1.nc
# Method 2: Obtain ensemble January average by averaging the raw data
# Employ ncra's &quot;subcycle&quot; feature (http://nco.sf.net/nco.html#ssc)
ncra --dbl -O -F -d time,1,,10,3 ~/foo.nc ~/foo_avg_mth2.nc
# Difference the two methods
ncbo -O ~/foo_avg_mth1.nc ~/foo_avg_mth2.nc ~/foo_avg_dff.nc
# Show differences
ncks --cdl ~/foo_avg_dff.nc
# tpt_dbl = 5.6843418860808e-14 ;
# tpt_flt = 0 ;
</pre></example>
<para>The <samp>--dbl</samp> switch has no effect on the results computed from
double-precision inputs.
But now the two methods produce bit-for-bit identical results from the
single-precision inputs!
This is due to the happenstance of rounding along with the effects of 
the <samp>--dbl</samp> switch.
The <samp>--flt</samp> and <samp>--rth_flt</samp> switches are provided for
symmetry.
They enforce the traditional <acronym><acronymword>NCO</acronymword></acronym> and Fortran convention of
keeping single-precision arithmetic in single-precision unless a
double-precision number is explicitly involved.
</para>
<para>As has been seen, forced promotion of single- to double-precision
prior to arithmetic has advantages and disadvantages. 
The primary disadvantages are speed and size. 
Double-precision arithmetic is 10&textndash;60% slower than, and requires
twice the memory of single-precision arithmetic. 
The primary advantage is that rounding errors in double-precision are 
much less likely to accumulate to values near the precision of the 
underlying geophysical variable. 
</para>
<para>For example, if we know temperature to five significant digits, then a
rounding error of 1-bit could affect the least precise digit of
temperature after 1,000&textndash;10,000 consecutive one-sided rounding
errors under the worst possible scenario.
Many geophysical grids have tens-of-thousands to millions of points
that must be summed prior to normalization to compute an average.
It is possible for single-precision rouding errors to accumulate and
degrade the precision in such situtations. 
Double-precision arithmetic mititgates this problem, so <samp>--dbl</samp>
would be warranted. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="809"><acronym><acronymword>TREFHT</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="810"><acronym><acronymword>CAM3</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="811"><acronym><acronymword>GCM</acronymword></acronym></indexterm></cindex>
<para>This can be seen with another example, averaging a global surface
temperature field with <command>ncwa</command>.
The input contains a single-precision global temperature field
(stored in <code>TREFHT</code>) produced by the <acronym><acronymword>CAM3</acronymword></acronym> general
circulation model (<acronym><acronymword>GCM</acronymword></acronym>) run and stored at <w>1.9 by 2.5</w>
degrees resolution. 
This requires <w>94 latitudes</w> and <w>144 longitudes</w>, or <math>13,824</math> 
total surface gridpoints, a typical GCM resolution these days.
These input characteristics are provided only to show the context
to the interested reader, equivalent results would be found in 
statistics of any dataset of comparable size.
Models often represent Earth on a spherical grid where global averages 
must be created by weighting each gridcell by its latitude-dependent
weight (i.e., the Gaussian weight stored in <code>gw</code>), or by the
surface area of each contributing gridpoint (stored in <code>area</code>).
</para>
<para>Like many geophysical models and most <acronym><acronymword>GCM</acronymword></acronym>s, <acronym><acronymword>CAM3</acronymword></acronym>
runs completely in double-precision yet stores its archival output in
single-precision to save space.
In practice such models usually save multi-dimensional prognostic and
diagnostic fields (like <code>TREFHT(lat,lon)</code> and <code>area(lat,lon)</code>)
as single-precision, while saving all one-dimensional coordinates and
weights (here <code>lat</code>, <code>lon</code>, and <code>gw(lon)</code>) as
double-precision.  
To obtain pure double-precision arithmetic <emph>and</emph> storage of the 
globla mean temperature, we first create and store double-precision
versions of the single-precision fields:
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -O -s 'TREFHT_dbl=double(TREFHT);area_dbl=double(area)' in.nc in.nc
</pre></example>
<para>The single- and double-precision temperatures may each be averaged
globally using four permutations for the precision of the weight
and of the intermediate arithmetic representation:
</para><enumerate first="1" endspaces=" ">
<listitem spaces=" "><para>Single-precision weight (<code>area</code>), single-precision arithmetic
</para></listitem><listitem spaces=" "><para>Double-precision weight (<code>gw</code>),   single-precision arithmetic
</para></listitem><listitem spaces=" "><para>Single-precision weight (<code>area</code>), double-precision arithmetic
</para></listitem><listitem spaces=" "><para>Double-precision weight (<code>gw</code>),   double-precision arithmetic
</para></listitem></enumerate>
<example endspaces=" ">
<pre xml:space="preserve"># NB: Values below are printed with C-format %5.6f using
# ncks -H -C -s '%5.6f' -v TREFHT,TREFHT_dbl out.nc
# Single-precision weight (area), single-precision arithmetic
ncwa --flt -O -a lat,lon -w area in.nc out.nc
# TREFHT     = 289.246735 
# TREFHT_dbl = 289.239964
# Double-precision weight (gw),   single-precision arithmetic
ncwa --flt -O -a lat,lon -w gw   in.nc out.nc
# TREFHT     = 289.226135
# TREFHT_dbl = 289.239964
# Single-precision weight (area), double-precision arithmetic
ncwa --dbl -O -a lat,lon -w area in.nc out.nc
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964
# Double-precision weight (gw),   double-precision arithmetic
ncwa --dbl -O -a lat,lon -w gw   in.nc out.nc
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964
</pre></example>
<para>First note that the <code>TREFHT_dbl</code> average never changes because 
<code>TREFHT_dbl(lat,lon)</code> is double-precision in the input file.
As described above, <acronym><acronymword>NCO</acronymword></acronym> automatically converts all operands
involving to the highest precision involved in the operation.
So specifying <samp>--dbl</samp> is redundant for double-precision inputs.
</para>
<para>Second, the single-precision arithmetic averages of the single-precision
input <code>TREFHT</code> differ by <math>289.246735 - 289.226135 = 0.0206</math>
from eachother, and, more importantly, by as much as 
<math>289.239964 - 289.226135 = 0.013829</math> from the correct
(double-precision) answer.
These averages differ in the fifth digit, i.e., they agree only to four
significant figures!
Given that climate scientists are concerned about global temperature
variations of a tenth of a degree or less, this difference is large.
It means that the global mean temperature changes scientists are looking
for are comparable in size to the numerical artifacts produced by the
averaging procedure. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="812">rounding</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="813">random walk</indexterm></cindex>
<para>Why are the single-precision numerical artifacts so large?
Each global average is the result of multiplying almost 15,000 elements
each by its weight, summing those, and then dividing by the summed 
weights.  
Thus about 50,000 single-precision floating point operations caused
the loss of two to three significant digits of precision.
The net error of a series of independent rounding errors is a random
walk phenomena
<footnote spaces="\n"><cindex index="cp" spaces=" "><indexterm index="cp" number="814">Michael Prather</indexterm></cindex>
<para>Thanks to <w>Michael J.</w> Prather for explaining this to me.</para></footnote>.
Successive rounding errors displace the answer further from the truth.
An ensemble of such averages will, on average, have no net bias.
In other words, the expectation value of a series of <acronym><acronymword>IEEE</acronymword></acronym>
rounding errors is zero.
And the error of any given sequence of rounding errors obeys, for large 
series, a Gaussian distribution centered on zero.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="815">mantissa</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="816">exponent</indexterm></cindex>
<para>Single-precision numbers use three of their four eight-bit bytes to
represent the mantissa so the smallest representable single-precision
mantissa is <math>\epsilon \equiv 2^{-23} = 1.19209 \times 10^{-7}</math>.
This <math>\epsilon</math> is the smallest <var>x</var> such that 
<math>1.0 + x \ne 1.0</math>.
This is the rounding error for non-exact precision-numbers.
Applying random walk theory to rounding, it can be shown that the
expected rounding error after <var>n</var> inexact operations is
<math>\sqrt{2n/\pi}</math> for <w>large <var>n</var></w>.
The expected (i.e., mean absolute) rounding error in our example with
<math>13,824</math> additions is about 
<math>\sqrt{2 \times 13824 / \pi} = 91.96</math>.
Hence, addition alone of about fifteen thousand single-precision floats
is expected to consume about two significant digits of precision.
This neglects the error due to the inner product (weights times values)
and normalization (division by tally) aspects of a weighted average.
the ratio of two numbers each containing a numerical bias can magnify
the size of the bias. 
In summary, a global mean number computed from about 15,000 gridpoints
each with weights can be expected to lose up to three significant digits.
Since single-precision starts with about seven significant digits, we
should not expect to retain more than four significant digits after
computing weighted averages in single-precision.
The above example with <code>TREFHT</code> shows the expected four digits of
agreement. 
<!-- c For example, 50,000 coin flips would lead to 25,500 or more ``heads'' -->
<!-- c only a small percentage of the time. -->
<!-- c P(k,n)= 50000_C_25500 p^k(1-p)^(n-k) -->
<!-- c P(25500,50000)= 50000_C_25500 (0.5)^(25500)(0.5)^(24500) -->
<!-- c P(>=25500,50000)= ? -->
<!-- c fxm: Use Gaussian distribution/Random Walk -->
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="817">beer</indexterm></cindex>
<para>The <acronym><acronymword>NCO</acronymword></acronym> results have been independently validated to the
extent possible in three other languages: 
<w>C</w>, Matlab, and <acronym><acronymword>NCL</acronymword></acronym>. 
C and <acronym><acronymword>NCO</acronymword></acronym> are the only languages that permit single-precision 
numbers to be treated with single precision arithmetic:
</para><example endspaces=" ">
<pre xml:space="preserve"># Double-precision weight (gw),   single-precision arithmetic (C)
ncwa_3528514.exe
# TREFHT     = 289.240112
# Double-precision weight (gw),   double-precision arithmetic (C)
# TREFHT     = 289.239964
# Single-precision weight (area), double-precision arithmetic (Matlab)
# TREFHT     = 289.239964
# Double-precision weight (gw),   double-precision arithmetic (Matlab)
# TREFHT     = 289.239964
# Single-precision weight (area), double-precision arithmetic (NCL)
ncl &lt; ncwa_3528514.ncl
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964
# Double-precision weight (gw),   double-precision arithmetic (NCL)
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964
</pre></example>
<para>All languages tested (C, Matlab, <acronym><acronymword>NCL</acronymword></acronym>, and <acronym><acronymword>NCO</acronymword></acronym>) agree
to machine precision with double-precision arithmetic.
Users are fortunate to have a variety of high quality software that 
liberates them from the drudgery of coding their own.
Many packages are free (as in beer)!
As shown above <acronym><acronymword>NCO</acronymword></acronym> permits one to shift to their
float-promotion preferences as desired. 
No other language allows this with a simple switch.
</para>
<para>To summarize, until version 4.3.6 (September, 2013), the default
arithmetic convention of <acronym><acronymword>NCO</acronymword></acronym> followed the behavior of Fortran,
and automatically promoted single-precision to double-precision in all
mixed-precision expressions, and left-alone pure single-precision
expressions.  
This is faster and more memory efficient than other conventions.
However, pure single-precision arithmetic can lose too much precision
when used to condense (e.g., average) large arrays.
Statistics involving about <math>n = 10,000</math> single-precision inputs
will lose about <w>2&textndash;3</w> digits if not promoted to double-precision
prior to arithmetic. 
The loss scales with the squareroot <w>of <var>n</var></w>. 
For larger <var>n</var>, users should promote floats with the <samp>--dbl</samp> 
option if they want to preserve more than four significant digits in
their results. 
</para>
<para>The <samp>--dbl</samp> and <samp>--flt</samp> switches are only available with the
<acronym><acronymword>NCO</acronymword></acronym> arithmetic operators that could potentially perform more
than a few  single-precision floating point operations per result.
These are <command>nces</command>, <command>ncra</command>, and <command>ncwa</command>.
Each is capable of thousands to millions or more operations per result. 
By contrast, the arithmetic operators <command>ncbo</command> and
<command>ncflint</command> perform at most one floating point operation per
result. 
Providing the <samp>--dbl</samp> option for such trivial operations makes 
little sense, so the option is not currently made available.
</para>
<para>At the time of this writing (September 2013), we are interested in
users&textrsquo; opinions on these matters. 
Currently the default behavior is <samp>--flt</samp>.
We are willing to change the default to <samp>--dbl</samp> if users prefer.
Or we could set a threshold (e.g., <math>n \ge 10000</math>) after which
single- to double-precision promotion is automatically invoked.
Or we could make the default promotion convention settable via an
environment variable (<acronym><acronymword>GSL</acronymword></acronym> does this a lot).
Please let us know what you think of the selected defaults and options.
</para>
</subsection>
<node name="Manual-type-conversion" spaces=" "><nodename>Manual type conversion</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">Promoting Single-precision to Double</nodeprev><nodeup spaces=" ">Type Conversion</nodeup></node>
<subsection spaces=" "><sectiontitle>Manual type conversion</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="818"><command>ncap2</command></indexterm></cindex>
<para><command>ncap2</command> provides intrinsic functions for performing manual type
conversions.
This, for example, converts variable <code>tpt</code> to external type
<code>NC_SHORT</code> (a C-type <code>short</code>), and variable <code>prs</code> to
external type <code>NC_DOUBLE</code> (a C-type <code>double</code>). 
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'tpt=short(tpt);prs=double(prs)' in.nc out.nc
</pre></example>
<para><xref label="ncap2-netCDF-Arithmetic-Processor"><xrefnodename>ncap2 netCDF Arithmetic Processor</xrefnodename></xref>, for more details.
</para>
<html endspaces=" ">
&lt;a name=&quot;ovr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ovr &textndash;&gt;
&lt;a name=&quot;-O&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-O &textndash;&gt;
</html>
</subsection>
</section>
<node name="Batch-Mode" spaces=" "><nodename>Batch Mode</nodename><nodenext spaces=" ">History Attribute</nodenext><nodeprev spaces=" ">Type Conversion</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Batch Mode</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="819">batch mode</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="820">overwriting files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="821">appending to files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="822">force overwrite</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="823">force append</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="824"><code>-O</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="825"><code>-A</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="826"><code>--overwrite</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="827"><code>--ovr</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="828"><code>--apn</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="829"><code>--append</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
Short options: <samp>-O</samp>, <samp>-A</samp>&linebreak;
Long options: <samp>--ovr</samp>, <samp>--overwrite</samp>, <samp>--apn</samp>, <samp>--append</samp>&linebreak;
</para></cartouche>
<para>If the <var>output-file</var> specified for a command is a pre-existing file,
then the operator will prompt the user whether to overwrite (erase) the
existing <var>output-file</var>, attempt to append to it, or abort the
operation. 
However, interactive questions reduce productivity when processing large
amounts of data.
Therefore <acronym><acronymword>NCO</acronymword></acronym> also implements two ways to override its own safety
features, the <samp>-O</samp> and <samp>-A</samp> switches.
Specifying <samp>-O</samp> tells the operator to overwrite any existing
<var>output-file</var> without prompting the user interactively.
Specifying <samp>-A</samp> tells the operator to attempt to append to any
existing <var>output-file</var> without prompting the user interactively.
These switches are useful in batch environments because they suppress
interactive keyboard input.
NB: As of 20120515, <command>ncap2</command> is unable to append to files that
already contain the appended dimensions. 
</para>
<html endspaces=" ">
&lt;a name=&quot;hst&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hst &textndash;&gt;
&lt;a name=&quot;history&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#history &textndash;&gt;
&lt;a name=&quot;-h&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-h &textndash;&gt;
</html>
</section>
<node name="History-Attribute" spaces=" "><nodename>History Attribute</nodename><nodenext spaces=" ">File List Attributes</nodenext><nodeprev spaces=" ">Batch Mode</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>History Attribute</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="830"><code>history</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="831">timestamp</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="832">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="833">attributes, global</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="834"><code>-h</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="835"><code>--hst</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="836"><code>--history</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
Short options: <samp>-h</samp>&linebreak;
Long options: <samp>--hst</samp>, <samp>--history</samp>&linebreak;
</para></cartouche>
<para>All operators automatically append a <code>history</code> global attribute to
any file they create or modify.
The <code>history</code> attribute consists of a timestamp and the full string
of the invocation command to the operator, e.g., <samp>Mon May 26 20:10:24
1997: ncks in.nc foo.nc</samp>.
The full contents of an existing <code>history</code> attribute are copied
from the first <var>input-file</var> to the <var>output-file</var>.
The timestamps appear in reverse chronological order, with the most
recent timestamp appearing first in the <code>history</code> attribute.
Since <acronym><acronymword>NCO</acronymword></acronym> and many other netCDF operators adhere to the
<code>history</code> convention, the entire data processing path of a given
netCDF file may often be deduced from examination of its <code>history</code>
attribute. 
As of May, 2002, <acronym><acronymword>NCO</acronymword></acronym> is case-insensitive to the spelling
of the <code>history</code> attribute name.
Thus attributes named <code>History</code> or <code>HISTORY</code> (which are
non-standard and not recommended) will be treated as valid history
attributes. 
When more than one global attribute fits the case-insensitive search
for &quot;history&quot;, the first one found will be used.
<code>history</code> attribute
<cindex index="cp" spaces=" "><indexterm index="cp" number="837"><command>ncatted</command></indexterm></cindex>
To avoid information overkill, all operators have an optional switch
(<samp>-h</samp>, <samp>--hst</samp>, or <samp>--history</samp>) to override
automatically appending the <code>history</code> attribute 
(<pxref label="ncatted-netCDF-Attribute-Editor"><xrefnodename>ncatted netCDF Attribute Editor</xrefnodename></pxref>).   
Note that the <samp>-h</samp> switch also turns off writing the
<code>nco_input_file_list</code> attribute for multi-file operators
(<pxref label="File-List-Attributes"><xrefnodename>File List Attributes</xrefnodename></pxref>).
</para>
<html endspaces=" ">
&lt;a name=&quot;fl_lst_in_att&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#fl_lst_in_att &textndash;&gt;
</html>
</section>
<node name="File-List-Attributes" spaces=" "><nodename>File List Attributes</nodename><nodenext spaces=" ">CF Conventions</nodenext><nodeprev spaces=" ">History Attribute</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>File List Attributes</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="838"><code>nco_input_file_list</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="839"><code>nco_input_file_number</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="840"><code>stdin</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="841">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="842">attributes, global</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="843"><code>-H</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="844"><code>--fl_lst_in</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="845"><code>--file_list</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>nces</command>, <command>ncecat</command>, <command>ncra</command>, <command>ncrcat</command>&linebreak;
Short options: <samp>-H</samp>&linebreak;
Long options: <samp>--fl_lst_in</samp>, <samp>--file_list</samp>&linebreak;
</para></cartouche>
<para>Many methods of specifying large numbers of input file names pass
these names via pipes, encodings, or argument transfer programs
(<pxref label="Large-Numbers-of-Files"><xrefnodename>Large Numbers of Files</xrefnodename></pxref>).
When these methods are used, the input file list is not explicitly
passed on the command line.
This results in a loss of information since the <code>history</code>
attribute no longer contains the exact command by which the file
was created.
</para>
<para><acronym><acronymword>NCO</acronymword></acronym> solves this dilemma by archiving input file list
attributes.
When the input file list to a multi-file operator is specified
via <code>stdin</code>, the operator, by default, attaches two global
attributes to any file they create or modify.
The <code>nco_input_file_number</code> global attribute contains the number of
input files, and <code>nco_input_file_list</code> contains the file names,
specified as standard input to the multi-file operator. 
This information helps to verify that all input files the user thinks
were piped through <code>stdin</code> actually arrived.
Without the <code>nco_input_file_list</code> attribute, the information is lost
forever and the &textldquo;chain of evidence&textrdquo; would be broken.
</para>
<para>The <samp>-H</samp> switch overrides (turns off) the default behavior of
writing the input file list global attributes when input is from
<code>stdin</code>. 
The <samp>-h</samp> switch does this too, and turns off the <code>history</code>
attribute as well (<pxref label="History-Attribute"><xrefnodename>History Attribute</xrefnodename></pxref>).
Hence both switches allows space-conscious users to avoid storing what
may amount to many thousands of filenames in a metadata attribute.
</para>
<html endspaces=" ">
&lt;a name=&quot;CF&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#CF &textndash;&gt;
&lt;a name=&quot;cnv&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnv &textndash;&gt;
&lt;a name=&quot;cnv_CF&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnv_CF &textndash;&gt;
&lt;a name=&quot;cnv_CCSM&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnv_CCSM &textndash;&gt;
</html>
</section>
<node name="CF-Conventions" spaces=" "><nodename>CF Conventions</nodename><nodenext spaces=" ">ARM Conventions</nodenext><nodeprev spaces=" ">File List Attributes</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle><acronym><acronymword>CF</acronymword></acronym> Conventions</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="846"><acronym><acronymword>CF</acronymword></acronym> conventions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="847"><acronym><acronymword>CCSM</acronymword></acronym> conventions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="848">UDUnits</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="849"><code>ORO</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="850"><code>area</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="851"><code>datesec</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="852"><code>date</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="853"><code>gw</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="854"><code>hyai</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="855"><code>hyam</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="856"><code>hybi</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="857"><code>hybm</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="858"><code>lat_bnds</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="859"><code>lon_bnds</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="860"><code>msk_*</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncbo</command>, <command>nces</command>, <command>ncecat</command>,
<command>ncflint</command>, <command>ncpdq</command>, <command>ncra</command>, <command>ncwa</command>&linebreak;
Short options: None&linebreak;
</para></cartouche>
<para><acronym><acronymword>NCO</acronymword></acronym> recognizes some Climate and Forecast (<acronym><acronymword>CF</acronymword></acronym>)
metadata conventions, and applies special rules to such data.
<acronym><acronymword>NCO</acronymword></acronym> was contemporaneous with <acronym><acronymword>COARDS</acronymword></acronym> and still
contains some rules to handle older <acronym><acronymword>NCAR</acronymword></acronym> model datasets, such as 
<acronym><acronymword>CCM</acronymword></acronym> and early <acronym><acronymword>CCSM</acronymword></acronym> datasets, that pre-date 
<acronym><acronymword>CF</acronymword></acronym>.
Such datasets may not contain an explicit <code>Conventions</code> attribute
(e.g., <samp>CF-1.0</samp>). 
Nevertheless, we refer to all such metadata collectively as <acronym><acronymword>CF</acronymword></acronym> 
metadata. 
Skip this section if you never work with <acronym><acronymword>CF</acronymword></acronym> metadata.
</para>
<para>The <acronym><acronymword>CF</acronymword></acronym> netCDF conventions are described 
<uref><urefurl>http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#coordinate-system</urefurl><urefdesc spaces=" ">here</urefdesc></uref>. 
Most <acronym><acronymword>CF</acronymword></acronym> netCDF conventions are transparent to <acronym><acronymword>NCO</acronymword></acronym>  
<footnote spaces="\n"><para>The exception is appending/altering the attributes <code>x_op</code>,
<code>y_op</code>, <code>z_op</code>, and <code>t_op</code> for variables which have been
averaged across space and time dimensions.
This feature is scheduled for future inclusion in <acronym><acronymword>NCO</acronymword></acronym>.
</para></footnote>.
There are no known pitfalls associated with using any <acronym><acronymword>NCO</acronymword></acronym>
operator on files adhering to these conventions
<footnote spaces="\n"><para>The <acronym><acronymword>CF</acronymword></acronym> conventions recommend <code>time</code> be stored in the
format <var>time</var> since <var>base_time</var>, e.g., the <code>units</code>
attribute of <code>time</code> might be 
<samp>days since 1992-10-8 15:15:42.5 -6:00</samp>. 
<w>A problem</w> with this format occurs when using <command>ncrcat</command> to
concatenate multiple files, each with a different <var>base_time</var>. 
That is, any <code>time</code> values from files following the first file to
be concatenated should be corrected to the <var>base_time</var> offset
specified in the <code>units</code> attribute of <code>time</code> from the first
file. 
The analogous problem has been fixed in <acronym><acronymword>ARM</acronymword></acronym> files 
(<pxref label="ARM-Conventions"><xrefnodename>ARM Conventions</xrefnodename></pxref>) and could be fixed for <acronym><acronymword>CF</acronymword></acronym> files if
there is sufficient lobbying.
</para></footnote>.
However, to facilitate maximum user friendliness, <acronym><acronymword>NCO</acronymword></acronym> applies 
special rules to certain variables in <acronym><acronymword>CF</acronymword></acronym> files.
The special functions are not required by the <acronym><acronymword>CF</acronymword></acronym> netCDF
conventions, yet experience shows that they simplify data analysis.
</para>
<html endspaces=" ">
&lt;a name=&quot;prc_xcp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#prc_xcp &textndash;&gt;
</html>
<para>Currently, <acronym><acronymword>NCO</acronymword></acronym> determines whether a datafile is a
<acronym><acronymword>CF</acronymword></acronym> output datafile simply by checking (case-insensitively)
whether the value of the global attribute <code>Conventions</code> (if any)
equals <samp>CF-1.0</samp> or <samp>NCAR-CSM</samp> 
Should <code>Conventions</code> equal either of these in the (first)
<var>input-file</var>, <acronym><acronymword>NCO</acronymword></acronym> will apply special rules to certain
variables because of their usual meaning in <acronym><acronymword>CF</acronymword></acronym> files. 
<acronym><acronymword>NCO</acronymword></acronym> will not average the following variables often found in
<acronym><acronymword>CF</acronymword></acronym> files: 
<code>ntrm</code>, <code>ntrn</code>, <code>ntrk</code>, <code>ndbase</code>, <code>nsbase</code>,
<code>nbdate</code>, <code>nbsec</code>, <code>mdt</code>, <code>mhisf</code>.
These variables contain scalar metadata such as the resolution of the
host geophysical model and it makes no sense to change their values.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="861">non-coordinate grid properties</indexterm></cindex>
<para>Furthermore, the <dfn>size and rank-preserving arithmetic operators</dfn> try
not to operate on certain grid properties.
These operators are <command>ncap2</command>, <command>ncbo</command>, <command>nces</command>,
<command>ncflint</command>, and <command>ncpdq</command> (when used for packing, not for
permutation).  
These operators do not operate, by default, on (i.e., add, subtract,
pack, etc.) the following variables:   
<code>ORO</code>, 
<code>area</code>, 
<code>datesec</code>, 
<code>date</code>, 
<code>gw</code>, 
<code>hyai</code>, 
<code>hyam</code>,
<code>hybi</code>. 
<code>hybm</code>, 
<code>lat_bnds</code>, 
<code>lon_bnds</code>,
<code>msk_*</code>.
These variables represent the Gaussian weights, the orography field,
time fields, hybrid pressure coefficients, and latititude/longitude
boundaries.
We call these fields non-coordinate <dfn>grid properties</dfn>.
Coordinate grid properties are easy to identify because they are 
coordinate variables such as <code>latitude</code> and <code>longitude</code>.
</para>
<para>Users usually want <emph>all</emph> grid properties to remain unaltered in the
output file. 
To be treated as a grid property, the variable name must <emph>exactly</emph>
match a name in the above list, or be a coordinate variable. 
The handling of <code>msk_*</code> is exceptional in that <emph>any</emph> variable 
name beginning with the string <code>msk_</code> is considered to be a
&textldquo;mask&textrdquo; and is thus preserved (not operated on arithmetically).
</para>
<para>You must spoof <acronym><acronymword>NCO</acronymword></acronym> if you would like any grid properties
or other special <acronym><acronymword>CF</acronymword></acronym> fields processed normally.
For example rename the variables first with <command>ncrename</command>, 
or alter the <code>Conventions</code> attribute.
</para>
<html endspaces=" ">
&lt;a name=&quot;cnv_CF_bnd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnv_CF_bnd &textndash;&gt;
&lt;a name=&quot;bnd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#bnd &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="862"><code>bounds</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="863">bounds convention</indexterm></cindex>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.0.8 (April, 2011), <acronym><acronymword>NCO</acronymword></acronym> 
supports the <acronym><acronymword>CF</acronymword></acronym> <code>bounds</code> convention for cell boundaries
described 
<uref><urefurl>http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#cell-boundaries</urefurl><urefdesc spaces=" ">here</urefdesc></uref>.
This convention allows coordinate variables (including multidimensional
coordinates) to describe the boundaries of their cells.
This is done by naming the variable which contains the bounds in
in the <code>bounds</code> attribute. 
Note that coordinates of rank <math>N</math> have bounds of rank <math>N+1</math>.
NCO-generated subsets of <acronym><acronymword>CF</acronymword></acronym>-compliant files with <code>bounds</code>
attributes will include the coordinates specified by the <code>bounds</code>
attribute, if any.  
Hence the subsets will themselves be <acronym><acronymword>CF</acronymword></acronym>-compliant.
</para>
<html endspaces=" ">
&lt;a name=&quot;cnv_CF_crd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnv_CF_crd &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="864"><code>coordinates</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="865">coordinates convention</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="866">coordinate variable</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="867">auxiliary coordinates</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="868">subsetting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="869"><code>-C</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="870"><code>-c</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="871"><code>--no-coords</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="872"><code>--no-crd</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="873"><code>--coords</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="874"><code>--crd</code></indexterm></cindex>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 3.9.6 (January, 2009), <acronym><acronymword>NCO</acronymword></acronym>
supports the <acronym><acronymword>CF</acronymword></acronym> <code>coordinates</code> convention described 
<uref><urefurl>http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#coordinate-system</urefurl><urefdesc spaces=" ">here</urefdesc></uref>. 
This convention allows variables to specify additional coordinates
(including multidimensional coordinates) in a space-separated string
attribute named <code>coordinates</code>. 
NCO attaches any such coordinates to the extraction list along with
variable and its usual (one-dimensional) coordinates, if any.
These auxiliary coordinates are subject to the user-specified overrides
described in <ref label="Subsetting-Coordinate-Variables"><xrefnodename>Subsetting Coordinate Variables</xrefnodename></ref>.
</para>
<html endspaces=" ">
&lt;a name=&quot;cnv_CF_cll_mth&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnv_CF_cll_mth &textndash;&gt;
&lt;a name=&quot;cll_mth&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cll_mth &textndash;&gt;
&lt;a name=&quot;no_cll_mth&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#no_cll_mth &textndash;&gt;
&lt;a name=&quot;cell_methods&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cell_methods &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="875"><code>cell_methods</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="876"><code>--cll_mth</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="877"><code>--no_cll_mth</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="878"><code>--cell_methods</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="879"><code>--no_cell_methods</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="880">cell methods convention</indexterm></cindex>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.4.2 (February, 2014), <acronym><acronymword>NCO</acronymword></acronym> 
supports some of the <acronym><acronymword>CF</acronymword></acronym> <code>cell_methods</code> 
<uref><urefurl>http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.7-draft1/cf-conventions.html#cell-methods</urefurl><urefdesc spaces=" ">convention</urefdesc></uref>
to describe the analysis procedures that have been applied to data.
The convention creates (or appends to an existing) <code>cell_methods</code>
attribute a space-separated list of couplets of the form <var>dmn: op</var>
where <var>dmn</var> is a comma-separated list of dimensions previously
contained in the variable that have been reduced by the arithmetic
operation <var>op</var>.
For example, the <code>cell_methods</code> value <code>time: mean</code> says that
the variable in question was averaged over the <code>time</code> dimension.
In such cases <code>time</code> will either be a scalar variable or a
degenerate dimension or coordinate. 
This simply means that it has been averaged-over.
The value <code>time, lon: mean lat: max</code> says that the variable in
question is the maximum zonal mean of the time averaged original
variable.
Which is to say that the variable was first averaged over time and
longitude, and then the residual latitudinal array was reduced by
choosing the maximum value.
Since the <code>cell methods</code> convention may alter metadata in an
undesirable (or possibly incorrect) fashion, we provide switches
to ensure it is always or never used.
Use long-options <samp>--cll_mth</samp> or <samp>--cell_methods</samp> to invoke the
algorithm (true by default), and options <samp>--no_cll_mth</samp> or 
<samp>--no_cell_methods</samp> to turn it off.
These options are only available in the operators <command>ncwa</command> and
<command>ncra</command>.
</para>
<html endspaces=" ">
&lt;a name=&quot;cnv_ARM&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnv_ARM &textndash;&gt;
</html>
</section>
<node name="ARM-Conventions" spaces=" "><nodename>ARM Conventions</nodename><nodenext spaces=" ">Operator Version</nodenext><nodeprev spaces=" ">CF Conventions</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle><acronym><acronymword>ARM</acronymword></acronym> Conventions</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="881"><acronym><acronymword>ARM</acronymword></acronym> conventions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="882"><code>time_offset</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="883"><code>base_time</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="884"><code>time</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: <command>ncrcat</command>&linebreak;
Short options: None&linebreak;
</para></cartouche>
<para><command>ncrcat</command> has been programmed to correctly handle data files
which utilize the Atmospheric Radiation Measurement (<acronym><acronymword>ARM</acronymword></acronym>)
Program <uref><urefurl>http://www.arm.gov/data/time.stm</urefurl><urefdesc>convention</urefdesc></uref> for
time and time offsets.
If you do not work with <acronym><acronymword>ARM</acronymword></acronym> data then you may skip this
section. 
<acronym><acronymword>ARM</acronymword></acronym> data files store time information in two variables, a
scalar, <code>base_time</code>, and a record variable, <code>time_offset</code>.
Subtle but serious problems can arise when these type of files are
just blindly concatenated.
Therefore <command>ncrcat</command> has been specially programmed to be able to
chain together consecutive <acronym><acronymword>ARM</acronymword></acronym> <var>input-files</var> and produce
and an <var>output-file</var> which contains the correct time information.
Currently, <command>ncrcat</command> determines whether a datafile is an
<acronym><acronymword>ARM</acronymword></acronym> datafile simply by testing for the existence of the
variables <code>base_time</code>, <code>time_offset</code>, and the dimension
<code>time</code>.  
If these are found in the <var>input-file</var> then <command>ncrcat</command> will 
automatically perform two non-standard, but hopefully useful,
procedures. 
First, <command>ncrcat</command> will ensure that values of <code>time_offset</code>
appearing in the <var>output-file</var> are relative to the <code>base_time</code>
appearing in the first <var>input-file</var> (and presumably, though not
necessarily, also appearing in the <var>output-file</var>).
Second, if a coordinate variable named <code>time</code> is not found in the
<var>input-files</var>, then <command>ncrcat</command> automatically creates the
<code>time</code> coordinate in the <var>output-file</var>.  
The values of <code>time</code> are defined by the <acronym><acronymword>ARM</acronymword></acronym> conventions 
<math><var>time</var> = <var>base_time</var> + <var>time_offset</var></math>.
Thus, if <var>output-file</var> contains the <code>time_offset</code>
variable, it will also contain the <code>time</code> coordinate.
<cindex index="cp" spaces=" "><indexterm index="cp" number="885"><code>history</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="886">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="887">attributes, global</indexterm></cindex>
<w>A short</w> message is added to the <code>history</code> global attribute
whenever these <acronym><acronymword>ARM</acronymword></acronym>-specific procedures are executed.
</para>
<html endspaces=" ">
&lt;a name=&quot;vrs&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#vrs &textndash;&gt;
&lt;a name=&quot;version&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#version &textndash;&gt;
</html>
</section>
<node name="Operator-Version" spaces=" "><nodename>Operator Version</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">ARM Conventions</nodeprev><nodeup spaces=" ">Common features</nodeup></node>
<section spaces=" "><sectiontitle>Operator Version</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="888">version</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="889"><acronym><acronymword>RCS</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="890"><code>-r</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="891"><code>--revision</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="892"><code>--version</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="893"><code>--vrs</code></indexterm></cindex>
<cartouche endspaces=" ">
<para>Availability: All operators&linebreak;
Short options: <samp>-r</samp>&linebreak;
Long options: <samp>--revision</samp>, <samp>--version</samp>, or <samp>--vrs</samp>&linebreak;
</para></cartouche>
<para>All operators can be told to print their version information,
library version, copyright notice, and compile-time configuration 
with the <samp>-r</samp> switch, or its long-option equivalent
<samp>revision</samp>. 
The <samp>--version</samp> or <samp>--vrs</samp> switches print the operator
version information only.
The internal version number varies between operators, and indicates the 
most recent change to a particular operator&textrsquo;s source code.
This is useful in making sure you are working with the most recent
operators.
The version of <acronym><acronymword>NCO</acronymword></acronym> you are using might be, e.g., <code>3.9.5</code>.
Using <samp>-r</samp> on, say, <command>ncks</command>, produces something like 
<samp>NCO netCDF Operators version &quot;3.9.5&quot; last modified 2008/05/11 built May 12 2008 on neige by zender 
Copyright (C) 1995--2008 Charlie Zender
ncks version 20090918</samp>.
This tells you that <command>ncks</command> contains all patches up to version 
<code>3.9.5</code>, which dates from <w>May 11</w>, 2008.
<html endspaces=" ">
&lt;a name=&quot;rfr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rfr &textndash;&gt;
</html>
</para></section>
</chapter>
<node name="Operator-Reference-Manual" spaces=" "><nodename>Operator Reference Manual</nodename><nodenext spaces=" ">Contributing</nodenext><nodeprev spaces=" ">Common features</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<chapter spaces=" "><sectiontitle>Operator Reference Manual</sectiontitle>

<para>This chapter presents reference pages for each of the operators
individually. 
The operators are presented in alphabetical order.
<cindex index="cp" spaces=" "><indexterm index="cp" number="894">command line switches</indexterm></cindex>
All valid command line switches are included in the syntax statement.
Recall that descriptions of many of these command line switches are
provided only in <ref label="Common-features"><xrefnodename>Common features</xrefnodename></ref>, to avoid redundancy.
Only options specific to, or most useful with, a particular operator are 
described in any detail in the sections below.  
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">ncap2 netCDF Arithmetic Processor</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ncatted netCDF Attribute Editor</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ncbo netCDF Binary Operator</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">nces netCDF Ensemble Statistics</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ncecat netCDF Ensemble Concatenator</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ncflint netCDF File Interpolator</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ncks netCDF Kitchen Sink</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ncpdq netCDF Permute Dimensions Quickly</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ncra netCDF Record Averager</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ncrcat netCDF Record Concatenator</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ncrename netCDF Renamer</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ncwa netCDF Weighted Averager</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncap&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap &textndash;&gt;
&lt;a name=&quot;ncap2&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap2 &textndash;&gt;
</html>
<node name="ncap2-netCDF-Arithmetic-Processor" spaces=" "><nodename>ncap2 netCDF Arithmetic Processor</nodename><nodenext spaces=" ">ncatted netCDF Attribute Editor</nodenext><nodeprev spaces=" ">Operator Reference Manual</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncap2</command> netCDF Arithmetic Processor</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="895">parser</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="896">lexer</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="897">arithmetic processor</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="14" mergedindex="cp">ncap</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="15" mergedindex="cp">ncap2</indexterm></findex>

<cartouche endspaces=" ">
<para><command>ncap2</command> understands a relatively full-featured 
language of operations, including loops, conditionals, arrays,
and math functions.
<command>ncap2</command> is the most rapidly changing <acronym><acronymword>NCO</acronymword></acronym> operator and
its documentation is incomplete.
The distribution file <file>data/ncap2_tst.nco</file> contains an up-to-date
overview of its syntax and capabilities. 
The <file>data/*.nco</file> distribution files (especially
<file>bin_cnt.nco</file>, <file>psd_wrf.nco</file>, and <file>rgr.nco</file>) contain
in-depth examples of <command>ncap2</command> solutions to complex problems.
</para></cartouche>

<!-- c fxm: TODO nco549 hyper-link all switches to explanatory sections? -->
<!-- c Problem is that only works well in HTML mode -->
<!-- c TeXInfo has no native mode for concise hyperlinks in text mode -->
<!-- c Currently in TeX/PDF mode, TeXInfo opens browser to find link, -->
<!-- c rather than jumping to internal link within document -->
<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 [-3] [-4] [-6] [-7] [<uref><urefurl>http://nco.sf.net/nco.html#-A</urefurl><urefreplacement>-A</urefreplacement></uref>] [-C] [-c] 
[-D <var>dbg</var>] [-F] [-f] [-h] [--hdf] [--hdr_pad <var>nbr</var>] [-L <var>dfl_lvl</var>] [-l <var>path</var>]
[--no_tmp_fl] [-O] [-o <var>output-file</var>] [-p <var>path</var>] [-R] [-r] [--ram_all]
[-s <var>algebra</var>] [-S <var>fl.nco</var>] [-t <var>thr_nbr</var>] [-v]
<var>input-file</var> [<var>output-file</var>]  
</pre></example>
 
<noindent></noindent>
<para>DESCRIPTION
</para>
<para><command>ncap2</command> arithmetically processes netCDF files
<footnote><para><command>ncap2</command> is the successor to <command>ncap</command> which was
put into maintenance mode in November, 2006. 
This documentation refers to <command>ncap2</command>, which has a superset of
the <command>ncap</command> functionality. 
Eventually <command>ncap</command> will be deprecated in favor <command>ncap2</command>.
<command>ncap2</command> may be renamed <command>ncap</command> in 2013.</para></footnote>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="898">script file</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="899"><code>--script-file</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="900"><code>--fl_spt</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="901"><code>--script</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="902"><code>--spt</code></indexterm></cindex>
The processing instructions are contained either in the <acronym><acronymword>NCO</acronymword></acronym>
script file <file>fl.nco</file> or in a sequence of command line arguments.
The options <samp>-s</samp> (or long options <samp>--spt</samp> or <samp>--script</samp>)
are used for in-line scripts and <samp>-S</samp> (or long options
<samp>--fl_spt</samp> or <samp>--script-file</samp>) are used to provide the
filename where (usually multiple) scripting commands are pre-stored.   
<command>ncap2</command> was written to perform arbitrary algebraic
transformations of data and archive the results as easily as possible.
<cindex index="cp" spaces=" "><indexterm index="cp" number="903">derived fields</indexterm></cindex>
<xref label="Missing-Values"><xrefnodename>Missing Values</xrefnodename></xref>, for treatment of missing values.
The results of the algebraic manipulations are called 
<dfn>derived fields</dfn>. 
</para>
<para>Unlike the other operators, <command>ncap2</command> does not accept a list of
variables to be operated on as an argument to <samp>-v</samp> 
(<pxref label="Subsetting-Files"><xrefnodename>Subsetting Files</xrefnodename></pxref>).
Rather, the <samp>-v</samp> switch takes no arguments and indicates
that <command>ncap2</command> should output <emph>only</emph> user-defined variables.
<command>ncap2</command> neither accepts nor understands the <var>-x</var> switch.
<cindex index="cp" spaces=" "><indexterm index="cp" number="904">appending variables</indexterm></cindex>
NB: As of 20120515, <command>ncap2</command> is unable to append to files that
already contain the appended dimensions. 
</para>
<!-- c @subsection Scripting Mathematical Processing with @command{ncap2} -->

<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Syntax of ncap2 statements</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Expressions</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Dimensions</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Left hand casting</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Arrays and hyperslabs</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Attributes</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Number literals</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">if statement</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">print statement</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Missing values ncap2</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Methods and functions</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">RAM variables</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Where statement</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Loops</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Include files</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Sort methods</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Irregular grids</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Bilinear interpolation</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">GSL special functions</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">GSL interpolation</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">GSL least-squares fitting</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">GSL statistics</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">GSL random number generation</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Examples ncap2</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Intrinsic mathematical methods</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Operator precedence and associativity </menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">ID Quoting</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<html endspaces=" ">
&lt;a name=&quot;att_prp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#att_prp &textndash;&gt;
</html>
<para>Defining new variables in terms of existing variables is a powerful
feature of <command>ncap2</command>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="905">derived fields</indexterm></cindex>
Derived fields inherit the metadata (i.e., attributes) of their
ancestors, if any, in the script or input file. 
When the derived field is completely new (no identically-named ancestors
exist), then it inherits the metadata (if any) of the left-most variable
on the right hand side of the defining expression.
This metadata inheritance is called <dfn>attribute propagation</dfn>.
Attribute propagation is intended to facilitate well-documented 
data analysis, and we welcome suggestions to improve this feature.
</para>
<para>The only exception to this rule of attribute propagation is in cases of
left hand casting (<pxref label="Left-hand-casting"><xrefnodename>Left hand casting</xrefnodename></pxref>).
The user must manually define the proper metadata for variables defined
using left hand casting. 
</para>
<html endspaces=" ">
&lt;a name=&quot;syn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#syn &textndash;&gt;
</html>
<node name="Syntax-of-ncap2-statements" spaces=" "><nodename>Syntax of ncap2 statements</nodename><nodenext spaces=" ">Expressions</nodenext><nodeprev spaces=" ">ncap2 netCDF Arithmetic Processor</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Syntax of <command>ncap2</command> statements</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="906">statement</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="907">syntax</indexterm></cindex>
<para>Mastering <command>ncap2</command> is relatively simple.
Each valid statement <var>statement</var> consists of standard forward
algebraic expression. 
The <file>fl.nco</file>, if present, is simply a list of such statements,
whitespace, and comments.
<cindex index="cp" spaces=" "><indexterm index="cp" number="908">C language</indexterm></cindex>
The syntax of statements is most like the computer <w>language C</w>.
The following characteristics <w>of C</w> are preserved:
</para><table commandarg="asis" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="asis">Array syntax</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="909">array syntax</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="910"><code>[]</code> (array delimiters)</indexterm></cindex>
<para>Arrays elements are placed within <code>[]</code> characters;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Array indexing</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="911">array indexing</indexterm></cindex>
<para>Arrays are 0-based;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Array storage</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="912">array storage</indexterm></cindex>
<para>Last dimension is most rapidly varying;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Assignment statements</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="913">assignment statement</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="914">semi-colon</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="915"><code>;</code> (end of statement)</indexterm></cindex>
<para><w>A semi</w>-colon <samp>;</samp> indicates the end of an assignment statement.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Comments</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="916">comments</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="917"><code>/*...*/</code> (comment)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="918"><code>//</code> (comment)</indexterm></cindex>
<para>Multi-line comments are enclosed within <code>/* */</code> characters.
Single line comments are preceded by <code>//</code> characters.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Nesting</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="919">including files</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="920">nesting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="921"><code>#include</code></indexterm></cindex>
<para>Files may be nested in scripts using <code>#include <var>script</var></code>.
Note that the <code>#include</code> command is not followed by a semi-colon
because it is a pre-processor directive, not an assignment statement.
The filename <file>script</file> is interpreted relative to the run directory.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Attribute syntax</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="922">attribute syntax</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="923"><code>&arobase;</code> (attribute)</indexterm></cindex>
<para>The at-sign <code>&arobase;</code> is used to delineate an attribute name from a
variable name.
</para></tableitem></tableentry></table> 

<html endspaces=" ">
&lt;a name=&quot;ncap_xpr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_xpr &textndash;&gt;
</html>
</subsection>
<node name="Expressions" spaces=" "><nodename>Expressions</nodename><nodenext spaces=" ">Dimensions</nodenext><nodeprev spaces=" ">Syntax of ncap2 statements</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<cindex index="cp" spaces=" "><indexterm index="cp" number="924">expressions</indexterm></cindex>
<subsection spaces=" "><sectiontitle>Expressions</sectiontitle>
<para>Expressions are the fundamental building block of <command>ncap2</command>. 
Expressions are composed of variables, numbers, literals, and
attributes. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="925">C language</indexterm></cindex>
The following <w>C operators</w> are &textldquo;overloaded&textrdquo; and work with scalars 
and multi-dimensional arrays:
</para><example endspaces=" ">
<pre xml:space="preserve">Arithmetic Operators: * / % + - ^
Binary Operators:     &gt; &gt;= &lt; &lt;= == != == || &amp;&amp; &gt;&gt; &lt;&lt;
Unary Operators:      + - ++ -- !
Conditional Operator: exp1 ? exp2 : exp3
Assign Operators:     = += -= /= *=
</pre></example>

<para>In the following section a <dfn>variable</dfn> also refers to a 
number literal which is read in as a scalar variable:
</para>
<para><strong>Arithmetic and Binary Operators </strong>
</para>
<para>Consider <emph>var1 &textrsquo;op&textrsquo; var2</emph>
</para>
<para><strong>Precision</strong>
</para><itemize commandarg="bullet" spaces=" " endspaces=" "><itemprepend><formattingcommand command="bullet"/></itemprepend>
<listitem spaces=" "><prepend>&bullet;</prepend><para>When both operands are variables, the result has the precision of the higher precision operand.
</para></listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>When one operand is a variable and the other an attribute, the result has the precision of the variable. 
</para></listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>When both operands are attributes, the result has the precision of the more precise attribute.
</para></listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>The exponentiation operator &textldquo;^&textrdquo; is an exception to the above rules. 
When both operands have type less than <code>NC_FLOAT</code>, the result is <code>NC_FLOAT</code>. 
When either type is <code>NC_DOUBLE</code>, the result is also <code>NC_DOUBLE</code>. 
</para></listitem></itemize>
<!-- c csz got to here editing -->

<cindex index="cp" spaces=" "><indexterm index="cp" number="926">broadcasting variables</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="927">rank</indexterm></cindex>
<para><strong>Rank</strong>
</para><itemize commandarg="bullet" spaces=" " endspaces=" "><itemprepend><formattingcommand command="bullet"/> </itemprepend>
<listitem spaces=" "><prepend>&bullet;</prepend><para>The Rank of the result is generally equal to Rank of the operand
that has the greatest number of dimensions.  
</para></listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>If the dimensions in var2 are a subset of the dimensions in var1
then its possible to  make var2 conform to var1 through broadcasting and
or dimension reordering.  
</para></listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>Broadcasting a variable means creating data in non-existing
dimensions by copying data in existing dimensions. 
</para></listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>More specifically: If the numbers of dimensions in var1 is greater 
than or equal to the number of dimensions in var2 then an attempt is
made to make var2 conform to var1 ,else var1 is made to conform to
var2. If conformance  is not possible then an error message will be
emitted and script execution will cease.&linebreak; 
</para></listitem></itemize>

<noindent></noindent> <para>Even though the logical operators return True(1) or False(0)
they are treated in the same way as the arithmetic operators with regard
to precision and rank.&linebreak; 
Examples:
</para>
<example endspaces=" ">
<pre xml:space="preserve">dimensions: time=10, lat=2, lon=4
Suppose we have the two variables:

double  P(time,lat,lon);
float   PZ0(lon,lat);  // PZ0=1,2,3,4,5,6,7,8;

Consider now the expression:
 PZ=P-PZ0

PZ0 is made to conform to P and the result is
PZ0 =
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,

Once the expression is evaluated then PZ will be of type double;

Consider now 
 start=four-att_var&arobase;double_att;  // start =-69  and is of type intger;
 four_pow=four^3.0f               // four_pow=64 and is of type float  
 three_nw=three_dmn_var_sht*1.0f; // type is now float
 start&arobase;n1=att_var&arobase;short_att*att_var&arobase;int_att; 
                                  // start&arobase;n1=5329 and is type int 
</pre></example>

<noindent></noindent> <para><strong>Binary Operators</strong> &linebreak; 
<cindex index="cp" spaces=" "><indexterm index="cp" number="928">binary Operators</indexterm></cindex>
Unlike C the binary operators return an array of values. 
There is no such thing as short circuiting with the AND/OR operators. 
Missing values are carried into the result in the same way they are with
the arithmetic operators. 
When an expression is evaluated in an if() the missing values are
treated as true.&linebreak;  
The binary operators are, in order of precedence: 
</para><example endspaces=" ">
<pre xml:space="preserve">	
!   Logical Not
----------------------------
&lt;&lt;  Less Than Selection
&gt;&gt;  Greater Than Selection
----------------------------
&gt;   Greater than
&gt;=  Greater than or equal to
&lt;   Less than
&lt;=  Less than or equal to
----------------------------
==  Equal to
!=  Not equal to
----------------------------
&amp;&amp;  Logical AND
----------------------------
||  Logical OR
----------------------------
</pre></example>

<para>To see all operators: <pxref label="Operator-precedence-and-associativity"><xrefnodename>Operator precedence and associativity</xrefnodename></pxref> 
Examples:
</para><example endspaces=" ">
<pre xml:space="preserve">tm1=time&gt;2 &amp;&amp; time &lt;7;  // tm1=0, 0, 1, 1, 1, 1, 0, 0, 0, 0 double
tm2=time==3 || time&gt;=6; // tm2=0, 0, 1, 0, 0, 1, 1, 1, 1, 1 double
tm3=int(!tm1);          // tm3=1, 1, 0, 0, 0, 0, 1, 1, 1, 1 int
tm4=tm1 &amp;&amp; tm2;         // tm4=0, 0, 1, 0, 0, 1, 0, 0, 0, 0 double
tm5=!tm4;               // tm5=1, 1, 0, 1, 1, 0, 1, 1, 1, 1 double
</pre></example>
	
<noindent></noindent> <para><strong>Regular Assign Operator</strong>&linebreak;
<noindent></noindent> <emph>var1 &textrsquo;=&textrsquo; exp1</emph> &linebreak;
If var1 does not already exist in Output then var1 is written to Output with the values and dimensions from expr1. If var1 already exists in Output, then the only requirement on expr1 is that the number of elements must match the number already on disk. The type of expr1 is converted if necessary to the disk type.
</para>
<noindent></noindent> <para><strong> Other Assign Operators +=,-=,*=./= </strong>&linebreak;
        <noindent></noindent>  <emph>var1 &textrsquo;ass_op&textrsquo; exp1 </emph>&linebreak;
if exp1 is a variable and it doesn&textrsquo;t conform to var1 then an attempt is made to make it conform to var1. If exp1 is an attribute it must have unity size or else have the same number of elements as var1. If expr1 has a different type to var1 the it is converted to the var1 type.
</para><example endspaces=" ">
<pre xml:space="preserve">z1=four+=one*=10 // z1=14 four=14 one=10;	
time-=2          // time= -1,0,1,2,3,4,5,6,7,8
</pre></example>

<noindent></noindent> <para><strong>Increment/ Decrement Operators &linebreak; </strong> 
These work in a similar fashion to their regular C counterparts. If say the variable &quot;four&quot; is input only then the statement &quot;++four&quot; effectively means -read four from input increment each element by one , then write the new values to Output;
</para>
<para>Example:
</para><example endspaces=" ">
<pre xml:space="preserve">n2=++four;   n2=5, four=5 
n3=one--+20; n3=21  one=0;	 
n4=--time;   n4=time=0.,1.,2.,3.,4.,5.,6.,7.,8.,9.;
</pre></example>

<noindent></noindent> <para><strong>Conditional Operator ?: </strong> &linebreak;
<cindex index="cp" spaces=" "><indexterm index="cp" number="929">conditional Operator</indexterm></cindex>
<noindent></noindent> <emph>exp1 ? exp2 : exp3 </emph> &linebreak;
The conditional operator (or ternary Operator) is a succinct way
of writing an if/then/else. If exp1 evaluates to true then exp2 is
returned else exp3 is returned. 
</para>
<para>Example:
</para><example endspaces=" ">
<pre xml:space="preserve">weight_avg=weight.avg();
weight_avg&arobase;units= (weight_avg == 1 ? &quot;kilo&quot; : &quot;kilos&quot;);  
PS_nw=PS-(PS.min() &gt; 100000 ? 100000 : 0);
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;clp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#clp &textndash;&gt;
&lt;a name=&quot;clipping&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#clipping &textndash;&gt;
</html>
<noindent></noindent> <para><strong>Clipping Operators</strong>
<cindex index="cp" spaces=" "><indexterm index="cp" number="930">clipping operators</indexterm></cindex>
</para><table commandarg="asis" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="asis">&lt;&lt; Less-than Clipping&linebreak;</itemformat></item>
</tableterm><tableitem><para>For arrays, the less-than selection operator selects all values in the
left operand that are less than the corresponding value in the right
operand. 
If the value of the left side is greater than or equal to the
corresponding value of the right side, then the right side value is 
placed in the result	 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">&gt;&gt; Greater-than Clipping&linebreak;</itemformat></item>
</tableterm><tableitem><para>For arrays, the greater-than selection operator selects all values in
the left operand that are greater than the corresponding value in the
right operand. 
If the value of the left side is less than or equal to the corresponding
value of the right side, then the right side value is placed in the
result.  
</para></tableitem></tableentry></table>

<para>Example:
</para><example endspaces=" ">
<pre xml:space="preserve">RDM2=RDM &gt;&gt; 100.0 // 100,100,100,100,126,126,100,100,100,100 double
RDM2=RDM &lt;&lt;  90s  // 1, 9, 36, 84, 90, 90, 84, 36, 9, 1 int
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_dims&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_dims &textndash;&gt;
&lt;a name=&quot;defdim&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#defdim &textndash;&gt;
</html>
</subsection>
<node name="Dimensions" spaces=" "><nodename>Dimensions</nodename><nodenext spaces=" ">Left hand casting</nodenext><nodeprev spaces=" ">Expressions</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Dimensions</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="931">defining dimensions in <command>ncap2</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="932"><code>defdim()</code></indexterm></cindex>
<para>Dimensions are defined in Output using the <code>defdim()</code> function.
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;cnt&quot;,10);
</pre></example>

<para>This dimension name must then be prefixed with a dollar-sign <samp>$</samp>
when referred to in method arguments or left-hand-casting, e.g.,
</para><example endspaces=" ">
<pre xml:space="preserve">new_var[$cnt]=time;
temperature[$time,$lat,$lon]=35.5;
temp_avg=temperature.avg($time);
</pre></example>

<para>The <code>size</code> method allows the dimension size to be used in an
arithmetic expression:
</para><example endspaces=" ">
<pre xml:space="preserve">time_avg=time.total() / $time.size;
</pre></example>

<para>Increase the size of a new variable by one and set new member to zero:
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;cnt_new&quot;,$cnt.size+1);
new_var[$cnt_new]=0.0;
new_var(0:($cnt_new.size-2))=old_var;
</pre></example>

<noindent></noindent> <para><strong>Dimension Abbreviations &linebreak;</strong>
It is possible to use dimension abbreviations as method arguments:&linebreak;
<code>$0</code> is the first dimension of a variable&linebreak;
<code>$1</code> is the second dimension of a variable&linebreak;
<code>$n</code> is the n+1 dimension of a variable&linebreak;
</para>
<example endspaces=" ">
<pre xml:space="preserve">float four_dmn_rec_var(time,lat,lev,lon);
double three_dmn_var_dbl(time,lat,lon);

four_nw=four_dmn_rev_var.reverse($time,$lon)
four_nw=four_dmn_rec_var.reverse($0,$3);

four_avg=four_dmn_rec_var.avg($lat,$lev);  
four_avg=four_dmn_rec_var.avg($1,$2);  

three_mw=three_dmn_var_dbl.permute($time,$lon,$lat);
three_mw=three_dmn_var_dbl.permute($0,$2,$1);
</pre></example>

<noindent></noindent> <para><strong>ID Quoting &linebreak;</strong>
If the dimension name contains non-regular characters use ID quoting. 
See <pxref label="ID-Quoting"><xrefnodename>ID Quoting</xrefnodename></pxref>
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;a--list.A&quot;,10);
A1['$a--list.A']=30.0;
</pre></example>

<noindent></noindent> <para><strong>GOTCHA &linebreak;</strong>
<noindent></noindent> It is not possible to manually define in Output any dimensions that exist in Input. When a variable from Input appears in an expression or statement its  dimensions in Input are  automagically copied to Output (if they are not already present)
</para>
<html endspaces=" ">
&lt;a name=&quot;lhc&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#lhc &textndash;&gt;
&lt;a name=&quot;lhs&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#lhs &textndash;&gt;
</html>
</subsection>
<node name="Left-hand-casting" spaces=" "><nodename>Left hand casting</nodename><nodenext spaces=" ">Arrays and hyperslabs</nodenext><nodeprev spaces=" ">Dimensions</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Left hand casting</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="933">hybrid coordinate system</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="934">left hand casting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="935"><acronym><acronymword>LHS</acronymword></acronym></indexterm></cindex>
<para>The following examples demonstrate the utility of the 
<dfn>left hand casting</dfn> ability of <command>ncap2</command>.
Consider first this simple, artificial, example.
If <var>lat</var> and <var>lon</var> are one dimensional coordinates of
dimensions <var>lat</var> and <var>lon</var>, respectively, then addition
of these two one-dimensional arrays is intrinsically ill-defined because 
whether <var>lat_lon</var> should be dimensioned <var>lat</var> by <var>lon</var>
or <var>lon</var> by <var>lat</var> is ambiguous (assuming that addition is to
remain a <dfn>commutative</dfn> procedure, i.e., one that does not depend on 
the order of its arguments).
Differing dimensions are said to be <dfn>orthogonal</dfn> to one another,
and sets of dimensions which are mutually exclusive are orthogonal
as a set and any arithmetic operation between variables in orthogonal
dimensional spaces is ambiguous without further information.
</para>
<para>The ambiguity may be resolved by enumerating the desired dimension 
ordering of the output expression inside square brackets on the
left hand side (<acronym><acronymword>LHS</acronymword></acronym>) of the equals sign.
This is called <dfn>left hand casting</dfn> because the user resolves the 
dimensional ordering of the <acronym><acronymword>RHS</acronymword></acronym> of the expression by
specifying the desired ordering on the <acronym><acronymword>LHS</acronymword></acronym>.
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'lat_lon[lat,lon]=lat+lon' in.nc out.nc
ncap2 -s 'lon_lat[lon,lat]=lat+lon' in.nc out.nc
</pre></example>
<para>The explicit list of dimensions on the <acronym><acronymword>LHS</acronymword></acronym>, <code>[lat,lon]</code>
resolves the otherwise ambiguous ordering of dimensions in
<var>lat_lon</var>. 
In effect, the <acronym><acronymword>LHS</acronymword></acronym> <dfn>casts</dfn> its rank properties onto the 
<acronym><acronymword>RHS</acronymword></acronym>.
Without <acronym><acronymword>LHS</acronymword></acronym> casting, the dimensional ordering of <var>lat_lon</var>
would be undefined and, hopefully, <command>ncap2</command> would print an error
message. 
</para>
<para>Consider now a slightly more complex example.
In geophysical models, a coordinate system based on 
a blend of terrain-following and density-following surfaces is 
called a <dfn>hybrid coordinate system</dfn>.
In this coordinate system, four variables must be manipulated to
obtain the pressure of the vertical coordinate:
<var>PO</var> is the domain-mean surface pressure offset (a scalar),
<var>PS</var> is the local (time-varying) surface pressure (usually two
horizontal spatial dimensions, i.e. latitude by longitude), <var>hyam</var>
is the weight given to surfaces of constant density (one spatial
dimension, pressure, which is orthogonal to the horizontal
dimensions), and <var>hybm</var> is the weight given to surfaces of
constant elevation (also one spatial dimension). 
This command constructs a four-dimensional pressure <code>prs_mdp</code>
from the four input variables of mixed rank and orthogonality:
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'prs_mdp[time,lat,lon,lev]=P0*hyam+PS*hybm' in.nc out.nc
</pre></example>
<para>Manipulating the four fields which define the pressure in a hybrid
coordinate system is easy with left hand casting.
</para>
<html endspaces=" ">
&lt;a name=&quot;ncap_arr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_arr &textndash;&gt;
</html>
</subsection>
<node name="Arrays-and-hyperslabs" spaces=" "><nodename>Arrays and hyperslabs</nodename><nodenext spaces=" ">Attributes</nodenext><nodeprev spaces=" ">Left hand casting</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Arrays and hyperslabs</sectiontitle>

<findex index="fn" spaces=" "><indexterm index="fn" number="16" mergedindex="cp">array</indexterm></findex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="936"><code>array</code> function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="937">arrays</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="938">findgen-equivalent</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="939">indgen-equivalent</indexterm></cindex>
<para>Generating a regularly spaced one-dimensional array with <command>ncap2</command>
is simple with the <code>array()</code> function. 
The syntax is 
</para><example endspaces=" ">
<pre xml:space="preserve">var_out=array(val_srt,val_ncr,$dmn_nm); // One-dimensional output
var_out=array(val_srt,val_ncr,var_tpl); // Multi-dimensional output
</pre></example>
<noindent></noindent>
<para>where the arguments are the starting value <var>val_srt</var>, 
incremental value <var>val_ncr</var>, and, for one-dimensional output, the
single dimension <code>$dmn_nm</code>, or, for multi-dimensional output, a
template variable <code>var_tpl</code>, i.e., a variable with the same shape
as the desired output. 
The type of <code>var_out</code> will be the same as <code>val_srt</code>.
Be sure to encode this type with the appropriate decimal point
and floating point suffix when <code>val_srt</code> is a &textldquo;naked constant&textrdquo;
rather than a variable.
For example, to produce an array of shorts (signed two-byte integers), 
integers (signed four-byte integers), unsigned 64-bit integers,
floats, or doubles use
</para><example endspaces=" ">
<pre xml:space="preserve">var_out=array(1s,val_ncr,$dmn_nm); // NC_SHORT array
var_out=array(1,val_ncr,$dmn_nm); // NC_INT array
var_out=array(1ull,val_ncr,$dmn_nm); // NC_UINT64 array
var_out=array(1f,val_ncr,$dmn_nm); // NC_FLOAT array
var_out=array(1.,val_ncr,$dmn_nm); // NC_DOUBLE array
</pre></example>

<para>Once the associated dimensions have been defined, the start and
increment arguments may be supplied as values, mathmatical expressions,
or variables:
</para><example endspaces=" ">
<pre xml:space="preserve">var_out=array(1,1,$time); // 1,2,3,4,5,6,7,8,9,10
var_out=array(1+2-2,one,$time); // 1,2,3,4,5,6,7,8,9,10
var_out=array(1,2,three_dmn_rec_var); // 1,3,5,...155,157,159
</pre></example>

<cindex index="cp" spaces=" "><indexterm index="cp" number="940">hyperslabs</indexterm></cindex>
<para>Hyperslabs in <command>ncap2</command> are more limited than hyperslabs with the
other <acronym><acronymword>NCO</acronymword></acronym> operators. 
<command>ncap2</command> does not understand the shell command-line syntax
used to specify multi-slabs, wrapped co-ordinates, negative stride or
coordinate value limits.
However with a bit of syntactic magic they are all are possible. 
<command>ncap2</command> accepts (in fact, it requires) <var>N</var>-hyperslab
arguments for a variable of rank <var>N</var>:
</para><example endspaces=" ">
<pre xml:space="preserve">var1(arg1,arg2 ... argN);
</pre></example>
<para>where each hyperslab argument is of the form
</para><example endspaces=" ">
<pre xml:space="preserve">start:end:stride 
</pre></example>
<para>and the arguments for different dimensions are separated by commas.
<noindent></noindent> 
If &quot;start&quot; is omitted, it defaults to 0.
If &quot;end&quot; is omitted, it defaults to dimension size minus one.
If &quot;stride&quot; is omitted, it defaults to 1.
</para><sp value="1" line=" 1"></sp>
<noindent></noindent> <para>If a single value is present then it is assumed that that
dimension collapses to a single value (i.e., a cross-section). 
The number of hyperslab arguments MUST equal the variable&textrsquo;s rank.
</para><sp value="1" line=" 1"></sp>

<noindent></noindent> <para><strong>Hyperslabs on the Right Hand Side of an assign&linebreak;</strong>
</para>
<para>A simple 1D example:
</para><example endspaces=" "> 
<pre xml:space="preserve">($time.size=10)
od[$time]=&lbrace;20,22,24,26,28,30,32,34,36,38&rbrace;;

od(7);     // 34
od(7:);    // 34,36,38
od(:7);    // 20,22,24,26,28,30,32,34 
od(::4);   // 20,28,36
od(1:6:2)  // 22,26,30
od(:)      // 20,22,24,26,28,30,32,34,36,38 
</pre></example>

<para>A more complex three dimensional example:
</para><example endspaces=" ">
<pre xml:space="preserve">($lat.size=2,$lon.size=4)
th[$time,$lat,$lon]=      
                          &lbrace;1, 2, 3, 4, 5, 6, 7, 8,
                          9,10,11,12,13,14,15,16,
                          17,18,19,20,21,22,23,24,
                          -99,-99,-99,-99,-99,-99,-99,-99,
                          33,34,35,36,37,38,39,40,
                          41,42,43,44,45,46,47,48,
                          49,50,51,52,53,54,55,56,
                          -99,58,59,60,61,62,63,64,
                          65,66,67,68,69,70,71,72,
                          -99,74,75,76,77,78,79,-99 &rbrace;;

th(1,1,3);        // 16
th(2,0,:);        // 17, 18, 19, 20
th(:,1,3);        // 8, 16, 24, -99, 40, 48, 56, 64, 72, -99 
th(::5,:,0:3:2); // 1, 3, 5, 7, 41, 43, 45, 47
</pre></example>

<para>If hyperslab arguments collapse to a single value (a cross-section has
been specified), then that dimension is removed from the returned
variable. 
If all the values collapse then a scalar variable is returned.
So, for example, the following is valid: 
</para><example endspaces=" ">
<pre xml:space="preserve">th_nw=th(0,:,:)+th(9,:,:); 
// th_nw has dimensions $lon,$lat 
// NB: the time dimension has become degenerate
</pre></example>

<para>The following is invalid:
</para><example endspaces=" ">
<pre xml:space="preserve">th_nw=th(0,:,0:1)+th(9,:,0:1);
</pre></example>
<para>because the <code>$lon</code> dimension now only has two elements.
The above can be calculated by using a LHS cast with 
<code>$lon_nw</code> as replacement dim for <code>$lon</code>: 
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;lon_nw&quot;,2);
th_nw[$lat,$lon_nw]=th(0,:,0:1) +th(9,:,0:1);
</pre></example>

<noindent></noindent> <para><strong>Hyperslabs on the Left Hand Side of an assign&linebreak;</strong>
<noindent></noindent> When hyperslabing on the LHS, the expression on the RHS must 
evaluate to a scalar or a variable/attribute with the same number of 
elements as the LHS hyperslab.
Set all elements of the last record to zero:
</para><example endspaces=" ">
<pre xml:space="preserve">th(9,:,:)=0.0;
</pre></example>
<para>Set first element of each lon element to 1.0:
</para><example endspaces=" ">
<pre xml:space="preserve">th(:,:,0)=1.0;
</pre></example>
<para>One may hyperslab on both sides of an assign.
For example, this sets the last record to the first record:
</para><example endspaces=" ">
<pre xml:space="preserve">th(9,:,:)=th(0,:,:);
</pre></example>
<para>Say <var>th0</var> represents pressure at height=0 and 
<var>th1</var> represents pressure at height=1.
Then it is possible to insert these hyperslabs into the records
</para><example endspaces=" ">
<pre xml:space="preserve">prs[$time,$height,$lat,$lon]=0.0;
prs(:,0,:,:)=th0;
prs(:,1,:,:)=th1
</pre></example>

<noindent></noindent> <para><strong>Reverse method</strong>&linebreak;
<cindex index="cp" spaces=" "><indexterm index="cp" number="941">reverse()</indexterm></cindex>
Use the <code>reverse()</code> method to reverse a dimension&textrsquo;s elements in a
variable with at least one dimension.
This is equivalent to a negative stride, e.g., 
</para><example endspaces=" "> 
<pre xml:space="preserve">th_rv=th(1 ,:,:).reverse($lon); // &lbrace;12,11,10,9 &rbrace;, &lbrace;16,15,14,13&rbrace;
od_rv=od.reverse($time);        // &lbrace;38,36,34,32,30,28,26,24,22,20&rbrace;
</pre></example>

<noindent></noindent> <para><strong>Permute method</strong>p&linebreak;
<cindex index="cp" spaces=" "><indexterm index="cp" number="942">permute()</indexterm></cindex>
Use the <code>permute()</code> method to swap the dimensions of a variable.
The number and names of dimension arguments must match the dimensions in
the variable. 
If the first dimension in the variable is of record type then this must
remain the first dimension. 
If you want to change the record dimension then consider using
<command>ncpdq</command>. 
</para>
<para>Consider the variable:
</para><example endspaces=" ">
<pre xml:space="preserve">float three_dmn_var(lat,lev,lon);
three_dmn_var_prm=three_dmn_var.permute($lon,$lat,$lev);
// The permuted values are
three_dmn_var_prm= 
  0,4,8,
  12,16,20,
  1,5,9,
  13,17,21,
  2,6,10,
  14,18,22,
  3,7,11,
  15,19,23;
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_att&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_att &textndash;&gt;
</html>
</subsection>
<node name="Attributes" spaces=" "><nodename>Attributes</nodename><nodenext spaces=" ">Number literals</nodenext><nodeprev spaces=" ">Arrays and hyperslabs</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Attributes</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="943">attributes<command>ncap2</command></indexterm></cindex>
<noindent></noindent> <para>Attributes are referred to by <emph>var_nm&arobase;att_nm</emph> &linebreak;
All the following are valid statements:
</para><example endspaces=" ">
<pre xml:space="preserve">global&arobase;text=&quot;Test Attributes&quot;; /* Assign a global variable attribute */
a1[$time]=time*20;
a1&arobase;long_name=&quot;Kelvin&quot;;
a1&arobase;min=a1.min();
a1&arobase;max=a1.max();
a1&arobase;min++;
--a1&arobase;max; q
a1(0)=a1&arobase;min;
a1($time.size-1)=a1&arobase;max;
</pre></example>

<para>A <emph>value list</emph> can be used on the RHS of an assign...
<cindex index="cp" spaces=" "><indexterm index="cp" number="944">value list</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">a1&arobase;trip1=&lbrace;1,2,3&rbrace; ;
a1&arobase;triplet=&lbrace;a1&arobase;min,(a1&arobase;min+a1&arobase;max)/2,a1&arobase;max&rbrace;; 
</pre></example>
<para>The netCDF specification allows all attribute types to have a size
greater than one. 
The maximum is defined by <code>NC_MAX_ATTRS</code>.
The following is an <command>ncdump</command> of the metadata for variable <var>a1</var> 
</para><example endspaces=" "> 
<pre xml:space="preserve">double a1(time) ;
  a1:long_name = &quot;Kelvin&quot; ;
  a1:max = 199. ;
  a1:min = 21. ;
  a1:trip1 = 1, 2, 3 ;
  a1:triplet = 21., 110., 199. ;
</pre></example>

<para>The <code>size()</code> method can be used with attributes.
For example, to save an attribute text string in a variable,
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;sng_len&quot;, a1&arobase;long_name.size());
sng_arr[$sng_len]=a1&arobase;long_name; // sng_arr now contains &quot;Kelvin&quot; 
</pre></example>
<para>Attributes defined in a script are stored in memory and are written to Output after script completion.
To stop the attribute being written use the ram_delete() method or use a bogus variable name.
</para>
<noindent></noindent> <para><strong>Attribute Propagation and Inheritance</strong>
<cindex index="cp" spaces=" "><indexterm index="cp" number="945">attribute propagation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="946">attribute inheritance</indexterm></cindex>
</para><itemize commandarg="bullet" spaces=" " endspaces=" "><itemprepend><formattingcommand command="bullet"/></itemprepend>
<beforefirstitem>  </beforefirstitem><listitem spaces=" "><prepend>&bullet;</prepend><para>Attribute propagation occurs in a regular  assign statement. The variable being defined on the LHS gets copies of the attributes from the leftermost variable on the RHS
</para>
  </listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>Attribute Inheritance: The LHS variable &quot;inherits&quot; attributes from an Input variable with the same name
</para>
  </listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>It is possible to have a regular assign statement for which both propagation and inheritance occur.
</para></listitem></itemize>

<example endspaces=" ">
<pre xml:space="preserve">// prs_mdp inherits attributes from P0:
prs_mdp[time,lat,lon,lev]=P0*hyam+hybm*PS;
// th_min inherits attributes from three_dmn_var_dbl:
th_min=1.0 + 2*three_dmn_var_dbl.min($time);
</pre></example>

<para>If the attribute name contains non-regular characters use ID quoting. See <pxref label="ID-Quoting"><xrefnodename>ID Quoting</xrefnodename></pxref>
</para><example endspaces=" ">
<pre xml:space="preserve">'b..m1&arobase;c--lost'=23;
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_num&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_num &textndash;&gt;
</html>
</subsection>
<node name="Number-literals" spaces=" "><nodename>Number literals</nodename><nodenext spaces=" ">if statement</nodenext><nodeprev spaces=" ">Attributes</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Number literals</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="947">number literals <command>ncap2</command></indexterm></cindex>
<para>The table below lists the postfix character(s) to add to a number
literal for type cohesion. 
To use the new netCDF4 types <acronym><acronymword>NCO</acronymword></acronym> must be compiled/linked to
the netCDF4 library and the output file must be <acronym><acronymword>HDF5</acronymword></acronym>.
</para>
<example endspaces=" ">
<pre xml:space="preserve">n1[$time]=1UL; // n1 will now by type <code>NC_UINT</code>
n2[$lon]=4b;   // n2 will be of type <code>NC_BYTE</code>
n3[$lat]=5ull; // n3 will be of type <code>NC_UINT64</code>  
n3&arobase;a1=6.0d;   // attribute will be type <code>NC_DOUBLE</code>
n3&arobase;a2=-666L;  // attribute will be type <code>NC_INT</code>
</pre></example>

<para>A floating point number without a postfix will default to
<code>NC_DOUBLE</code>. 
An integer without a postfix will default to type <code>NC_INT</code>. 
There is no postfix for characters, use a quoted string instead.
</para><example endspaces=" ">
<pre xml:space="preserve">n4[$rlev]=0.1      // n4 will be of type <code>NC_DOUBLE</code>
n5[$lon_grd]=2.0   // n5 will be of type <code>NC_DOUBLE</code>
n6[$gds_crd]=2e3;  // n6 will be of type <code>NC_DOUBLE</code>
n7[$gds_crd]=2e3f;  // n7 will be of type <code>NC_FLOAT</code>
n6&arobase;a1=41;         // attribute will be type <code>NC_INT</code>
n6&arobase;a2=-21;        // attribute will be type <code>NC_INT</code>  
n6&arobase;units=&quot;kelvin&quot; // attribute will be type <code>NC_CHAR</code>
</pre></example> 

<table commandarg="asis" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="asis"><strong>netCDF3/4 Types</strong></itemformat></item>
</tableterm></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">b|B	 </itemformat></item>
</tableterm><tableitem>  <para><code>NC_BYTE</code>, a signed 1-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">none	 </itemformat></item>
</tableterm><tableitem>  <para><code>NC_CHAR</code>, an ISO/ASCII character 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">s|S	 </itemformat></item>
</tableterm><tableitem>  <para><code>NC_SHORT</code>, a signed 2-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">l|L	 </itemformat></item>
</tableterm><tableitem>  <para><code>NC_INT</code>, a signed 4-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">f|F	 </itemformat></item>
</tableterm><tableitem>  <para><code>NC_FLOAT</code>, a single-precision (4-byte) floating point number 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">d|D</itemformat></item>
</tableterm><tableitem>  <para><code>NC_DOUBLE</code>, a double-precision (8-byte) floating point number 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis"><strong>netCDF4 Types</strong></itemformat></item>
</tableterm></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">ub|UB	 </itemformat></item>
</tableterm><tableitem>  <para><code>NC_UBYTE</code>, an unsigned 1-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">us|US </itemformat></item>
</tableterm><tableitem>  <para><code>NC_USHORT</code>, an unsigned 2-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">u|U|ul|UL	 </itemformat></item>
</tableterm><tableitem>  <para><code>NC_UINT</code>, an unsigned 4-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">ll|LL	 </itemformat></item>
</tableterm><tableitem>  <para><code>NC_INT64</code>, a signed 8-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">ull|ULL </itemformat></item>
</tableterm><tableitem>  <para><code>NC_UINT64</code>, an unsigned 8-byte integer 
</para></tableitem></tableentry></table>

<html endspaces=" ">
&lt;a name=&quot;ncap_if&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_if &textndash;&gt;
</html>
</subsection>
<node name="if-statement" spaces=" "><nodename>if statement</nodename><nodenext spaces=" ">print statement</nodenext><nodeprev spaces=" ">Number literals</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>if statement</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="948">if()</indexterm></cindex>
<para>The syntax of the if statement is similar to its C counterpart. 
The <emph>Conditional Operator (ternary operator)</emph> has also been
implemented. 
</para><example endspaces=" ">
<pre xml:space="preserve">if(exp1)
   stmt1;
else if(exp2)     
   stmt2;
else
   stmt3;

# Can use code blocks as well:
if(exp1)&lbrace;
   stmt1;
   stmt1a;
   stmt1b;
&rbrace; else if(exp2)     
   stmt2; 
else &lbrace;
   stmt3;
   stmt3a;
   stmt3b;
&rbrace;     
</pre></example>

<!-- comment Truth -->
<noindent></noindent> <para>For a variable or attribute expression to be logically true
all its non-missing value elements must be logically true, i.e.,
non-zero. 
The expression can be of any type. 
<w>Unlike C</w> there is no short-circuiting of an expression with the 
OR (<code>||</code>) and AND (<code>&amp;&amp;</code>) operators. 
The whole expression is evaluated regardless if one of the AND/OR
operands are True/False.
</para><example endspaces=" ">
<pre xml:space="preserve"># Simple example
if(time&gt;0)
  print(&quot;All values of time are greater than zero\n&quot;);
else if( time&lt;0)
  print(&quot;All values of time are less than zero\n&quot;);   
else &lbrace;
  time_max=time.max();
  time_min=time.min();
  print(&quot;min value of time=&quot;);print(time_min,&quot;%f&quot;);
  print(&quot;max value of time=&quot;);print(time_max,&quot;%f&quot;);
&rbrace;

# Example from ddra.nco
if(fl_typ==fl_typ_gcm)&lbrace;
  var_nbr_apx=32;
  lmn_nbr=1.0*var_nbr_apx*varsz_gcm_4D; /* [nbr] Variable size */
  if(nco_op_typ==nco_op_typ_avg)&lbrace;
    lmn_nbr_avg=1.0*var_nbr_apx*varsz_gcm_4D; // Block size
    lmn_nbr_wgt=dmnsz_gcm_lat; /* [nbr] Weight size */
  &rbrace; // !nco_op_typ_avg
&rbrace;else if(fl_typ==fl_typ_stl)&lbrace;
  var_nbr_apx=8;
  lmn_nbr=1.0*var_nbr_apx*varsz_stl_2D; /* [nbr] Variable size */
  if(nco_op_typ==nco_op_typ_avg)&lbrace;
    lmn_nbr_avg=1.0*var_nbr_apx*varsz_stl_2D; // Block size
    lmn_nbr_wgt=dmnsz_stl_lat; /* [nbr] Weight size */
  &rbrace; // !nco_op_typ_avg
&rbrace; // !fl_typ
</pre></example>

<noindent></noindent> <para><strong>Conditional Operator &linebreak;</strong>
</para><example endspaces=" ">
<pre xml:space="preserve">// netCDF4 needed for this example
th_nw=(three_dmn_var_sht &gt;= 0 ? three_dmn_var_sht.uint() : \
       three_dmn_var_sht.int()); 
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_prn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_prn &textndash;&gt;
</html>
</subsection>
<node name="print-statement" spaces=" "><nodename>print statement</nodename><nodenext spaces=" ">Missing values ncap2</nodenext><nodeprev spaces=" ">if statement</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>print statement</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="949">print() <command>ncap2</command></indexterm></cindex>
<example endspaces=" ">
<pre xml:space="preserve">print(variable_name/attribute name/string, format string);
</pre></example>  

<noindent></noindent> <para>The print function takes a variable name or attribute name or
a quoted string and prints the contents in a in a similar fashion to
<code>ncks -H</code>.
There is also an optional C-language style format string argument. 
Currently the print function cannot print <acronym><acronymword>RAM</acronymword></acronym> variables or expressions 
such as <code>'print(var_msk*3+4)'</code>. 
To print an expression, first evaluate it as a non-<acronym><acronymword>RAM</acronymword></acronym> variable (so it
will be saved and can be printed), and then print the variable.
</para>
<noindent></noindent> <para>examples
</para><example endspaces=" ">
<pre xml:space="preserve">print(lon);
lon[0]=0 
lon[1]=90 
lon[2]=180 
lon[3]=270 

print(lon_2D_rrg,&quot;%3.2f,&quot;);
0.00,0.00,180.00,0.00,180.00,0.00,180.00,0.00,

print(mss_val_fst&arobase;_FillValue);
mss_val_fst&arobase;_FillValue, size = 1 NC_FLOAT, value = -999

print(&quot;This function \t is monotonic\n&quot;);
This function is 	  monotonic
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_miss&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_miss &textndash;&gt;
</html>
</subsection>
<node name="Missing-values-ncap2" spaces=" "><nodename>Missing values ncap2</nodename><nodenext spaces=" ">Methods and functions</nodenext><nodeprev spaces=" ">print statement</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Missing values ncap2</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="950">missing values ncap2</indexterm></cindex>
<para>Missing values operate slightly differently in <command>ncap2</command> 
Consider the expression where op is any of the following operators (excluding &textrsquo;=&textrsquo;)
</para><example endspaces=" ">
<pre xml:space="preserve">Arithmetic operators ( * / % + - ^ )
Binary Operators     ( &gt;, &gt;= &lt;, &lt;= ==, !=,==,||,&amp;&amp;, &gt;&gt;,&lt;&lt; ) 
Assign Operators     ( +=,-=,/=, *= ) 

var1 'op' var2
</pre></example>

<noindent></noindent> <para>If var1 has a missing value then this is the value used in the 
operation, otherwise the missing value for var2 is used. 
If during the element-by-element operation an element from either
operand is equal to the missing value then the missing value is carried 
through. 
In this way missing values &textrsquo;percolate&textrsquo; or propagate through an
expression.&linebreak;  
Missing values associated with Output variables are stored in memory and
are written to disk after the script finishes. 
During script execution its possible (and legal) for the missing value
of a variable to take on several different values. 
</para><example endspaces=" "> 
<pre xml:space="preserve"># Consider the variable:
int rec_var_int_mss_val_int(time); =-999,2,3,4,5,6,7,8,-999,-999;
rec_var_int_mss_val_int:_FillValue = -999;

n2=rec_var_int_mss_val_int + rec_var_int_mss_val_int.reverse($time); 

n2=-999,-999,11,11,11,11,11,11,999,-999;
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;set_miss&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#set_miss &textndash;&gt;
&lt;a name=&quot;get_miss&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#get_miss &textndash;&gt;
&lt;a name=&quot;change_miss&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#change_miss &textndash;&gt;
&lt;a name=&quot;number_miss&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#number_miss &textndash;&gt;
</html>
<para>The following methods manipulate missing value information associated
with a variable. 
They only work on variables in Output. 
</para><table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">set_miss(expr)</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="951"><code>set_miss()</code></indexterm></cindex>
 <para>The numeric argument <var>expr</var> becomes the new missing value,
 overwriting the old missing value, if any.
 The argument given is converted if necessary to the variable type.
 NB: This only changes the missing value attribute. 
 Missing values in the original variable remain unchanged, and thus 
 are no long considered missing values.
 They are &textldquo;orphaned&textrdquo;.
 Thus <code>set_miss()</code> is normally used only when creating new
 variables.
 The intrinsic function <code>change_miss()</code> (see below) is typically 
 used to edit values of existing variables.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">change_miss(expr)</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="952"><code>change_miss()</code></indexterm></cindex>
 <para>Sets or changes (any pre-existing) missing value attribute and missing 
 data values to <var>expr</var>. 
 NB: This is an expensive function since all values must be examined. 
 Use this function when changing missing values for pre-existing
 variables. 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">get_miss() </itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="953"><code>get_miss()</code></indexterm></cindex>
 <para>Returns the missing value of a variable. 
 If the variable exists in Input and Output then the missing value of
 the variable in Output is returned. 
 If the variable has no missing value then an error is returned.   
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">delete_miss()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="954"><code>delete_miss()</code></indexterm></cindex>
 <para>Deletes the missing value associated with a variable.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">number_miss()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="955"><code>number_miss()</code></indexterm></cindex>
 <para>Counts the number of missing values a variable contains.
</para></tableitem></tableentry></table>

<example endspaces=" ">
<pre xml:space="preserve">th=three_dmn_var_dbl;
th.change_miss(-1e10d);
/* Set values less than 0 or greater than 50 to missing value */
where(th &lt; 0.0 || th &gt; 50.0) th=th.get_miss();

# Another example:
new[$time,$lat,$lon]=1.0;
new.set_miss(-997.0);

// Extract only elements divisible by 3
where (three_dmn_var_dbl%3 == 0)
     new=three_dmn_var_dbl; 
elsewhere
     new=new.get_miss();   

// Print missing value and variable summary
mss_val_nbr=three_dmn_var_dbl.number_miss();
print(three_dmn_var_dbl&arobase;_FillValue);
print(&quot;Number of missing values in three_dmn_var_dbl: &quot;);
print(mss_val_nbr,&quot;%d&quot;);
print(three_dmn_var_dbl);
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_mtd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_mtd &textndash;&gt;
</html>
</subsection>
<node name="Methods-and-functions" spaces=" "><nodename>Methods and functions</nodename><nodenext spaces=" ">RAM variables</nodenext><nodeprev spaces=" ">Missing values ncap2</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Methods and functions</sectiontitle>

<para>The convention within this document is that methods can be used as 
functions. 
However, functions are not and cannot be used as methods.
Methods can be daisy-chained d and their syntax is cleaner than functions. 
Method names are reserved words and CANNOT be used as variable names.  
The command <code>ncap2 -f</code> shows the complete list of methods available
on your build. 
</para><example endspaces=" ">
<pre xml:space="preserve">n2=sin(theta) 
n2=theta.sin() 
n2=sin(theta)^2 + cos(theta)^2 
n2=theta.sin().pow(2) + theta.cos()^2
</pre></example>

<para>This statement chains together methods to convert three_dmn_var_sht to
type double, average it, then convert this back to type short: 
</para><example endspaces=" ">
<pre xml:space="preserve">three_avg=three_dmn_var_sht.double().avg().short();
</pre></example>

<sp value="1" line=" 1"></sp>
<noindent></noindent> <para><strong>Aggregate Methods &linebreak;</strong> 
<noindent></noindent> These methods mirror the averaging types available in <command>ncwa</command>. The arguments to the methods are the dimensions to average over. Specifying no dimensions is equivalent to specifying all dimensions i.e., averaging over all dimensions. A masking variable and a weighting variable can be manually created and applied as needed.
</para>
<table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">avg()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="956">avg()</indexterm></cindex>
<para>Mean value 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">sqravg()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="957">sqravg()</indexterm></cindex>
<para>Square of the mean
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">avgsqr()</itemformat></item>
</tableterm><tableitem><para>Mean of sum of squares
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">max()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="958">max()</indexterm></cindex>
<para>Maximum value
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">min()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="959">min()</indexterm></cindex>
<para>Minimum value
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">rms()</itemformat></item>
</tableterm><tableitem><para>Root-mean-square (normalize by <var>N</var>)
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">rmssdn()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="960">rmssdn()</indexterm></cindex>
<para>Root-mean square (normalize by <var>N-1</var>)
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ttl() or total()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="961">ttl()</indexterm></cindex>
<para>Sum of values
</para></tableitem></tableentry></table>

<example endspaces=" ">
<pre xml:space="preserve">// Average a variable over time
four_time_avg=four_dmn_rec_var($time);
</pre></example>

<sp value="1" line=" 1"></sp>
<noindent></noindent> <para><strong> Packing Methods &linebreak; </strong> 
For more information see <pxref label="Packed-data"><xrefnodename>Packed data</xrefnodename></pxref> and <pxref label="ncpdq-netCDF-Permute-Dimensions-Quickly"><xrefnodename>ncpdq netCDF Permute Dimensions Quickly</xrefnodename></pxref>&linebreak;
</para><table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">pack() &amp; pack_short()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="962">pack()</indexterm></cindex>
<para>The default packing algorithm is applied and variable is packed to <code>NC_SHORT</code>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">pack_byte()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="963">pack_byte()</indexterm></cindex>
<para>Variable is packed to <code>NC_BYTE</code>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">pack_short()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="964">pack_short()</indexterm></cindex>
<para>Variable is packed to <code>NC_SHORT</code>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">pack_int()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="965">pack_int()</indexterm></cindex>
<para>Variable is packed to <code>NC_INT</code>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">unpack()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="966">unpack()</indexterm></cindex>
<para>The standard unpacking algorithm is applied. 
</para></tableitem></tableentry></table>

<noindent></noindent> <para><strong>Basic Methods &linebreak;</strong>
These methods work with variables and attributes. They have no arguments
</para>
<table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">size()	</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="967">size()</indexterm></cindex>
<para>Total number of elements 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ndims()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="968">ndims()</indexterm></cindex>
<para>Number of dimensions in variable
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">type() </itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="969">type()</indexterm></cindex>
<para>Returns the netcdf type (see previous section)
</para></tableitem></tableentry></table>

<sp value="1" line=" 1"></sp>
<noindent></noindent> <para><strong>Utility Methods &linebreak;</strong>
These functions are used to manipulate missing values and <acronym><acronymword>RAM</acronymword></acronym> variables.
<pxref label="Missing-values-ncap2"><xrefnodename>Missing values ncap2</xrefnodename></pxref> 
</para>
<table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">set_miss(expr)</itemformat></item>
</tableterm><tableitem> <para>Takes one argument the missing value. Sets or overwrites the existing missing value. The argument given is converted if necessary to the variable type
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">change_miss(expr)</itemformat></item>
</tableterm><tableitem> <para>Changes the missing value elements of the variable to the new missing value (n.b. an expensive function).
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">get_miss() </itemformat></item>
</tableterm><tableitem> <para>Returns the missing value of a variable in Input or Output  
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">delete_miss()</itemformat></item>
</tableterm><tableitem> <para>Deletes the missing value associated with a variable.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ram_write()</itemformat></item>
</tableterm><tableitem> <para>Writes a <acronym><acronymword>RAM</acronymword></acronym> variable to disk i.e., converts it to a regular disk type variable
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ram_delete()</itemformat></item>
</tableterm><tableitem> <para>Deletes a <acronym><acronymword>RAM</acronymword></acronym> variable or an attribute 
</para></tableitem></tableentry></table>

<sp value="1" line=" 1"></sp>
<noindent></noindent> <para><strong>PDQ Methods &linebreak;</strong>
See <pxref label="ncpdq-netCDF-Permute-Dimensions-Quickly"><xrefnodename>ncpdq netCDF Permute Dimensions Quickly</xrefnodename></pxref>
</para><table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">reverse(dim args)</itemformat></item>
</tableterm><tableitem> <para>Reverses the dimension ordering of elements in a variable. 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">permute(dim args)</itemformat></item>
</tableterm><tableitem> <para>Re-shapes variables by re-ordering the dimensions. All the dims of the variable must be specified in the arguments. A limitation of this permute (unlike ncpdq) is that the record dimension cannot be re-assigned. 
</para></tableitem></tableentry></table> 
<para>// Swap dimensions about and reorder along lon
</para><example endspaces=" ">
<pre xml:space="preserve">lat_2D_rrg_new=lat_2D_rrg.permute($lon,$lat).reverse($lon);
lat_2D_rrg_new=0,90,-30,30,-30,30,-90,0
</pre></example>

<sp value="1" line=" 1"></sp>
<noindent></noindent> <para><strong>Type Conversion Methods &linebreak;</strong>
<noindent></noindent> These methods allow <command>ncap2</command> to convert variables and attributes to the different netcdf types. For more details on automatic and manual type conversion see (<pxref label="Type-Conversion"><xrefnodename>Type Conversion</xrefnodename></pxref>). You can only use the new netCDF4 types if you have compiled/links <acronym><acronymword>NCO</acronymword></acronym> with the netCDF4 library and the Output file is HDF5.
</para>
<table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code"><strong>netCDF3/4 Types</strong></itemformat></item>
</tableterm></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">byte()	 </itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="970">byte()</indexterm></cindex>
 <para>convert to <code>NC_BYTE</code>,  a signed 1-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">char()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="971">char()</indexterm></cindex>
 <para>convert to <code>NC_CHAR</code>,  an ISO/ASCII character
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">short()	</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="972">sshort()</indexterm></cindex>
 <para>convert to <code>NC_SHORT</code>, a signed 2-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">int()	 </itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="973">int()</indexterm></cindex>
 <para>convert to <code>NC_INT</code>,   a signed 4-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">float()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="974">float()</indexterm></cindex>
 <para>convert to <code>NC_FLOAT</code>, a single-precision (4-byte) floating point number 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">double() </itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="975">double()</indexterm></cindex>
 <para>convert to <code>NC_DOUBLE</code>, a double-precision (8-byte) floating point number 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code"><strong>netCDF4 Types</strong></itemformat></item>
</tableterm></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ubyte()	 </itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="976">ubyte()</indexterm></cindex>
 <para>convert to <code>NC_UBYTE</code>, an unsigned 1-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ushort() </itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="977">ushort()</indexterm></cindex>
 <para>convert to <code>NC_USHORT</code>, an unsigned 2-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">uint()</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="978">uint()</indexterm></cindex>
 <para>convert to <code>NC_UINT</code>, an unsigned 4-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">int64()	</itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="979">int64()</indexterm></cindex>
 <para>convert to <code>NC_INT64</code>, a signed 8-byte integer 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">uint64() </itemformat></item>
</tableterm><tableitem><cindex index="cp" spaces=" "><indexterm index="cp" number="980">unit64()</indexterm></cindex>
 <para>convert to <code>NC_UINT64</code>, an unsigned 8-byte integer
</para></tableitem></tableentry></table>

<noindent></noindent> <para><strong>Intrinsic Mathematical Methods &linebreak;</strong>
The list of mathematical methods is system dependant.
For the full list <pxref label="Intrinsic-mathematical-methods"><xrefnodename>Intrinsic mathematical methods</xrefnodename></pxref> 
</para>
<para>All the mathematical methods take a single argument except <code>atan2()</code>
and <code>pow()</code> which take two. 
If the operand type is less than <emph>float</emph> then the result will be of
type <emph>float</emph>. 
Arguments of type <emph>double</emph> yield results of type <emph>double</emph>. 
Like the other methods, you are free to use the mathematical methods as functions. 
</para>
<example endspaces=" ">
<pre xml:space="preserve">n1=pow(2,3.0f)    // n1 type float
n2=atan2(2,3.0)   // n2 type double
n3=1/(three_dmn_var_dbl.cos().pow(2))-tan(three_dmn_var_dbl)^2; // n3 type double
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_ram&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_ram &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="981"><acronym><acronymword>RAM</acronymword></acronym> variables</indexterm></cindex>
</subsection>
<node name="RAM-variables" spaces=" "><nodename>RAM variables</nodename><nodenext spaces=" ">Where statement</nodenext><nodeprev spaces=" ">Methods and functions</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle><acronym><acronymword>RAM</acronymword></acronym> variables</sectiontitle>
<para>Unlike regular variables, <acronym><acronymword>RAM</acronymword></acronym> variables are never written to disk.
Hence using <acronym><acronymword>RAM</acronymword></acronym> variables in place of regular variables (especially
within loops) significantly increases execution speed.
Variables that are frequently accessed within <code>for</code> or <code>where</code>
clauses provide the greatest opportunities for optimization. 
To declare and define a <acronym><acronymword>RAM</acronymword></acronym> variable simply prefix the variable name
with an asterisk (<code>*</code>) when the variable is declared/initialized.
To delete a <acronym><acronymword>RAM</acronymword></acronym> variables (and recover their memory) use the
<code>ram_delete()</code> method. 
To write a <acronym><acronymword>RAM</acronymword></acronym> variable to disk (like a regular variable) use
<code>ram_write()</code>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="982">ram_write()</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="983">ram_delete()</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">*temp[$time,$lat,$lon]=10.0;     // Cast
*temp_avg=temp.avg($time);      // Regular assign
temp.ram_delete();              // Delete RAM variable
temp_avg.ram_write();           // Write Variable to output

// Create and increment a RAM variable from &quot;one&quot; in Input
*one++;   
// Create RAM variables from the variables three and four in Input.
// Multiply three by 10 and add it to four. 
*four+=*three*=10; // three=30, four=34 
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;where&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#where &textndash;&gt;
&lt;a name=&quot;ncap_whr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_whr &textndash;&gt;
&lt;a name=&quot;ncap_where&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_where &textndash;&gt;
</html>
</subsection>
<node name="Where-statement" spaces=" "><nodename>Where statement</nodename><nodenext spaces=" ">Loops</nodenext><nodeprev spaces=" ">RAM variables</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Where statement</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="984">where()</indexterm></cindex>
<para>A <code>where()</code> combines the definition and application of a mask all in one go and can lead to succinct code. 
The full syntax of a <code>where()</code> statement is as follows:
</para>
<example endspaces=" ">
<pre xml:space="preserve">// Single assign (the 'elsewhere' block is optional)
where(mask) 
   var1=expr1;
elsewhere
   var1=expr2;	   	

// Multiple assigns
where(mask)&lbrace;
    var1=expr1;
    var2=expr2;
    ...
&rbrace;elsewhere&lbrace;
    var1=expr3
    var2=expr4
    var3=expr5;
    ...
&rbrace;
</pre></example>

<itemize commandarg="bullet" spaces=" " endspaces=" "><itemprepend><formattingcommand command="bullet"/></itemprepend>
<listitem spaces=" "><prepend>&bullet;</prepend><para>The only expression allowed in the predicate of a where is assign,
i.e., &textrsquo;var=expr&textrsquo;. 
This assign differs from a regular <command>ncap2</command> assign. 
The LHS var must already exist in Input or Output. 
The RHS expression must evaluate to a scalar or a variable/attribute of
the same size as the LHS variable.
</para></listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>Consider when both the LHS and RHS are variables: 
For every element where mask condition is True, the corresponding LHS
variable element is re-assigned to its partner element on the RHS. 
In the elsewhere part the mask is logically inverted and the assign
process proceeds as before.
</para></listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>If the mask dimensions are a subset of the LHS variable&textrsquo;s
dimensions, then it is made to conform; if it cannot be made to conform 
then script execution halts.   
</para></listitem><listitem spaces=" "><prepend>&bullet;</prepend><para>Missing values in the mask evaluate to False in the where 
code/block statement and to True in the elsewhere block/statement. 
LHS variable elements set to missing value are not re-assigned.
For these reasons, do not explicitly reference missing values in the
masking condition, e.g., <code>where(foo=foo.get_missing()) foo=1;</code>
will not work as expected.
</para></listitem></itemize>

<para>Example:
Consider the variables <code>float lon_2D_rct(lat,lon);</code> and
<code>float var_msk(lat,lon);</code>. 
Suppose we wish to multiply by two the elements for which <code>var_msk</code> 
<w>equals 1</w>: 
</para><example endspaces=" ">
<pre xml:space="preserve">where(var_msk==1) lon_2D_rct=2*lon_2D_rct;
</pre></example>
<para>Suppose that we have the variable <code>int RDM(time)</code> and that we want
to set its values less than 8 or greater than 80 <w>to 0</w>:
</para><example endspaces=" ">
<pre xml:space="preserve">where(RDM &lt; 8 || RDM &gt; 80) RDM=0;          
</pre></example>

<para>Consider irregularly gridded data, described using <w>rank 2</w> coordinates: 
<code>double lat(south_north,east_west)</code>,
<code>double lon(south_north,east_west)</code>, 
<code>double temperature(south_north,east_west)</code>.
To find the average temperature in a region bounded by
[<var>lat_min</var>,<var>lat_max</var>] and [<var>lon_min</var>,<var>lon_max</var>]:
</para><example endspaces=" ">
<pre xml:space="preserve">temperature_msk[$south_north,$east_west]=0.0;
where(lat &gt;= lat_min &amp;&amp; lat &lt;= lat_max) &amp;&amp; (lon &gt;= lon_min &amp;&amp; lon &lt;= lon_max)
  temperature_msk=temperature;	
elsewhere
  temperature_msk=temperature&arobase;_FillValue;

temp_avg=temperature_msk.avg();
temp_max=temperature.max();
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_lop&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_lop &textndash;&gt;
</html>
</subsection>
<node name="Loops" spaces=" "><nodename>Loops</nodename><nodenext spaces=" ">Include files</nodenext><nodeprev spaces=" ">Where statement</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Loops</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="985">while()</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="986">for()</indexterm></cindex>
<para><command>ncap2</command> supplies <command>for()</command> loops and <command>while()</command> loops. 
They are completely unoptimized so use them only with <acronym><acronymword>RAM</acronymword></acronym> 
variables unless you want thrash your disk to death. 
To break out of a loop use the <command>break</command> command. 
To iterate to the next cycle use the <command>continue</command> command. 
</para>
<example endspaces=" ">
<pre xml:space="preserve">// Set elements in variable double temp(time,lat) 
// If element &lt; 0 set to 0, if element &gt; 100 set to 100
*sz_idx=$time.size;
*sz_jdx=$lat.size;

for(*idx=0;idx&lt;sz_idx;idx++)
  for(*jdx=0;jdx&lt;sz_jdx;jdx++)
    if(temp(idx,jdx) &gt; 100) temp(idx,jdx)=100.0; 
      else if(temp(idx,jdx) &lt; 0) temp(idx,jdx)=0.0;

// Are values of co-ordinate variable double lat(lat) monotonic?
*sz=$lat.size;

for(*idx=1;idx&lt;sz;idx++)
  if(lat(idx)-lat(idx-1) &lt; 0.0) break;

if(idx == sz) print(&quot;lat co-ordinate is monotonic\n&quot;);
   else print(&quot;lat co-ordinate is NOT monotonic\n&quot;);

// Sum odd elements	
*idx=0;
*sz=$lat_nw.size;
*sum=0.0;

while(idx&lt;sz)&lbrace;
  if(lat(idx)%2) sum+=lat(idx);
  idx++;
&rbrace;

ram_write(sum);
print(&quot;Total of odd elements &quot;);print(sum);print(&quot;\n&quot;); 
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_inc&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_inc &textndash;&gt;
</html>
</subsection>
<node name="Include-files" spaces=" "><nodename>Include files</nodename><nodenext spaces=" ">Sort methods</nodenext><nodeprev spaces=" ">Loops</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Include files</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="987"><command>include</command></indexterm></cindex>
<para>The syntax of an <var>include-file</var> is:
</para><example endspaces=" ">
<pre xml:space="preserve">#include &quot;script.nco&quot;
</pre></example>
<para>The script filename is searched relative to the run directory. 
It is possible to nest include files to an arbitrary depth. 
A handy use of inlcude files is to store often used constants. 
Use <acronym><acronymword>RAM</acronymword></acronym> variables if you do not want these constants written to
<var>output-file</var>.  
</para><example endspaces=" ">
<pre xml:space="preserve">// script.nco
// Sample file to #include in ncap2 script
*pi=3.1415926535; // RAM variable, not written to output
*h=6.62607095e-34; // RAM variable, not written to output
e=2.71828; // Regular (disk) variable, written to output
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;srt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#srt &textndash;&gt;
&lt;a name=&quot;sort&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#sort &textndash;&gt;
&lt;a name=&quot;remap&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#remap &textndash;&gt;
</html>
</subsection>
<node name="Sort-methods" spaces=" "><nodename>Sort methods</nodename><nodenext spaces=" ">Irregular grids</nodenext><nodeprev spaces=" ">Include files</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle><command>sort</command> methods</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="988"><command>sort</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="989"><command>asort</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="990"><command>dsort</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="991"><command>remap</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="992"><command>unmap</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="993"><command>invert_map</command></indexterm></cindex>
<para>In <acronym><acronymword>ncap2</acronymword></acronym> there are multiple ways to sort data. 
Beginning with <acronym><acronymword>NCO</acronymword></acronym> 4.1.0 (March, 2012), <acronym><acronymword>ncap2</acronymword></acronym> 
support six sorting functions:
</para><example endspaces=" ">
<pre xml:space="preserve">var_out=sort(var_in,&amp;srt_map); // Ascending sort
var_out=asort(var_in,&amp;srt_map); // Accending sort 
var_out=dsort(var_in,&amp;srt_map); // Desending sort     
var_out=remap(var_in,srt_map); // Apply srt_map to var_in
var_out=unmap(var_in,srt_map); // Reverse what srt_map did to var_in
dsr_map=invert_map(srt_map); // Produce &quot;de-sort&quot; map that inverts srt_map
</pre></example>
<para>The first two functions, <command>sort()</command> and <command>asort()</command>
sort, in ascending order, all the elements of <var>var_in</var> (which can be
a variable or attribute) without regard to any dimensions.
The third function, <command>dsort()</command> does the same but sorts in
descending order. 
Remember that ascending and descending sorts are specified by
<command>asort()</command> and <command>dsort()</command>, respectively.
</para>
<para>These three functions are overloaded to take a second, optional argument 
called the sort map <var>srt_map</var>, which should be supplied as a
call-by-reference variable, i.e., preceded with an ampersand.
If the sort map does not yet exist, then it will be created and 
returned as an integer type the same shape as the input variable.
</para>
<para>The output <var>var_out</var> of each sort function is a sorted version of
the input, <var>var_in</var>.
The output <var>var_out</var> of the two mapping functions the result of
applying (with <command>remap()</command> or un-applying (with <command>unmap()</command>) 
the sort map <var>srt_map</var> to the input <var>var_in</var>.
To apply the sort map with <command>remap()</command> the size of the variable
must be exactly divisible by the size of the sort map. 
</para>
<para>The final function <command>invert_map()</command> returns the so-called
de-sorting map <var>dsr_map</var> which is inverse map of the input map
<var>srt_map</var>. 
This gives the user access to both the forward and inverse sorting maps
which can be useful in special situations.
</para><example endspaces=" ">
<pre xml:space="preserve">a1[$time]=&lbrace;10,2,3,4,6,5,7,3,4,1&rbrace;;
a1_sort=sort(a1);
print(a1_sort);
// 1, 2, 3, 3, 4, 4, 5, 6, 7, 10;

a2[$lon]=&lbrace;2,1,4,3&rbrace;;
a2_sort=sort(a2,&amp;a2_map);
print(a2);
// 1, 2, 3, 4
print(a2_map);
// 1, 0, 3, 2;
</pre></example>  

<para>If the map variable does not exist prior to the <command>sort()</command> call,
then it will be created with the same shape as the input variable and be
of type <code>NC_INT</code>. 
If the map variable already exists, then the only restriction is that it
be of at least the same size as the input variable. 
To apply a map use <code>remap(var_in,srt_map)</code>. 
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;nlat&quot;,5);

a3[$lon]=&lbrace;2,5,3,7&rbrace;;
a4[$nlat,$lon]=&lbrace;
 1, 2, 3, 4, 
 5, 6, 7, 8,
 9,10,11,12,
 13,14,15,16,
 17,18,19,20&rbrace;;

a3_sort=sort(a3,&amp;a3_map);
print(a3_map);
// 0, 2, 1, 3;

a4_sort=remap(a4,a3_map);
print(a4_sort);
// 1, 3, 2, 4,
// 5, 7, 6, 8,
// 9,11,10,12,
// 13,15,14,16,
// 17,19,18,20;

a3_map2[$nlat]=&lbrace;4,3,0,2,1&rbrace;;

a4_sort2=remap(a4,a3_map2);
print(a4_sort2);
// 3, 5, 4, 2, 1
// 8, 10, 9,7, 6, 
// 13,15,14,12,11, 
// 18,20,19,17,16
</pre></example>
<para>As in the above example you may create your own sort map.
To sort in descending order, apply the <code>reverse()</code> method after the
<command>sort()</command>.    
</para>
<para>Here is an extended example of how to use <command>ncap2</command> features to
hyperslab an irregular region based on the values of a variable not a
coordinate. 
The distinction is crucial: hyperslabbing based on dimensional indices
or coordinate values is straightforward.
Using the values of single or multi-dimensional variable to define a
hyperslab is quite different.
</para><example endspaces=" ">
<pre xml:space="preserve">cat &gt; ~/ncap2_foo.nco &lt;&lt; 'EOF'
// Purpose: Save irregular 1-D regions based on variable values

// Included in NCO User Guide at http://nco.sf.net/nco.html#sort

/* NB: Single quotes around EOF above turn off shell parameter 
    expansion in &quot;here documents&quot;. This in turn prevents the
    need for protecting dollarsign characters in NCO scripts with
    backslashes when the script is cut-and-pasted (aka &quot;moused&quot;) 
    from an editor or e-mail into a shell console window */

/* Copy coordinates and variable(s) of interest into RAM variable(s)
   Benefits:
   1. ncap2 defines writes all variables on LHS of expression to disk
      Only exception is RAM variables, which are stored in RAM only
      Repeated operations on regular variables takes more time, 
      because changes are written to disk copy after every change.
      RAM variables are only changed in RAM so script works faster
      RAM variables can be written to disk at end with ram_write()
   2. Script permutes variables of interest during processing
      Safer to work with copies that have different names
      This discourages accidental, mistaken use of permuted versions
   3. Makes this script a more generic template:
      var_in instead of specific variable names everywhere */
*var_in=one_dmn_rec_var;
*crd_in=time;
*dmn_in_sz=$time.size; // [nbr] Size of input arrays

/* Create all other &quot;intermediate&quot; variables as RAM variables 
   to prevent them from cluttering the output file.
   Mask flag and sort map are same size as variable of interest */
*msk_flg=var_in;
*srt_map=var_in;

/* In this example we mask for all values evenly divisible by 3
   This is the key, problem-specific portion of the template
   Replace this where() condition by that for your problem
   Mask variable is Boolean: 1=Meets condition, 0=Fails condition */
where(var_in % 3 == 0) msk_flg=1; elsewhere msk_flg=0;

// print(&quot;msk_flg = &quot;);print(msk_flg); // For debugging...

/* The sort() routine is overloaded, and takes one or two arguments
   The second argument (optional) is the &quot;sort map&quot; (srt_map below)
   Pass the sort map by reference, i.e., prefix with an ampersand
   If the sort map does not yet exist, then it will be created and 
   returned as an integer type the same shape as the input variable.
   The output of sort(), on the LHS, is a sorted version of the input
   msk_flg is not needed in its original order after sort()
   Hence we use msk_flg as both input to and output from sort()
   Doing this prevents the need to define a new, unneeded variable */
msk_flg=sort(msk_flg,&amp;srt_map);

// Count number of valid points in mask by summing the one's
*msk_nbr=msk_flg.total();

// Define output dimension equal in size to number of valid points
defdim(&quot;crd_out&quot;,msk_nbr);

/* Now sort the variable of interest using the sort map and remap()
   The output, on the LHS, is the input re-arranged so that all points
   meeting the mask condition are contiguous at the end of the array
   Use same srt_map to hyperslab multiple variables of the same shape
   Remember to apply srt_map to the coordinate variables */
crd_in=remap(crd_in,srt_map);
var_in=remap(var_in,srt_map);

/* Hyperslab last msk_nbr values of variable(s) of interest */
crd_out[crd_out]=crd_in((dmn_in_sz-msk_nbr):(dmn_in_sz-1));
var_out[crd_out]=var_in((dmn_in_sz-msk_nbr):(dmn_in_sz-1));

/* NB: Even though we created all variables possible as RAM variables,
   the original coordinate of interest, time, is written to the ouput.
   I'm not exactly sure why. For now, delete it from the output with: 
   ncks -O -x -v time ~/foo.nc ~/foo.nc
   */ 
EOF
ncap2 -O -v -S ~/ncap2_foo.nco ~/nco/data/in.nc ~/foo.nc
ncks -O -x -v time ~/foo.nc ~/foo.nc
ncks ~/foo.nc
</pre></example>

<para>Here is an extended example of how to use <command>ncap2</command> features to
sort multi-dimensional arrays based on the coordinate values along a
single dimension. 
</para><example endspaces=" ">
<pre xml:space="preserve">cat &gt; ~/ncap2_foo.nco &lt;&lt; 'EOF'
/* Purpose: Sort multi-dimensional array based on coordinate values
   This example sorts the variable three_dmn_rec_var(time,lat,lon)
   based on the values of the time coordinate. */

// Included in NCO User Guide at http://nco.sf.net/nco.html#sort

// Randomize the time coordinate
time=10.0*gsl_rng_uniform(time);
//print(&quot;original randomized time =\n&quot;);print(time);

/* The sort() routine is overloaded, and takes one or two arguments
   The first argument is a one dimensional array
   The second argument (optional) is the &quot;sort map&quot; (srt_map below)
   Pass the sort map by reference, i.e., prefix with an ampersand
   If the sort map does not yet exist, then it will be created and 
   returned as an integer type the same shape as the input variable.
   The output of sort(), on the LHS, is a sorted version of the input */

time=sort(time,&amp;srt_map);
//print(&quot;sorted time (ascending order) and associated sort map =\n&quot;);print(time);print(srt_map);

/* sort() always sorts in ascending order
   The associated sort map therefore re-arranges the original,
   randomized time array into ascending order.
   There are two methods to obtain the descending order the user wants
   1) We could solve the problem in ascending order (the default)
   and then apply the reverse() method to re-arrange the results.
   2) We could change the sort map to return things in descending
   order of time and solve the problem directly in descending order. */

// Following shows how to do method one:

/* Expand the sort map to srt_map_3d, the size of the data array
   1. Use data array to provide right shape for the expanded sort map
   2. Coerce data array into an integer so srt_map_3d is an integer
   3. Multiply data array by zero so 3-d map elements are all zero
   4. Add the 1-d sort map to the 3-d sort map (NCO automatically resizes)
   5. Add the spatial (lat,lon) offsets to each time index 
   6. de-sort using the srt_map_3d
   7. Use reverse to obtain descending in time order
   Loops could accomplish the same thing (exercise left for reader)
   However, loops are slow for large datasets */

/* Following index manipulation requires understanding correspondence
   between 1-d (unrolled, memory order of storage) and access into that
   memory as a multidimensional (3-d, in this case) rectangular array.
   Key idea to understand is how dimensionality affects offsets */ 
// Copy 1-d sort map into 3-d sort map
srt_map_3d=(0*int(three_dmn_rec_var))+srt_map;
// Multiply base offset by factorial of lesser dimensions
srt_map_3d*=$lat.size*$lon.size;
lon_idx=array(0,1,$lon);
lat_idx=array(0,1,$lat)*$lon.size;
lat_lon_idx[$lat,$lon]=lat_idx+lon_idx;
srt_map_3d+=lat_lon_idx;

print(&quot;sort map 3d =\n&quot;);print(srt_map_3d);

// Use remap() to re-map the data
three_dmn_rec_var=remap(three_dmn_rec_var,srt_map_3d);

// Finally, reverse data so time coordinate is descending
time=time.reverse($time);
//print(&quot;sorted time (descending order) =\n&quot;);print(time);
three_dmn_rec_var=three_dmn_rec_var.reverse($time);

// Method two: Key difference is srt_map=$time.size-srt_map-1;
EOF
ncap2 -O -v -S ~/ncap2_foo.nco ~/nco/data/in.nc ~/foo.nc
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;grd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#grd &textndash;&gt;
&lt;a name=&quot;rrg&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rrg &textndash;&gt;
&lt;a name=&quot;rct&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rct &textndash;&gt;
</html>

</subsection>
<node name="Irregular-grids" spaces=" "><nodename>Irregular grids</nodename><nodenext spaces=" ">Bilinear interpolation</nodenext><nodeprev spaces=" ">Sort methods</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Irregular Grids</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="994">irregular grids</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="995">rectangular grids</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="996">non-rectangular grids</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="997">non-standard grids</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="998">mask</indexterm></cindex>
<!-- c fxm need to edit rrg sxn beginning here -->
<para><acronym><acronymword>NCO</acronymword></acronym> is capable of analyzing datasets for many different
underlying coordinate grid types.
netCDF was developed for and initially used with grids comprised of
orthogonal dimensions forming a rectangular coordinate system.
We call such grids <emph>standard</emph> grids.
It is increasingly common for datasets to use metadata to describe
much more complex grids.
Let us first define three important coordinate grid properties:
rectangularity, regularity, and fxm.
</para>
<para>Grids are <emph>regular</emph> if the spacing between adjacent is constant. 
For example, a 4-by-5 degree latitude-longitude grid is regular
because the spacings between adjacent latitudes (<w>4 degrees</w>) are
constant as are the (<w>5 degrees</w>) spacings between adjacent
longitudes. 
Spacing in <emph>irregular</emph> grids depends on the location along the
coordinate. 
Grids such as Gaussian grids have uneven spacing in latitude (points 
cluster near the equator) and so are irregular.
</para>
<para>Grids are <emph>rectangular</emph> if the number of elements in any
dimension is not a function of any other dimension.
For example, a T42 Gaussian latitude-longitude grid is rectangular
because there are the same number of longitudes (128) for each of the 
(64) latitudes.
Grids are <emph>non-rectangular</emph> if the elements in any dimension
depend on another dimension.
Non-rectangular grids present many special challenges to 
analysis software like <acronym><acronymword>NCO</acronymword></acronym>.
</para>
<para>Wrapped coordinates (<pxref label="Wrapped-Coordinates"><xrefnodename>Wrapped Coordinates</xrefnodename></pxref>), such as longitude,
are independent of these grid properties (regularity,
rectangularity). 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="999">wrapped coordinates</indexterm></cindex>
<para>The preferred <acronym><acronymword>NCO</acronymword></acronym> technique to analyze data on non-standard
coordinate grids is to create a region mask with <command>ncap2</command>, and
then to use the mask within <command>ncap2</command> for variable-specific
processing, and/or with other operators (e.g., <command>ncwa</command>,
<command>ncdiff</command>) for entire file processing. 
</para>
<para>Before describing the construction of masks, let us review how
irregularly gridded geoscience data are described.
Say that latitude and longitude are stored as <var>R</var>-dimensional
arrays and the product of the dimension sizes is the total number of  
elements N in the other variables.
Geoscience applications tend to use <math><var>R</var>=1</math>, 
<math><var>R</var>=2</math>, and <math><var>R</var>=3</math>.
</para>
<para>If the grid is has no simple representation (e.g., discontinuous) then
it makes sense to store all coordinates as 1D arrays with the same
size as the number of grid points. 
These gridpoints can be completely independent of all the other (own
weight, area, etc.).  
</para>
<para><var>R</var>=1: lat(number_of_gridpoints) and lon(number_of_gridpoints)
</para>
<para>If the horizontal grid is time-invariant then <var>R</var>=2 is common:
</para>
<para><var>R</var>=2: lat(south_north,east_west) and lon(south_north,east_west)
</para>
<para>The Weather and Research Forecast (<acronym><acronymword>WRF</acronymword></acronym>) model uses <var>R</var>=3:
</para>
<para><var>R</var>=3: lat(time,south_north,east_west), lon(time,south_north,east_west)
</para>
<para>and so supports grids that change with time.
</para>
<para>Grids with <var>R</var> &gt; 1 often use missing values to indicated empty points.
For example, so-called &quot;staggered grids&quot; will use fewer east_west
points near the poles and more near the equator. netCDF only accepts
rectangular arrays so space must be allocated for the maximum number
of east_west points at all latitudes. Then the application writes
missing values into the unused points near the poles.
</para>
<para>We demonstrate the <command>ncap2</command> analysis technique for irregular
regions by constructing a mask for an <var>R</var>=2 grid.
We wish to find, say, the mean temperature within 
[<var>lat_min</var>,<var>lat_max</var>] and [<var>lon_min</var>,<var>lon_max</var>]: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'mask_var= (lat &gt;= lat_min &amp;&amp; lat &lt;= lat_max) &amp;&amp; \
                    (lon &gt;= lon_min &amp;&amp; lon &lt;= lon_max);' in.nc out.nc
</pre></example>
<para>Arbitrarily shaped regions can be defined by more complex conditional
statements. 
Once defined, masks can be applied to specific variables,
and to entire files:
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'temperature_avg=(temperature*mask_var).avg()' in.nc out.nc
ncwa -a lat,lon -m mask_var -w area in.nc out.nc
</pre></example>
<para>Crafting such commands on the command line is possible though unwieldy.
In such cases, a script is often cleaner and allows you to document the
procedure:
</para><example endspaces=" ">
<verbatim xml:space="preserve">
cat &gt; ncap2.in &lt;&lt; 'EOF'
mask_var = (lat &gt;= lat_min &amp;&amp; lat &lt;= lat_max) &amp;&amp; (lon &gt;= lon_min &amp;&amp; &gt; lon &lt;= lon_max);
if(mask_var.total() &gt; 0){ // Check that mask contains some valid values
  temperature_avg=(temperature*mask_var).avg(); // Average temperature
  temperature_max=(temperature*mask_var).max(); // Maximum temperature
}
EOF
ncap2 -S ncap2.in in.nc out.nc
</verbatim>
</example>

<html endspaces=" ">
&lt;a name=&quot;wrf&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#wrf &textndash;&gt;
&lt;a name=&quot;WRF&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#WRF &textndash;&gt;
</html>
<ignore>
http://foehn.colorado.edu/wrfout_to_cf/wrfout_to_cf.ncl
ncl 'file_in=&quot;wrfout.nc&quot;' 'file_out=&quot;wrfpost.nc&quot;' wrfout_to_cf.ncl
ncl 'file_in=&quot;wrfout_d02_2013-10-04_20:00:00&quot;' 'file_out=&quot;wrfout_d02_2013-10-04_20:00:00_cf.nc&quot;' wrfout_to_cf.ncl
ncl 'file_in=&quot;wrfout_v2_Lambert&quot;' 'file_out=&quot;wrfout_v2_Lambert.nc&quot;' wrfout_to_cf.ncl
</ignore>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1000"><acronym><acronymword>WRF</acronymword></acronym></indexterm></cindex>
<para>Grids like those produced by the <acronym><acronymword>WRF</acronymword></acronym> model are complex because
one must use global metadata to determine the grid staggering and
offsets to translate <code>XLAT</code> and <code>XLONG</code> into real latitudes, 
longitudes, and missing points. 
The <acronym><acronymword>WRF</acronymword></acronym> grid documentation should describe this.
For <acronym><acronymword>WRF</acronymword></acronym> files creating regional masks looks like 
</para><example endspaces=" ">
<pre xml:space="preserve">mask_var = (XLAT &gt;= lat_min &amp;&amp; XLAT &lt;= lat_max) &amp;&amp; (XLONG &gt;= lon_min &amp;&amp; XLONG &lt;= lon_max);
</pre></example>

<para>A few notes:
Irregular regions are the union of arrays lat/lon_min/max&textrsquo;s. 
The mask procedure is identical for all <var>R</var>.
<!-- c fxm need to edit rrg sxn down to here -->
</para>
<html endspaces=" ">
&lt;a name=&quot;bln_ntp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#bln_ntp &textndash;&gt;
&lt;a name=&quot;bil_int&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#bil_int &textndash;&gt;
</html>
</subsection>
<node name="Bilinear-interpolation" spaces=" "><nodename>Bilinear interpolation</nodename><nodenext spaces=" ">GSL special functions</nodenext><nodeprev spaces=" ">Irregular grids</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Bilinear interpolation</sectiontitle>
<noindent></noindent> <para>As of version 4.0.0 <acronym><acronymword>NCO</acronymword></acronym> has internal routines to
perform bilinear interpolation on gridded data sets.
In mathematics, bilinear interpolation is an extension of linear
interpolation for interpolating functions of two variables on a regular
grid. 
The idea is to perform linear interpolation first in one direction, and
then again in the other direction.
</para>
<para>Suppose we have an irregular grid of data <code>temperature[lat,lon]</code>,
with co-ordinate vars <code>lat[lat], lon[lon]</code>. 
We wish to find the temperature at an arbitary point [<var>X</var>,<var>Y</var>]
within the grid. 
If we can locate lat_min,lat_max and lon_min,lon_max such that 
<code>lat_min &lt;= X &lt;= lat_max</code> and <code>lon_min &lt;= Y &lt;= lon_max</code> 
then we can interpolate in two dimensions the temperature at
[<var>X</var>,<var>Y</var>]. 
</para>
<para>The general form of the <command>ncap2</command> interpolation function is
</para><example endspaces=" ">
<pre xml:space="preserve">var_out=bilinear_interp(grid_in,grid_out,grid_out_x,grid_out_y,grid_in_x,grid_in_y)
</pre></example>
<para>where
</para><table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">grid_in</itemformat></item>
</tableterm><tableitem><para>Input function data. 
Usually a two dimensional variable. 
It must be of size <code>grid_in_x.size()*grid_in_y.size()</code>        
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">grid_out</itemformat></item>
</tableterm><tableitem><para>This variable is the shape of <code>var_out</code>. 
Usually a two dimensional variable. 
It must be of size <code>grid_out_x.size()*grid_out_y.size()</code>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">grid_out_x</itemformat></item>
</tableterm><tableitem><para><var>X</var> output values 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">grid_out_y</itemformat></item>
</tableterm><tableitem><para><var>Y</var> output values 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">grid_in_x</itemformat></item>
</tableterm><tableitem><para><var>X</var> input values values. Must be monotonic (increasing or decreasing).
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">grid_in_y</itemformat></item>
</tableterm><tableitem><para><var>Y</var> input values values. Must be monotonic (increasing or decreasing).
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>Prior to calculations all arguments are converted to type
<code>NC_DOUBLE</code>.
After calculations <code>var_out</code> is converted to the input type of
<code>grid_in</code>. 
</para>
<para>Suppose the first part of an <command>ncap2</command> script is
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;X&quot;,4);
defdim(&quot;Y&quot;,5);

// Temperature
T_in[$X,$Y]=
 &lbrace;100, 200, 300, 400, 500,
  101, 202, 303, 404, 505,
  102, 204, 306, 408, 510,
  103, 206, 309, 412, 515.0 &rbrace;;

// Coordinate variables
x_in[$X]=&lbrace;0.0,1.0,2.0,3.01&rbrace;;
y_in[$Y]=&lbrace;1.0,2.0,3.0,4.0,5&rbrace;;
</pre></example>
<para>Now we interpolate with the following variables:
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;Xn&quot;,3);
defdim(&quot;Yn&quot;,4); 
T_out[$Xn,$Yn]=0.0;
x_out[$Xn]=&lbrace;0.0,0.02,3.01&rbrace;;
y_out[$Yn]=&lbrace;1.1,2.0,3,4&rbrace;;

var_out=bilinear_interp(T_in,T_out,x_out,y_out,x_in,y_in);
print(var_out);
// 110, 200, 300, 400,
// 110.022, 200.04, 300.06, 400.08,
// 113.3, 206, 309, 412 ;
</pre></example> 

<para>It is possible to interpolate a single point:
</para><example endspaces=" ">
<pre xml:space="preserve">var_out=bilinear_interp(T_in,0.0,3.0,4.99,x_in,y_in);
print(var_out);
// 513.920594059406
</pre></example>

<noindent></noindent> <para><strong>Wrapping and Extrapolation</strong> &linebreak;
<noindent></noindent> The function <code>bilinear_interp_wrap()</code> takes the same
arguments as <code>bilinear_interp()</code> but performs wrapping (<var>Y</var>)
and extrapolation (<var>X</var>) for points off the edge of the grid.
If the given range of longitude is say (25-335) and we have a point at
20 degrees, then the endpoints of the range are used for the
interpolation. 
This is what wrapping means.   
For wrapping to occur <var>Y</var> must be longitude and must be in the range
(0,360) or (-180,180). 
There are no restrictions on the longitude (<var>X</var>) values, though
typically these are in the range (-90,90).
This <command>ncap2</command> script illustrates both wrapping and extrapolation
of end points:
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;lat_in&quot;,6);
defdim(&quot;lon_in&quot;,5);

// Coordinate input vars
lat_in[$lat_in]=&lbrace;-80,-40,0,30,60.0,85.0&rbrace;;
lon_in[$lon_in]=&lbrace;30, 110, 190, 270, 350.0&rbrace;;

T_in[$lat_in,$lon_in]=
  &lbrace;10,40,50,30,15,   
    12,43,52,31,16,   
    14,46,54,32,17,   
    16,49,56,33,18,   
    18,52,58,34,19,   
    20,55,60,35,20.0 &rbrace;;
   
defdim(&quot;lat_out&quot;,4);
defdim(&quot;lon_out&quot;,3);

// Coordinate variables
lat_out[$lat_out]=&lbrace;-90,0,70,88.0&rbrace;;   
lon_out[$lon_out]=&lbrace;0,190,355.0&rbrace;;

T_out[$lat_out,$lon_out]=0.0;

T_out=bilinear_interp_wrap(T_in,T_out,lat_out,lon_out,lat_in,lon_in);
print(T_out); 
// 13.4375, 49.5, 14.09375,
// 16.25, 54, 16.625,
// 19.25, 58.8, 19.325,
// 20.15, 60.24, 20.135 ;
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;gsl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#gsl &textndash;&gt;
</html>
</subsection>
<node name="GSL-special-functions" spaces=" "><nodename>GSL special functions</nodename><nodenext spaces=" ">GSL interpolation</nodenext><nodeprev spaces=" ">Bilinear interpolation</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>GSL special functions</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1001"><acronym><acronymword>GSL</acronymword></acronym></indexterm></cindex>
<noindent></noindent> <para>As of version 3.9.6 (released January, 2009), <acronym><acronymword>NCO</acronymword></acronym> 
can link to the GNU Scientific Library (<acronym><acronymword>GSL</acronymword></acronym>). 
<command>ncap2</command> can access most <acronym><acronymword>GSL</acronymword></acronym> special functions including
Airy, Bessel, error, gamma, beta, hypergeometric, and Legendre functions
and elliptical integrals. 
<acronym><acronymword>GSL</acronymword></acronym> must be <w>version 1.4</w> or later. 
To list the <acronym><acronymword>GSL</acronymword></acronym> functions available with your <acronym><acronymword>NCO</acronymword></acronym> 
build, use <command>ncap2 -f | grep ^gsl</command>.
</para>
<noindent></noindent> <para>The function names used by <acronym><acronymword>ncap2</acronymword></acronym> mirror their
<acronym><acronymword>GSL</acronymword></acronym> names.
The <acronym><acronymword>NCO</acronymword></acronym> wrappers for <acronym><acronymword>GSL</acronymword></acronym> functions automatically
call the error-handling version of the <acronym><acronymword>GSL</acronymword></acronym> function when
available  
<footnote spaces="   \n"><para>These are the <acronym><acronymword>GSL</acronymword></acronym> standard function names postfixed with
<code>_e</code>.  
<acronym><acronymword>NCO</acronymword></acronym> calls these functions automatically, without the 
<acronym><acronymword>NCO</acronymword></acronym> command having to specifically indicate the <code>_e</code>
function suffix.
</para></footnote>.
This allows <acronym><acronymword>NCO</acronymword></acronym> to return a missing value when the
<acronym><acronymword>GSL</acronymword></acronym> library encounters a domain error or a floating point 
exception. 
The slow-down due to calling the error-handling version of the 
<acronym><acronymword>GSL</acronymword></acronym> numerical functions was found to be negligible (please let
us know if you find otherwise).
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1002">gamma function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1003"><var>gsl_sf_gamma</var></indexterm></cindex>
<noindent></noindent> <para>Consider the gamma function.&linebreak;
<noindent></noindent> The <acronym><acronymword>GSL</acronymword></acronym> function prototype is &linebreak;
<code>int gsl_sf_gamma_e(const double x, gsl_sf_result * result)</code>
The <command>ncap2</command> script would be:
</para><example endspaces=" ">
<pre xml:space="preserve">lon_in[lon]=&lbrace;-1,0.1,0,2,0.3&rbrace;;
lon_out=gsl_sf_gamma(lon_in);
lon_out= _, 9.5135, 4.5908, 2.9915 
</pre></example>

<noindent></noindent> <para>The first value is set to <code>_FillValue</code> since the gamma
function is undefined for negative integers.
If the input variable has a missing value then this value is used.
Otherwise, the default double fill value is used
(defined in the netCDF header <file>netcdf.h</file> as 
<code>NC_FILL_DOUBLE = 9.969e+36</code>).
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1004">Bessel function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1005"><var>gsl_sf_bessel_Jn</var></indexterm></cindex>
<noindent></noindent> <para>Consider a call to a Bessel function with <acronym><acronymword>GSL</acronymword></acronym>
prototype&linebreak; 
<code>int gsl_sf_bessel_Jn_e(int n, double x, gsl_sf_result * result)</code> 
</para>
<para>An <command>ncap2</command> script would be
</para><example endspaces=" ">
<pre xml:space="preserve">lon_out=gsl_sf_bessel_Jn(2,lon_in);  
lon_out=0.11490, 0.0012, 0.00498, 0.011165
</pre></example>
<para>This computes the Bessel function of order <var>n=2</var> for every value in
<code>lon_in</code>.
The Bessel order argument, an integer, can also be a non-scalar
variable, i.e., an array.  
</para><example endspaces=" ">
<pre xml:space="preserve">n_in[lon]=&lbrace;0,1,2,3&rbrace;;
lon_out=gsl_sf_bessel_Jn(n_in,0.5);
lon_out= 0.93846, 0.24226, 0.03060, 0.00256
</pre></example>

<noindent></noindent> <para>Arguments to <acronym><acronymword>GSL</acronymword></acronym> wrapper functions in <command>ncap2</command>
must conform to one another, i.e., they must share the same sub-set of
dimensions.  
For example: <code>three_out=gsl_sf_bessel_Jn(n_in,three_dmn_var_dbl)</code>
is valid because the variable <code>three_dmn_var_dbl</code> has a <var>lon</var> 
dimension, so <code>n_in</code> in can be broadcast to conform to
<code>three_dmn_var_dbl</code>.  
However <code>time_out=gsl_sf_bessel_Jn(n_in,time)</code> is invalid.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1006">Elliptic integrals</indexterm></cindex>
<para>Consider the elliptical integral with prototype
<code>int gsl_sf_ellint_RD_e(double x, double y, double z, gsl_mode_t mode, gsl_sf_result * result)</code>
</para><example endspaces=" ">
<pre xml:space="preserve">three_out=gsl_sf_ellint_RD(0.5,time,three_dmn_var_dbl);
</pre></example>

<noindent></noindent> <para>The three arguments are all conformable so the above <command>ncap2</command> call is valid. The mode argument in the function prototype controls the convergence of the algorithm. It also appears  in the Airy Function prototypes. It can be set by defining the environment variable <code>GSL_PREC_MODE</code>. If unset it defaults to the value <code>GSL_PREC_DOUBLE</code>. See the <acronym><acronymword>GSL</acronymword></acronym> manual for more details. 
</para><example endspaces=" ">
<pre xml:space="preserve">export GSL_PREC_MODE=0 // GSL_PREC_DOUBLE
export GSL_PREC_MODE=1 // GSL_PREC_SINGLE
export GSL_PREC_MODE=2 // GSL_PREC_APPROX
</pre></example>

<noindent></noindent> <para>The <command>ncap2</command> wrappers to the array functions are
slightly different. 
Consider the following <acronym><acronymword>GSL</acronymword></acronym> prototype &linebreak; 
<code>int gsl_sf_bessel_Jn_array(int nmin, int nmax, double x, double *result_array)</code>
</para><example endspaces=" ">
<pre xml:space="preserve">b1=lon.double();
x=0.5;
status=gsl_sf_bessel_Jn_array(1,4,x,&amp;b1);
print(status);
b1=0.24226,0.0306,0.00256,0.00016;
</pre></example>
<noindent></noindent> <para>This calculates the Bessel function of <var>x</var>=0.5 for
<var>n</var>=1 to 4. 
The first three arguments are scalar values. 
If a non-scalar variable is supplied as an argument then only the first
value is used. 
The final argument is the variable where the results are stored (NB: the
<code>&amp;</code> indicates this is a call by reference). 
This final argument must be of type <code>double</code> and must be of least
size <var>nmax-nmin+1</var>. 
If either of these conditions is not met then then the function 
returns an error message. 
The function/wrapper returns a status flag. 
Zero indicates success. 
</para>
<noindent></noindent> <para>Consider another array function &linebreak; 
<code>int gsl_sf_legendre_Pl_array( int lmax, double x, double *result_array);</code>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1007">Legendre polynomial</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="17" mergedindex="cp">gsl_sf_legendre_Pl</indexterm></findex>
</para><example endspaces=" ">
<pre xml:space="preserve">a1=time.double();
x=0.3;
status=gsl_sf_legendre_Pl_array(a1.size()-1, x,&amp;a1);  
print(status);
</pre></example>
<noindent></noindent> <para>This call calculates <var>P_l</var>(0.3) for <var>l</var>=0..9. 
Note that <var>|x|&lt;=1</var>, otherwise there will be a domain error. 
See the <acronym><acronymword>GSL</acronymword></acronym> 
documentation for more details.  
</para>
<noindent></noindent> <para>The <acronym><acronymword>GSL</acronymword></acronym> functions implemented in <acronym><acronymword>NCO</acronymword></acronym> are 
listed in the table below. 
This table is correct for <acronym><acronymword>GSL</acronymword></acronym> version 1.10. 
To see what functions are available on your build run the command
<command>ncap2 -f |grep ^gsl</command> . 
To see this table along with the <acronym><acronymword>GSL</acronymword></acronym> <w>C-function</w>
prototypes look at the spreadsheet <strong>doc/nco_gsl.ods</strong>. &linebreak; &linebreak; 
</para>
<multitable spaces=" " endspaces=" "><columnfractions line=" .35 .05 .60 "><columnfraction value=".35"></columnfraction><columnfraction value=".05"></columnfraction><columnfraction value=".60"></columnfraction></columnfractions>
<tbody><row><entry command="item" spaces=" "><para><strong>GSL NAME</strong> </para></entry><entry command="tab" spaces=" "><para><strong>I</strong> </para></entry><entry command="tab" spaces=" "><para><strong>NCAP FUNCTION CALL</strong>
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_Ai_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_Ai(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_Bi_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_Bi(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_Ai_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_Ai_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_Bi_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_Bi_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_Ai_deriv_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_Ai_deriv(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_Bi_deriv_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_Bi_deriv(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_Ai_deriv_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_Ai_deriv_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_Bi_deriv_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_Bi_deriv_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_zero_Ai_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_zero_Ai(uint_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_zero_Bi_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_zero_Bi(uint_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_zero_Ai_deriv_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_zero_Ai_deriv(uint_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_airy_zero_Bi_deriv_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_airy_zero_Bi_deriv(uint_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_J0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_J0(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_J1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_J1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Jn_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Jn(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Jn_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_bessel_Jn_array(int,int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Y0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Y0(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Y1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Y1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Yn_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Yn(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Yn_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Yn_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_I0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_I0(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_I1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_I1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_In_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_In(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_In_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_bessel_In_array(int,int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_I0_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_I0_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_I1_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_I1_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_In_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_In_scaled(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_In_scaled_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>staus=gsl_sf_bessel_In_scaled_array(int,int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_K0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_K0(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_K1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_K1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Kn_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Kn(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Kn_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_bessel_Kn_array(int,int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_K0_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_K0_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_K1_scaled_e  </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_K1_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Kn_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Kn_scaled(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Kn_scaled_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_bessel_Kn_scaled_array(int,int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_j0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_J0(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_j1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_J1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_j2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_j2(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_jl_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_jl(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_jl_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_bessel_jl_array(int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_jl_steed_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_jl_steed_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_y0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Y0(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_y1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Y1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_y2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_y2(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_yl_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_yl(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_yl_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_bessel_yl_array(int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_i0_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_I0_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_i1_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_I1_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_i2_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_i2_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_il_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_il_scaled(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_il_scaled_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_bessel_il_scaled_array(int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_k0_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_K0_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_k1_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_K1_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_k2_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_k2_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_kl_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_kl_scaled(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_kl_scaled_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_bessel_kl_scaled_array(int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Jnu_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Jnu(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Ynu_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Ynu(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_sequence_Jnu_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_sequence_Jnu
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Inu_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Inu_scaled(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Inu_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Inu(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Knu_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Knu_scaled(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_Knu_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_Knu(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_lnKnu_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_lnKnu(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_zero_J0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_zero_J0(uint_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_zero_J1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_zero_J1(uint_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_bessel_zero_Jnu_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_bessel_zero_Jnu
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_clausen_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_clausen(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hydrogenicR_1_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hydrogenicR_1
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hydrogenicR_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hydrogenicR
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coulomb_wave_FG_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coulomb_wave_FG
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coulomb_wave_F_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coulomb_wave_F_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coulomb_wave_FG_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coulomb_wave_FG_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coulomb_wave_FGp_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coulomb_wave_FGp_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coulomb_wave_sphF_array  </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coulomb_wave_sphF_array 
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coulomb_CL_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coulomb_CL
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coulomb_CL_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coulomb_CL_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coupling_3j_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coupling_3j
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coupling_6j_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coupling_6j
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coupling_RacahW_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coupling_RacahW
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coupling_9j_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coupling_9j
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_coupling_6j_INCORRECT_e </para></entry><entry command="tab" spaces=" "><para>N  </para></entry><entry command="tab" spaces=" "><para>gsl_sf_coupling_6j_INCORRECT
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_dawson_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_dawson(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_debye_1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_debye_1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_debye_2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_debye_2(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_debye_3_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_debye_3(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_debye_4_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_debye_4(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_debye_5_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_debye_5(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_debye_6_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_debye_6(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_dilog_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_dilog
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_complex_dilog_xy_e  </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_complex_dilog_xy_e 
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_complex_dilog_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_complex_dilog
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_complex_spence_xy_e   </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_complex_spence_xy_e  
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_multiply_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_multiply
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_multiply_err_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_multiply_err
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_Kcomp_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_Kcomp(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_Ecomp_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_Ecomp(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_Pcomp_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_Pcomp(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_Dcomp_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_Dcomp(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_F_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_F(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_E_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_E(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_P_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_P(dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_D_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_D(dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_RC_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_RC(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_RD_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_RD(dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_RF_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_RF(dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_ellint_RJ_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_ellint_RJ(dbl_expr,dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_elljac_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_elljac
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_erfc_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_erfc(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_log_erfc_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_log_erfc(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_erf_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_erf(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_erf_Z_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_erf_Z(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_erf_Q_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_erf_Q(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hazard_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hazard(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exp_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exp(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exp_e10_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exp_e10
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exp_mult_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exp_mult(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exp_mult_e10_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exp_mult_e10
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_expm1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_expm1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exprel_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exprel(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exprel_2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exprel_2(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exprel_n_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exprel_n(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exp_err_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exp_err(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exp_err_e10_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exp_err_e10
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exp_mult_err_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exp_mult_err
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_exp_mult_err_e10_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_exp_mult_err_e10
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_expint_E1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_expint_E1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_expint_E2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_expint_E2(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_expint_En_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_expint_En(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_expint_E1_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_expint_E1_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_expint_E2_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_expint_E2_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_expint_En_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_expint_En_scaled(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_expint_Ei_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_expint_Ei(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_expint_Ei_scaled_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_expint_Ei_scaled(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_Shi_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_Shi(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_Chi_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_Chi(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_expint_3_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_expint_3(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_Si_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_Si(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_Ci_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_Ci(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_atanint_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_atanint(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_fermi_dirac_m1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_fermi_dirac_m1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_fermi_dirac_0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_fermi_dirac_0(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_fermi_dirac_1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_fermi_dirac_1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_fermi_dirac_2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_fermi_dirac_2(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_fermi_dirac_int_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_fermi_dirac_int(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_fermi_dirac_mhalf_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_fermi_dirac_mhalf(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_fermi_dirac_half_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_fermi_dirac_half(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_fermi_dirac_3half_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_fermi_dirac_3half(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_fermi_dirac_inc_0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_fermi_dirac_inc_0(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lngamma_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lngamma(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lngamma_sgn_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lngamma_sgn 
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gamma_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gamma(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gammastar_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gammastar(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gammainv_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gammainv(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lngamma_complex_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lngamma_complex
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_taylorcoeff_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_taylorcoeff(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_fact_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_fact(uint_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_doublefact_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_doublefact(uint_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lnfact_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lnfact(uint_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lndoublefact_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lndoublefact(uint_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lnchoose_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lnchoose
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_choose_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_choose
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lnpoch_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lnpoch(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lnpoch_sgn_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lnpoch_sgn
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_poch_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_poch(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_pochrel_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_pochrel(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gamma_inc_Q_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gamma_inc_Q(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gamma_inc_P_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gamma_inc_P(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gamma_inc_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gamma_inc(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lnbeta_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lnbeta(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lnbeta_sgn_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lnbeta_sgn
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_beta_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_beta(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_beta_inc_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_beta_inc
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gegenpoly_1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gegenpoly_1(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gegenpoly_2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gegenpoly_2(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gegenpoly_3_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gegenpoly_3(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gegenpoly_n_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gegenpoly_n
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_gegenpoly_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_gegenpoly_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_0F1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_0F1(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_1F1_int_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_1F1_int(int_expr,int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_1F1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_1F1(dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_U_int_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_U_int(int_expr,int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_U_int_e10_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_U_int_e10
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_U_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_U(dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_U_e10_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_U_e10
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_2F1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_2F1(dbl_expr,dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_2F1_conj_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_2F1_conj(dbl_expr,dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_2F1_renorm_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_2F1_renorm(dbl_expr,dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_2F1_conj_renorm_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_2F1_conj_renorm(dbl_expr,dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hyperg_2F0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hyperg_2F0(dbl_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_laguerre_1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_laguerre_1(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_laguerre_2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_laguerre_2(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_laguerre_3_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_laguerre_3(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_laguerre_n_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_laguerre_n(int_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lambert_W0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lambert_W0(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lambert_Wm1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lambert_Wm1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_Pl_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_Pl(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_Pl_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_legendre_Pl_array(int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_Pl_deriv_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_Pl_deriv_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_P1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_P1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_P2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_P2(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_P3_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_P3(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_Q0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_Q0(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_Q1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_Q1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_Ql_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_Ql(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_Plm_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_Plm(int_expr,int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_Plm_array  </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_legendre_Plm_array(int,int,double,&amp;var_out) 
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_Plm_deriv_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_Plm_deriv_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_sphPlm_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_sphPlm(int_expr,int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_sphPlm_array </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>status=gsl_sf_legendre_sphPlm_array(int,int,double,&amp;var_out)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_sphPlm_deriv_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_sphPlm_deriv_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_array_size </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_array_size
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_conicalP_half_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_conicalP_half(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_conicalP_mhalf_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_conicalP_mhalf(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_conicalP_0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_conicalP_0(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_conicalP_1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_conicalP_1(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_conicalP_sph_reg_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_conicalP_sph_reg(int_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_conicalP_cyl_reg_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_conicalP_cyl_reg(int_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_H3d_0_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_H3d_0(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_H3d_1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_H3d_1(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_H3d_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_H3d(int_expr,dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_H3d_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_H3d_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_legendre_array_size </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_legendre_array_size
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_log_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_log(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_log_abs_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_log_abs(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_complex_log_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_complex_log
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_log_1plusx_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_log_1plusx(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_log_1plusx_mx_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_log_1plusx_mx(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_a_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_a_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_b_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_b_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_a </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_a
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_b </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_b
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_a_coeff </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_a_coeff
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_b_coeff </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_b_coeff
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_ce </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_ce
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_se </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_se
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_ce_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_ce_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_se_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_se_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_Mc </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_Mc
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_Ms </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_Ms
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_Mc_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_Mc_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_mathieu_Ms_array </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_mathieu_Ms_array
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_pow_int_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_pow_int
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_psi_int_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_psi_int(int_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_psi_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_psi(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_psi_1piy_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_psi_1piy(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_complex_psi_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_complex_psi
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_psi_1_int_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_psi_1_int(int_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_psi_1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_psi_1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_psi_n_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_psi_n(int_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_synchrotron_1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_synchrotron_1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_synchrotron_2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_synchrotron_2(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_transport_2_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_transport_2(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_transport_3_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_transport_3(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_transport_4_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_transport_4(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_transport_5_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_transport_5(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_sin_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_sin
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_cos_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_cos
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hypot_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hypot
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_complex_sin_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_complex_sin
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_complex_cos_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_complex_cos
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_complex_logsin_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_complex_logsin
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_sinc_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_sinc
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lnsinh_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lnsinh
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_lncosh_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_lncosh
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_polar_to_rect </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_polar_to_rect
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_rect_to_polar </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_rect_to_polar
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_sin_err_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_sin_err
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_cos_err_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_cos_err
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_angle_restrict_symm_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_angle_restrict_symm
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_angle_restrict_pos_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_angle_restrict_pos
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_angle_restrict_symm_err_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_angle_restrict_symm_err
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_angle_restrict_pos_err_e </para></entry><entry command="tab" spaces=" "><para>N </para></entry><entry command="tab" spaces=" "><para>gsl_sf_angle_restrict_pos_err
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_zeta_int_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_zeta_int(int_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_zeta_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_zeta(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_zetam1_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_zetam1(dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_zetam1_int_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_zetam1_int(int_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_hzeta_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_hzeta(dbl_expr,dbl_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_eta_int_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_eta_int(int_expr)
</para></entry></row><row><entry command="item" spaces=" "><para>gsl_sf_eta_e </para></entry><entry command="tab" spaces=" "><para>Y </para></entry><entry command="tab" spaces=" "><para>gsl_sf_eta(dbl_expr)
</para></entry></row></tbody></multitable>

<html endspaces=" ">
&lt;a name=&quot;gsl_int&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#gsl_int &textndash;&gt;
</html>
</subsection>
<node name="GSL-interpolation" spaces=" "><nodename>GSL interpolation</nodename><nodenext spaces=" ">GSL least-squares fitting</nodenext><nodeprev spaces=" ">GSL special functions</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>GSL interpolation</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1008"><acronym><acronymword>GSL</acronymword></acronym></indexterm></cindex>
<noindent></noindent> <para>As of version 3.9.9 (released July, 2009), <acronym><acronymword>NCO</acronymword></acronym> has wrappers to the <acronym><acronymword>GSL</acronymword></acronym> interpolation functions.
</para>
<noindent></noindent>  <para>Given a set of data points (x1,y1)...(xn, yn) the <acronym><acronymword>GSL</acronymword></acronym> functions computes a continuous interpolating function <acronym><acronymword>Y(x)</acronymword></acronym> such that <acronym><acronymword>Y(xi) = yi</acronymword></acronym>. The interpolation is piecewise smooth, and its behavior at the end-points is determined by the type of interpolation used. For more information consult the <acronym><acronymword>GSL</acronymword></acronym> manual. 
</para>
<noindent></noindent> <para>Interpolation with <command>ncap2</command> is a two stage process. In the first stage, a <acronym><acronymword>RAM</acronymword></acronym> variable is created from the chosen interpolating function and the data set. This <acronym><acronymword>RAM</acronymword></acronym> variable holds in memory a <acronym><acronymword>GSL</acronymword></acronym> interpolation object. In the second stage, points along the interpolating function are calculated. If you have a very large data set or are interpolating many sets then consider deleting the <acronym><acronymword>RAM</acronymword></acronym> variable when it is redundant. Use the command <command>ram_delete(var_nm)</command>.   
</para>
<noindent></noindent> <para>A simple example
</para>
<example endspaces=" ">
<pre xml:space="preserve">x_in[$lon]=&lbrace;1.0,2.0,3.0,4.0&rbrace;;
y_in[$lon]=&lbrace;1.1,1.2,1.5,1.8&rbrace;;

// Ram variable is declared and defined here 
gsl_interp_cspline(&amp;ram_sp,x_in,y_in);

x_out[$lon_grd]=&lbrace;1.1,2.0,3.0,3.1,3.99&rbrace;;

y_out=gsl_spline_eval(ram_sp,x_out);
y2=gsl_spline_eval(ram_sp,1.3);
y3=gsl_spline_eval(ram_sp,0.0);
ram_delete(ram_sp);

print(y_out); // 1.10472, 1.2, 1.4, 1.42658, 1.69680002 
print(y2);    // 1.12454 
print(y3);    // '_' 
</pre></example>

<noindent></noindent> <para>Note in the above example y3 is set to &textrsquo;missing value&textrsquo; because 0.0 isn&textrsquo;t within the input X range.  
</para>
<para><strong><acronym><acronymword>GSL</acronymword></acronym> Interpolation Types</strong>&linebreak;
<noindent></noindent> All the interpolation functions have been implemented. These are:&linebreak;
gsl_interp_linear() &linebreak; gsl_interp_polynomial() &linebreak; gsl_interp_cspline()&linebreak;
gsl_interp_cspline_periodic()&linebreak; gsl_interp_akima() &linebreak; gsl_interp_akima_periodic() &linebreak;
</para>
&linebreak; &linebreak;
<para><strong> Evaluation of Interpolating Types </strong> &linebreak;
<noindent></noindent> <strong>Implemented</strong> &linebreak;
gsl_spline_eval() &linebreak;
<noindent></noindent> <strong>Unimplemented</strong> &linebreak;
gsl_spline_deriv()&linebreak;
gsl_spline_deriv2()&linebreak;
gsl_spline_integ()&linebreak;
</para>
<html endspaces=" ">
&lt;a name=&quot;ncap_lsqf&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_lsqf &textndash;&gt;
</html>
</subsection>
<node name="GSL-least_002dsquares-fitting" spaces=" "><nodename>GSL least-squares fitting</nodename><nodenext spaces=" ">GSL statistics</nodenext><nodeprev spaces=" ">GSL interpolation</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces="  "><sectiontitle>GSL least-squares fitting       </sectiontitle>
<noindent></noindent> <para>Least Squares fitting is a method of calculating a straight line through a set of experimental data points in the XY plane. The data maybe weighted or unweighted. For more information please refer to the <acronym><acronymword>GSL</acronymword></acronym> manual.
</para>
<noindent></noindent> <para>These <acronym><acronymword>GSL</acronymword></acronym> functions fall into three categories:&linebreak;
<strong>A)</strong> Fitting data to Y=c0+c1*X&linebreak;
<strong>B)</strong> Fitting data (through the origin) Y=c1*X&linebreak;
<strong>C)</strong> Multi-parameter fitting (not yet implemented)&linebreak; 
</para>
<para><strong>Section A</strong> &linebreak; &linebreak;
<code>status=<strong>gsl_fit_linear</strong> (data_x,stride_x,data_y,stride_y,n,&amp;co,&amp;c1,&amp;cov00,&amp;cov01,&amp;cov11,&amp;sumsq) </code>
</para>
<noindent></noindent> <para><strong>Input variables</strong>: data_x, stride_x, data_y, stride_y, n &linebreak; 
From the above variables an X and Y vector both of length &textrsquo;n&textrsquo; are derived. 
If data_x or data_y is less than type double then it is converted to type <code>double</code>. 
It is up to you to do bounds checking on the input data. 
For example if stride_x=3 and n=8 then the size of data_x must be at least 24
</para>
<noindent></noindent> <para><strong>Output variables</strong>: c0, c1, cov00, cov01, cov11,sumsq &linebreak; 
The &textrsquo;&amp;&textrsquo; prefix indicates that these are call-by-reference variables.
If any of the output variables don&textrsquo;t exist prior to the call then they are created on the fly as scalar variables of type <code>double</code>. If they already exist then their existing value is overwritten. If the function call is successful then <code>status=0</code>. 
</para>
<para><code>status= <strong>gsl_fit_wlinear</strong>(data_x,stride_x,data_w,stride_w,data_y,stride_y,n,&amp;co,&amp;c1,&amp;cov00,&amp;cov01,&amp;cov11,&amp;chisq) </code>
</para>
<noindent></noindent> <para>Similar to the above call except it creates an additional weighting vector from the variables data_w, stride_w, n  
</para>

<para><code> data_y_out=<strong>gsl_fit_linear_est</strong>(data_x,c0,c1,cov00,cov01,cov11) </code>
</para>
<noindent></noindent> <para>This function calculates y values along the line Y=c0+c1*X &linebreak; &linebreak;
</para>


<para><strong>Section B</strong> &linebreak; &linebreak;
<code>status=<strong>gsl_fit_mul</strong>(data_x,stride_x,data_y,stride_y,n,&amp;c1,&amp;cov11,&amp;sumsq) </code>
</para>
<noindent></noindent> <para><strong>Input variables</strong>: data_x, stride_x, data_y, stride_y, n &linebreak; 
From the above variables an X and Y vector both of length &textrsquo;n&textrsquo; are derived. 
If data_x or data_y is less than type <code>double</code> then it is converted to type <code>double</code>. &linebreak;
</para>
<noindent></noindent> <para><strong>Output variables</strong>: c1,cov11,sumsq &linebreak; 
</para>

<para><code>status= <strong>gsl_fit_wmul</strong>(data_x,stride_x,data_w,stride_w,data_y,stride_y,n,&amp;c1,&amp;cov11,&amp;sumsq) </code>
</para>
<noindent></noindent> <para>Similar to the above call except it creates an additional weighting vector from the variables data_w, stride_w, n  
</para>

<para><code> data_y_out=<strong>gsl_fit_mul_est</strong>(data_x,c0,c1,cov11) </code>
</para>
<noindent></noindent> <para>This function calculates y values along the line Y=c1*X &linebreak; &linebreak;
</para>
<noindent></noindent> <para>The below example shows <strong>gsl_fit_linear()</strong> in action
</para>
<example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;d1&quot;,10);
xin[d1]=&lbrace;1,2,3,4,5,6,7,8,9,10.0&rbrace;;
yin[d1]=&lbrace;3.1,6.2,9.1,12.2,15.1,18.2,21.3,24.0,27.0,30.0&rbrace;;
gsl_fit_linear(xin,1,yin,1,$d1.size,&amp;c0,&amp;c1,&amp;cov00,&amp;cov01,&amp;cov11,&amp;sumsq);
print(c0);  // 0.2
print(c1);  // 2.98545454545


defdim(&quot;e1&quot;,4);
xout[e1]=&lbrace;1.0,3.0,4.0,11&rbrace;;
yout[e1]=0.0;

yout=gsl_fit_linear_est(xout, c0,c1, cov00,cov01, cov11, sumsq);

print(yout);  // 3.18545454545 ,9.15636363636, ,12.1418181818 ,33.04
</pre></example>
&linebreak; &linebreak;



<html endspaces=" ">
&lt;a name=&quot;ncap_stat&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_stat &textndash;&gt;
</html>

</subsection>
<node name="GSL-statistics" spaces=" "><nodename>GSL statistics</nodename><nodenext spaces=" ">GSL random number generation</nodenext><nodeprev spaces=" ">GSL least-squares fitting</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>GSL statistics</sectiontitle>

<noindent></noindent> <para>Wrappers for most of the <acronym><acronymword>GSL</acronymword></acronym> Statistical functions have been implemented. The <acronym><acronymword>GSL</acronymword></acronym> function names include a type specifier (except for type double functions). To obtain the equivalent <acronym><acronymword>NCO</acronymword></acronym> name simply remove the type specifier; then depending on the data type the appropriate <acronym><acronymword>GSL</acronymword></acronym> function  is called. The weighed statistical functions e.g., <code> gsl_stats_wvariance()</code> are only defined in <acronym><acronymword>GSL</acronymword></acronym> for floating point types; so your data must of type <code>float</code> or <code>double</code> otherwise ncap2 will emit an error message. To view the implemented functions use the shell command <command>ncap2 -f|grep _stats</command>   
</para>
<noindent></noindent> <para><acronym><acronymword>GSL</acronymword></acronym> Functions
</para><example endspaces=" ">
<pre xml:space="preserve">short gsl_stats_max (short data[], size_t stride, size_t n);
double gsl_stats_int_mean (int data[], size_t stride, size_t n);
double gsl_stats_short_sd_with_fixed_mean (short data[], size_t stride, size_t n, double mean);
double gsl_stats_wmean (double w[], size_t wstride, double data[], size_t stride, size_t n);
double gsl_stats_quantile_from_sorted_data (double sorted_data[], size_t stride, size_t n, double f) ;
</pre></example>


<noindent></noindent> <para>Equivalent ncap2 wrapper functions
</para><example endspaces=" ">
<pre xml:space="preserve">short gsl_stats_max (var_data, data_stride, n);
double gsl_stats_mean (var_data, data_stride, n);
double gsl_stats_sd_with_fixed_mean (var_data, data_stride, n, var_mean);
double gsl_stats_wmean (var_weight, weight_stride, var_data, data_stride, n, var_mean);
double gsl_stats_quantile_from_sorted_data (var_sorted_data, data_stride, n, var_f) ;
</pre></example>

<noindent></noindent> <para><acronym><acronymword>GSL</acronymword></acronym> has no notion of missing values or dimensionality beyond one. If your data has missing values which you want ignored in the calculations then use the <command>ncap2</command> built in aggregate functions( <ref label="Methods-and-functions"><xrefnodename>Methods and functions</xrefnodename></ref> ). The <acronym><acronymword>GSL</acronymword></acronym> functions operate on a vector of values created from the var_data/stride/n arguments. The ncap wrappers check that there is no bounding error with regard to the size of the data and the final value in the vector. 
</para>
<para>Some examples
</para>
<example endspaces=" ">
<pre xml:space="preserve">a1[time]=&lbrace;1,2,3,4,5,6,7,8,9,10 &rbrace;;

a1_avg=gsl_stats_mean(a1,1,10);
print(a1_avg); // 5.5

a1_var=gsl_stats_variance(a1,4,3);
print(a1_var); // 16.0

// bounding error, vector attempts to access element a1(10)
a1_sd=gsl_stats_sd(a1,5,3); 

</pre></example>

<noindent></noindent> <para>For functions with the signature 
<strong>func_nm(var_data,data_stride,n)</strong>, 
one may omit the second or third arguments. 
The default value for <var>stride</var> is <code>1</code>. 
The default value for <var>n</var> is <code>1+(data.size()-1)/stride</code>.
</para>
<example endspaces=" ">
<pre xml:space="preserve">// Following statements are equvalent
n2=gsl_stats_max(a1,1,10)
n2=gsl_stats_max(a1,1);
n2=gsl_stats_max(a1);

// Following statements are equvalent
n3=gsl_stats_median_from_sorted_data(a1,2,5);
n3=gsl_stats_median_from_sorted_data(a1,2);

// Following statements are NOT equvalent
n4=gsl_stats_kurtosis(a1,3,2);
n4=gsl_stats_kurtosis(a1,3); //default n=4
</pre></example> 

<para>The following example illustrates some of the weighted functions.
The data are randomly generated. 
In this case the value of the weight for each datum is either 0.0 or 1.0   
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;r1&quot;,2000);
data[r1]=1.0;

// Fill with random numbers [0.0,10.0)
data=10.0*gsl_rng_uniform(data);

// Create a weighting variable
weight=(data&gt;4.0);

wmean=gsl_stats_wmean(weight,1,data,1,$r1.size);
print(wmean);

wsd=gsl_stats_wsd(weight,1,data,1,$r1.size);
print(wsd);

// number of values in data that are greater than 4
weight_size=weight.total();
print(weight_size);

// print min/max of data 
dmin=data.gsl_stats_min();
dmax=data.gsl_stats_max();
print(dmin);print(dmax);
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_rng&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_rng &textndash;&gt;
</html>

</subsection>
<node name="GSL-random-number-generation" spaces=" "><nodename>GSL random number generation</nodename><nodenext spaces=" ">Examples ncap2</nodenext><nodeprev spaces=" ">GSL statistics</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>GSL random number generation</sectiontitle>
<para>The <acronym><acronymword>GSL</acronymword></acronym> library has a large number of random number generators. In addition there are a large set of functions for turning uniform random numbers into discrete or continuous probabilty distributions. The random number generator algorithms vary in terms of quality numbers output, speed of execution and maximium number output. For more information see the <acronym><acronymword>GSL</acronymword></acronym> documentation. The algorithm and seed are set via environment variables, these are picked up by the <code>ncap2</code> code.
</para>
<noindent></noindent> <para><strong>Setup</strong> &linebreak;  
The number algorithm is set by the environment variable <code>GSL_RNG_TYPE</code>. If this variable isn&textrsquo;t set then the default rng algorithm is gsl_rng_19937. The seed is set with the environment variable <code>GSL_RNG_SEED</code>. The following wrapper functions in ncap2 provide information about the chosen algorithm. &linebreak; 
</para>
<table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">gsl_rng_min() </itemformat></item>
</tableterm><tableitem><para>the minimium value returned by the rng algorithm.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">gsl_rng_max()</itemformat></item>
</tableterm><tableitem><para>the maximium value returned by the rng algorithm. 
</para></tableitem></tableentry></table>

<noindent></noindent> <para><strong>Uniformly Distributed Random Numbers</strong> 
</para><table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">gsl_rng_get(var_in)</itemformat></item>
</tableterm><tableitem><para>This function returns var_in with integers from the chosen rng algorithm. The min and max values depend uoon the chosen rng algorthm.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">gsl_rng_uniform_int(var_in)</itemformat></item>
</tableterm><tableitem><para>This function returns var_in with random integers from 0 to n-1. The value n must be less than or equal to the maximium value of the chosen rng algorithm. 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">gsl_rng_uniform(var_in)</itemformat></item>
</tableterm><tableitem><para>This function returns var_in with double-precision numbers in the range [0.0,1). The range includes 0.0 and excludes 1.0.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">gsl_rng_uniform_pos(var_in)</itemformat></item>
</tableterm><tableitem><para>This function returns var_in with double-precision numbers in the range (0.0,1), excluding both 0.0 and 1.0.
</para></tableitem></tableentry></table>

<noindent></noindent> <para>Below are examples of <code>gsl_rng_get()</code> and <code>gsl_rng_uniform_int()</code> in action.
</para>
<example endspaces=" ">
<pre xml:space="preserve">export GSL_RNG_TYPE=ranlux
export GSL_RNG_SEED=10
ncap2 -v -O -s 'a1[time]=0;a2=gsl_rng_get(a1);' in.nc foo.nc 
// 10 random numbers from the range 0 - 16777215
// a2=9056646, 12776696, 1011656, 13354708, 5139066, 1388751, 11163902, 7730127, 15531355, 10387694 ;

ncap2 -v -O -s 'a1[time]=21;a2=gsl_rng_uniform_int(a1).sort();' in.nc foo.nc
// 10 random numbers from the range 0 - 20
a2 = 1, 1, 6, 9, 11, 13, 13, 15, 16, 19 ;

</pre></example>

<noindent></noindent> <para>The following example produces an <code>ncap2</code> runtime error. This is because the chose rng algorithm has a maximium value greater than <code> NC_MAX_INT=2147483647 </code>; the wrapper functions to <code>gsl_rng_get()</code> and <code>gsl_rng_uniform_int()</code> return variable of type <code>NC_INT</code>. Please be aware of this when using random number distribution functions functions from the <acronym><acronymword>GSL</acronymword></acronym> library which return <code>unsigned int</code>. Examples of these are <code>gsl_ran_geometric()</code> and <code>gsl_ran_pascal()</code>. 
</para>
<example endspaces=" ">
<pre xml:space="preserve">export GSL_RNG_TYPE=mt19937
ncap2 -v -O -s 'a1[time]=0;a2=gsl_rng_get(a1);' in.nc foo.nc 
</pre></example>

<noindent></noindent> <para>To find the maximium value of the chosen rng algorithm use the following code snippet.
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -v -O -s 'rng_max=gsl_rng_max();print(rng_max)' in.nc foo.nc
</pre></example>

<noindent></noindent> <para><strong>Random Number Distributions</strong> &linebreak;
The <acronym><acronymword>GSL</acronymword></acronym> library has a rich set of random number disribution functions. The library also provides cumulative distribution functions and inverse cumulative distribution functions sometimes referred to a quantile functions. To see whats available on your build use the shell command <code>ncap2 -f|grep -e _ran -e _cdf</code>.
</para>
<noindent></noindent> <para>The following examples all return variables of type <code>NC_INT</code> &linebreak;
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;out&quot;,15);
a1[$out]=0.5;
a2=gsl_ran_binomial(a1,30).sort();
//a2 = 10, 11, 12, 12, 13, 14, 14, 15, 15, 16, 16, 16, 16, 17, 22 ;
a3=gsl_ran_geometric(a2).sort();
//a2 = 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 4, 5 ;
a4=gsl_ran_pascal(a2,50);
//a5 = 37, 40, 40, 42, 43, 45, 46, 49, 52, 58, 60, 62, 62, 65, 67 ;
</pre></example>

<noindent></noindent> <para>The following all return variables of type <code>NC_DOUBLE</code>;
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;b1&quot;,1000);
b1[$b1]=0.8;
b2=gsl_ran_exponential(b1);
b2_avg=b2.avg();
print(b2_avg);
// b2_avg = 0.756047976787

b3=gsl_ran_gaussian(b1);
b3_avg=b3.avg();
b3_rms=b3.rms();
print(b3_avg);
// b3_avg = -0.00903446534258;
print(b3_rms);
// b3_rms = 0.81162979889;

b4[$b1]=10.0;
b5[$b1]=20.0;
b6=gsl_ran_flat(b4,b5);
b6_avg=b6.avg();
print(b6_avg);
// b6_avg=15.0588129413
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ncap_emp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_emp &textndash;&gt;
</html>
</subsection>
<node name="Examples-ncap2" spaces=" "><nodename>Examples ncap2</nodename><nodenext spaces=" ">Intrinsic mathematical methods</nodenext><nodeprev spaces=" ">GSL random number generation</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Examples ncap2</sectiontitle>

<para>See the <file>ncap.in</file> and <file>ncap2.in</file> scripts released with <acronym><acronymword>NCO</acronymword></acronym> 
for more complete demonstrations of <command>ncap2</command> functionality
(script available on-line at <url><urefurl>http://nco.sf.net/ncap2.in</urefurl></url>).
</para>
<para>Define new attribute <var>new</var> for existing variable <var>one</var>
as twice the existing attribute <var>double_att</var> of variable
<var>att_var</var>: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'one&arobase;new=2*att_var&arobase;double_att' in.nc out.nc
</pre></example>

<para>Average variables of mixed types (result is of type <code>double</code>):
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'average=(var_float+var_double+var_int)/3' in.nc out.nc 
</pre></example>

<para>Multiple commands may be given to <command>ncap2</command> in three ways.
First, the commands may be placed in a script which is executed, e.g.,
<file>tst.nco</file>. 
Second, the commands may be individually specified with multiple
<samp>-s</samp> arguments to the same <command>ncap2</command> invocation.
Third, the commands may be chained into a single <samp>-s</samp>
argument to <command>ncap2</command>.
Assuming the file <file>tst.nco</file> contains the commands
<code>a=3;b=4;c=sqrt(a^2+b^2);</code>, then the following <command>ncap2</command>
invocations produce identical results:
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -v -S tst.nco in.nc out.nc
ncap2 -v -s 'a=3' -s 'b=4' -s 'c=sqrt(a^2+b^2)' in.nc out.nc
ncap2 -v -s 'a=3;b=4;c=sqrt(a^2+b^2)' in.nc out.nc
</pre></example>
<para>The second and third examples show that <command>ncap2</command> does not require
that a trailing semi-colon <samp>;</samp> be placed at the end of a <samp>-s</samp>
argument, although a trailing semi-colon <samp>;</samp> is always allowed.
However, semi-colons are required to separate individual assignment
statements chained together as a single <samp>-s</samp> argument. 
</para>
<html endspaces=" ">
&lt;a name=&quot;xmp_grw&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_grw &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1009">growing dimensions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1010">dimensions, growing</indexterm></cindex>
<para><command>ncap2</command> may be used to &textldquo;grow&textrdquo; dimensions, i.e., to increase
dimension sizes without altering existing data.
Say <file>in.nc</file> has <code>ORO(lat,lon)</code> and the user wishes a new
file with <code>new_ORO(new_lat,new_lon)</code> that contains zeros in the
undefined portions of the new grid.
</para><example endspaces=" ">
<pre xml:space="preserve">defdim(&quot;new_lat&quot;,$lat.size+1); // Define new dimension sizes
defdim(&quot;new_lon&quot;,$lon.size+1);
new_ORO[$new_lat,$new_lon]=0.0f; // Initialize to zero
new_ORO(0:$lat.size-1,0:$lon.size-1)=ORO; // Fill valid data
</pre></example>
<para>The commands to define new coordinate variables <code>new_lat</code>
and <code>new_lon</code> in the output file follow a similar pattern.
One would might store these commands in a script <file>grow.nco</file>
and then execute the script with
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -v -S grow.nco in.nc out.nc
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;flg&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#flg &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1011">flags</indexterm></cindex>
<para>Imagine you wish to create a binary flag based on the value of 
an array.
The flag should have <w>value 1.0</w> where the array <w>exceeds 1.0</w>,
and <w>value 0.0</w> elsewhere.
This example creates the binary flag <code>ORO_flg</code> in <file>out.nc</file>
from the continuous array named <code>ORO</code> in <file>in.nc</file>.
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'ORO_flg=(ORO &gt; 1.0)' in.nc out.nc
</pre></example>
<para>Suppose your task is to change all values of <code>ORO</code> which 
<w>equal 2.0</w> to the new <w>value 3.0</w>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'ORO_msk=(ORO==2.0);ORO=ORO_msk*3.0+!ORO_msk*ORO' in.nc out.nc
</pre></example>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1012">mask</indexterm></cindex>
<para>This creates and uses <code>ORO_msk</code> to mask the subsequent arithmetic
operation.
Values of <code>ORO</code> are only changed where <code>ORO_msk</code> is true,
i.e., where <code>ORO</code> <w>equals 2.0</w> &linebreak;
Using the <code>where</code> statement the above code simplifies to :
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'where(ORO==2.0) ORO=3.0;' in.nc foo.nc
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;cvr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cvr &textndash;&gt;
</html>
<para>This example uses <command>ncap2</command> to compute the covariance of two
variables. 
Let the variables <var>u</var> and <var>v</var> be the horizontal 
wind components. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1013">covariance</indexterm></cindex>
<!-- c fxm 20030423: texi2html 1.64 has problems with this legal syntax but makeinfo -html does not -->
The <dfn>covariance</dfn> of <var>u</var> and <var>v</var> is defined
as the time mean product of the deviations of <var>u</var> and
<var>v</var> from their respective time means.
Symbolically, the covariance 
<set name="flg" line=" flg"></set>
<tex endspaces=" ">
$[\wndznl^{\prime} \wndmrd^{\prime}] =
[\wndznl\wndmrd]-[\wndznl][\wndmrd]$ where $[\xxx]$ denotes the
time-average of~$\xxx$, 
$[\xxx] \equiv {1 \over \tau} \int_{\tm=0}^{\tm=\tau} \xxx(\tm) \,\dfr\tm$
and $\xxx^{\prime}$ 
<clear name="flg" line=" flg "></clear>
</tex>
<math>[<var>u'v'</var>] =
[<var>uv</var>]-[<var>u</var>][<var>v</var>]</math> 
where <math>[<var>x</var>]</math> denotes the time-average of
<math><var>x</var></math> and <math><var>x'</var></math> 
<clear name="flg" line=" flg"></clear>
denotes the deviation from the time-mean. 
The covariance tells us how much of the correlation of two signals
arises from the signal fluctuations versus the mean signals.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1014">eddy covariance</indexterm></cindex>
Sometimes this is called the <dfn>eddy covariance</dfn>.
We will store the covariance in the variable <code>uprmvprm</code>.
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -O -a time -v u,v in.nc foo.nc # Compute time mean of u,v
ncrename -O -v u,uavg -v v,vavg foo.nc # Rename to avoid conflict
ncks -A -v uavg,vavg foo.nc in.nc # Place time means with originals
ncap2 -O -s 'uprmvprm=u*v-uavg*vavg' in.nc in.nc # Covariance
ncra -O -v uprmvprm in.nc foo.nc # Time-mean covariance
</pre></example>
<para>The mathematically inclined will note that the same covariance would be
obtained by replacing the step involving <command>ncap2</command> with
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -O -s 'uprmvprm=(u-uavg)*(v-vavg)' foo.nc foo.nc # Covariance
</pre></example>

<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 3.1.8 (December, 2006), <command>ncap2</command>
can compute averages, and thus covariances, by itself:
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'uavg=u.avg($time);vavg=v.avg($time);uprmvprm=u*v-uavg*vavg' \
      -s 'uprmvrpmavg=uprmvprm.avg($time)' in.nc foo.nc
</pre></example>
<para>We have not seen a simpler method to script and execute powerful
arithmetic than <command>ncap2</command>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1015">globbing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1016">shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1017">quotes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1018">extended regular expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1019">regular expressions</indexterm></cindex>
<para><command>ncap2</command> utilizes many meta-characters 
(e.g., <samp>$</samp>, <samp>?</samp>, <samp>;</samp>, <samp>()</samp>, <samp>[]</samp>)
that can confuse the command-line shell if not quoted properly.
The issues are the same as those which arise in utilizing extended
regular expressions to subset variables (<pxref label="Subsetting-Files"><xrefnodename>Subsetting Files</xrefnodename></pxref>).
The example above will fail with no quotes and with double quotes.
This is because shell globbing tries to <dfn>interpolate</dfn> the value of
<code>$time</code> from the shell environment unless it is quoted:
</para><example endspaces=" ">
<pre xml:space="preserve">ncap2 -s 'uavg=u.avg($time)'  in.nc foo.nc # Correct (recommended)
ncap2 -s  uavg=u.avg('$time') in.nc foo.nc # Correct (and dangerous)
ncap2 -s  uavg=u.avg($time)   in.nc foo.nc # Fails ($time = '')
ncap2 -s &quot;uavg=u.avg($time)&quot;  in.nc foo.nc # Fails ($time = '')
</pre></example>
<para>Without the single quotes, the shell replaces <code>$time</code> with an
empty string.
The command <command>ncap2</command> receives from the shell is
<code>uavg=u.avg()</code>. 
This causes <command>ncap2</command> to average over all dimensions rather than
just the <var>time</var> dimension, and unintended consequence.
</para>
<para>We recommend using single quotes to protect <command>ncap2</command>
command-line scripts from the shell, even when such protection is not
strictly necessary. 
Expert users may violate this rule to exploit the ability to use shell
variables in <command>ncap2</command> command-line scripts 
(<pxref label="CCSM-Example"><xrefnodename>CCSM Example</xrefnodename></pxref>). 
In such cases it may be necessary to use the shell backslash character
<samp>\</samp> to protect the <command>ncap2</command> meta-character.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1020">appending data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1021">time-averaging</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="18" mergedindex="cp">ncks</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="19" mergedindex="cp">ncwa</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="20" mergedindex="cp">ncra</indexterm></findex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1022">degenerate dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1023"><samp>-b</samp></indexterm></cindex>
<para>A dimension of size one is said to be <emph>degenerate</emph>.
Whether a degenerate record dimension is desirable or not
depends on the application.
Often a degenerate <var>time</var> dimension is useful, e.g., for
concatentating, but it may cause problems with arithmetic.
Such is the case in the above example, where the first step employs
<command>ncwa</command> rather than <command>ncra</command> for the time-averaging.
Of course the numerical results are the same with both operators.
The difference is that, unless <samp>-b</samp> is specified, <command>ncwa</command>
writes no <var>time</var> dimension to the output file, while <command>ncra</command>
defaults to keeping <var>time</var> as a degenerate (<w>size 1</w>) dimension. 
Appending <code>u</code> and <code>v</code> to the output file would cause
<command>ncks</command> to try to expand the degenerate time axis of <code>uavg</code>
and <code>vavg</code> to the size of the non-degenerate <var>time</var> dimension
in the input file.
Thus the append (<command>ncks -A</command>) command would be undefined (and
should fail) in this case. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1024"><code>-C</code></indexterm></cindex>
Equally important is the <samp>-C</samp> argument 
(<pxref label="Subsetting-Coordinate-Variables"><xrefnodename>Subsetting Coordinate Variables</xrefnodename></pxref>) to <command>ncwa</command> to prevent
any scalar <var>time</var> variable from being written to the output file.  
Knowing when to use <command>ncwa -a time</command> rather than the default
<command>ncra</command> for time-averaging takes, well, time.
</para>
<html endspaces=" ">
&lt;a name=&quot;mth&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mth &textndash;&gt;
</html>
</subsection>
<node name="Intrinsic-mathematical-methods" spaces=" "><nodename>Intrinsic mathematical methods</nodename><nodenext spaces=" " trailingspaces=" ">Operator precedence and associativity</nodenext><nodeprev spaces=" ">Examples ncap2</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Intrinsic mathematical methods</sectiontitle>
<para><command>ncap2</command> supports the standard mathematical functions supplied with
most operating systems.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1025">addition</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1026">subtraction</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1027">multiplication</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1028">division</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1029">exponentiation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1030">power</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1031">modulus</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1032"><code>+</code> (addition)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1033"><code>-</code> (subtraction)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1034"><code>*</code> (multiplication)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1035"><code>/</code> (division)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1036"><code>^</code> (power)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1037"><code>%</code> (modulus)</indexterm></cindex>
Standard calculator notation is used for addition <kbd>+</kbd>, subtraction
<kbd>-</kbd>, multiplication <kbd>*</kbd>, division <kbd>/</kbd>, exponentiation
<kbd>^</kbd>, and modulus <kbd>%</kbd>.
The available elementary mathematical functions are: 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1038"><var>abs</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1039"><var>acosh</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1040"><var>acos</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1041"><var>asinh</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1042"><var>asin</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1043"><var>atanh</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1044"><var>atan</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1045"><var>ceil</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1046"><var>cosh</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1047"><var>cos</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1048"><var>erfc</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1049"><var>erf</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1050"><var>exp</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1051"><var>floor</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1052"><var>gamma</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1053"><var>ln</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1054"><var>log10</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1055"><var>log</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1056"><var>nearbyint</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1057"><var>pow</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1058"><var>rint</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1059"><var>round</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1060"><var>sinh</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1061"><var>sin</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1062"><var>sqrt</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1063"><var>tanh</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1064"><var>tan</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1065"><var>trunc</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1066">mathematical functions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1067">nearest integer function (inexact)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1068">nearest integer function (exact)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1069">rounding functions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1070">truncation function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1071">absolute value</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1072">arccosine function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1073">arcsine function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1074">arctangent function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1075">ceiling function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1076">complementary error function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1077">cosine function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1078">error function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1079">exponentiation function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1080">floor function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1081">gamma function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1082">hyperbolic arccosine function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1083">hyperbolic arcsine function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1084">hyperbolic arctangent function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1085">hyperbolic cosine function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1086">hyperbolic sine function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1087">hyperbolic tangent</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1088">logarithm, base 10</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1089">logarithm, natural</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1090">power function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1091">sine function</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1092">square root function</indexterm></cindex>
</para><table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">abs(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Absolute value</dfn>
<tex endspaces=" ">
Absolute value of $x$, $|x|$.
</tex>
Absolute value of <var>x</var>.
Example: 
<tex endspaces=" ">
abs$(-1) = 1$
</tex>
<math>abs(-1) = 1</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">acos(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Arc-cosine</dfn>
Arc-cosine of <var>x</var> where <var>x</var> is specified in radians.
Example: 
<tex endspaces=" ">
acos$(1.0) = 0.0$
</tex>
<math>acos(1.0) = 0.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">acosh(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Hyperbolic arc-cosine</dfn>
Hyperbolic arc-cosine of <var>x</var> where <var>x</var> is specified in radians.
Example: 
<tex endspaces=" ">
acosh$(1.0) = 0.0$
</tex>
<math>acosh(1.0) = 0.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">asin(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Arc-sine</dfn>
Arc-sine of <var>x</var> where <var>x</var> is specified in radians.
Example: 
<tex endspaces=" ">
asin$(1.0) = 1.57079632679489661922$
</tex>
<math>asin(1.0) = 1.57079632679489661922</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">asinh(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Hyperbolic arc-sine</dfn>
Hyperbolic arc-sine of <var>x</var> where <var>x</var> is specified in radians.
Example: 
<tex endspaces=" ">
asinh$(1.0) = 0.88137358702$
</tex>
<math>asinh(1.0) = 0.88137358702</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">atan(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Arc-tangent</dfn>
Arc-tangent of <var>x</var> where <var>x</var> is specified in radians between 
<tex endspaces=" ">
$-\pi/2$ and $\pi/2$.
</tex>
<math>-pi/2</math> and <math>pi/2</math>.
Example: 
<tex endspaces=" ">
atan$(1.0) = 0.78539816339744830961$
</tex>
<math>atan(1.0) = 0.78539816339744830961</math>
</para>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">atan2(y,x)</itemformat></item>
</tableterm><tableitem><para><dfn>Arc-tangent2</dfn>
Arc-tangent of <var>y/x</var> 
<math>:Example atan2(1,3) =  0.321689857</math>
</para>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">atanh(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Hyperbolic arc-tangent</dfn>
Hyperbolic arc-tangent of <var>x</var> where <var>x</var> is specified in radians between 
<tex endspaces=" ">
$-\pi/2$ and $\pi/2$.
</tex>
<math>-pi/2</math> and <math>pi/2</math>.
Example:
<tex endspaces=" ">
atanh$(3.14159265358979323844) = 1.0$
</tex>
<math>atanh(3.14159265358979323844) = 1.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ceil(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Ceil</dfn>
Ceiling of <var>x</var>. Smallest integral value not less than argument.
Example: 
<tex endspaces=" ">
ceil$(0.1) = 1.0$
</tex>
<math>ceil(0.1) = 1.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">cos(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Cosine</dfn>
Cosine of <var>x</var> where <var>x</var> is specified in radians.
Example: 
<tex endspaces=" ">
cos$(0.0) = 1.0$
</tex>
<math>cos(0.0) = 1.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">cosh(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Hyperbolic cosine</dfn>
Hyperbolic cosine of <var>x</var> where <var>x</var> is specified in radians.
Example: 
<tex endspaces=" ">
cosh$(0.0) = 1.0$
</tex>
<math>cosh(0.0) = 1.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">erf(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Error function</dfn>
Error function of <var>x</var> where <var>x</var> is specified between
<tex endspaces=" ">
$-1$ and $1$.
</tex>
<math>-1</math> and <math>1</math>.
Example: 
<tex endspaces=" ">
erf$(1.0) = 0.842701$
</tex>
<math>erf(1.0) = 0.842701</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">erfc(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Complementary error function</dfn>
Complementary error function of <var>x</var> where <var>x</var> is specified between
<tex endspaces=" ">
$-1$ and $1$.
</tex>
<math>-1</math> and <math>1</math>.
Example: 
<tex endspaces=" ">
erfc$(1.0) = 0.15729920705$
</tex>
<math>erfc(1.0) = 0.15729920705</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">exp(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Exponential</dfn>
Exponential of <var>x</var>,
<tex endspaces=" ">
$e^{x}$.
</tex>
<math>e^x</math>.
Example: 
<tex endspaces=" ">
exp$(1.0) = 2.71828182845904523536$
</tex>
<math>exp(1.0) = 2.71828182845904523536</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">floor(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Floor</dfn>
Floor of <var>x</var>. Largest integral value not greater than argument.
Example: 
<tex endspaces=" ">
floor$(1.9) = 1$
</tex>
<math>floor(1.9) = 1</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">gamma(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Gamma function</dfn>
Gamma function of <var>x</var>,
<tex endspaces=" ">
$\Gamma(x)$.
</tex>
<math>Gamma(x)</math>.
The well-known and loved continuous factorial function.
Example: 
<tex endspaces=" ">
gamma$(0.5) = \sqrt{\pi}$
</tex>
<math>gamma(0.5) = sqrt(pi)</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">gamma_inc_P(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Incomplete Gamma function</dfn>
Incomplete Gamma function of parameter <var>a</var> and variable <var>x</var>,
<tex endspaces=" ">
$P(a,x)$.
</tex>
<math>gamma_inc_P(a,x)</math>.
One of the four incomplete gamma functions.
Example: 
<tex endspaces=" ">
gamma\_inc\_P$(1,1) = 1-\me^{-1}$
</tex>
<math>gamma_inc_P(1,1) = 1-1/e</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ln(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Natural Logarithm</dfn>
Natural logarithm of <var>x</var>,
<tex endspaces=" ">
$\ln(x)$.
</tex>
<math>ln(x)</math>.
Example: 
<tex endspaces=" ">
ln$(2.71828182845904523536) = 1.0$
</tex>
<math>ln(2.71828182845904523536) = 1.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">log(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Natural Logarithm</dfn>
Exact synonym for <code>ln(x)</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">log10(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Base 10 Logarithm</dfn>
<w>Base 10</w> logarithm of <var>x</var>, 
<tex endspaces=" ">
$\log_{10}(x)$.
</tex>
<math>log10(x)</math>.
Example: 
<tex endspaces=" ">
log$(10.0) = 1.0$
</tex>
<math>log(10.0) = 1.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">nearbyint(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Round inexactly</dfn>
Nearest integer to <var>x</var> is returned in floating point format.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1093">inexact conversion</indexterm></cindex>
No exceptions are raised for <dfn>inexact conversions</dfn>.
Example: 
<tex endspaces=" ">
nearbyint$(0.1) = 0.0$
</tex>
<math>nearbyint(0.1) = 0.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">pow(x,y)</itemformat></item>
</tableterm><tableitem><para><dfn>Power</dfn>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1094">promotion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1095">automatic type conversion</indexterm></cindex>
Value of <var>x</var> is raised to the power of <var>y</var>.
Exceptions are raised for <dfn>domain errors</dfn>.
Due to type-limitations in the <w>C language</w> <code>pow</code> function,
integer arguments are promoted (<pxref label="Type-Conversion"><xrefnodename>Type Conversion</xrefnodename></pxref>) to type
<code>NC_FLOAT</code> before evaluation. 
Example: 
<tex endspaces=" ">
pow$(2,3) = 8$
</tex>
<math>pow(2,3) = 8</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">rint(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Round exactly</dfn>
Nearest integer to <var>x</var> is returned in floating point format.
Exceptions are raised for <dfn>inexact conversions</dfn>.
Example: 
<tex endspaces=" ">
rint$(0.1) = 0$
</tex>
<math>rint(0.1) = 0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">round(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Round</dfn>
Nearest integer to <var>x</var> is returned in floating point format.
Round halfway cases away from zero, regardless of current IEEE rounding direction. 
Example: 
<tex endspaces=" ">
round$(0.5) = 1.0$
</tex>
<math>round(0.5) = 1.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">sin(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Sine</dfn>
Sine of <var>x</var> where <var>x</var> is specified in radians.
Example: 
<tex endspaces=" ">
sin$(1.57079632679489661922) = 1.0$
</tex>
<math>sin(1.57079632679489661922) = 1.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">sinh(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Hyperbolic sine</dfn>
Hyperbolic sine of <var>x</var> where <var>x</var> is specified in radians.
Example: 
<tex endspaces=" ">
sinh$(1.0) = 1.1752$
</tex>
<math>sinh(1.0) = 1.1752</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">sqrt(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Square Root</dfn>
Square Root of <var>x</var>,
<tex endspaces=" ">
$\sqrt{x}$.
</tex>
<math>sqrt(x)</math>.
Example: 
<tex endspaces=" ">
sqrt$(4.0) = 2.0$
</tex>
<math>sqrt(4.0) = 2.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">tan(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Tangent</dfn>
Tangent of <var>x</var> where <var>x</var> is specified in radians.
Example: 
<tex endspaces=" ">
tan$(0.78539816339744830961) = 1.0$
</tex>
<math>tan(0.78539816339744830961) = 1.0</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">tanh(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Hyperbolic tangent</dfn>
Hyperbolic tangent of <var>x</var> where <var>x</var> is specified in radians.
Example: 
<tex endspaces=" ">
tanh$(1.0) = 0.761594155956$
</tex>
<math>tanh(1.0) = 0.761594155956</math>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">trunc(x)</itemformat></item>
</tableterm><tableitem><para><dfn>Truncate</dfn>
Nearest integer to <var>x</var> is returned in floating point format.
Round halfway cases toward zero, regardless of current IEEE rounding direction. 
Example: 
<tex endspaces=" ">
trunc$(0.5) = 0.0$
</tex>
<math>trunc(0.5) = 0.0</math>
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>The complete list of mathematical functions supported is
platform-specific.  
Functions mandated by <w>ANSI C</w> are <emph>guaranteed</emph> to be present
and are indicated with an asterisk 
<!-- c fxm No they're not, not yet -->
<cindex index="cp" spaces=" "><indexterm index="cp" number="1096"><code>ANSI C</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1097"><code>float</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1098">precision</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1099">quadruple-precision</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1100">single-precision</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1101">double-precision</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1102"><code>long double</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1103"><code>NC_DOUBLE</code></indexterm></cindex>
<footnote spaces="\n"><para><w>ANSI C</w> compilers are guaranteed to support double-precision versions
of these functions.
These functions normally operate on netCDF variables of type <code>NC_DOUBLE</code>
without having to perform intrinsic conversions.
For example, <acronym><acronymword>ANSI</acronymword></acronym> compilers provide <code>sin</code> for the sine of C-type
<code>double</code> variables. 
The <acronym><acronymword>ANSI</acronymword></acronym> standard does not require, but many compilers provide,
an extended set of mathematical functions that apply to single
(<code>float</code>) and quadruple (<code>long double</code>) precision variables. 
Using these functions (e.g., <code>sinf</code> for <code>float</code>, 
<code>sinl</code> for <code>long double</code>), when available, is (presumably)
more efficient than casting variables to type <code>double</code>,
performing the operation, and then re-casting.
<acronym><acronymword>NCO</acronymword></acronym> uses the faster intrinsic functions when they are
available, and uses the casting method when they are not.
</para></footnote>.
and are indicated with an asterisk. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1104"><code>-f</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1105"><code>--prn_fnc_tbl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1106"><code>--fnc_tbl</code></indexterm></cindex>
Use the <samp>-f</samp> (or <samp>fnc_tbl</samp> or <samp>prn_fnc_tbl</samp>) switch
to print a complete list of functions supported on your platform.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1107">Linux</indexterm></cindex>
<footnote><para>Linux supports more of these intrinsic functions than
other OSs.</para></footnote>
</para>
<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncap&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncap &textndash;&gt;
&lt;a name=&quot;xmp_ncap2&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncap2 &textndash;&gt;
</html>

<!-- c Begin HMB documentation -->

<html endspaces=" ">
&lt;a name=&quot;ncap_opts&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_opts &textndash;&gt;
</html>
</subsection>
<node name="Operator-precedence-and-associativity" spaces=" "><nodename trailingspaces=" ">Operator precedence and associativity</nodename><nodenext spaces=" ">ID Quoting</nodenext><nodeprev spaces=" ">Intrinsic mathematical methods</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>Operator precedence and associativity</sectiontitle>
<para>This page lists the <command>ncap2</command> operators in order of precedence (highest to lowest). Their associativity indicates in what order operators of equal precedence in an expression are applied.
</para>
<multitable spaces=" " endspaces=" "><columnfractions line=" .18 .62 .20"><columnfraction value=".18"></columnfraction><columnfraction value=".62"></columnfraction><columnfraction value=".20"></columnfraction></columnfractions>
<thead><row><entry command="headitem" spaces=" "><para>Operator </para></entry><entry command="tab" spaces=" "><para>Description </para></entry><entry command="tab" spaces=" "><para>Associativity
</para></entry></row></thead><tbody><row><entry command="item" spaces=" "><para><code>++ --</code> </para></entry><entry command="tab" spaces=" "><para>Postfix Increment/Decrement </para></entry><entry command="tab" spaces=" "><para>Right to Left
</para></entry></row><row><entry command="item" spaces=" "><para><code>()</code> </para></entry><entry command="tab" spaces=" "><para>Parentheses (function call)
</para></entry></row><row><entry command="item" spaces=" "><para><code>.</code>	</para></entry><entry command="tab" spaces=" "><para>Method call
</para></entry></row><row><entry command="item">
</entry></row><row><entry command="item" spaces=" "><para><code>++ --</code> </para></entry><entry command="tab" spaces=" "><para>Prefix Increment/Decrement </para></entry><entry command="tab" spaces=" "><para>Right to Left
</para></entry></row><row><entry command="item" spaces=" "><para><code>+ -</code> </para></entry><entry command="tab" spaces="  "><para>Unary  Plus/Minus
</para></entry></row><row><entry command="item" spaces=" "><para><code>!</code> </para></entry><entry command="tab" spaces=" "><para>Logical Not
</para></entry></row><row><entry command="item">
</entry></row><row><entry command="item" spaces=" "><para><code>^</code> </para></entry><entry command="tab" spaces=" "><para>Power of Operator </para></entry><entry command="tab" spaces=" "><para>Right to Left
</para></entry></row><row><entry command="item">
</entry></row><row><entry command="item" spaces=" "><para><code>* / %</code> </para></entry><entry command="tab" spaces=" "><para>Multiply/Divide/Modulus </para></entry><entry command="tab" spaces=" "><para>Left To Right
</para></entry></row><row><entry command="item">
</entry></row><row><entry command="item" spaces=" "><para><code>+ -</code> </para></entry><entry command="tab" spaces=" "><para>Addition/Subtraction </para></entry><entry command="tab" spaces=" "><para>Left To Right
</para></entry></row><row><entry command="item">
</entry></row><row><entry command="item" spaces=" "><para><code>&gt;&gt; &lt;&lt;</code> </para></entry><entry command="tab" spaces=" "><para>Fortran style array clipping </para></entry><entry command="tab" spaces=" "><para>Left to Right
</para></entry></row><row><entry command="item">
</entry></row><row><entry command="item">
</entry></row><row><entry command="item" spaces=" "><para><code>&lt; &lt;=</code> </para></entry><entry command="tab" spaces=" "><para>Less than/Less than or equal to </para></entry><entry command="tab" spaces=" "><para>Left to Right
</para></entry></row><row><entry command="item" spaces=" "><para><code>&gt; &gt;=</code> </para></entry><entry command="tab" spaces=" "><para>Greater than/Greater than or equal to
</para></entry></row><row><entry command="item">
</entry></row><row><entry command="item" spaces=" "><para><code>== !=</code> </para></entry><entry command="tab" spaces=" "><para>Equal to/Not equal to </para></entry><entry command="tab" spaces=" "><para>Left to Right
</para></entry></row><row><entry command="item"> 
</entry></row><row><entry command="item" spaces=" "><para><code>&amp;&amp;</code>	</para></entry><entry command="tab" spaces=" "><para>Logical AND </para></entry><entry command="tab" spaces=" "><para>Left to Right
</para></entry></row><row><entry command="item">
</entry></row><row><entry command="item" spaces=" "><para><code>||</code>	</para></entry><entry command="tab" spaces=" "><para>Logical OR </para></entry><entry command="tab" spaces=" "><para>Left to Right
</para></entry></row><row><entry command="item">
</entry></row><row><entry command="item" spaces=" "><para><code>?:</code> </para></entry><entry command="tab" spaces=" "><para>Ternary Operator </para></entry><entry command="tab" spaces=" "><para>Right to Left
</para></entry></row><row><entry command="item">
</entry></row><row><entry command="item" spaces=" "><para><code>=</code>	</para></entry><entry command="tab" spaces=" "><para>Assignment	</para></entry><entry command="tab" spaces=" "><para>Right to Left
</para></entry></row><row><entry command="item" spaces=" "><para><code>+= -=</code> </para></entry><entry command="tab" spaces="	"><para>Addition/subtraction assignment	
</para></entry></row><row><entry command="item" spaces=" "><para><code>*= /=</code> </para></entry><entry command="tab" spaces="	"><para>Multiplication/division assignment
</para></entry></row></tbody></multitable>

<html endspaces=" ">
&lt;a name=&quot;ncap_nmc&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncap_nmc &textndash;&gt;
</html>
</subsection>
<node name="ID-Quoting" spaces=" "><nodename>ID Quoting</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" " trailingspaces=" ">Operator precedence and associativity</nodeprev><nodeup spaces=" ">ncap2 netCDF Arithmetic Processor</nodeup></node>
<subsection spaces=" "><sectiontitle>ID Quoting</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1108">ID Quoting</indexterm></cindex>
<para>In this section when I refer to a name I mean a variable name, attribute name or a dimension name
The allowed characters in a valid netCDF name vary from release to release. (See end section). If you want to use metacharacters in a name or use a method name as a variable name then the name has to be quoted wherever it occurs. 
</para>
<noindent></noindent> <para>The default <acronym><acronymword>NCO</acronymword></acronym> name is specified by the regular expressions:
</para>
<example endspaces=" ">
<pre xml:space="preserve">DGT:     ('0'..'9');
LPH:     ( 'a'..'z' | 'A'..'Z' | '_' );
name:    (LPH)(LPH|DGT)+
</pre></example>

<noindent></noindent> <para>The first character of a valid name must be alphabetic or the underscore. Any subsequent characters must be alphanumeric or underscore. ( e.g., a1,_23, hell_is_666 )
</para>
<noindent></noindent> <para>The valid characters in a quoted name are specified by the regular expressions:
</para><example endspaces=" ">
<pre xml:space="preserve">LPHDGT:  ( 'a'..'z' | 'A'..'Z' | '_' | '0'..'9');
name:    (LPHDGT|'-'|'+'|'.'|'('|')'|':' )+  ;      
</pre></example>

<noindent></noindent> <para>Quote a variable:&linebreak;
&textrsquo;avg&textrsquo; , &textrsquo;10_+10&textrsquo;,&textrsquo;set_miss&textrsquo; &textrsquo;+-90field&textrsquo; , &textrsquo;&textndash;test&textrsquo;=10.0d&linebreak; &linebreak; 
Quote a attribute: &linebreak;
&textrsquo;three&arobase;10&textrsquo;, &textrsquo;set_mss&arobase;+10&textrsquo;, &textrsquo;666&arobase;hell&textrsquo;, &textrsquo;t1&arobase;+units&textrsquo;=&quot;kelvin&quot; &linebreak; &linebreak;
Quote a dimension: &linebreak;
&textrsquo;$10&textrsquo;, &textrsquo;$t1&textndash;&textrsquo;, &textrsquo;$&textndash;odd&textrsquo;, c1[&textrsquo;$10&textrsquo;,&textrsquo;$t1&textndash;&textrsquo;]=23.0d &linebreak; 
</para>
<sp value="1" line=" 1"></sp>
<para>The following comments are from the netCDF library definitions and
detail the naming conventions for each release. 
<noindent></noindent> netcdf-3.5.1 &linebreak;
netcdf-3.6.0-p1 &linebreak;
netcdf-3.6.1 &linebreak;
netcdf-3.6.2 &linebreak;
</para><example endspaces=" ">
<pre xml:space="preserve">/*
 * ( [a-zA-Z]|[0-9]|'_'|'-'|'+'|'.'|'|':'|'&arobase;'|'('|')' )+
 * Verify that name string is valid CDL syntax, i.e., all characters are
 * alphanumeric, '-', '_', '+', or '.'.
 * Also permit ':', '&arobase;', '(', or ')' in names for chemists currently making 
 * use of these characters, but don't document until ncgen and ncdump can 
 * also handle these characters in names.
 */
</pre></example>

<noindent></noindent> <para>netcdf-3.6.3&linebreak;
netcdf-4.0 Final  2008/08/28&linebreak;
</para><example endspaces=" ">
<pre xml:space="preserve">/*
 * Verify that a name string is valid syntax.  The allowed name
 * syntax (in RE form) is:
 *
 * ([a-zA-Z_]|&lbrace;UTF8&rbrace;)([^\x00-\x1F\x7F/]|&lbrace;UTF8&rbrace;)*
 *
 * where UTF8 represents a multibyte UTF-8 encoding.  Also, no
 * trailing spaces are permitted in names.  This definition
 * must be consistent with the one in ncgen.l.  We do not allow '/'
 * because HDF5 does not permit slashes in names as slash is used as a
 * group separator.  If UTF-8 is supported, then a multi-byte UTF-8
 * character can occur anywhere within an identifier.  We later
 * normalize UTF-8 strings to NFC to facilitate matching and queries.
 */ 
</pre></example>
<!-- c End HMB documentation -->

<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncatted&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncatted &textndash;&gt;
</html>
</subsection>
</section>
<node name="ncatted-netCDF-Attribute-Editor" spaces=" "><nodename>ncatted netCDF Attribute Editor</nodename><nodenext spaces=" ">ncbo netCDF Binary Operator</nodenext><nodeprev spaces=" ">ncap2 netCDF Arithmetic Processor</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncatted</command> netCDF Attribute Editor</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1109">attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1110">attribute names</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1111">editing attributes</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="21" mergedindex="cp">ncatted</indexterm></findex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted [-a <var>att_dsc</var>] [-a &dots;] [-D <var>dbg</var>] [-h] [--hdr_pad <var>nbr</var>]
[-l <var>path</var>] [-O] [-o <var>output-file</var>] [-p <var>path</var>] [-R] [-r] [--ram_all]
<var>input-file</var> [[<var>output-file</var>]]
</pre></example>
 
<noindent></noindent>
<para>DESCRIPTION
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1112"><command>ncattget</command></indexterm></cindex>
<para><command>ncatted</command> edits attributes in a netCDF file.  
If you are editing attributes then you are spending too much time in the
world of metadata, and <command>ncatted</command> was written to get you back out as
quickly and painlessly as possible.
<command>ncatted</command> can <dfn>append</dfn>, <dfn>create</dfn>, <dfn>delete</dfn>,
<dfn>modify</dfn>, and <dfn>overwrite</dfn> attributes (all explained below).  
<command>ncatted</command> allows each editing operation to be applied
to every variable in a file.
This saves time when changing attribute conventions throughout a file. 
<command>ncatted</command> is for <emph>writing</emph> attributes.
To <emph>read</emph> attribute values in plain text, use <command>ncks -m -M</command>,
or define something like <command>ncattget</command> as a shell command
(<pxref label="Filters-for-ncks"><xrefnodename>Filters for <command>ncks</command></xrefnodename></pxref>). 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1113"><code>history</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1114"><code>-h</code></indexterm></cindex>
<para>Because repeated use of <command>ncatted</command> can considerably increase the size
of the <code>history</code> global attribute (<pxref label="History-Attribute"><xrefnodename>History Attribute</xrefnodename></pxref>), the
<samp>-h</samp> switch is provided to override automatically appending the
command to the <code>history</code> global attribute in the <var>output-file</var>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1115">missing values</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1116">data, missing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1117"><code>_FillValue</code></indexterm></cindex>
<para>When <command>ncatted</command> is used to change the <code>_FillValue</code> attribute,
it changes the associated missing data self-consistently.
If the internal floating point representation of a missing value, 
e.g., 1.0e36, differs between two machines then netCDF files produced 
on those machines will have incompatible missing values.
This allows <command>ncatted</command> to change the missing values in files from 
different machines to a single value so that the files may then be 
concatenated, e.g., by <command>ncrcat</command>, without losing information.   
<xref label="Missing-Values"><xrefnodename>Missing Values</xrefnodename></xref>, for more information.
</para>
<para>To master <command>ncatted</command> one must understand the meaning of the
structure that describes the attribute modification, <var>att_dsc</var> 
specified by the required option <samp>-a</samp> or <samp>--attribute</samp>. 
Each <var>att_dsc</var> contains five elements, which makes using
<command>ncatted</command> somewhat complicated, yet powerful.
The <var>att_dsc</var> fields are in the following order:&linebreak; 
</para>
<para><var>att_dsc</var> = <var>att_nm</var>, <var>var_nm</var>, <var>mode</var>, <var>att_type</var>,
<var>att_val</var>&linebreak;
</para>
<table commandarg="var" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="var">att_nm</itemformat></item>
</tableterm><tableitem><para>Attribute name. 
Example: <code>units</code>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="var">var_nm</itemformat></item>
</tableterm><tableitem><para>Variable name. 
Example: <code>pressure</code>, <code>'^H2O'</code>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1118">extended regular expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1119">regular expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1120">pattern matching</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1121">wildcards</indexterm></cindex>
Regular expressions (<pxref label="Subsetting-Files"><xrefnodename>Subsetting Files</xrefnodename></pxref>) are accepted and will 
select any matching variable names.
The names <code>global</code> and <code>group</code> have special meaning.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="var">mode</itemformat></item>
</tableterm><tableitem><para>Edit mode abbreviation. 
Example: <code>a</code>. 
See below for complete listing of valid values of <var>mode</var>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="var">att_type</itemformat></item>
</tableterm><tableitem><para>Attribute type abbreviation. 
Example: <code>c</code>. 
See below for complete listing of valid values of <var>att_type</var>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="var">att_val</itemformat></item>
</tableterm><tableitem><para>Attribute value. 
Example: <code>pascal</code>. 
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>There should be no empty space between these five consecutive
arguments. 
The description of these arguments follows in their order of
appearance. 
</para>
<para>The value of <var>att_nm</var> is the name of the attribute you want to
edit. 
This meaning of this should be clear to all <command>ncatted</command> users.
Recall, as mentioned above, that <var>var_nm</var> may be specified as a
regular expression. 
If <var>att_nm</var> is omitted (i.e., left blank) and <dfn>Delete</dfn> mode is 
selected, then all attributes associated with the specified variable
will be deleted. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1122">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1123">attributes, global</indexterm></cindex>
<para>The value of <var>var_nm</var> is the name of the variable containing the
attribute (named <var>att_nm</var>) that you want to edit.
There are three very important and useful exceptions to this rule.
The value of <var>var_nm</var> can also be used to direct <command>ncatted</command>
to edit global attributes, or to repeat the editing operation for every 
group or variable in a file.
<w>A value</w> of <var>var_nm</var> of <code>global</code> indicates that <var>att_nm</var>
refers to a global (i.e., root-level) attribute, rather than to a
particular variable&textrsquo;s attribute. 
This is the method <command>ncatted</command> supports for editing global
attributes.
<w>A value</w> of <var>var_nm</var> of <code>group</code> indicates that <var>att_nm</var>
refers to all groups, rather than to a particular variable&textrsquo;s or group&textrsquo;s
attribute.  
The operation will proceed to edit group metadata for every group.
Finally, if <var>var_nm</var> is left blank, then <command>ncatted</command> 
attempts to perform the editing operation on every variable in the file.
This option may be convenient to use if you decide to change the
conventions you use for describing the data.
</para>
<html endspaces=" ">
&lt;a name=&quot;mode&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mode &textndash;&gt;
&lt;a name=&quot;att_mode&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#att_mode &textndash;&gt;
</html>
<para>The value of <var>mode</var> is a single character abbreviation (<code>a</code>,
<code>c</code>, <code>d</code>, <code>m</code>, or <code>o</code>) standing for one of
five editing modes:&linebreak;
<cindex index="cp" spaces=" "><indexterm index="cp" number="1124">attributes, appending</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1125">attributes, creating</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1126">attributes, deleting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1127">attributes, modifying</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1128">attributes, editing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1129">attributes, overwriting</indexterm></cindex>
</para><table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">a </itemformat></item>
</tableterm><tableitem><para><dfn>Append</dfn>.
Append value <var>att_val</var> to current <var>var_nm</var> attribute
<var>att_nm</var> value <var>att_val</var>, if any.  
If <var>var_nm</var> does not have an attribute <var>att_nm</var>, there is no
effect.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">c</itemformat></item>
</tableterm><tableitem><para><dfn>Create</dfn>.
Create variable <var>var_nm</var> attribute <var>att_nm</var> with <var>att_val</var>
if <var>att_nm</var> does not yet exist.  
If <var>var_nm</var> already has an attribute <var>att_nm</var>, there is no
effect. 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">d</itemformat></item>
</tableterm><tableitem><para><dfn>Delete</dfn>.
Delete current <var>var_nm</var> attribute <var>att_nm</var>.
If <var>var_nm</var> does not have an attribute <var>att_nm</var>, there is no
effect. 
If <var>att_nm</var> is omitted (left blank), then all attributes associated
with the specified variable are automatically deleted. 
When <dfn>Delete</dfn> mode is selected, the <var>att_type</var> and <var>att_val</var>
arguments are superfluous and may be left blank.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">m</itemformat></item>
</tableterm><tableitem><para><dfn>Modify</dfn>.
Change value of current <var>var_nm</var> attribute <var>att_nm</var> to value
<var>att_val</var>. 
If <var>var_nm</var> does not have an attribute <var>att_nm</var>, there is no
effect. 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">o</itemformat></item>
</tableterm><tableitem><para><dfn>Overwrite</dfn>.
Write attribute <var>att_nm</var> with value <var>att_val</var> to variable
<var>var_nm</var>, overwriting existing attribute <var>att_nm</var>, if any. 
This is the default mode.
</para></tableitem></tableentry></table>

<html endspaces=" ">
&lt;a name=&quot;att_typ&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#att_typ &textndash;&gt;
</html>
<para>The value of <var>att_type</var> is a single character abbreviation 
(<code>f</code>, <code>d</code>, <code>l</code>, <code>i</code>, <code>s</code>, <code>c</code>, 
<code>b</code>, <code>u</code>) or a short string standing for one of the twelve
primitive netCDF data types:&linebreak;  
</para><table commandarg="code" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="code">f</itemformat></item>
</tableterm><tableitem><para><dfn>Float</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_FLOAT</code>. 
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">d</itemformat></item>
</tableterm><tableitem><para><dfn>Double</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_DOUBLE</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">i, l</itemformat></item>
</tableterm><tableitem><para><dfn>Integer</dfn> or <dfn>Long</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_INT</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">s</itemformat></item>
</tableterm><tableitem><para><dfn>Short</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_SHORT</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">c</itemformat></item>
</tableterm><tableitem><para><dfn>Char</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_CHAR</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">b</itemformat></item>
</tableterm><tableitem><para><dfn>Byte</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_BYTE</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ub</itemformat></item>
</tableterm><tableitem><para><dfn>Unsigned Byte</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_UBYTE</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">us</itemformat></item>
</tableterm><tableitem><para><dfn>Unsigned Short</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_USHORT</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">u, ui, ul</itemformat></item>
</tableterm><tableitem><para><dfn>Unsigned Int</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_UINT</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ll, int64</itemformat></item>
</tableterm><tableitem><para><dfn>Int64</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_INT64</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">ull, uint64</itemformat></item>
</tableterm><tableitem><para><dfn>Uint64</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_UINT64</code>.
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="code">sng, string</itemformat></item>
</tableterm><tableitem><para><dfn>String</dfn>.
Value(s) specified in <var>att_val</var> will be stored as netCDF intrinsic
type <code>NC_STRING</code>.
Note that <command>ncatted</command> handles type <code>NC_STRING</code> attributes
correctly beginning with version 4.3.3 released in July, 2013. 
Earlier versions fail when asked to handle <code>NC_STRING</code> attributes.  
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>In <dfn>Delete</dfn> mode the specification of <var>att_type</var> is optional
(and is ignored if supplied).
</para>
<para>The value of <var>att_val</var> is what you want to change attribute
<var>att_nm</var> to contain.
The specification of <var>att_val</var> is optional in <dfn>Delete</dfn> (and is
ignored) mode. 
Attribute values for all types besides <code>NC_CHAR</code> must have an
attribute length of at least one.
Thus <var>att_val</var> may be a single value or one-dimensional array of
elements of type <code>att_type</code>.
If the <var>att_val</var> is not set or is set to empty space,
and the <var>att_type</var> is <code>NC_CHAR</code>, e.g., <code>-a units,T,o,c,&quot;&quot;</code>
or <code>-a units,T,o,c,</code>, then the corresponding attribute is set to 
have zero length.
When specifying an array of values, it is safest to enclose
<var>att_val</var> in single or double quotes, e.g., 
<code>-a levels,T,o,s,&quot;1,2,3,4&quot;</code> or   
<code>-a levels,T,o,s,'1,2,3,4'</code>.
The quotes are strictly unnecessary around <var>att_val</var> except 
when <var>att_val</var> contains characters which would confuse the calling
shell, such as spaces, commas, and wildcard characters. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1130">Perl</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1131"><acronym><acronymword>ASCII</acronymword></acronym></indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> processing of <code>NC_CHAR</code> attributes is a bit like Perl in
that it attempts to do what you want by default (but this sometimes
causes unexpected results if you want unusual data storage).
<cindex index="cp" spaces=" "><indexterm index="cp" number="1132"><code>printf()</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1133"><code>\n</code> (<acronym><acronymword>ASCII</acronymword></acronym> LF, linefeed)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1134">characters, special</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1135"><code>\t</code> (<acronym><acronymword>ASCII</acronymword></acronym> HT, horizontal tab)</indexterm></cindex>
If the <var>att_type</var> is <code>NC_CHAR</code> then the argument is interpreted as a
string and it may contain C-language escape sequences, e.g., <code>\n</code>,
which <acronym><acronymword>NCO</acronymword></acronym> will interpret before writing anything to disk.
<acronym><acronymword>NCO</acronymword></acronym> translates valid escape sequences and stores the
appropriate <acronym><acronymword>ASCII</acronymword></acronym> code instead.
Since two byte escape sequences, e.g., <code>\n</code>, represent one-byte
<acronym><acronymword>ASCII</acronymword></acronym> codes, e.g., <acronym><acronymword>ASCII</acronymword></acronym> 10 (decimal), the stored
string attribute is one byte shorter than the input string length for
each embedded escape sequence. 
The most frequently used C-language escape sequences are <code>\n</code> (for
linefeed) and <code>\t</code> (for horizontal tab).
These sequences in particular allow convenient editing of formatted text
attributes. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1136"><code>\a</code> (<acronym><acronymword>ASCII</acronymword></acronym> BEL, bell)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1137"><code>\b</code> (<acronym><acronymword>ASCII</acronymword></acronym> BS, backspace)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1138"><code>\f</code> (<acronym><acronymword>ASCII</acronymword></acronym> FF, formfeed)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1139"><code>\r</code> (<acronym><acronymword>ASCII</acronymword></acronym> CR, carriage return)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1140"><code>\v</code> (<acronym><acronymword>ASCII</acronymword></acronym> VT, vertical tab)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1141"><code>\\</code> (<acronym><acronymword>ASCII</acronymword></acronym> \, backslash)</indexterm></cindex>
The other valid <acronym><acronymword>ASCII</acronymword></acronym> codes are <code>\a</code>, <code>\b</code>, <code>\f</code>,
<code>\r</code>, <code>\v</code>, and <code>\\</code>. 
<xref label="ncks-netCDF-Kitchen-Sink"><xrefnodename>ncks netCDF Kitchen Sink</xrefnodename></xref>, for more examples of string formatting
(with the <command>ncks</command> <samp>-s</samp> option) with special characters. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1142"><code>\'</code> (protected end quote)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1143"><code>\&quot;</code> (protected double quote)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1144"><code>\?</code> (protected question mark)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1145"><code>\\</code> (protected backslash)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1146"><code>'</code> (end quote)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1147"><code>&quot;</code> (double quote)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1148"><code>?</code> (question mark)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1149"><code>\</code> (backslash)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1150">special characters</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1151"><acronym><acronymword>ASCII</acronymword></acronym></indexterm></cindex>
<para>Analogous to <code>printf</code>, other special characters are also allowed by 
<command>ncatted</command> if they are &quot;protected&quot; by a backslash.
The characters <code>&quot;</code>, <code>'</code>, <code>?</code>, and <code>\</code> may be 
input to the shell as <code>\&quot;</code>, <code>\'</code>, <code>\?</code>, and <code>\\</code>.
<acronym><acronymword>NCO</acronymword></acronym> simply strips away the leading backslash from these
characters before editing the attribute.
No other characters require protection by a backslash.
Backslashes which precede any other character (e.g., <code>3</code>, <code>m</code>,
<code>$</code>, <code>|</code>, <code>&amp;</code>, <code>&arobase;</code>, <code>%</code>, <code>&lbrace;</code>, and
<code>&rbrace;</code>) will not be filtered and will be included in the attribute.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1152">strings</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1153">NUL-termination</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1154">NUL</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1155">C language</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1156"><code>0</code> (NUL)</indexterm></cindex>
<para>Note that the NUL character <code>\0</code> which terminates <w>C language</w>
strings is assumed and need not be explicitly specified.
<!-- comment If @code{\0} is input, it will not be translated (because it would -->
<!-- comment terminate the string in an additional location). -->
<!-- comment 20101007 Before today, \0 was not translated to NUL -->
<!-- comment 20101007 As of today,  \0 is      translated to NUL -->
If <code>\0</code> is input, it is translated to the NUL character.
However, this will make the subsequent portion of the string, if any,
invisible to <w>C standard</w> library string functions. 
And that may cause unintended consequences.
Because of these context-sensitive rules, one must use <command>ncatted</command>
with care in order to store data, rather than text strings, in an 
attribute of type <code>NC_CHAR</code>.
</para>
<para>Note that <command>ncatted</command> interprets character attributes
(i.e., attributes of type <code>NC_CHAR</code>) as strings.
<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncatted&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncatted &textndash;&gt;
</html>
EXAMPLES
</para>
<para>Append the string &quot;Data version 2.0.\n&quot; to the global attribute
<code>history</code>: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -a history,global,a,c,&quot;Data version 2.0\n&quot; in.nc 
</pre></example>
<para>Note the use of embedded <w>C language</w> <code>printf()</code>-style escape 
sequences. 
</para>
<para>Change the value of the <code>long_name</code> attribute for variable <code>T</code>
from whatever it currently is to &quot;temperature&quot;:
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -a long_name,T,o,c,temperature in.nc
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;NaN&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#NaN &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1157">NaN</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1158">IEEE NaN</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1159">Not-a-Number</indexterm></cindex>
<para><acronym><acronymword>NCO</acronymword></acronym> arithmetic operators will not work as expected on IEEE
NaN (short for Not-a-Number) and NaN-like numbers such as positive
infinity and negative infinity.  
One way to work-around this problem is to change IEEE NaNs to normal
missing values. 
As of <acronym><acronymword>NCO</acronymword></acronym> 4.1.0 (March, 2012), <command>ncatted</command> works with
NaNs.
First set the missing value (i.e., the value of the <code>_FillValue</code>
attribute) for the variable(s) in question to the IEEE NaN value. 
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -a _FillValue,,o,f,NaN in.nc
</pre></example>
<para>Then change the missing value from the IEEE NaN value to a normal IEEE
number, like 1.0e36 (or to whatever the original missing value was).
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -a _FillValue,,m,f,1.0e36 in.nc
</pre></example>

<para>Delete all existing <code>units</code> attributes:
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -a units,,d,, in.nc
</pre></example>
<noindent></noindent>
<para>The value of <var>var_nm</var> was left blank in order to select all
variables in the file.
The values of <var>att_type</var> and <var>att_val</var> were left blank because
they are superfluous in <dfn>Delete</dfn> mode. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1160"><code>global</code> attribute</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1161">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1162">attributes, global</indexterm></cindex>
<para>Delete all attributes associated with the <code>tpt</code> variable, and
delete all global attributes
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -a ,tpt,d,, -a ,global,d,, in.nc
</pre></example>
<noindent></noindent>
<para>The value of <var>att_nm</var> was left blank in order to select all
attributes associated with the variable.
To delete all global attributes, simply replace <code>tpt</code> with
<code>global</code> in the above.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1163"><code>units</code></indexterm></cindex>
<para>Modify all existing <code>units</code> attributes to &quot;meter second-1&quot;:
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -a units,,m,c,&quot;meter second-1&quot; in.nc
</pre></example>

<cindex index="cp" spaces=" "><indexterm index="cp" number="1164"><code>units</code></indexterm></cindex>
<para>Add a <code>units</code> attribute of &quot;kilogram kilogram-1&quot; to all variables 
whose first three characters are <samp>H2O</samp>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -a units,'^H2O',c,c,&quot;kilogram kilogram-1&quot; in.nc
</pre></example>

<para>Overwrite the <code>quanta</code> attribute of variable
<code>energy</code> to an array of four integers. 
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -O -a quanta,energy,o,s,&quot;010,101,111,121&quot; in.nc
</pre></example>

<cindex index="cp" spaces=" "><indexterm index="cp" number="1165">extended regular expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1166">regular expressions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1167">pattern matching</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1168">wildcards</indexterm></cindex>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> 3.9.6 (January, 2009), <command>ncatted</command> accepts
<dfn>extended regular expressions</dfn> as arguments for variable names.
Create <code>isotope</code> attributes for all variables containing <samp>H2O</samp> 
in their names.
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -O -a isotope,'^H2O*',c,s,&quot;18&quot; in.nc
</pre></example>
<para>See <ref label="Subsetting-Files"><xrefnodename>Subsetting Files</xrefnodename></ref> for more details.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1169">groups</indexterm></cindex>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> 4.3.8 (November, 2013), <command>ncatted</command> 
accepts full and partial group paths in names of attributes,
variables, dimensions, and groups.
<!-- c ncks -m -v lon ~/in_grp.nc -->
</para><example endspaces=" ">
<pre xml:space="preserve"># Overwrite units attribute of specific 'lon' variable
ncatted -O -a units,/g1/lon,o,c,&quot;degrees_west&quot; in_grp.nc
# Overwrite units attribute of all 'lon' variables
ncatted -O -a units,lon,o,c,&quot;degrees_west&quot; in_grp.nc
# Delete units attribute of all 'lon' variables
ncatted -O -a units,lon,d,, in_grp.nc
# Overwrite units attribute with new type for specific 'lon' variable
ncatted -O -a units,/g1/lon,o,sng,&quot;degrees_west&quot; in_grp.nc
# Add new_att attribute to all variables
ncatted -O -a new_att,,c,sng,&quot;new variable attribute&quot; in_grp.nc
# Add new_grp_att group attribute to all groups
ncatted -O -a new_grp_att,group,c,sng,&quot;new group attribute&quot; in_grp.nc
# Add new_grp_att group attribute to single group
ncatted -O -a g1_grp_att,g1,c,sng,&quot;new group attribute&quot; in_grp.nc
# Add new_glb_att global attribute to root group
ncatted -O -a new_glb_att,global,c,sng,&quot;new global attribute&quot; in_grp.nc
</pre></example>

<para>Demonstrate input of C-language escape sequences (e.g., <code>\n</code>) and
other special characters (e.g., <code>\&quot;</code>) 
</para><example endspaces=" ">
<pre xml:space="preserve">ncatted -h -a special,global,o,c,
'\nDouble quote: \&quot;\nTwo consecutive double quotes: \&quot;\&quot;\n
Single quote: Beyond my shell abilities!\nBackslash: \\\n
Two consecutive backslashes: \\\\\nQuestion mark: \?\n' in.nc
</pre></example>
<para>Note that the entire attribute is protected from the shell by single
quotes. 
These outer single quotes are necessary for interactive use, but may be
omitted in batch scripts.
</para>
<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncbo&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncbo &textndash;&gt;
&lt;a name=&quot;ncdiff&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncdiff &textndash;&gt;
&lt;a name=&quot;ncadd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncadd &textndash;&gt;
&lt;a name=&quot;ncsub&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncsub &textndash;&gt;
&lt;a name=&quot;ncsubtract&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncsubtract &textndash;&gt;
&lt;a name=&quot;ncmult&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncmult &textndash;&gt;
&lt;a name=&quot;ncmultiply&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncmultiply &textndash;&gt;
&lt;a name=&quot;ncdivide&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncdivide &textndash;&gt;
</html>
</section>
<node name="ncbo-netCDF-Binary-Operator" spaces=" "><nodename>ncbo netCDF Binary Operator</nodename><nodenext spaces=" ">nces netCDF Ensemble Statistics</nodenext><nodeprev spaces=" ">ncatted netCDF Attribute Editor</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncbo</command> netCDF Binary Operator</sectiontitle>
<findex index="fn" spaces=" "><indexterm index="fn" number="22" mergedindex="cp">ncbo</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="23" mergedindex="cp">ncdiff</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="24" mergedindex="cp">ncadd</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="25" mergedindex="cp">ncsub</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="26" mergedindex="cp">ncsubtract</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="27" mergedindex="cp">ncmult</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="28" mergedindex="cp">ncmultiply</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="29" mergedindex="cp">ncdivide</indexterm></findex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1170">binary operations</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1171">addition</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1172">subtraction</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1173">multiplication</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1174">adding data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1175">subtracting data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1176">multiplying data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1177">dividing data</indexterm></cindex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" "> 
<pre xml:space="preserve">ncbo [-3] [-4] [-6] [-7] [-A] [-C] [-c]
[--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz]
[-D <var>dbg</var>] [-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]] [-F] 
[-G <var>gpe_dsc</var>] [-g <var>grp</var>[,&dots;]] [-h] [--hdr_pad <var>nbr</var>]
[-L <var>dfl_lvl</var>] [-l <var>path</var>] [--no_tmp_fl]
[-O] [-o <var>file_3</var>] [-p <var>path</var>] [-R] [-r] [--ram_all]
[-t <var>thr_nbr</var>] [--unn] [-v <var>var</var>[,&dots;]] [-X ...] [-x] [-y <var>op_typ</var>]
<var>file_1</var> <var>file_2</var> [<var>file_3</var>]
</pre></example>

<noindent></noindent>
<para>DESCRIPTION
</para>
<para><command>ncbo</command> performs binary operations on variables in <var>file_1</var>
and the corresponding variables (those with the same name) in
<var>file_2</var> and stores the results in <var>file_3</var>. 
The binary operation operates on the entire files (modulo any excluded
variables). 
<xref label="Missing-Values"><xrefnodename>Missing Values</xrefnodename></xref>, for treatment of missing values.
One of the four standard arithmetic binary operations currently
supported must be selected with the <samp>-y <var>op_typ</var></samp> switch (or
long options <samp>--op_typ</samp> or <samp>--operation</samp>).
<cindex index="cp" spaces=" "><indexterm index="cp" number="1178"><code>add</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1179"><code>subtract</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1180"><code>multiply</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1181"><code>divide</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1182"><code>+</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1183"><code>-</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1184"><code>*</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1185"><code>/</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1186"><code>-y <var>op_typ</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1187"><code>--operation <var>op_typ</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1188"><code>--op_typ <var>op_typ</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1189">alternate invocations</indexterm></cindex>
The valid binary operations for <command>ncbo</command>, their definitions, 
corresponding values of the <var>op_typ</var> key, and alternate invocations
are:  
</para><table commandarg="dfn" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="dfn">Addition</itemformat></item>
</tableterm><tableitem><!-- c Internal operation code: @{nco_op_add}@* -->
<para>Definition: <var>file_3</var> = <var>file_1</var> + <var>file_2</var>&linebreak;
Alternate invocation: <command>ncadd</command>&linebreak;
<var>op_typ</var> key values: <samp>add</samp>, <samp>+</samp>, <samp>addition</samp>&linebreak;
Examples: <samp>ncbo --op_typ=add 1.nc 2.nc 3.nc</samp>, <samp>ncadd 1.nc 2.nc 3.nc</samp>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Subtraction</itemformat></item>
</tableterm><tableitem><para>Definition: <var>file_3</var> = <var>file_1</var> - <var>file_2</var>&linebreak;
Alternate invocations: <command>ncdiff</command>, <command>ncsub</command>, <command>ncsubtract</command>&linebreak;
<var>op_typ</var> key values: <samp>sbt</samp>, <samp>-</samp>, <samp>dff</samp>, <samp>diff</samp>, <samp>sub</samp>, <samp>subtract</samp>, <samp>subtraction</samp>&linebreak;
Examples: <samp>ncbo --op_typ=- 1.nc 2.nc 3.nc</samp>, <samp>ncdiff 1.nc 2.nc 3.nc</samp>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Multiplication</itemformat></item>
</tableterm><tableitem><para>Definition: <var>file_3</var> = <var>file_1</var> * <var>file_2</var>&linebreak; 
Alternate invocations: <command>ncmult</command>, <command>ncmultiply</command>&linebreak; 
<var>op_typ</var> key values: <samp>mlt</samp>, <samp>*</samp>, <samp>mult</samp>, <samp>multiply</samp>, <samp>multiplication</samp>&linebreak;
Examples: <samp>ncbo --op_typ=mlt 1.nc 2.nc 3.nc</samp>, <samp>ncmult 1.nc 2.nc 3.nc</samp>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Division</itemformat></item>
</tableterm><tableitem><para>Definition: <var>file_3</var> = <var>file_1</var> / <var>file_2</var>&linebreak; 
Alternate invocation: <command>ncdivide</command>&linebreak;
<var>op_typ</var> key values: <samp>dvd</samp>, <samp>/</samp>, <samp>divide</samp>, <samp>division</samp>&linebreak;
Examples: <samp>ncbo --op_typ=/ 1.nc 2.nc 3.nc</samp>, <samp>ncdivide 1.nc 2.nc 3.nc</samp>&linebreak;
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>Care should be taken when using the shortest form of key values,
i.e., <samp>+</samp>, <samp>-</samp>, <samp>*</samp>, <w>and <samp>/</samp></w>.
Some of these single characters may have special meanings to the shell
<cindex index="cp" spaces=" "><indexterm index="cp" number="1190">naked characters</indexterm></cindex>
<footnote><para><w>A naked</w> (i.e., unprotected or unquoted) <samp>*</samp> is a
wildcard character.  
<w>A naked</w> <samp>-</samp> may confuse the command line parser.
<w>A naked</w> <samp>+</samp> and <samp>/</samp> are relatively harmless.</para></footnote>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1191">Bash shell</indexterm></cindex>
Place these characters inside quotes to keep them from being interpreted 
(globbed) by the shell
<footnote><para>The widely used shell Bash correctly interprets all these
special characters even when they are not quoted. 
That is, Bash does not prevent <acronym><acronymword>NCO</acronymword></acronym> from correctly interpreting 
the intended arithmetic operation when the following arguments are given
(without quotes) to <command>ncbo</command>:
<samp>--op_typ=+</samp>, <samp>--op_typ=-</samp>, <samp>--op_typ=*</samp>,
and <samp>--op_typ=/</samp></para></footnote>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1192">globbing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1193">shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1194">quotes</indexterm></cindex>
For example, the following commands are equivalent
</para><example endspaces=" ">
<pre xml:space="preserve">ncbo --op_typ=* 1.nc 2.nc 3.nc # Dangerous (shell may try to glob)
ncbo --op_typ='*' 1.nc 2.nc 3.nc # Safe ('*' protected from shell)
ncbo --op_typ=&quot;*&quot; 1.nc 2.nc 3.nc # Safe ('*' protected from shell)
ncbo --op_typ=mlt 1.nc 2.nc 3.nc
ncbo --op_typ=mult 1.nc 2.nc 3.nc
ncbo --op_typ=multiply 1.nc 2.nc 3.nc
ncbo --op_typ=multiplication 1.nc 2.nc 3.nc
ncmult 1.nc 2.nc 3.nc # First do 'ln -s ncbo ncmult'
ncmultiply 1.nc 2.nc 3.nc # First do 'ln -s ncbo ncmultiply'
</pre></example>
<para>No particular argument or invocation form is preferred.
Users are encouraged to use the forms which are most intuitive to them.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1195"><command>alias</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1196"><command>ln -s</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1197">symbolic links</indexterm></cindex>
<para>Normally, <command>ncbo</command> will fail unless an operation type is specified
with <samp>-y</samp> (equivalent to <samp>--op_typ</samp>).
You may create exceptions to this rule to suit your particular tastes,
in conformance with your site&textrsquo;s policy on <dfn>symbolic links</dfn> to
executables (files of a different name point to the actual executable).
For many years, <command>ncdiff</command> was the main binary file operator.
As a result, many users prefer to continue invoking <command>ncdiff</command>
rather than memorizing a new command (<samp>ncbo -y <var>sbt</var></samp>) which
behaves identically to the original <command>ncdiff</command> command.
However, from a software maintenance standpoint, maintaining a distinct 
executable for each binary operation (e.g., <command>ncadd</command>) is untenable,
and a single executable, <command>ncbo</command>, is desirable.
To maintain backward compatibility, therefore, <acronym><acronymword>NCO</acronymword></acronym>
automatically creates a symbolic link from <command>ncbo</command> to
<command>ncdiff</command>.  
Thus <command>ncdiff</command> is called an <dfn>alternate invocation</dfn> of
<command>ncbo</command>. 
<command>ncbo</command> supports many additional alternate invocations which must
be manually activated.
Should users or system adminitrators decide to activate them, the
procedure is simple. 
For example, to use <samp>ncadd</samp> instead of <samp>ncbo --op_typ=add</samp>, 
simply create a symbolic link from <command>ncbo</command> to <command>ncadd</command>
<footnote><para>The command to do this is <samp>ln -s -f ncbo ncadd</samp></para></footnote>.
The alternatate invocations supported for each operation type are listed
above. 
Alternatively, users may always define <samp>ncadd</samp> as an <dfn>alias</dfn> to 
<samp>ncbo --op_typ=add</samp>
<footnote><para>The command to do this is <samp>alias ncadd='ncbo --op_typ=add'</samp></para></footnote>.
</para>
<para>It is important to maintain portability in <acronym><acronymword>NCO</acronymword></acronym> scripts.
Therefore we recommend that site-specfic invocations (e.g.,
<samp>ncadd</samp>) be used only in interactive sessions from the
command-line.
For scripts, we recommend using the full invocation (e.g., 
<samp>ncbo --op_typ=add</samp>).
This ensures portability of scripts between users and sites.
</para>
<html endspaces=" ">
&lt;a name=&quot;brd_var&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#brd_var &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1198">broadcasting variables</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1199">rank</indexterm></cindex>
<para><command>ncbo</command> operates (e.g., adds) variables in <var>file_2</var> with the
corresponding variables (those with the same name) in <var>file_1</var> and
stores the results in <var>file_3</var>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1200">broadcasting variables</indexterm></cindex>
Variables in <var>file_1</var> or <var>file_2</var> are <dfn>broadcast</dfn> to conform
to the corresponding variable in the other input file if
necessary<footnote spaces="\n"><para>Prior to <acronym><acronymword>NCO</acronymword></acronym> version 4.3.1 (May, 2013), <command>ncbo</command>
would only broadcast variables in <var>file_2</var> to conform to
<var>file_1</var>. 
Variables in <var>file_1</var> were <emph>never</emph> broadcast to conform to the 
dimensions in <var>file_2</var>.</para></footnote>. 
Now <command>ncbo</command> is completely symmetric with respect to <var>file_1</var>
and <var>file_2</var>, i.e., 
<set name="flg" line=" flg"></set>
<tex endspaces=" ">
$\rm{file}_1 - \rm{file}_2 = -(\rm{file}_2-\rm{file}_1)$.
<clear name="flg" line=" flg"></clear>
</tex>
<math><var>file_1</var> - <var>file_2</var> = - (<var>file_2</var> - <var>file_1</var></math>.
<clear name="flg" line=" flg"></clear>
</para>
<para>Broadcasting a variable means creating data in non-existing dimensions
by copying data in existing dimensions.
For example, a two dimensional variable in <var>file_2</var> can be
subtracted from a four, three, or two (not one or zero)
dimensional variable (of the same name) in <code>file_1</code>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1201">anomalies</indexterm></cindex>
This functionality allows the user to compute anomalies from the mean.
In the future, we will broadcast variables in <var>file_1</var>, if necessary
to conform to their counterparts in <var>file_2</var>.
<!-- c TODO #268 -->
<cindex index="cp" spaces=" "><indexterm index="cp" number="1202">rank</indexterm></cindex>
Thus, presently, the number of dimensions, or <dfn>rank</dfn>, of any
processed variable in <var>file_1</var> must be greater than or equal to the
rank of the same variable in <var>file_2</var>. 
Of course, the size of all dimensions common to both <var>file_1</var> and
<var>file_2</var> must be equal. 
</para>
<para>When computing anomalies from the mean it is often the case that
<var>file_2</var> was created by applying an averaging operator to a file
with initially the same dimensions as <var>file_1</var> (often <var>file_1</var>
itself).  
In these cases, creating <var>file_2</var> with <command>ncra</command> rather than
<command>ncwa</command> will cause the <command>ncbo</command> operation to fail.
For concreteness say the record dimension in <code>file_1</code> is
<code>time</code>.  
If <var>file_2</var> were created by averaging <var>file_1</var> over the
<code>time</code> dimension with the <command>ncra</command> operator rather than with
the <command>ncwa</command> operator, then <var>file_2</var> will have a <code>time</code>
dimension of <w>size 1</w> rather than having no <code>time</code> dimension at
all 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1203">degenerate dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1204"><samp>-b</samp></indexterm></cindex>
<footnote><para>This is because <command>ncra</command> collapses the record dimension
to a size <w>of 1</w> (making it a <dfn>degenerate</dfn> dimension), but does
not remove it, while, unless <samp>-b</samp> is given, <command>ncwa</command> removes
all averaged dimensions.
In other words, by default <command>ncra</command> changes variable size though
not rank, while, <command>ncwa</command> changes both variable size and rank.</para></footnote>.   
In this case the input files to <command>ncbo</command>, <var>file_1</var> and
<var>file_2</var>, will have unequally sized <code>time</code> dimensions which
causes <command>ncbo</command> to fail.
To prevent this from occuring, use <command>ncwa</command> to remove the
<code>time</code> dimension from <var>file_2</var>.
See the example below.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1205">coordinate variable</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1206"><code>NC_CHAR</code></indexterm></cindex>
<para><command>ncbo</command> never operates on coordinate variables or variables
of type <code>NC_CHAR</code> or <code>NC_STRING</code>. 
This ensures that coordinates like (e.g., latitude and longitude) are 
physically meaningful in the output file, <var>file_3</var>. 
This behavior is hardcoded.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1207"><acronym><acronymword>CF</acronymword></acronym> conventions</indexterm></cindex>
<command>ncbo</command> applies special rules to some 
<acronym><acronymword>CF</acronymword></acronym>-defined (and/or <acronym><acronymword>NCAR CCSM</acronymword></acronym> or <acronym><acronymword>NCAR CCM</acronymword></acronym> 
fields) such as <code>ORO</code>.
See <ref label="CF-Conventions"><xrefnodename>CF Conventions</xrefnodename></ref> for a complete description.
Finally, we note that <command>ncflint</command> (<pxref label="ncflint-netCDF-File-Interpolator"><xrefnodename>ncflint netCDF File
Interpolator</xrefnodename></pxref>) is designed for file interpolation.
As such, it also performs file subtraction, addition, multiplication,
albeit in a more convoluted way than <command>ncbo</command>.
</para>
<html endspaces=" ">
&lt;a name=&quot;brd_grp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#brd_grp &textndash;&gt;
&lt;a name=&quot;gb&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#gb &textndash;&gt;
&lt;a name=&quot;GB&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#GB &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1208">broadcasting groups</indexterm></cindex>
<para>Beginning with <acronym><acronymword>NCO</acronymword></acronym> version 4.3.1 (May, 2013), <command>ncbo</command> 
supports <dfn>group broadcasting</dfn>.
Group broadcasting means processing data based on group patterns in the
input file(s) and automatically transferring or transforming groups to
the output file. 
Consider the case where <var>file_1</var> contains multiple groups each with
the variable <var>v1</var>, while <var>file_2</var> contains <var>v1</var> only in its 
top-level (i.e., root) group.
Then <command>ncbo</command> will replicate the group structure of <var>file_1</var>
in the output file, <var>file_3</var>.
Each group in <var>file_3</var> contains the output of the corresponding
group in <var>file_1</var> operating on the data in the single group in
<var>file_2</var>. 
An example is provided below.
</para>
<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncbo&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncbo &textndash;&gt;
&lt;a name=&quot;xmp_ncdiff&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncdiff &textndash;&gt;
</html>
<para>EXAMPLES
</para>
<para>Say files <file>85_0112.nc</file> and <file>86_0112.nc</file> each contain 12 months
of data.
Compute the change in the monthly averages from 1985 to 1986:
</para><example endspaces=" ">
<pre xml:space="preserve">ncbo   86_0112.nc 85_0112.nc 86m85_0112.nc
ncdiff 86_0112.nc 85_0112.nc 86m85_0112.nc
ncbo --op_typ=sub 86_0112.nc 85_0112.nc 86m85_0112.nc
ncbo --op_typ='-' 86_0112.nc 85_0112.nc 86m85_0112.nc
</pre></example>
<noindent></noindent>
<para>These commands are all different ways of expressing the same thing.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1209">broadcasting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1210">rank</indexterm></cindex>
<para>The following examples demonstrate the broadcasting feature of
<command>ncbo</command>.  
Say we wish to compute the monthly anomalies of <code>T</code> from the yearly
average of <code>T</code> for the year 1985.
First we create the 1985 average from the monthly data, which is stored
with the record dimension <code>time</code>.
</para><example endspaces=" ">
<pre xml:space="preserve">ncra 85_0112.nc 85.nc
ncwa -O -a time 85.nc 85.nc
</pre></example>
<noindent></noindent>
<para>The second command, <command>ncwa</command>, gets rid of the <code>time</code> dimension
of <w>size 1</w> that <command>ncra</command> left in <file>85.nc</file>. 
Now none of the variables in <file>85.nc</file> has a <code>time</code> dimension.
<w>A quicker</w> way to accomplish this is to use <command>ncwa</command> from the
beginning:  
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -a time 85_0112.nc 85.nc
</pre></example>
<noindent></noindent>
<para>We are now ready to use <command>ncbo</command> to compute the anomalies for 1985:
</para><example endspaces=" ">
<pre xml:space="preserve">ncdiff -v T 85_0112.nc 85.nc t_anm_85_0112.nc
</pre></example>
<noindent></noindent>
<para>Each of the 12 records in <file>t_anm_85_0112.nc</file> now contains the
monthly deviation of <code>T</code> from the annual mean of <code>T</code> for each 
gridpoint. 
</para>
<para>Say we wish to compute the monthly gridpoint anomalies from the zonal
annual mean. 
<w>A <dfn>zonal mean</dfn></w> is a quantity that has been averaged over the
longitudinal (or <var>x</var>) direction.
First we use <command>ncwa</command> to average over longitudinal direction
<code>lon</code>, creating <file>85_x.nc</file>, the zonal mean of <file>85.nc</file>. 
Then we use <command>ncbo</command> to subtract the zonal annual means from the
monthly gridpoint data:
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -a lon 85.nc 85_x.nc
ncdiff 85_0112.nc 85_x.nc tx_anm_85_0112.nc
</pre></example>
<noindent></noindent>
<para>This examples works assuming <file>85_0112.nc</file> has dimensions
<code>time</code> and <code>lon</code>, and that <file>85_x.nc</file> has no <code>time</code>
or <code>lon</code> dimension.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1211">broadcasting groups</indexterm></cindex>
<para>Group broadcasting simplifies evaluation of multiple models against
observations.
Consider the input file <file>cmip5.nc</file> which contains multiple
top-level groups <code>cesm</code>, <code>ecmwf</code>, and <code>giss</code>, each of
which contains the surface air temperature field <code>tas</code>.
We wish to compare these models to observations stored in <file>obs.nc</file> 
which contains <code>tas</code> only in its top-level (i.e., root) group.
It is often the case that many models and/or model simulations exist,
whereas only one observational dataset does.
We evaluate the models and obtain the bias (difference) between models
and observations by subtracting <file>obs.nc</file> from <file>cmip5.nc</file>.
Then <command>ncbo</command> &textldquo;broadcasts&textrdquo; (i.e., replicates) the observational
data to match the group structure of <file>cmip5.nc</file>, subtracts,
and then stores the results in the output file, <file>bias.nc</file>
which has the same group structure as <file>cmip5.nc</file>.
</para><example endspaces=" ">
<pre xml:space="preserve">% ncbo -O cmip5.nc obs.nc bias.nc
% ncks -H -v tas -d time,3 bias.nc
/cesm/tas
time[3] tas[3]=-1 
/ecmwf/tas
time[3] tas[3]=0 
/giss/tas
time[3] tas[3]=1 
</pre></example>
<noindent></noindent>

<html endspaces=" ">
&lt;a name=&quot;csn_anm&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#csn_anm &textndash;&gt;
</html>
<para>As a final example, say we have five years of monthly data (i.e., 
<w>60 months</w>) stored in <file>8501_8912.nc</file> and we wish to create a
file which contains the twelve month seasonal cycle of the average
monthly anomaly from the five-year mean of this data. 
The following method is just one permutation of many which will
accomplish the same result.
First use <command>ncwa</command> to create the five-year mean: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -a time 8501_8912.nc 8589.nc
</pre></example>
<noindent></noindent>
<para>Next use <command>ncbo</command> to create a file containing the difference of
each month&textrsquo;s data from the five-year mean:
</para><example endspaces=" ">
<pre xml:space="preserve">ncbo 8501_8912.nc 8589.nc t_anm_8501_8912.nc
</pre></example>
<noindent></noindent>
<para>Now use <command>ncks</command> to group together the five January anomalies in
one file, and use <command>ncra</command> to create the average anomaly for all
five Januarys. 
These commands are embedded in a shell loop so they are repeated for all
twelve months:
<cindex index="cp" spaces=" "><indexterm index="cp" number="1212">Bash Shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1213">Bourne Shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1214">C Shell</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">for idx in &lbrace;1..12&rbrace;; do # Bash Shell (version 3.0+) 
  idx=`printf &quot;%02d&quot; $&lbrace;idx&rbrace;` # Zero-pad to preserve order
  ncks -F -d time,$&lbrace;idx&rbrace;,,12 t_anm_8501_8912.nc foo.$&lbrace;idx&rbrace;
  ncra foo.$&lbrace;idx&rbrace; t_anm_8589_$&lbrace;idx&rbrace;.nc
done
for idx in 01 02 03 04 05 06 07 08 09 10 11 12; do # Bourne Shell
  ncks -F -d time,$&lbrace;idx&rbrace;,,12 t_anm_8501_8912.nc foo.$&lbrace;idx&rbrace;
  ncra foo.$&lbrace;idx&rbrace; t_anm_8589_$&lbrace;idx&rbrace;.nc
done
foreach idx (01 02 03 04 05 06 07 08 09 10 11 12) # C Shell
  ncks -F -d time,$&lbrace;idx&rbrace;,,12 t_anm_8501_8912.nc foo.$&lbrace;idx&rbrace;
  ncra foo.$&lbrace;idx&rbrace; t_anm_8589_$&lbrace;idx&rbrace;.nc
end
</pre></example>
<noindent></noindent>
<para>Note that <command>ncra</command> understands the <code>stride</code> argument so the
two commands inside the loop may be combined into the single command 
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -F -d time,$&lbrace;idx&rbrace;,,12 t_anm_8501_8912.nc foo.$&lbrace;idx&rbrace;
</pre></example>
<noindent></noindent>
<para>Finally, use <command>ncrcat</command> to concatenate the <w>12 average</w> monthly  
anomaly files into one twelve-record file which contains the entire
seasonal cycle of the monthly anomalies:
</para><example endspaces=" ">
<pre xml:space="preserve">ncrcat t_anm_8589_??.nc t_anm_8589_0112.nc
</pre></example>
<noindent></noindent>

<page></page>
<html endspaces=" ">
&lt;a name=&quot;nces&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#nces &textndash;&gt;
&lt;a name=&quot;ncea&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncea &textndash;&gt;
</html>
</section>
<node name="nces-netCDF-Ensemble-Statistics" spaces=" "><nodename>nces netCDF Ensemble Statistics</nodename><nodenext spaces=" ">ncecat netCDF Ensemble Concatenator</nodenext><nodeprev spaces=" ">ncbo netCDF Binary Operator</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>nces</command> netCDF Ensemble Statistics</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1215">averaging data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1216">ensemble average</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="30" mergedindex="cp">nces</indexterm></findex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">nces [-3] [-4] [-6] [-7] [-A] [-C] [-c]
[--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz]
[-D <var>dbg</var>] [-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]] [-F]
[-G <var>gpe_dsc</var>] [-g <var>grp</var>[,&dots;]] [-h] [--hdf] [--hdr_pad <var>nbr</var>] 
[-L <var>dfl_lvl</var>] [-l <var>path</var>] [-n <var>loop</var>] [--no_tmp_fl] [--nsm_fl|grp] [--nsm_sfx sfx]
[-O] [-o <var>output-file</var>] [-p <var>path</var>] [-R] [-r] [--ram_all] [--rth_dbl|flt]
[-t <var>thr_nbr</var>] [--unn] [-v <var>var</var>[,&dots;]] [-X ...] [-x] [-y <var>op_typ</var>]
[<var>input-files</var>] [<var>output-file</var>]
</pre></example>

<noindent></noindent>
<para>DESCRIPTION
</para>
<para><command>nces</command> performs gridpoint statistics on variables across an
arbitrary number (an <dfn>ensemble</dfn>) of <var>input-files</var> and/or of
input groups within each file.
Each file (or group) receives an equal weight.
<command>nces</command> was formerly (until <acronym><acronymword>NCO</acronymword></acronym> version 4.3.9,
released December, 2013) known as <command>ncea</command> (netCDF Ensemble
Averager)<footnote spaces="\n"><para>The old ncea command was deprecated in <acronym><acronymword>NCO</acronymword></acronym> version 4.3.9,  
released December, 2013.
<acronym><acronymword>NCO</acronymword></acronym> will attempt to maintain back-compatibility and work
as expected with invocations of <command>ncea</command> for as long as possible.
Please replace <command>ncea</command> by <command>nces</command> in all future work.</para></footnote>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1217">ensemble</indexterm></cindex>
For example, <command>nces</command> will average a set of files or groups,
weighting each file or group evenly. 
This is distinct from <command>ncra</command>, which performs statistics only
over the record dimension(s) (e.g., <var>time</var>), and weights each record 
in each record dimension evenly.
</para>
<para>The file or group is the logical unit of organization for the results of
many scientific studies.
Often one wishes to generate a file or group which is the statistical
product (e.g., average) of many separate files or groups. 
This may be to reduce statistical noise by combining the results of a
large number of experiments, or it may simply be a step in a procedure 
whose goal is to compute anomalies from a mean state. 
In any case, when one desires to generate a file whose statistical
properties are equally influenced by all the inputs, then <command>nces</command>
is the operator to use. 
</para>
<para>Variables in the <var>output-file</var> are the same size as the variable
hyperslab in each input file or group, and each input file or group
must be the same size after hyperslabbing
<footnote><para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.4.2 (released February, 2014)
<command>nces</command> allows hyperslabs in all dimensions so long as the
hyperslabs resolve to the same size. 
The fixed (i.e., non-record) dimensions should be the same size in
all ensemble members both before and after hyperslabbing, although
the hypserslabs may (and usually do) change the size of the dimensions
from the input to the output files.
Prior to this, <command>nces</command> was only guaranteed to work on hyperslabs
in the record dimension that resolved to the same size.</para></footnote>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1218">record dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1219">hyperslab</indexterm></cindex>
<command>nces</command> does allow files to differ in the record dimension size
if the requested record hyperslab (<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>) resolves to the
same size for all files.  
<command>nces</command> recomputes the record dimension hyperslab limits for
each input file so that coordinate limits may be used to select equal
length timeseries from unequal length files.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1220"><acronym><acronymword>IPCC</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1221"><acronym><acronymword>AR4</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1222"><acronym><acronymword>CMIP</acronymword></acronym></indexterm></cindex>
This simplifies analysis of unequal length timeseries from simulation
ensembles (e.g., the <acronym><acronymword>CMIP3</acronymword></acronym> <acronym><acronymword>IPCC</acronymword></acronym> <acronym><acronymword>AR4</acronymword></acronym>
archive).   
</para>
<html endspaces=" ">
&lt;a name=&quot;nsm_fl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#nsm_fl &textndash;&gt;
&lt;a name=&quot;nsm_grp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#nsm_grp &textndash;&gt;
&lt;a name=&quot;nsm_sfx&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#nsm_sfx &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1223"><code>--nsm_fl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1224"><code>--nsm_grp</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1225"><code>--ensemble_file</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1226"><code>--ensemble_group</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1227"><code>--nsm_sfx</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1228"><code>--ensemble_suffix</code></indexterm></cindex>
<para><command>nces</command> works in one of two modes, file ensembles 
or group ensembles.
File ensembles are the default (equivalent to the old <command>ncea</command>) 
and may also be explicitly specified by the <samp>--nsm_fl</samp> or
<samp>--ensemble_file</samp> switches.
To perform statistics on ensembles of groups, a newer feature, use
<samp>--nsm_grp</samp> or <samp>--ensemble_group</samp>.
Members of a group ensemble are groups that share the same structure,
parent group, and nesting level. 
Members must be <dfn>leaf groups</dfn>, i.e., not contain any sub-groups.
Their contents usually have different values because they are
realizations of replicated experiments.  
In group ensemble mode <command>nces</command> computes the statistics across 
the ensemble, which may span multiple input files. 
Files may contain members of multiple, distinct ensembles. 
However, all ensembles must have at least one member in the first input 
file. 
Group ensembles behave as an unlimited dimension of datasets: 
they may contain an arbitrary and extensible number of realizations in
each file, and may be composed from multiple files. 
</para>
<para>Output statistics in group ensemble mode are stored in the parent group
by default. 
If the ensemble members are <file>/cesm/cesm_01</file> and
<file>/cesm/cesm_02</file>, then the computed statistic will be in
<file>/cesm</file> in the output file.  
The <samp>--nsm_sfx</samp> option instructs nces to instead store output in  
a new child group of the parent created by attaching the suffix
to the parent group&textrsquo;s name, e.g., <samp>--nsm_sfx='_avg'</samp> would store
results in the output group <file>/cesm/cesm_avg</file>:
</para><example endspaces=" ">
<pre xml:space="preserve">nces --nsm_grp                  mdl1.nc mdl2.nc mdl3.nc out.nc
nces --nsm_grp --nsm_sfx='_avg' mdl1.nc mdl2.nc mdl3.nc out.nc
</pre></example>

<para><xref label="Statistics-vs_002e-Concatenation"><xrefnodename>Statistics vs. Concatenation</xrefnodename></xref>, for a description of the
distinctions between the statistics tools and concatenators. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1229">multi-file operators</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1230">standard input</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1231"><code>stdin</code></indexterm></cindex>
As a multi-file operator, <command>nces</command> will read the list of
<var>input-files</var> from <code>stdin</code> if they are not specified 
as positional arguments on the command line 
(<pxref label="Large-Numbers-of-Files"><xrefnodename>Large Numbers of Files</xrefnodename></pxref>).
</para>
<para>Like <command>ncra</command> and <command>ncwa</command>, <command>nces</command> treats coordinate
variables as a special case.
Coordinate variables are assumed to be the same in all ensemble members,
so <command>nces</command> simply copies the coordinate variables that appear in 
ensemble members directly to the output file.
This has the same effect as averaging the coordinate variable across the
ensemble, yet does not incur the time- or precision- penalties of
actually averaging them.
<command>ncra</command> and <command>ncwa</command> allow coordinate variables to be
processed only by the linear average operation, regardless of the
arithmetic operation type performed on the non-coordinate variables
(<pxref label="Operation-Types"><xrefnodename>Operation Types</xrefnodename></pxref>). 
Thus it can be said that the three operators (<command>ncra</command>,
<command>ncwa</command>, and <command>nces</command>) all average coordinate variables
(even though <command>nces</command> simply copies them).
All other requested arithmetic operations (e.g., maximization,
square-root, RMS) are applied only to non-coordinate variables.
In these cases the linear average of the coordinate variable will be
returned.
</para>
<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncea&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncea &textndash;&gt;
&lt;a name=&quot;xmp_nces&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_nces &textndash;&gt;
</html>
<para>EXAMPLES
</para>
<para>Consider a model experiment which generated five realizations of one
year of data, say 1985.
Imagine that the experimenter slightly perturbs the initial conditions
of the problem before generating each new solution.   
Assume each file contains all twelve months (a seasonal cycle) of data
and we want to produce a single file containing the ensemble average
(mean) seasonal cycle.  
Here the numeric filename suffix denotes the realization number
(<emph>not</emph> the month):
</para><example endspaces=" ">
<pre xml:space="preserve">nces 85_01.nc 85_02.nc 85_03.nc 85_04.nc 85_05.nc 85.nc
nces 85_0[1-5].nc 85.nc
nces -n 5,2,1 85_01.nc 85.nc
</pre></example>
<noindent></noindent>
<para>These three commands produce identical answers.
<xref label="Specifying-Input-Files"><xrefnodename>Specifying Input Files</xrefnodename></xref>, for an explanation of the distinctions
between these methods.
The output file, <file>85.nc</file>, is the same size as the inputs files.
It contains 12 months of data (which might or might not be stored in the 
record dimension, depending on the input files), but each value in the
output file is the average of the five values in the input files.
</para>
<para>In the previous example, the user could have obtained the ensemble
average values in a particular spatio-temporal region by adding a 
hyperslab argument to the command, e.g.,
</para><example endspaces=" ">
<pre xml:space="preserve">nces -d time,0,2 -d lat,-23.5,23.5 85_??.nc 85.nc
</pre></example>
<noindent></noindent>
<para>In this case the output file would contain only three slices of data in
the <var>time</var> dimension. 
These three slices are the average of the first three slices from the
input files.
Additionally, only data inside the tropics is included.
</para>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.3.9 (released December, 2013)
<command>nces</command> also works with groups (rather than files) as the
fundamental unit of the ensemble.
Consider two ensembles, <code>/ecmwf</code> and <code>/cesm</code> stored across
three input files <file>mdl1.nc</file>, <file>mdl2.nc</file>, and <file>mdl3.nc</file>.
Ensemble members would be leaf groups with names like <code>/ecmwf/01</code>,
<code>/ecmwf/02</code> etc. and <code>/cesm/01</code>, <code>/cesm/02</code>, etc.
These commands average both ensembles: 
</para><example endspaces=" ">
<pre xml:space="preserve">nces --nsm_grp mdl1.nc mdl2.nc mdl3.nc out.nc
nces --nsm_grp --nsm_sfx='_min' --op_typ=min -n 3,1,1 mdl1.nc out.nc
nces --nsm_grp -g cesm -v tas -d time,0,3 -n 3,1,1 mdl1.nc out.nc
</pre></example>
<para>The first command stores averages in the output groups <file>/cesm</file> and 
<file>/ecmwf</file>, while the second stores minima in the output groups
<file>/cesm/cesm_min</file> and <file>/ecmwf/ecmwf_min</file>:
The third command demonstrates that sub-setting and hyperslabbing work
as expected.
Note that each input file may contain different numbers of members
of each ensemble, as long as all distinct ensembles contain at least one
member in the first file.
</para>
<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncecat&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncecat &textndash;&gt;
</html>
</section>
<node name="ncecat-netCDF-Ensemble-Concatenator" spaces=" "><nodename>ncecat netCDF Ensemble Concatenator</nodename><nodenext spaces=" ">ncflint netCDF File Interpolator</nodenext><nodeprev spaces=" ">nces netCDF Ensemble Statistics</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncecat</command> netCDF Ensemble Concatenator</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1232">concatenation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1233">ensemble concatenation</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="31" mergedindex="cp">ncecat</indexterm></findex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">ncecat [-3] [-4] [-6] [-7] [-A] [-C] [-c]
[--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz]
[-D <var>dbg</var>] [-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]] [-F]
[-G <var>gpe_dsc</var>] [-g <var>grp</var>[,&dots;]] [--gag] [-h] [--hdf] [--hdr_pad <var>nbr</var>]
[-L <var>dfl_lvl</var>] [-l <var>path</var>] [-M] [--md5_digest] [--mrd] [-n <var>loop</var>] [--no_tmp_fl] 
[-O] [-o <var>output-file</var>] [-p <var>path</var>] [-R] [-r] [--ram_all] 
[-t <var>thr_nbr</var>] [-u <var>ulm_nm</var>] [--unn] [-v <var>var</var>[,&dots;]] [-X ...] [-x] 
[<var>input-files</var>] [<var>output-file</var>]
</pre></example>

<noindent></noindent>
<para>DESCRIPTION
</para>
<para><command>ncecat</command> aggregates an arbitrary number of input files into a 
single output file using using one of two methods.
<dfn>Record AGgregation</dfn> (<acronym><acronymword>RAG</acronymword></acronym>), the traditional method employed on
netCDF3 files and still the default method, stores <var>input-files</var> as
consecutive records in the <var>output-file</var>.
<dfn>Group AGgregation</dfn> (<acronym><acronymword>GAG</acronymword></acronym>) stores <var>input-files</var> as top-level
groups in the netCDF4 <var>output-file</var>.
Record Aggregation (<acronym><acronymword>RAG</acronymword></acronym>) makes numerous assumptions about the structure
of input files and Group Aggregation (<acronym><acronymword>GAG</acronymword></acronym>) makes none.
Both methods are described in detail below.
Since <command>ncecat</command> aggregates all the contents of the input files,
it can easily produce large output files so it is often helpful to invoke
subsetting simultaneously (<pxref label="Subsetting-Files"><xrefnodename>Subsetting Files</xrefnodename></pxref>).
</para>
<html endspaces=" ">
&lt;a name=&quot;rag&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rag &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1234">record aggregation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1235"><acronym><acronymword>RAG</acronymword></acronym></indexterm></cindex>
<para><acronym><acronymword>RAG</acronymword></acronym> makes each variable (except coordinate variables) in each
input file into a single record of the same variable in the output file.  
Coordinate variables are not concatenated, they are instead simply
copied from the first input file to the <var>output-file</var>.
All <var>input-files</var> must contain all extracted variables (or else
there would be &quot;gaps&quot; in the output file).
</para>
<para>A new record dimension is the glue which binds together the input file
data.
The new record dimension is defined in the root group of the output file
so it is visible to all sub-groups.
Its name is, by default, &textldquo;record&textrdquo;.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1236">unlimited dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1237">record dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1238"><samp>-u <var>ulm_nm</var></samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1239"><samp>--ulm_nm <var>ulm_nm</var></samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1240"><samp>--rcd_nm <var>ulm_nm</var></samp></indexterm></cindex>
This default name can be overridden with the <samp>-u <var>ulm_nm</var></samp>
short option (or the <samp>--ulm_nm</samp> or <samp>rcd_nm</samp> long options).
</para>
<para>Each extracted variable must be constant in size and rank across all
<var>input-files</var>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1241">record dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1242">hyperslab</indexterm></cindex>
The only exception is that <command>ncecat</command> allows files to differ in
the record dimension size if the requested record hyperslab
(<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>) resolves to the same size for all files. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1243"><acronym><acronymword>CMIP</acronymword></acronym></indexterm></cindex>
This allows easier gluing/averaging of unequal length timeseries from 
simulation ensembles (e.g., the <acronym><acronymword>CMIP</acronymword></acronym> rchive). 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1244">fixed dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1245">fix record dimension</indexterm></cindex>
<para>Classic (i.e., all netCDF3 and <code>NETCDF4_CLASSIC</code>) output files
can contain only one record dimension.
<command>ncecat</command> makes room for the new glue record dimension by
changing the pre-existing record dimension, if any, in the input files
into a fixed dimension in the output file. 
netCDF4 output files may contain any number of record dimensions, so
<command>ncecat</command> need not and does not alter the record dimensions,
if any, of the input files as it copies them to the output file. 
</para>
<html endspaces=" ">
&lt;a name=&quot;gag&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#gag &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1246">group aggregation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1247"><acronym><acronymword>GAG</acronymword></acronym></indexterm></cindex>
<para><dfn>Group AGgregation</dfn> (<acronym><acronymword>GAG</acronymword></acronym>) stores <var>input-files</var> as
top-level groups in the <var>output-file</var>.
No assumption is made about the size or shape or type of a given 
object (variable or dimension or group) in the input file.
The entire contents of the extracted portion of each input file
is placed in its own top-level group in <var>output-file</var>, which
is automatically made as a netCDF4-format file.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1248"><option>--gag</option></indexterm></cindex>
<para><acronym><acronymword>GAG</acronymword></acronym> has two methods to specify group names for the
<var>output-file</var>.    
The <samp>-G</samp> option, or its long-option equivalent <samp>--gpe</samp>,
takes as argument a group path editing description <var>gpe_dsc</var> of
where to place the results.
Each input file needs a distinct output group name to avoid namespace
conflicts in the <var>output-file</var>. 
Hence <command>ncecat</command> automatically creates unique output group names
based on either the input filenames or the <var>gpe_dsc</var> arguments.
When the user provides <var>gpe_dsc</var> (i.e., with <samp>-G</samp>), then the
output groups are formed by enumerating sequential two-digit numeric
suffixes starting with zero, and appending them to the specified group
path (<pxref label="Group-Path-Editing"><xrefnodename>Group Path Editing</xrefnodename></pxref>).
When <var>gpe_dsc</var> is not provided (i.e., user requests <acronym><acronymword>GAG</acronymword></acronym> with
<samp>--gag</samp> instead of <samp>-G</samp>), then <command>ncecat</command> forms the
output groups by stripping the input file name of any type-suffix
(e.g., <code>.nc</code>), and all but the final component of the full
filename. 
</para><example endspaces=" ">
<pre xml:space="preserve">ncecat --gag 85.nc 86.nc 87.nc 8587.nc # Output groups 85, 86, 87
ncecat -G 85_ a.nc b.nc c.nc 8589.nc # Output groups 85_00, 85_01, 85_02
ncecat -G 85/ a.nc b.nc c.nc 8589.nc # Output groups 85/00, 85/01, 85/02
</pre></example>

<para>With both <acronym><acronymword>RAG</acronymword></acronym> and <acronym><acronymword>GAG</acronymword></acronym> the <var>output-file</var> size is
the sum of the sizes of the extracted variables in the input files. 
<xref label="Statistics-vs_002e-Concatenation"><xrefnodename>Statistics vs. Concatenation</xrefnodename></xref>, for a description of the
distinctions between the various statistics tools and concatenators. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1249">multi-file operators</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1250">standard input</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1251"><code>stdin</code></indexterm></cindex>
As a multi-file operator, <command>ncecat</command> will read the list of
<var>input-files</var> from <code>stdin</code> if they are not specified 
as positional arguments on the command line 
(<pxref label="Large-Numbers-of-Files"><xrefnodename>Large Numbers of Files</xrefnodename></pxref>).
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1252"><code>-M</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1253"><code>--glb_mtd_spp</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1254">metadata, global</indexterm></cindex>
<para>Suppress global metadata copying.
By default <acronym><acronymword>NCO</acronymword></acronym>&textrsquo;s multi-file operators copy the global metadata
from the first input file into <var>output-file</var>.  
This helps to preserve the provenance of the output data.
However, the use of metadata is burgeoning and is not uncommon to
encounter files with excessive amounts of extraneous metadata.
Extracting small bits of data from such files leads to output files
which are much larger than necessary due to the automatically copied
metadata.
<command>ncecat</command> supports turning off the default copying of global
metadata via the <samp>-M</samp> switch (or its long option equivalents,
<samp>--glb_mtd_spp</samp> and <samp>--global_metadata_suppress</samp>). 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1255">climate model</indexterm></cindex>
<para>Consider five realizations, <file>85a.nc</file>, <file>85b.nc</file>, 
<w>&dots; <file>85e.nc</file></w> of 1985 predictions from the same climate
model. 
Then <code>ncecat 85?.nc 85_ens.nc</code> glues together the individual
realizations into the single file, <file>85_ens.nc</file>. 
If an input variable was dimensioned [<code>lat</code>,<code>lon</code>], it will
by default have dimensions [<code>record</code>,<code>lat</code>,<code>lon</code>] in
the output file. 
<w>A restriction</w> of <command>ncecat</command> is that the hyperslabs of the
processed variables must be the same from file to file.
Normally this means all the input files are the same size, and contain 
data on different realizations of the same variables.
</para>
<findex index="fn" spaces=" "><indexterm index="fn" number="32" mergedindex="cp">ncpdq</indexterm></findex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1256">packing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1257">unpacking</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1258"><code>add_offset</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1259"><code>scale_factor</code></indexterm></cindex>
<para>Concatenating a variable packed with different scales across multiple
datasets is beyond the capabilities of <command>ncecat</command> (and
<command>ncrcat</command>, the other concatenator (<ref label="Concatenation"><xrefnodename>Concatenation</xrefnodename></ref>).
<command>ncecat</command> does not unpack data, it simply <emph>copies</emph> the data
from the <var>input-files</var>, and the metadata from the <emph>first</emph>
<var>input-file</var>, to the <var>output-file</var>. 
This means that data compressed with a packing convention must use
the identical packing parameters (e.g., <code>scale_factor</code> and
<code>add_offset</code>) for a given variable across <emph>all</emph> input files.
Otherwise the concatenated dataset will not unpack correctly.
The workaround for cases where the packing parameters differ across
<var>input-files</var> requires three steps:
First, unpack the data using <command>ncpdq</command>.
Second, concatenate the unpacked data using <command>ncecat</command>, 
Third, re-pack the result with <command>ncpdq</command>.
</para>
<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncecat&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncecat &textndash;&gt;
</html>
<para>EXAMPLES
</para>
<para>Consider a model experiment which generated five realizations of one
year of data, say 1985.
You can imagine that the experimenter slightly perturbs the
initial conditions of the problem before generating each new solution.  
Assume each file contains all twelve months (a seasonal cycle) of data
and we want to produce a single file containing all the seasonal
cycles. 
Here the numeric filename suffix denotes the experiment number
(<emph>not</emph> the month):
</para><example endspaces=" ">
<pre xml:space="preserve">ncecat 85_01.nc 85_02.nc 85_03.nc 85_04.nc 85_05.nc 85.nc
ncecat 85_0[1-5].nc 85.nc
ncecat -n 5,2,1 85_01.nc 85.nc
</pre></example>
<noindent></noindent>
<para>These three commands produce identical answers.
<xref label="Specifying-Input-Files"><xrefnodename>Specifying Input Files</xrefnodename></xref>, for an explanation of the distinctions
between these methods.
The output file, <file>85.nc</file>, is five times the size as a single
<var>input-file</var>. 
It contains <w>60 months</w> of data.
</para>
<html endspaces=" ">
&lt;a name=&quot;ncecat_rnm&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncecat_rnm &textndash;&gt;
</html>
<para>One often prefers that the (new) record dimension have a more
descriptive, context-based name than simply &textldquo;record&textrdquo;. 
This is easily accomplished with the <samp>-u <var>ulm_nm</var></samp> switch:
</para><example endspaces=" ">
<pre xml:space="preserve">ncecat -u realization 85_0[1-5].nc 85.nc
</pre></example>
<noindent></noindent>
<para>Users are more likely to understand the data processing history when
such descriptive coordinates are used. 
</para>
<html endspaces=" ">
&lt;a name=&quot;dmn_rcd_rm&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dmn_rcd_rm &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1260">record dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1261">fixed dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1262">fix record dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1263"><code>--mk_rec_dmn <var>dim</var></code></indexterm></cindex>
<para>Consider a file with an existing record dimension named <code>time</code>. 
and suppose the user wishes to convert <code>time</code> from a record
dimension to a non-record dimension.
This may be useful, for example, when the user has another use for the
record variable.
The simplest method is to use <samp>ncks --fix_rec_dmn</samp> but another
possibility is to use <command>ncecat</command> followed by 
<command>ncwa</command>: 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1264">degenerate dimension</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">ncecat in.nc out.nc # Convert time to non-record dimension
ncwa -a record in.nc out.nc # Remove new degenerate record dimension
</pre></example>
<noindent></noindent>
<para>The second step removes the degenerate record dimension.
See <ref label="ncpdq-netCDF-Permute-Dimensions-Quickly"><xrefnodename>ncpdq netCDF Permute Dimensions Quickly</xrefnodename></ref> and
<ref label="ncks-netCDF-Kitchen-Sink"><xrefnodename>ncks netCDF Kitchen Sink</xrefnodename></ref> for other methods of
of changing variable dimensionality, including the record dimension.
</para>
<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncflint&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncflint &textndash;&gt;
</html>
</section>
<node name="ncflint-netCDF-File-Interpolator" spaces=" "><nodename>ncflint netCDF File Interpolator</nodename><nodenext spaces=" ">ncks netCDF Kitchen Sink</nodenext><nodeprev spaces=" ">ncecat netCDF Ensemble Concatenator</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncflint</command> netCDF File Interpolator</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1265">interpolation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1266">adding data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1267">multiplying data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1268">addition</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="33" mergedindex="cp">ncflint</indexterm></findex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">ncflint [-3] [-4] [-6] [-7] [-A] [-C] [-c]
[--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz]
[-D <var>dbg</var>] [-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]
[-F] [--fix_rec_crd] [-G <var>gpe_dsc</var>] [-g <var>grp</var>[,&dots;]] [-h] [--hdr_pad <var>nbr</var>]
[-i <var>var</var>,<var>val3</var>] [-L <var>dfl_lvl</var>] [-l <var>path</var>] [--no_tmp_fl] 
[-O] [-o <var>file_3</var>] [-p <var>path</var>] [-R] [-r] [--ram_all] 
[-t <var>thr_nbr</var>] [--unn] [-v <var>var</var>[,&dots;]] [-w <var>wgt1</var>[,<var>wgt2</var>]] [-X ...] [-x]
<var>file_1</var> <var>file_2</var> [<var>file_3</var>]
</pre></example>

<noindent></noindent>
<para>DESCRIPTION
</para>
<para><command>ncflint</command> creates an output file that is a linear combination of 
the input files.
This linear combination is a weighted average, a normalized weighted
average, or an interpolation of the input files.
Coordinate variables are not acted upon in any case, they are simply
copied from <var>file_1</var>.
</para>
<para>There are two conceptually distinct methods of using <command>ncflint</command>.
The first method is to specify the weight each input file contributes to 
the output file.
In this method, the value <var>val3</var> of a variable in the output file
<var>file_3</var> is determined from its values <var>val1</var> and <var>val2</var> in
the two input files according to 
<set name="flg" line=" flg"></set>
<tex endspaces=" ">
$val3 = wgt1 \times val1 + wgt2 \times val2$
<clear name="flg" line=" flg"></clear>
</tex>
<math><var>val3</var> = <var>wgt1</var>*<var>val1</var> + <var>wgt2</var>*<var>val2</var></math> 
<clear name="flg" line=" flg"></clear>
.
Here at least <var>wgt1</var>, and, optionally, <var>wgt2</var>, are specified on 
the command line with the <samp>-w</samp> (or <samp>--weight</samp> or
<samp>--wgt_var</samp>) switch.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1269"><code>-w <var>wgt1</var>[,<var>wgt2</var>]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1270"><code>--weight <var>wgt1</var>[,<var>wgt2</var>]</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1271"><code>--wgt_var <var>wgt1</var>[,<var>wgt2</var>]</code></indexterm></cindex>
If only <var>wgt1</var> is specified then <var>wgt2</var> is automatically
computed as <math><var>wgt2</var> = 1 &minus; <var>wgt1</var></math>.
Note that weights larger <w>than 1</w> are allowed. 
Thus it is possible to specify <math><var>wgt1</var> = 2</math> and
<math><var>wgt2</var> = -3</math>.
One can use this functionality to multiply all the values in a given
file by a constant.
</para>
<para>The second method of using <command>ncflint</command> is to specify the
interpolation option <w>with <samp>-i</samp></w> (or with the <samp>--ntp</samp> or 
<samp>--interpolate</samp> long options). 
This is the inverse of the first method in the following sense: 
When the user specifies the weights directly, <command>ncflint</command> has no
work to do besides multiplying the input values by their respective
weights and adding together the results to produce the output values.  
It makes sense to use this when the weights are known 
<emph><w>a priori</w></emph>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1272">arrival value</indexterm></cindex>
<para>Another class of problems has the <dfn>arrival value</dfn> (i.e., <var>val3</var>)
of a particular variable <var>var</var> known <emph><w>a priori</w></emph>. 
In this case, the implied weights can always be inferred by examining
the values of <var>var</var> in the input files. 
This results in one equation in two unknowns, <var>wgt1</var> and <var>wgt2</var>:  
<set name="flg" line=" flg"></set>
<tex endspaces=" ">
$val3 = wgt1 \times val1 + wgt2 \times val2$
<clear name="flg" line=" flg"></clear>
</tex>
<math><var>val3</var> = <var>wgt1</var>*<var>val1</var> + <var>wgt2</var>*<var>val2</var></math> 
<clear name="flg" line=" flg"></clear>
.
Unique determination of the weights requires imposing the additional
constraint of normalization on the weights:
<math><var>wgt1</var> + <var>wgt2</var> = 1</math>.
Thus, to use the interpolation option, the user specifies <var>var</var>
and <var>val3</var> with the <samp>-i</samp> option.
<command>ncflint</command> then computes <var>wgt1</var> and <var>wgt2</var>, and uses these
weights on all variables to generate the output file.
Although <var>var</var> may have any number of dimensions in the input
files, it must represent a single, scalar value.  
<cindex index="cp" spaces=" "><indexterm index="cp" number="1273">degenerate dimension</indexterm></cindex>
Thus any dimensions associated with <var>var</var> must be <dfn>degenerate</dfn>,
i.e., of size one.
</para>
<para>If neither <samp>-i</samp> nor <samp>-w</samp> is specified on the command line,
<command>ncflint</command> defaults to weighting each input file equally in the
output file.
This is equivalent to specifying <samp>-w 0.5</samp> or <samp>-w 0.5,0.5</samp>.
Attempting to specify both <samp>-i</samp> and <samp>-w</samp> methods in the same
command is an error. 
</para>
<para><command>ncflint</command> does not interpolate variables of type <code>NC_CHAR</code>
and <code>NC_STRING</code>. 
This behavior is hardcoded.
</para>
<para>By default <command>ncflint</command> interpolates or multiplies record
coordinate variables (e.g., time is often stored as a record coordinate) 
not other coordinate variables (e.g., latitude and longitude). 
This is because <command>ncflint</command> is often used to time-interpolate
between existing files, but is rarely used to spatially interpolate.
Sometimes however, users wish to multiply entire files by a constant
that does not multiply any coordinate variables.
The <samp>--fix_rec_crd</samp> switch was implemented for this purpose
in <acronym><acronymword>NCO</acronymword></acronym> version 4.2.6 (March, 2013).
It prevents <command>ncflint</command> from multiplying or interpolating any
coordinate variables, including record coordinate variables. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1274">missing values</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1275"><code>_FillValue</code></indexterm></cindex>
<para>Depending on your intuition, <command>ncflint</command> may treat missing values
unexpectedly.
Consider a point where the value in one input file, say <var>val1</var>,
equals the missing value <var>mss_val_1</var> and, at the same point,
the corresponding value in the other input file <var>val2</var> is not
misssing (i.e., does not equal <var>mss_val_2</var>).
There are three plausible answers, and this creates ambiguity.
</para>
<para><w>Option one</w> is to set <math><var>val3</var> = <var>mss_val_1</var></math>.
The rationale is that <command>ncflint</command> is, at heart, an interpolator
and interpolation involving a missing value is intrinsically undefined.
<command>ncflint</command> currently implements this behavior since it is the
most conservative and least likely to lead to misinterpretation.
</para>
<para><w>Option two</w> is to output the weighted valid data point, i.e.,
<set name="flg" line=" flg"></set>
<tex endspaces=" ">
$val3 = wgt2 \times val2$
<clear name="flg" line=" flg"></clear>
</tex>
<math><var>val3</var> = <var>wgt2</var>*<var>val2</var></math> 
<clear name="flg" line=" flg"></clear>
.
The rationale for this behavior is that interpolation is really a
weighted average of known points, so <command>ncflint</command> should weight the
valid point. 
</para>
<para><w>Option three</w> is to return the <emph>unweighted</emph> valid point, i.e.,
<math><var>val3</var> = <var>val2</var></math>.
This behavior would appeal to those who use <command>ncflint</command> to
estimate data using the closest available data. 
When a point is not bracketed by valid data on both sides, it is better
to return the known datum than no datum at all.
</para>
<para>The current implementation uses the first approach, <w>Option one</w>.
If you have strong opinions on this matter, let us know, since we are
willing to implement the other approaches as options if there is enough
interest. 
</para>
<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncflint&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncflint &textndash;&gt;
</html>
<para>EXAMPLES
</para>
<para>Although it has other uses, the interpolation feature was designed 
to interpolate <var>file_3</var> to a time between existing files.
Consider input files <file>85.nc</file> and <file>87.nc</file> containing variables 
describing the state of a physical system at times <math><code>time</code> =
85</math> and <math><code>time</code> = 87</math>.
Assume each file contains its timestamp in the scalar variable
<code>time</code>.  
Then, to linearly interpolate to a file <file>86.nc</file> which describes
the state of the system at time at <code>time</code> = 86, we would use
</para><example endspaces=" ">
<pre xml:space="preserve">ncflint -i time,86 85.nc 87.nc 86.nc
</pre></example>

<para>Say you have observational data covering January and April 1985 in two
files named <file>85_01.nc</file> and <file>85_04.nc</file>, respectively.
Then you can estimate the values for February and March by interpolating
the existing data as follows.
Combine <file>85_01.nc</file> and <file>85_04.nc</file> in a 2:1 ratio to make
<file>85_02.nc</file>:  
</para><example endspaces=" ">
<pre xml:space="preserve">ncflint -w 0.667 85_01.nc 85_04.nc 85_02.nc
ncflint -w 0.667,0.333 85_01.nc 85_04.nc 85_02.nc
</pre></example>

<para>Multiply <file>85.nc</file> <w>by 3</w> and <w>by &minus;2</w> and add them
together to make <file>tst.nc</file>: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncflint -w 3,-2 85.nc 85.nc tst.nc
</pre></example>
<noindent></noindent>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1276">null operation</indexterm></cindex>
<para>This is an example of a null operation, so <file>tst.nc</file> should be
identical (within machine precision) to <file>85.nc</file>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1277">multiplication</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1278">file multiplication</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1279">scaling</indexterm></cindex>
<para>Multiply all the variables except the coordinate variables in the file
<file>emissions.nc</file> by <w>by 0.8</w>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncflint --fix_rec_crd -w 0.8,0.0 emissions.nc emissions.nc scaled_emissions.nc
</pre></example>
<noindent></noindent>
<para>The use of <samp>--fix_rec_crd</samp> ensures, e.g., that the <code>time</code>
coordinate, if any, is not scaled (i.e., multiplied).
</para>
<para>Add <file>85.nc</file> to <file>86.nc</file> to obtain <file>85p86.nc</file>,
then subtract <file>86.nc</file> from <file>85.nc</file> to obtain <file>85m86.nc</file> 
</para><example endspaces=" ">
<pre xml:space="preserve">ncflint -w 1,1 85.nc 86.nc 85p86.nc
ncflint -w 1,-1 85.nc 86.nc 85m86.nc
ncdiff 85.nc 86.nc 85m86.nc
</pre></example>
<noindent></noindent>
<para>Thus <command>ncflint</command> can be used to mimic some <command>ncbo</command>
operations. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1280">broadcasting variables</indexterm></cindex>
However this is not a good idea in practice because <command>ncflint</command>
does not broadcast (<pxref label="ncbo-netCDF-Binary-Operator"><xrefnodename>ncbo netCDF Binary Operator</xrefnodename></pxref>) conforming
variables during arithmetic. 
Thus the final two commands would produce identical results except that    
<command>ncflint</command> would fail if any variables needed to be broadcast.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1281"><code>units</code></indexterm></cindex>
<para>Rescale the dimensional units of the surface pressure <code>prs_sfc</code>
from Pascals to hectopascals (millibars)
</para><example endspaces=" ">
<pre xml:space="preserve">ncflint -C -v prs_sfc -w 0.01,0.0 in.nc in.nc out.nc
ncatted -a units,prs_sfc,o,c,millibar out.nc
</pre></example>
<noindent></noindent>

<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncks&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncks &textndash;&gt;
</html>
</section>
<node name="ncks-netCDF-Kitchen-Sink" spaces=" "><nodename>ncks netCDF Kitchen Sink</nodename><nodenext spaces=" ">ncpdq netCDF Permute Dimensions Quickly</nodenext><nodeprev spaces=" ">ncflint netCDF File Interpolator</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncks</command> netCDF Kitchen Sink</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1282">kitchen sink</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1283">printing files contents</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1284">printing variables</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="34" mergedindex="cp">ncks</indexterm></findex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">ncks [-3] [-4] [-5] [-6] [-7] [-A] [-a] [-b <var>binary-file</var>] [-C] [-c] [--cdl]
[--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz]
[-D <var>dbg</var>] [-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]] [-F] [--fix_rec_dmn <var>dim</var>] 
[-G <var>gpe_dsc</var>] [-g <var>grp</var>[,&dots;]] [-H] [-h] [--hdn] [--hdr_pad <var>nbr</var>]
[-L <var>dfl_lvl</var>] [-l <var>path</var>] [-M] [-m] [--mk_rec_dmn <var>dim</var>] [--md5_digest]
[--no_blank] [--no_tmp_fl] [-O] [-o <var>output-file</var>] [-P] [-p <var>path</var>] 
[-Q] [-q] [-R] [-r] [--ram_all] [-s <var>format</var>] 
[-u] [--unn] [-v <var>var</var>[,&dots;]] [-X ...] [-x] [--xml]
<var>input-file</var> [[<var>output-file</var>]]
</pre></example>

<noindent></noindent>
<para>DESCRIPTION
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1285"><command>ncextr</command></indexterm></cindex>
<para>The nickname &textldquo;kitchen sink&textrdquo; is a catch-all because <command>ncks</command>
combines most features of <command>ncdump</command> and <command>nccopy</command> with
extra features to extract, hyperslab, multi-slab, sub-set, and translate  
into one versatile utility. 
<command>ncks</command> extracts (a subset of the) data from <var>input-file</var> and 
and writes (or pastes) it in netCDF format to <var>output-file</var>, and 
optionally writes it in flat binary format to <file>binary-file</file>, and
optionally prints it to screen.
</para>
<para><command>ncks</command> prints netCDF input data in <acronym><acronymword>ASCII</acronymword></acronym>,
<acronym><acronymword>CDL</acronymword></acronym>, or <acronym><acronymword>NcML</acronymword></acronym> text formats <code>stdout</code>, like (an
extended version of) <command>ncdump</command>.
By default <command>ncks</command> prints data in a tabular format intended to be
easy to search for the data you want, one datum per screen line, with
all dimension subscripts and coordinate values (if any) preceding the
datum. 
Option <samp>-s</samp> (or long options <samp>--sng_fmt</samp> and <samp>--string</samp>) 
permits the user to format data using C-style format strings, while
option <samp>--cdl</samp> outputs <acronym><acronymword>CDL</acronymword></acronym> and option <samp>--xml</samp>
outputs <acronym><acronymword>NcML</acronymword></acronym>. 
<command>ncks</command> exposes many flexible controls over printed output,
including <acronym><acronymword>CDL</acronymword></acronym> and <acronym><acronymword>NcML</acronymword></acronym>.
</para>
<para>Options <samp>-5</samp>, <samp>-a</samp>, <samp>--cdl</samp>, <samp>-F</samp> , <samp>-H</samp>,
<samp>--hdn</samp>, <samp>-M</samp>, <samp>-m</samp>, <samp>-P</samp>, <samp>-Q</samp>, <samp>-q</samp>,
<samp>-s</samp>, <samp>-u</samp>, <samp>--xml</samp> (and their long option counterparts)
control the formatted appearance of the data.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1286">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1287">attributes, global</indexterm></cindex>
<para><command>ncks</command> extracts (and optionally creates a new netCDF file
comprised of) only selected variables from the input file
(similar to the old <command>ncextr</command> specification).
Only variables and coordinates may be specifically included or
excluded&textmdash;all global attributes and any attribute associated with an
extracted variable are copied to the screen and/or output netCDF file. 
Options <samp>-c</samp>, <samp>-C</samp>, <samp>-v</samp>, and <samp>-x</samp> (and their long 
option synonyms) control which variables are extracted.
</para>
<para><command>ncks</command> extracts hyperslabs from the specified variables
(<command>ncks</command> implements the original <command>nccut</command> specification). 
Option <samp>-d</samp> controls the hyperslab specification.
Input dimensions that are not associated with any output variable do
not appear in the output netCDF.
This feature removes superfluous dimensions from netCDF files. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1288">appending data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1289">merging files</indexterm></cindex>
<para><command>ncks</command> will append variables and attributes from the
<var>input-file</var> to <var>output-file</var> if <var>output-file</var> is a
pre-existing netCDF file whose relevant dimensions conform to dimension
sizes of <var>input-file</var>. 
The append features of <command>ncks</command> are intended to provide a
rudimentary means of adding data from one netCDF file to another,
conforming, netCDF file. 
If naming conflicts exist between the two files, data in
<var>output-file</var> is usually overwritten by the corresponding data from 
<var>input-file</var>.  
Thus, when appending, the user should backup <var>output-file</var> in case 
valuable data are inadvertantly overwritten.
</para>
<para>If <var>output-file</var> exists, the user will be queried whether to
<dfn>overwrite</dfn>, <dfn>append</dfn>, or <dfn>exit</dfn> the <command>ncks</command> call
completely.  
Choosing <dfn>overwrite</dfn> destroys the existing <var>output-file</var> and
create an entirely new one from the output of the <command>ncks</command> call.  
Append has differing effects depending on the uniqueness of the
variables and attributes output by <command>ncks</command>: If a variable or
attribute extracted from <var>input-file</var> does not have a name conflict
with the members of <var>output-file</var> then it will be added to
<var>output-file</var> without overwriting any of the existing contents of
<var>output-file</var>.  
In this case the relevant dimensions must agree (conform) between the
two files; new dimensions are created in <var>output-file</var> as required. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1290">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1291">attributes, global</indexterm></cindex>
When a name conflict occurs, a global attribute from <var>input-file</var>
will overwrite the corresponding global attribute from
<var>output-file</var>.  
If the name conflict occurs for a non-record variable, then the
dimensions and type of the variable (and of its coordinate dimensions,
if any) must agree (conform) in both files. 
Then the variable values (and any coordinate dimension values)
from <var>input-file</var> will overwrite the corresponding variable values
(and coordinate dimension values, if any) in <var>output-file</var> 
<footnote spaces="\n"><para>Those familiar with netCDF mechanics might wish to know what is
happening here: <command>ncks</command> does not attempt to redefine the variable
in <var>output-file</var> to match its definition in <var>input-file</var>,
<command>ncks</command> merely copies the values of the variable and its
coordinate dimensions, if any, from <var>input-file</var> to
<var>output-file</var>. 
</para></footnote>.
</para>
<para>Since there can only be one record dimension in a file, the record
dimension must have the same name (though not necessarily the same size) in 
both files if a record dimension variable is to be appended. 
If the record dimensions are of differing sizes, the record dimension of
<var>output-file</var> will become the greater of the two record dimension
sizes, the record variable from <var>input-file</var> will overwrite any
counterpart in <var>output-file</var> and fill values will be written to any
gaps left in the rest of the record variables (I think). 
In all cases variable attributes in <var>output-file</var> are superseded by
attributes of the same name from <var>input-file</var>, and left alone if
there is no name conflict. 
</para>
<para>Some users may wish to avoid interactive <command>ncks</command> queries about
whether to overwrite existing data.
For example, batch scripts will fail if <command>ncks</command> does not receive 
responses to its queries. 
Options <samp>-O</samp> and <samp>-A</samp> are available to force overwriting
existing files and variables, respectively. 
</para>
<unnumberedsubsec spaces=" "><sectiontitle>Options specific to <command>ncks</command></sectiontitle>

<para>The following list provides a short summary of the features unique to
<command>ncks</command>.  
Features common to many operators are described in 
<ref label="Common-features"><xrefnodename>Common features</xrefnodename></ref>. 
</para>
<table commandarg="samp" spaces=" " endspaces=" ">
<beforefirstitem>
<html endspaces=" ">
&lt;a name=&quot;-5&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-5 &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1292"><code>-5</code></indexterm></cindex>
</beforefirstitem><tableentry><tableterm><item spaces=" "><itemformat command="samp">-5 </itemformat></item>
</tableterm><tableitem><para>Print data to screen alphabetically by group, and alphabetically by
variable within each group.
This ordering here is used by default in <acronym><acronymword>CDL</acronymword></acronym>-mode printing,
and may be selected for traditional mode printing with <samp>-5</samp>
(The switch for invocation may change to something more descriptive in
the future). 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1293">alphabetization</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1294">sort alphabetically</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1295"><code>-a</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1296"><code>--abc</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1297"><code>--alphabetize</code></indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-a</itemformat></item>
</tableterm><tableitem><para>Do not alphabetize extracted fields. 
By default, the specified output variables are extracted, printed, and
written to disk in alphabetical order.
This tends to make long output lists easier to search for particular
variables. 
Specifying <code>-a</code> results in the variables being extracted, printed,
and written to disk in the order in which they were saved in the input
file.
Thus <code>-a</code> retains the original ordering of the variables.
Also <samp>--abc</samp> and <samp>--alphabetize</samp>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1298">binary format</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1299"><code>-b</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1300"><code>--fl_bnr</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1301"><code>--bnr</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1302"><code>--binary</code></indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-b <file>file</file></itemformat></item>
</tableterm><tableitem><para>Activate native machine binary output writing to binary file
<file>file</file>.
Also <samp>--fl_bnr</samp> and <samp>--binary-file</samp>.
Writing packed variables in binary format is not supported.
</para>
<html endspaces=" ">
&lt;a name=&quot;dmn_fix_mk&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dmn_fix_mk &textndash;&gt;
&lt;a name=&quot;fix_rec_dmn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#fix_rec_dmn &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1303">record dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1304">fixed dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1305"><code>--fix_rec_dmn <var>dim</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1306"><code>--no_rec_dmn <var>dim</var></code></indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">--fix_rec_dmn</itemformat></item>
</tableterm><tableitem><para>Change record dimension <var>dim</var> in the input file into a fixed
dimension in the output file. 
Also <samp>--no_rec_dmn</samp>.
Before <acronym><acronymword>NCO</acronymword></acronym> version 4.2.5 (January, 2013), the syntax for 
<code>--fix_rec_dmn</code> did not permit or require the specification of
the dimension name <var>dim</var>. 
This is because the feature only worked on netCDF3 files, which support
only one record dimension, so specifying its name was not necessary.
netCDF4 files allow an arbitrary number of record dimensions, so the
user must specify which record dimension to fix.
The decision was made that starting with <acronym><acronymword>NCO</acronymword></acronym> version 4.2.5
(January, 2013), it is always required to specify the dimension name to
fix regardless of the netCDF file type.
This keeps the code simple, and is symmetric with the syntax for
<code>--mk_rec_dmn</code>, described next.
</para>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.4.0 (January, 2014), the argument
<code>all</code> may be given to <samp>--fix_rec_dmn</samp> to convert <emph>all</emph> 
record dimensions to fixed dimensions in the output file. 
Previously, <samp>--fix_rec_dmn</samp> only allowed one option, the name of a
single record dimension to be fixed. 
Now it is simple to simultaneously fix all record dimensions.
This is useful (and nearly mandatory) when flattening netCDF4 files that
have multiple record dimensions per group into netCDF3 files (which are
limited to at most one record dimension) (<pxref label="Group-Path-Editing"><xrefnodename>Group Path Editing</xrefnodename></pxref>). 
</para>
<html endspaces=" ">
&lt;a name=&quot;hdn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hdn &textndash;&gt;
&lt;a name=&quot;hidden&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hidden &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1307">hidden attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1308">special attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1309"><code>--hdn</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1310"><code>--hidden</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1311"><code>_Format</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1312"><code>_DeflateLevel</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1313"><code>_Shuffle</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1314"><code>_Storage</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1315"><code>_ChunkSizes</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1316"><code>_Endianness</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1317"><code>_Fletcher32</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1318"><code>_NOFILL</code></indexterm></cindex>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.4.0 (January, 2014), the <samp>--hdn</samp>
or <samp>--hidden</samp> options print hidden (aka special) attributes.
This is equivalent to <samp>ncdump -s</samp>.
Hidden attributes include: <code>_Format</code>, <code>_DeflateLevel</code>,
<code>_Shuffle</code>, <code>_Storage</code>, <code>_ChunkSizes</code>,
<code>_Endianness</code>, <code>_Fletcher32</code>, and <code>_NOFILL</code>. 
Previously <command>ncks</command> ignored all these attributes in
<acronym><acronymword>CDL</acronymword></acronym>/<acronym><acronymword>XML</acronymword></acronym> modes. 
Now it prints these attributes as appropriate. 
Users are referred to the
<uref><urefurl>http://www.unidata.ucar.edu/software/netcdf/docs</urefurl><urefdesc spaces=" ">Unidata netCDF Documentation</urefdesc></uref>,
or the man pages for <command>ncgen</command> or <command>ncdump</command>, for
detailed descriptions of the meanings of these attributes. 
</para>
<html endspaces=" ">
&lt;a name=&quot;cdl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cdl &textndash;&gt;
&lt;a name=&quot;hdp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hdp &textndash;&gt;
&lt;a name=&quot;hncgen&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#hncgen &textndash;&gt;
&lt;a name=&quot;ncgen-hdf&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncgen-hdf &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1319"><command>hdp</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1320"><command>ncgen</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1321"><command>ncgen-hdf</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1322"><command>hncgen</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1323"><command>ncdump</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1324"><code>--cdl</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1325"><acronym><acronymword>CDL</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1326"><acronym><acronymword>HDF</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1327"><acronym><acronymword>HDF4</acronymword></acronym></indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">--cdl </itemformat></item>
</tableterm><tableitem><para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.3.3 (July, 2013), <command>ncks</command> can
print extracted data and metadata to screen (i.e., <code>stdout</code>) as
valid <acronym><acronymword>CDL</acronymword></acronym> (network Common data form Description Language). 
<acronym><acronymword>CDL</acronymword></acronym> is the human-readable &textldquo;lingua franca&textrdquo; of netCDF ingested by
<command>ncgen</command> and excreted by <command>ncdump</command>.
Compare <command>ncks</command> &textldquo;traditional&textrdquo; with <acronym><acronymword>CDL</acronymword></acronym> printing:
</para><example endspaces=" ">
<pre xml:space="preserve">zender&arobase;roulee:~$ ncks -v one ~/nco/data/in.nc
one: type NC_FLOAT, 0 dimensions, 1 attribute, chunked? no, compressed? no, packed? no
one size (RAM) = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes
one attribute 0: long_name, size = 3 NC_CHAR, value = one

one = 1 

zender&arobase;roulee:~$ ncks --cdl -v one ~/nco/data/in.nc
netcdf in &lbrace;

  variables:
    float one ;
    one:long_name = &quot;one&quot; ;

  data:
    one = 1 ;

&rbrace; // group /
</pre></example>
<para><command>ncgen</command> converts <acronym><acronymword>CDL</acronymword></acronym>-mode output into a netCDF file:
</para><example endspaces=" ">
<pre xml:space="preserve">ncks --cdl -v one ~/nco/data/in.nc &gt; ~/in.cdl
ncgen -k netCDF-4 -b -o ~/in.nc ~/in.cdl
ncks -v one ~/in.nc
</pre></example>
<para>The <acronym><acronymword>HDF</acronymword></acronym> version of <command>ncgen</command>, often named
<command>hncgen</command> or <command>ncgen-hdf</command>, converts netCDF3 <acronym><acronymword>CDL</acronymword></acronym>
into an <acronym><acronymword>HDF</acronymword></acronym> file:  
</para><example endspaces=" ">
<pre xml:space="preserve">/usr/hdf4/bin/ncgen -b -o ~/in.hdf ~/in.cdl # HDF ncgen (local builds)
/usr/bin/hncgen     -b -o ~/in.hdf ~/in.cdl # Same as HDF ncgen (RPM packages?)
/usr/bin/ncgen-hdf  -b -o ~/in.hdf ~/in.cdl # Same as HDF ncgen (Debian packages?)
hdp dumpsds ~/in.hdf                        # ncdump-equivalent for HDF4 files
</pre></example>
<para>Note that <acronym><acronymword>HDF4</acronymword></acronym> does not support netCDF-style groups, so the
above commands fail when the input file contains groups.
Only netCDF4 and <acronym><acronymword>HDF5</acronymword></acronym> support groups.
In our experience the <acronym><acronymword>HDF</acronymword></acronym> <command>ncgen</command> command, by whatever 
name installed, is not robust and can fail on valid netCDF3
<acronym><acronymword>CDL</acronymword></acronym>.  
</para>
<html endspaces=" ">
&lt;a name=&quot;dmn_rec_mk&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dmn_rec_mk &textndash;&gt;
&lt;a name=&quot;mk_rec_dmn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mk_rec_dmn &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1328">record dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1329">fixed dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1330">fix record dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1331"><code>--mk_rec_dmn <var>dim</var></code></indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">--mk_rec_dmn <var>dim</var></itemformat></item>
</tableterm><tableitem><para>Change existing dimension <var>dim</var> to a record dimension in the output file.
This is the most straightforward way of changing a dimension to a/the
record dimension, and works fine in most cases.
See <ref label="ncecat-netCDF-Ensemble-Concatenator"><xrefnodename>ncecat netCDF Ensemble Concatenator</xrefnodename></ref> and 
<ref label="ncpdq-netCDF-Permute-Dimensions-Quickly"><xrefnodename>ncpdq netCDF Permute Dimensions Quickly</xrefnodename></ref> for other methods of
changing variable dimensionality, including the record dimension. 
</para>
<html endspaces=" ">
&lt;a name=&quot;prn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#prn &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1332"><code>-H</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1333"><code>--data</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1334"><code>--hieronymus</code></indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-H </itemformat></item>
</tableterm><tableitem><para>Turn-on printing to screen or turn-off copying data (not metadata).
Also activated using <samp>--print</samp> or <samp>--prn</samp>.
By default <command>ncks</command> prints all metadata and data to screen if
no netCDF output file is specified.
Use <samp>-H</samp> to print data to screen if a netCDF output is specified,
or to restrict printing to data (no metadata) when no netCDF output is 
specified.
Also use <samp>-H</samp> to turn-off copying data (not metadata) to an output
file. 
Unless otherwise specified (with <code>-s</code>), each element of the data
hyperslab prints on a separate line containing the names, indices,
and, values, if any, of all of the variables dimensions.
The dimension and variable indices refer to the location of the
corresponding data element with respect to the variable as stored on
disk (i.e., not the hyperslab).
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0 
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1 
lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2 
...
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21 
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22 
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23 
</pre></example>
<para>Printing the same variable with the <samp>-F</samp> option shows the same
variable indexed with Fortran conventions
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -F -C -v three_dmn_var in.nc
lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0 
lon(2)=90 lev(1)=100 lat(1)=-90 three_dmn_var(2)=1 
lon(3)=180 lev(1)=100 lat(1)=-90 three_dmn_var(3)=2 
...
</pre></example>
<para>Printing a hyperslab does not affect the variable or dimension indices
since these indices are relative to the full variable (as stored in the
input file), and the input file has not changed.
However, if the hyperslab is saved to an output file and those values
are printed, the indices will change:
<!-- c fxm: replace with new MSA output style -->
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -H -d lat,90.0 -d lev,1000.0 -v three_dmn_var in.nc out.nc
...
lat[1]=90 lev[2]=1000 lon[0]=0 three_dmn_var[20]=20 
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21 
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22 
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23 
% ncks -C -v three_dmn_var out.nc
lat[0]=90 lev[0]=1000 lon[0]=0 three_dmn_var[0]=20 
lat[0]=90 lev[0]=1000 lon[1]=90 three_dmn_var[1]=21 
lat[0]=90 lev[0]=1000 lon[2]=180 three_dmn_var[2]=22 
lat[0]=90 lev[0]=1000 lon[3]=270 three_dmn_var[3]=23 
</pre></example>

<cindex index="cp" spaces=" "><indexterm index="cp" number="1335"><code>-M</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1336"><code>--Mtd</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1337"><code>--Metadata</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1338">metadata, global</indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-M</itemformat></item>
</tableterm><tableitem><para>Turn-on printing to screen or turn-off copying global and group metadata.
This includes file summary information and global and group attributes.
Also <samp>--Mtd</samp> and <samp>--Metadata</samp>.
By default <command>ncks</command> prints global metadata to screen if no netCDF
output file and no variable extraction list is specified (with <samp>-v</samp>).  
Use <samp>-M</samp> to print global metadata to screen if a netCDF output is 
specified, or if a variable extraction list is specified (with <samp>-v</samp>). 
Use <samp>-M</samp> to turn-off copying of global and group metadata when
copying, subsetting, or appending to an output file.
</para>
<html endspaces=" ">
&lt;a name=&quot;prn_tbl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#prn_tbl &textndash;&gt;
</html>
<para>The various combinations of printing switches can be confusing.
In an attempt to anticipate what most users want to do, <command>ncks</command>
uses context-sensitive defaults for printing.
Our goal is to minimize the use of switches required to accomplish the
common operations.
We assume that users creating a new file or overwriting (e.g., with
<samp>-O</samp>) an existing file usually wish to copy all global and
variable-specific attributes to the new file.
In contrast, we assume that users appending (e.g., with <samp>-A</samp> an
explicit variable list from one file to another usually wish to copy
only the variable-specific attributes to the output file.
The switches <samp>-H</samp>, <samp>-M</samp>, and <samp>-m</samp> switches are
implemented as toggles which reverse the default behavior.
The most confusing aspect of this is that <samp>-M</samp> inhibits copying
global metadata in overwrite mode and causes copying of global
metadata in append mode.
</para><example endspaces=" ">
<pre xml:space="preserve">ncks                 in.nc        # Print  VAs and GAs
ncks          -v one in.nc        # Print  VAs not GAs
ncks    -M    -v one in.nc        # Print  GAs only
ncks       -m -v one in.nc        # Print  VAs only
ncks    -M -m -v one in.nc        # Print  VAs and GAs
ncks -O              in.nc out.nc # Copy   VAs and GAs
ncks -O       -v one in.nc out.nc # Copy   VAs and GAs
ncks -O -M    -v one in.nc out.nc # Copy   VAs not GAs
ncks -O    -m -v one in.nc out.nc # Copy   GAs not VAs
ncks -O -M -m -v one in.nc out.nc # Copy   only data (no atts)
ncks -A              in.nc out.nc # Append VAs and GAs
ncks -A       -v one in.nc out.nc # Append VAs not GAs
ncks -A -M    -v one in.nc out.nc # Append VAs and GAs
ncks -A    -m -v one in.nc out.nc # Append only data (no atts)
ncks -A -M -m -v one in.nc out.nc # Append GAs not VAs
</pre></example>
<para>where <code>VAs</code> and <code>GAs</code> denote variable and group/global
attributes, respectively. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1339"><command>ncdump</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1340"><code>-m</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1341"><code>--mtd</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1342"><code>--metadata</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1343">metadata</indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-m</itemformat></item>
</tableterm><tableitem><para>Turn-on printing to screen or turn-off copying variable metadata.
Using <samp>-m</samp> will print variable metadata to screen (similar to
<kbd>ncdump -h</kbd>).  
This displays all metadata pertaining to each variable, one variable
at a time.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1344">chunking</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1345">compression</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1346">deflation</indexterm></cindex>
This includes information on the storage properties of the variable,
such as whether it employs chunking, compression, or packing.
Also activated using <samp>--mtd</samp> and <samp>--metadata</samp>.
The <command>ncks</command> default behavior is to print variable metadata to
screen if no netCDF output file is specified.
Use <samp>-m</samp> to print variable metadata to screen if a netCDF output is  
specified. 
Also use <samp>-m</samp> to turn-off copying of variable metadata to an output
file.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1347"><code>--no_blank</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1348"><code>--noblank</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1349"><code>--no-blank</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1350">blank</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1351">missing values</indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">--no_blank</itemformat></item>
</tableterm><tableitem><para>Print numeric representation of missing values.
As of <acronym><acronymword>NCO</acronymword></acronym>
As of <acronym><acronymword>NCO</acronymword></acronym> version 4.2.2 (October, 2012), <acronym><acronymword>NCO</acronymword></acronym> prints
missing values as blanks (i.e., the underscore character <samp>_</samp>) by default.
To enable the old behavior of printing the numeric representation of
missing values (e.g., <code>1.0e36</code>), use the <samp>--no_blank</samp> switch.
Also activated using <samp>--noblank</samp> or <samp>--no-blank</samp>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1352"><code>-P</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1353"><code>--print</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1354"><code>--prn</code></indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-P </itemformat></item>
</tableterm><tableitem><para>Print data, metadata, and units to screen.
The <samp>-P</samp> switch is a convenience abbreviation for 
<samp>-C -H -M -m -u</samp>.
Also activated using <samp>--print</samp> or <samp>--prn</samp>.
This set of switches is useful for exploring file contents.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1355"><code>-Q</code></indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-Q </itemformat></item>
</tableterm><tableitem><para>Toggle printing of dimension indices and coordinate values when printing
arrays. 
Each variable&textrsquo;s name appears flush left in the output.
This helps locate specific variables in lists with many variables and 
different dimensions. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1356"><code>-q</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1357"><code>--quiet</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1358">quiet</indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-q </itemformat></item>
</tableterm><tableitem><para>Turn off all printing to screen.
This overrides the setting of all print-related switches, equivalent to
<kbd>-H -M -m</kbd> when in single-file printing mode. 
When invoked with <code>-R</code> (<pxref label="Retaining-Retrieved-Files"><xrefnodename>Retaining Retrieved Files</xrefnodename></pxref>), <command>ncks</command>
automatically sets <code>-q</code>. 
This allows <command>ncks</command> to retrieve remote files without
automatically trying to print them.
Also <samp>--quiet</samp>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1359"><code>-s</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1360"><code>--string</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1361"><code>--sng_fmt</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1362"><code>printf()</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1363">C language</indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-s <var>format</var></itemformat></item>
</tableterm><tableitem><para>String format for text output. 
Accepts <w>C language</w> escape sequences and <code>printf()</code> formats. 
Also <samp>--string</samp>  and <samp>--sng_fmt</samp>. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1364"><code>-u</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1365"><code>--units</code></indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-u </itemformat></item>
</tableterm><tableitem><para>Toggle the printing of a variable&textrsquo;s <code>units</code> attribute, if any, 
with its values.
Also <samp>--units</samp>.
</para>
<html endspaces=" ">
&lt;a name=&quot;xml&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xml &textndash;&gt;
&lt;a name=&quot;ncmnl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncml &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1366"><code>--xml</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1367"><code>--ncml</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1368"><command>ncdump</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1369"><acronym><acronymword>XML</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1370"><acronym><acronymword>NcML</acronymword></acronym></indexterm></cindex>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">--xml, --ncml</itemformat></item>
</tableterm><tableitem><para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.3.3 (July, 2013), <command>ncks</command> can
print extracted metadata to screen (i.e., <code>stdout</code>) as
<acronym><acronymword>XML</acronymword></acronym> in <acronym><acronymword>NcML</acronymword></acronym>, the netCDF Markup Language.  
<command>ncks</command> supports for <acronym><acronymword>XML</acronymword></acronym> more completely than 
of <samp>ncdump -x</samp>.
With <command>ncks</command> one can translate entire netCDF3 and netCDF4 files
into <acronym><acronymword>NcML</acronymword></acronym>, including metadata and data, using all 
<acronym><acronymword>NCO</acronymword></acronym>&textrsquo;s subsetting and hyperslabbing capabilities.
Compare <command>ncks</command> &textldquo;traditional&textrdquo; with <acronym><acronymword>XML</acronymword></acronym> printing:
</para><example endspaces=" ">
<verbatim xml:space="preserve">
zender@@roulee:~$ ncks -v one ~/nco/data/in.nc
one: type NC_FLOAT, 0 dimensions, 1 attribute, chunked? no, compressed? no, packed? no
one size (RAM) = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes
one attribute 0: long_name, size = 3 NC_CHAR, value = one

one = 1 

zender@roulee:~$ ncks --xml -v one ~/nco/data/in.nc
&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
&lt;netcdf xmlns=&quot;http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2&quot; location=&quot;/home/zender/nco/data/in.nc&quot;&gt;
  &lt;variable name=&quot;one&quot; type=&quot;float&quot; shape=&quot;&quot;&gt;
    &lt;attribute name=&quot;long_name&quot; separator=&quot;*&quot; value=&quot;one&quot; /&gt;
    &lt;values&gt;1.&lt;/values&gt;
  &lt;/variable&gt;
&lt;/netcdf&gt;
</verbatim>
</example>
<para><acronym><acronymword>XML</acronymword></acronym>-mode prints variable metadata and, as of 
<acronym><acronymword>NCO</acronymword></acronym> version 4.3.7 (October, 2013), variable data and, as of
<acronym><acronymword>NCO</acronymword></acronym> version 4.4.0 (January, 2014), hidden attributes.
That <acronym><acronymword>ncks</acronymword></acronym> produces correct <acronym><acronymword>NcML</acronymword></acronym> translations of
<acronym><acronymword>CDM</acronymword></acronym> files for all supported datatypes is verified by
comparison to output from Unidata&textrsquo;s <command>toolsUI</command> Java program.
Please let us know how to improve <acronym><acronymword>XML</acronymword></acronym>/<acronym><acronymword>NcML</acronymword></acronym>
features. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1371"><code>--xml_no_location</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1372"><code>--xml_spr_chr</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1373"><code>--xml_spr_nmr</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1374">separator</indexterm></cindex>
<para><command>ncks</command> provides additional options to configure <acronym><acronymword>NcML</acronymword></acronym>
output: <samp>--xml_no_location</samp>, <samp>--xml_spr_chr</samp>, and
<samp>--xml_spr_nmr</samp>. 
Every <acronym><acronymword>NcML</acronymword></acronym> configuration option automatically triggers
<acronym><acronymword>NcML</acronymword></acronym> printing, so that specifying <samp>--xml</samp> in addition
to a configuration option is redundant and unnecessary.
The <samp>--xml_no_location</samp> switch prevents output of the
<acronym><acronymword>NcML</acronymword></acronym> <code>location</code> element.
By default the location element is printed with a value equal to the
location of the input dataset, e.g.,
<code>location=&quot;/home/zender/in.nc&quot;</code>.
The <samp>--xml_spr_chr</samp> and <samp>--xml_spr_nmr</samp> options customize
the strings used as <acronym><acronymword>NcML</acronymword></acronym> separators for attributes and
variables of character-type and numeric-type, respectively.
Their default separators are &quot;*&quot; and &quot; &quot;:
</para><example endspaces=" ">
<verbatim xml:space="preserve">
zender@@roulee:~$ ncks --xml -d time,0,3 -v two_dmn_rec_var_sng in.nc
...
   &lt;values separator=&quot;*&quot;&gt;abc*bcd*cde*def&lt;/values&gt;
 ...
 zender@@roulee:~$ ncks --xml_spr_chr=', ' -v two_dmn_rec_var_sng in.nc
...
&lt;values separator=&quot;, &quot;&gt;abc, bcd, cde, def, efg, fgh, ghi, hij, jkl, klm&lt;/values&gt;
...
zender@@roulee:~$ ncks --xml -v one_dmn_rec_var in.nc
...
&lt;values&gt;1 2 3 4 5 6 7 8 9 10&lt;/values&gt;
...
zender@@roulee:~$ ncks --xml_spr_nmr=', ' -v one_dmn_rec_var in.nc
...
&lt;values separator=&quot;, &quot;&gt;1, 2, 3, 4, 5, 6, 7, 8, 9, 10&lt;/values&gt;
...
</verbatim>
</example>
<para>Separator elements for strings are a thorny issue.
One must be sure that the separator element is not mistaken as a portion
of the string. 
<acronym><acronymword>NCO</acronymword></acronym> attempts to produce valid <acronym><acronymword>NcML</acronymword></acronym> and supplies the
<samp>--xml_spr_chr</samp> option to work around any difficulties.
<acronym><acronymword>NCO</acronymword></acronym> performs precautionary checks with
<code>strstr(<var>val</var>,<var>spr</var>)</code> to identify presence of the separator
string (<var>spr</var>) in data (<var>val</var>) and, when it detects a match,
automatically switches to a backup separator string (<code>*|*</code>). 
However limitations of <code>strstr()</code> may lead to false negatives 
when the separator string occurs in data beyond the first string in
multi-dimensional <code>NC_CHAR</code> arrays. 
Hence, results may be ambiguous to NcML parsers. 
If problems arise, use <samp>--xml_spr_chr</samp> to specify a multi-character
separator that does not appear in the string array and that does not
include an NcML formatting characters (e.g., commas, angles, quotes).
</para></tableitem></tableentry></table>

<html endspaces=" ">
&lt;a name=&quot;ncattget&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncattget &textndash;&gt;
&lt;a name=&quot;nclist&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#nclist &textndash;&gt;
&lt;a name=&quot;ncdmnsz&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncdmnsz &textndash;&gt;
&lt;a name=&quot;ncrecsz&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncrecsz &textndash;&gt;
&lt;a name=&quot;ncmax&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncmax &textndash;&gt;
&lt;a name=&quot;ncmdn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncmdn &textndash;&gt;
&lt;a name=&quot;ncavg&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncavg &textndash;&gt;
&lt;a name=&quot;ncrng&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncrng &textndash;&gt;
&lt;a name=&quot;ncunits&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncunits &textndash;&gt;
&lt;a name=&quot;alias&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#alias &textndash;&gt;
&lt;a name=&quot;filters&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#filters &textndash;&gt;
&lt;a name=&quot;filter&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#filter &textndash;&gt;
&lt;a name=&quot;flt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#flt &textndash;&gt;
</html>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Filters for <command>ncks</command></menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

</unnumberedsubsec>
<node name="Filters-for-ncks" spaces=" "><nodename>Filters for <command>ncks</command></nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">ncks netCDF Kitchen Sink</nodeprev><nodeup spaces=" ">ncks netCDF Kitchen Sink</nodeup></node>
<subsection spaces=" "><sectiontitle>Filters for <command>ncks</command></sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1375"><acronym><acronymword>UNIX</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1376"><command>ncattget</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1377"><command>ncavg</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1378"><command>ncdmnsz</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1379"><command>nclist</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1380"><command>ncmax</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1381"><command>ncmdn</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1382"><command>ncmin</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1383"><command>ncrecsz</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1384"><command>ncrng</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1385"><command>ncunits</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1386"><file>.bashrc</file></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1387">filters</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1388">alias</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1389">shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1390">Bash shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1391">Csh shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1392">Sh shell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1393"><command>bash</command></indexterm></cindex>
<para>We encourage the use of standard <acronym><acronymword>UNIX</acronymword></acronym> pipes and filters to
narrow the verbose output of <command>ncks</command> into more precise targets.
For example, to obtain an uncluttered listing of the variables in a file
try 
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -m in.nc | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort
</pre></example>
<para>A Bash user could alias the previous filter to the shell command
<command>nclist</command> as shown below.
More complex examples could involve command line arguments.
For example, a user may frequently be interested in obtaining the value
of an attribute, e.g., for textual file examination or for passing to
another shell command.
Say the attribute is <code>purpose</code>, the variable is <code>z</code>, and the
file is <code>in.nc</code>.
In this example, <command>ncks -m -v z</command> is too verbose so a robust
<command>grep</command> and <command>cut</command> filter is desirable, such as
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -M -m in.nc | grep -E -i &quot;^z attribute [0-9]+: purpose&quot; | cut -f 11- -d ' ' | sort
</pre></example>
<para>The filters are clearly too complex to remember on-the-fly so the entire 
procedure could be implemented as a shell command or function called,
say, <command>ncattget</command>
</para><example endspaces=" ">
<pre xml:space="preserve">function ncattget &lbrace; ncks -M -m $&lbrace;3&rbrace; | grep -E -i &quot;^$&lbrace;2&rbrace; attribute [0-9]+: $&lbrace;1&rbrace;&quot; | cut -f 11- -d ' ' | sort ; &rbrace;
</pre></example>
<para>The shell <command>ncattget</command> is invoked with three arugments that are,
in order, the names of the attribute, variable, and file to examine.
Global attributes are indicated by using a variable name of <code>global</code>.
This definition yields the following results
</para><example endspaces=" ">
<pre xml:space="preserve">% ncattget purpose z in.nc
Height stored with a monotonically increasing coordinate
% ncattget Purpose Z in.nc
Height stored with a monotonically increasing coordinate
% ncattget history z in.nc
% ncattget history global in.nc
History global attribute.
</pre></example>
<para>Note that case sensitivity has been turned off for the variable and
attribute names (and could be turned on by removing the <samp>-i</samp> switch
to <command>grep</command>).
Furthermore, extended regular expressions may be used for both the
variable and attribute names.
The next two commands illustrate this by searching for the values
of attribute <code>purpose</code> in all variables, and then for all
attributes of the variable <code>z</code>:
</para><example endspaces=" ">
<pre xml:space="preserve">% ncattget purpose .+ in.nc
1-D latitude coordinate referred to by geodesic grid variables
1-D longitude coordinate referred to by geodesic grid variables
...
% ncattget .+ Z in.nc
Height
Height stored with a monotonically increasing coordinate
meter
</pre></example>

<para>Extended filters are best stored as shell commands if they are
used frequently.
Shell commands may be re-used when they are defined in shell
configuration files.
These files are usually named <file>.bashrc</file>, <file>.cshrc</file>, and
<file>.profile</file> for the Bash, Csh, and Sh shells, respectively.
</para><example endspaces=" ">
<verbatim xml:space="preserve">
# NB: Untested on Csh, Ksh, Sh, Zsh! Send us feedback!
# Bash shell (/bin/bash) users place these in .bashrc
# ncattget $att_nm $var_nm $fl_nm : What attributes does variable have?
function ncattget { ncks -M -m ${3} | grep -E -i &quot;^${2} attribute [0-9]+: ${1}&quot; | cut -f 11- -d ' ' | sort ; }
# ncunits $att_val $fl_nm : Which variables have given units?
function ncunits { ncks -M -m ${2} | grep -E -i &quot; attribute [0-9]+: units.+ ${1}&quot; | cut -f 1 -d ' ' | sort ; }
# ncavg $var_nm $fl_nm : What is mean of variable?
function ncavg { ncwa -y avg -O -C -v ${1} ${2} ~/foo.nc ; ncks -H -C -v ${1} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncavg $var_nm $fl_nm : What is mean of variable?
function ncavg { ncap2 -O -C -v -s &quot;foo=${1}.avg();print(foo)&quot; ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncdmnsz $dmn_nm $fl_nm : What is dimension size?
function ncdmnsz { ncks -m -M ${2} | grep -E -i &quot;: ${1}, size =&quot; | cut -f 7 -d ' ' | uniq ; }
# nclist $fl_nm : What variables are in file?
function nclist { ncks -m ${1} | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort ; }
# ncmax $var_nm $fl_nm : What is maximum of variable?
function ncmax { ncwa -y max -O -C -v ${1} ${2} ~/foo.nc ; ncks -H -C -v ${1} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncmax $var_nm $fl_nm : What is maximum of variable?
function ncmax { ncap2 -O -C -v -s &quot;foo=${1}.max();print(foo)&quot; ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncmdn $var_nm $fl_nm : What is median of variable?
function ncmdn { ncap2 -O -C -v -s &quot;foo=gsl_stats_median_from_sorted_data(${1}.sort());print(foo)&quot; ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncrng $var_nm $fl_nm : What is range of variable?
function ncrng { ncap2 -O -C -v -s &quot;foo_min=${1}.min();foo_max=${1}.max();print(foo_min,\&quot;%f\&quot;);print(\&quot; to \&quot;);print(foo_max,\&quot;%f\&quot;)&quot; ${2} ~/foo.nc ; }
# ncmode $var_nm $fl_nm : What is mode of variable?
function ncmode { ncap2 -O -C -v -s &quot;foo=gsl_stats_median_from_sorted_data(${1}.sort());print(foo)&quot; ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
# ncrecsz $fl_nm : What is record dimension size?
function ncrecsz { ncks -M ${1} | grep -E -i &quot;^Record dimension:&quot; | cut -f 8- -d ' ' ; }
# Csh shell (/bin/csh) users place these in .cshrc
ncattget() { ncks -M -m ${3} | grep -E -i &quot;^${2} attribute [0-9]+: ${1}&quot; | cut -f 11- -d ' ' | sort ; }
ncdmnsz() { ncks -m -M ${2} | grep -E -i &quot;: ${1}, size =&quot; | cut -f 7 -d ' ' | uniq ; }
nclist() { ncks -m ${1} | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort ; }
ncrecsz() { ncks -M ${1} | grep -E -i &quot;^Record dimension:&quot; | cut -f 8- -d ' ' ; }
# Sh shell (/bin/sh) users place these in .profile
ncattget() { ncks -M -m ${3} | grep -E -i &quot;^${2} attribute [0-9]+: ${1}&quot; | cut -f 11- -d ' ' | sort ; }
ncdmnsz() { ncks -m -M ${2} | grep -E -i &quot;: ${1}, size =&quot; | cut -f 7 -d ' ' | uniq ; }
nclist() { ncks -m ${1} | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort ; }
ncrecsz() { ncks -M ${1} | grep -E -i &quot;^Record dimension:&quot; | cut -f 8- -d ' ' ; }
</verbatim>
</example>

<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncks&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncks &textndash;&gt;
</html>
<para>EXAMPLES
</para>
<para>View all data in netCDF <file>in.nc</file>, printed with Fortran indexing
conventions: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -F in.nc
</pre></example>

<para>Copy the netCDF file <file>in.nc</file> to file <file>out.nc</file>.
</para><example endspaces=" ">
<pre xml:space="preserve">ncks in.nc out.nc
</pre></example>
<para>Now the file <file>out.nc</file> contains all the data from <file>in.nc</file>.
There are, however, two differences between <file>in.nc</file> and
<file>out.nc</file>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1394"><code>history</code></indexterm></cindex>
First, the <code>history</code> global attribute (<pxref label="History-Attribute"><xrefnodename>History Attribute</xrefnodename></pxref>)
will contain the command used to create <file>out.nc</file>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1395">alphabetize output</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1396">sort alphabetically</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1397"><code>-a</code></indexterm></cindex>
Second, the variables in <file>out.nc</file> will be defined in alphabetical
order.
Of course the internal storage of variable in a netCDF file should be
transparent to the user, but there are cases when alphabetizing a file 
is useful (see description of <code>-a</code> switch).
</para>
<html endspaces=" ">
&lt;a name=&quot;xmp_att_glb_cpy&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_att_glb_cpy &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1398">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1399">attributes, global</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1400">subsetting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1401">exclusion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1402">extraction</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1403"><code>-v <var>var</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1404"><code>--variable <var>var</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1405"><code>-x</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1406"><code>--exclude</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1407"><code>--xcl</code></indexterm></cindex>
<para>Copy all global attributes (and no variables) from <file>in.nc</file> to
<file>out.nc</file>: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -A -x ~/nco/data/in.nc ~/out.nc
</pre></example>
<para>The <samp>-x</samp> switch tells <acronym><acronymword>NCO</acronymword></acronym> to use the complement of the extraction
list (<pxref label="Subsetting-Files"><xrefnodename>Subsetting Files</xrefnodename></pxref>). 
Since no extraction list is explicitly specified (with <samp>-v</samp>),
the default is to extract all variables.
The complement of all variables is no variables.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1408"><code>-A</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1409"><code>--apn</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1410"><code>--append</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1411">appending to files</indexterm></cindex>
Without any variables to extract, the append (<samp>-A</samp>) command
(<pxref label="Appending-Variables"><xrefnodename>Appending Variables</xrefnodename></pxref>) has only to extract and copy
(i.e., append) global attributes to the output file.
</para>
<html endspaces=" ">
&lt;a name=&quot;xmp_att_glb_cpy&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_att_var_cpy &textndash;&gt;
</html>
<para>Copy/append metadata (not data) from variables in one file to
variables in a second file.
When copying/subsetting/appending files (as opposed to printing them),
the copying of data, variable metadata, and global/group metadata are
now turned OFF by <samp>-H</samp>, <samp>-m</samp>, and <samp>-M</samp>, respectively. 
This is the opposite sense in which these switches work when
<emph>printing</emph> a file. 
One can use these switches to easily replace data or metadata in one
file with data or metadata from another:
</para><example endspaces=" ">
<pre xml:space="preserve"># Extract naked (data-only) copies of two variables
ncks -h -M -m -O -C -v one,three_dmn_rec_var ~/nco/data/in.nc ~/out.nc
# Change values to be sure original values are not copied in following step
ncap2 -O -v -s 'one*=2;three_dmn_rec_var*=0' ~/nco/data/in.nc ~/in2.nc
# Append in2.nc metadata (not data!) to out.nc
ncks -A -C -H -v one,three_dmn_rec_var ~/in2.nc ~/out.nc
</pre></example>
<para>Variables in <file>out.nc</file> now contain data (not metadata) from
<file>in.nc</file> and metadata (not data) from <file>in2.nc</file>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1412"><code>printf()</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1413"><code>\n</code> (linefeed)</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1414"><code>\t</code> (horizontal tab)</indexterm></cindex>
<para>Print variable <code>three_dmn_var</code> from file <file>in.nc</file> with
default notations. 
Next print <code>three_dmn_var</code> as an un-annotated text column.
Then print <code>three_dmn_var</code> signed with very high precision.
Finally, print <code>three_dmn_var</code> as a comma-separated list.
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0 
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1 
...
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23 
% ncks -s '%f\n' -C -v three_dmn_var in.nc
0.000000
1.000000
...
23.000000
% ncks -s '%+16.10f\n' -C -v three_dmn_var in.nc
   +0.0000000000
   +1.0000000000
...
  +23.0000000000
% ncks -s '%f, ' -C -v three_dmn_var in.nc
0.000000, 1.000000, ..., 23.000000,
</pre></example>
<noindent></noindent>
<para>Programmers will recognize these as the venerable <w>C language</w> 
<code>printf()</code> formatting strings. 
The second and third options are useful when pasting data into text
files like reports or papers.  
<xref label="ncatted-netCDF-Attribute-Editor"><xrefnodename>ncatted netCDF Attribute Editor</xrefnodename></xref>, for more details on string
formatting and special characters. 
</para>
<para>As of <acronym><acronymword>NCO</acronymword></acronym> version 4.2.2 (October, 2012), <acronym><acronymword>NCO</acronymword></acronym> prints
missing values as blanks (i.e., the underscore character <samp>_</samp>) by
default: 
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -C -H -v mss_val in.nc
lon[0]=0 mss_val[0]=73 
lon[1]=90 mss_val[1]=_ 
lon[2]=180 mss_val[2]=73 
lon[3]=270 mss_val[3]=_ 
% ncks -s '%+5.1f, ' -H -C -v mss_val in.nc
+73.0, _, +73.0, _, 
</pre></example>

<para>One dimensional arrays of characters stored as netCDF variables are 
automatically printed as strings, whether or not they are
NUL-terminated, e.g.,
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -v fl_nm in.nc
</pre></example>
<noindent></noindent>
<para>The <code>%c</code> formatting code is useful for printing 
multidimensional arrays of characters representing fixed length strings
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -s '%c' -v fl_nm_arr in.nc
</pre></example>
<noindent></noindent>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1415"><code>core dump</code></indexterm></cindex>
<para>Using the <code>%s</code> format code on strings which are not NUL-terminated 
(and thus not technically strings) is likely to result in a core dump.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1416">subsetting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1417">exclusion</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1418">extraction</indexterm></cindex>
<para>Create netCDF <file>out.nc</file> containing all variables, and any associated 
coordinates, except variable <code>time</code>, from netCDF <file>in.nc</file>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -x -v time in.nc out.nc
</pre></example>
<para>As a special case of this, consider how to remove a 
<acronym><acronymword>CF</acronymword></acronym> Convention comliant <code>bounds</code> or <code>coordinates</code>
variable (<pxref label="CF-Conventions"><xrefnodename>CF Conventions</xrefnodename></pxref>) such as <code>time_bounds</code>.
<acronym><acronymword>NCO</acronymword></acronym> subsetting assumes the user wants all coordinates 
and bounds and axes associated with all extracted variables 
(<pxref label="Subsetting-Coordinate-Variables"><xrefnodename>Subsetting Coordinate Variables</xrefnodename></pxref>).
Hence to exclude a <code>bounds</code> or <code>coordinates</code> variable while
retaining the &textldquo;parent&textrdquo; variable (here <code>time</code>), one must use the
<samp>-C</samp> switch: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -C -x -v time_bounds in.nc out.nc
</pre></example>
<para>The <samp>-C</samp> switch tells the operator <emph>NOT</emph> to necessarily
include all the <acronym><acronymword>CF</acronymword></acronym> coordinates and bounds and axes.
Hence the output file will contain <code>time</code> and not
<code>time_bounds</code>. 
</para>
<para>Extract variables <code>time</code> and <code>pressure</code> from netCDF
<file>in.nc</file>.  
If <file>out.nc</file> does not exist it will be created.
Otherwise the you will be prompted whether to append to or to
overwrite <file>out.nc</file>: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -v time,pressure in.nc out.nc
ncks -C -v time,pressure in.nc out.nc
</pre></example>
<noindent></noindent>
<para>The first version of the command creates an <file>out.nc</file> which contains
<code>time</code>, <code>pressure</code>, and any coordinate variables associated
with <var>pressure</var>. 
The <file>out.nc</file> from the second version is guaranteed to contain only 
two variables <code>time</code> and <code>pressure</code>.  
</para>
<para>Create netCDF <file>out.nc</file> containing all variables from file
<file>in.nc</file>.  
Restrict the dimensions of these variables to a hyperslab. 
Print (with <code>-H</code>) the hyperslabs to the screen for good measure.  
The specified hyperslab is: the fifth value in dimension <code>time</code>;
the 
half-open range <math><var>lat</var> &gt; 0.</math> in coordinate <code>lat</code>; the
half-open range <math><var>lon</var> &lt; 330.</math> in coordinate <code>lon</code>; the
closed interval <math>0.3 &lt; <var>band</var> &lt; 0.5</math> in coordinate <code>band</code>;
and cross-section closest to 1000.&noeos; in coordinate <code>lev</code>.  
Note that limits applied to coordinate values are specified with a
decimal point, and limits applied to dimension indices do not have a 
decimal point <xref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></xref>.
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -H -d time,5 -d lat,,0.0 -d lon,330.0, -d band,0.3,0.5 
-d lev,1000.0 in.nc out.nc 
</pre></example>

<cindex index="cp" spaces=" "><indexterm index="cp" number="1419">wrapped coordinates</indexterm></cindex>
<para>Assume the domain of the monotonically increasing longitude coordinate
<code>lon</code> is <math>0 &lt; <var>lon</var> &lt; 360</math>. 
Here, <code>lon</code> is an example of a wrapped coordinate.
<command>ncks</command> will extract a hyperslab which crosses the Greenwich
meridian simply by specifying the westernmost longitude as <var>min</var> and 
the easternmost longitude as <var>max</var>, as follows:
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -d lon,260.0,45.0 in.nc out.nc
</pre></example>
<para>For more details <xref label="Wrapped-Coordinates"><xrefnodename>Wrapped Coordinates</xrefnodename></xref>.
</para>
<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncpdq&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncpdq &textndash;&gt;
&lt;a name=&quot;ncpack&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncpack &textndash;&gt;
&lt;a name=&quot;ncunpack&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncunpack &textndash;&gt;
</html>
</subsection>
</section>
<node name="ncpdq-netCDF-Permute-Dimensions-Quickly" spaces=" "><nodename>ncpdq netCDF Permute Dimensions Quickly</nodename><nodenext spaces=" ">ncra netCDF Record Averager</nodenext><nodeprev spaces=" ">ncks netCDF Kitchen Sink</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncpdq</command> netCDF Permute Dimensions Quickly</sectiontitle>
<findex index="fn" spaces=" "><indexterm index="fn" number="35" mergedindex="cp">ncpdq</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="36" mergedindex="cp">ncpack</indexterm></findex>
<findex index="fn" spaces=" "><indexterm index="fn" number="37" mergedindex="cp">ncunpack</indexterm></findex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1420">reshape variables</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1421">permute dimensions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1422">reverse dimensions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1423">re-order dimensions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1424">re-dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1425">packing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1426">unpacking</indexterm></cindex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">ncpdq [-3] [-4] [-6] [-7] [-A] [-a [-]<var>dim</var>[,&dots;]] [-C] [-c]
[--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz]
[-D <var>dbg</var>] [-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]]
[-F] [-G <var>gpe_dsc</var>] [-g <var>grp</var>[,&dots;]] [-h] [--hdf] [--hdr_pad <var>nbr</var>]
[-L <var>dfl_lvl</var>] [-l <var>path</var>] [-M <var>pck_map</var>] [--mrd] [--no_tmp_fl] 
[-O] [-o <var>output-file</var>] [-P <var>pck_plc</var>] [-p <var>path</var>] 
[-R] [-r] [--ram_all] [-t <var>thr_nbr</var>] [-U] [--unn] [-v <var>var</var>[,&dots;]] [-X ...] [-x]
<var>input-file</var> [<var>output-file</var>]
</pre></example>

<noindent></noindent>
<para>DESCRIPTION
</para>
<para><command>ncpdq</command> performs one (not both) of two distinct functions:
packing or dimension permutation.
<command>ncpdq</command> is optimized to perform these actions in a parallel
fashion with a minimum of time and memory.
The <dfn>pdq</dfn> may stand for &textldquo;Permute Dimensions Quickly&textrdquo;, 
&textldquo;Pack Data Quietly&textrdquo;, &textldquo;Pillory Dan Quayle&textrdquo;, or other silly uses.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1427"><code>add_offset</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1428"><code>scale_factor</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1429"><command>ncap2</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1430">packing policy</indexterm></cindex>
<unnumberedsubsec spaces=" "><sectiontitle>Packing and Unpacking Functions</sectiontitle>
<para>The <command>ncpdq</command> packing (and unpacking) algorithms are described 
in <ref label="Methods-and-functions"><xrefnodename>Methods and functions</xrefnodename></ref>, and are also implemented in
<command>ncap2</command>. 
<command>ncpdq</command> extends the functionality of these algorithms by 
providing high level control of the <dfn>packing policy</dfn> so that
users can consistently pack (and unpack) entire files with one command. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1431"><var>pck_plc</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1432"><code>-P <var>pck_plc</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1433"><code>--pck_plc <var>pck_plc</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1434"><code>--pack_policy <var>pck_plc</var></code></indexterm></cindex>
The user specifies the desired packing policy with the <samp>-P</samp> switch
(or its long option equivalents, <samp>--pck_plc</samp> and
<samp>--pack_policy</samp>) and its <var>pck_plc</var> argument.
Four packing policies are currently implemented:&linebreak;   
</para><table commandarg="dfn" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="dfn">Packing (and Re-Packing) Variables [<emph>default</emph>]</itemformat></item>
</tableterm><tableitem><para>Definition: Pack unpacked variables, re-pack packed variables&linebreak;
Alternate invocation: <code>ncpack</code>&linebreak;
<var>pck_plc</var> key values: <samp>all_new</samp>, <samp>pck_all_new_att</samp>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Packing (and not Re-Packing) Variables</itemformat></item>
</tableterm><tableitem><para>Definition: Pack unpacked variables, copy packed variables&linebreak;
Alternate invocation: none&linebreak;
<var>pck_plc</var> key values: <samp>all_xst</samp>, <samp>pck_all_xst_att</samp>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Re-Packing Variables</itemformat></item>
</tableterm><tableitem><para>Definition: Re-pack packed variables, copy unpacked variables&linebreak;
Alternate invocation: none&linebreak;
<var>pck_plc</var> key values: <samp>xst_new</samp>, <samp>pck_xst_new_att</samp>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Unpacking</itemformat></item>
</tableterm><tableitem><para>Definition: Unpack packed variables, copy unpacked variables&linebreak;
Alternate invocation: <code>ncunpack</code>&linebreak;
<var>pck_plc</var> key values: <samp>upk</samp>, <samp>unpack</samp>, <samp>pck_upk</samp>&linebreak;
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>Equivalent key values are fully interchangeable.
Multiple equivalent options are provided to satisfy disparate needs
and tastes of <acronym><acronymword>NCO</acronymword></acronym> users working with scripts and from the
command line.
</para>
<para>Regardless of the packing policy selected, <command>ncpdq</command> 
no longer (as of <acronym><acronymword>NCO</acronymword></acronym> version 4.0.4 in October, 2010)
packs coordinate variables, or the special variables, weights, 
and other grid properties described in <ref label="CF-Conventions"><xrefnodename>CF Conventions</xrefnodename></ref>.
Prior <command>ncpdq</command> versions treated coordinate variables and
grid properties no differently from other variables.
However, coordinate variables are one-dimensional, so packing saves
little space on large files, and the resulting files are difficult for
humans to read. 
<command>ncpdq</command> will, of course, <emph>unpack</emph> coordinate variables and
weights, for example, in case some other, non-<acronym><acronymword>NCO</acronymword></acronym> software
packed them in the first place.
</para>
<para>Concurrently, Gaussian and area weights and other grid properties are
often used to derive fields in re-inflated (unpacked) files, so packing
such grid properties causes a considerable loss of precision in 
downstream data processing.
If users express strong wishes to pack grid properties, we will
implement new packing policies.
An immediate workaround for those needing to pack grid properties
now, is to use the <command>ncap2</command> packing functions or to rename the
grid properties prior to calling <command>ncpdq</command>. 
We welcome your feedback. 
</para>
<para>To reduce required memorization of these complex policy switches, 
<command>ncpdq</command> may also be invoked via a synonym or with switches
that imply a particular policy.
<command>ncpack</command> is a synonym for <command>ncpdq</command> and behaves the same 
in all respects.
Both <command>ncpdq</command> and <command>ncpack</command> assume a default packing
policy request of <samp>all_new</samp>.
Hence <command>ncpack</command> may be invoked without any <samp>-P</samp> switch,
unlike <command>ncpdq</command>.
Similarly, <command>ncunpack</command> is a synonym for <command>ncpdq</command> 
except that <command>ncpack</command> implicitly assumes a request to unpack, 
i.e., <samp>-P pck_upk</samp>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1435"><code>-U</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1436"><code>--upk</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1437"><code>--unpack</code></indexterm></cindex>
Finally, the <command>ncpdq</command> <samp>-U</samp> switch (or its long option
equivalents, <samp>--upk</samp> and <samp>--unpack</samp>) requires no argument.
It simply requests unpacking.
</para>
<para>Given the menagerie of synonyms, equivalent options, and implied
options, a short list of some equivalent commands is appropriate.
The following commands are equivalent for packing:
<code>ncpdq -P all_new</code>, <code>ncpdq --pck_plc=all_new</code>, and
<code>ncpack</code>.
The following commands are equivalent for unpacking:
<code>ncpdq -P upk</code>, <code>ncpdq -U</code>, <code>ncpdq --pck_plc=unpack</code>, 
and <code>ncunpack</code>.
Equivalent commands for other packing policies, e.g., <samp>all_xst</samp>, 
follow by analogy. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1438"><command>alias</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1439"><command>ln -s</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1440">symbolic links</indexterm></cindex>
Note that <command>ncpdq</command> synonyms are subject to the same constraints 
and recommendations discussed in the secion on <command>ncbo</command> synonyms
(<pxref label="ncbo-netCDF-Binary-Operator"><xrefnodename>ncbo netCDF Binary Operator</xrefnodename></pxref>).
That is, symbolic links must exist from the synonym to <command>ncpdq</command>,
or else the user must define an <command>alias</command>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1441">packing map</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1442"><var>pck_map</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1443"><code>-M <var>pck_map</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1444"><code>--pck_map <var>pck_map</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1445"><code>--map <var>pck_map</var></code></indexterm></cindex>
<para>The <command>ncpdq</command> packing algorithms must know to which type
particular types of input variables are to be packed.
The correspondence between the input variable type and the output,
packed type, is called the <dfn>packing map</dfn>.
The user specifies the desired packing map with the <samp>-M</samp> switch
(or its long option equivalents, <samp>--pck_map</samp> and
<samp>--map</samp>) and its <var>pck_map</var> argument.
Five packing maps are currently implemented:&linebreak;
<cindex index="cp" spaces=" "><indexterm index="cp" number="1446"><samp>hgh_sht</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1447"><samp>hgh_byt</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1448"><samp>flt_sht</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1449"><samp>flt_byt</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1450"><samp>nxt_lsr</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1451"><code>NC_DOUBLE</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1452"><code>NC_FLOAT</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1453"><code>NC_INT64</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1454"><code>NC_UINT64</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1455"><code>NC_INT</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1456"><code>NC_UINT</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1457"><code>NC_SHORT</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1458"><code>NC_USHORT</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1459"><code>NC_CHAR</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1460"><code>NC_BYTE</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1461"><code>NC_UBYTE</code></indexterm></cindex>
</para><table commandarg="dfn" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="dfn">Pack Floating Precisions to <code>NC_SHORT</code> [<emph>default</emph>]</itemformat></item>
</tableterm><tableitem><para>Definition: Pack floating precision types to <code>NC_SHORT</code>&linebreak;
Map: Pack [<code>NC_DOUBLE</code>,<code>NC_FLOAT</code>] to <code>NC_SHORT</code>&linebreak;
Types copied instead of packed: [<code>NC_INT64</code>,<code>NC_UINT64</code>,<code>NC_INT</code>,<code>NC_UINT</code>,<code>NC_SHORT</code>,<code>NC_USHORT</code>,<code>NC_CHAR</code>,<code>NC_BYTE</code>,<code>NC_UBYTE</code>]&linebreak;
<var>pck_map</var> key values: <samp>flt_sht</samp>, <samp>pck_map_flt_sht</samp>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Pack Floating Precisions to <code>NC_BYTE</code></itemformat></item>
</tableterm><tableitem><para>Definition: Pack floating precision types to <code>NC_BYTE</code>&linebreak;
Map: Pack [<code>NC_DOUBLE</code>,<code>NC_FLOAT</code>] to <code>NC_BYTE</code>&linebreak; 
Types copied instead of packed: [<code>NC_INT64</code>,<code>NC_UINT64</code>,<code>NC_INT</code>,<code>NC_UINT</code>,<code>NC_SHORT</code>,<code>NC_USHORT</code>,<code>NC_CHAR</code>,<code>NC_BYTE</code>,<code>NC_UBYTE</code>]&linebreak;
<var>pck_map</var> key values: <samp>flt_byt</samp>, <samp>pck_map_flt_byt</samp>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Pack Higher Precisions to <code>NC_SHORT</code></itemformat></item>
</tableterm><tableitem><para>Definition: Pack higher precision types to <code>NC_SHORT</code>&linebreak;
Map: 
Pack [<code>NC_DOUBLE</code>,<code>NC_FLOAT</code>,<code>NC_INT64</code>,<code>NC_UINT64</code>,<code>NC_INT</code>,<code>NC_UINT</code>] to <code>NC_SHORT</code>&linebreak;
Types copied instead of packed: [<code>NC_SHORT</code>,<code>NC_USHORT</code>,<code>NC_CHAR</code>,<code>NC_BYTE</code>,<code>NC_UBYTE</code>]&linebreak;
<var>pck_map</var> key values: <samp>hgh_sht</samp>, <samp>pck_map_hgh_sht</samp>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Pack Higher Precisions to <code>NC_BYTE</code></itemformat></item>
</tableterm><tableitem><para>Definition: Pack higher precision types to <code>NC_BYTE</code>&linebreak;
Map: 
Pack [<code>NC_DOUBLE</code>,<code>NC_FLOAT</code>,<code>NC_INT64</code>,<code>NC_UINT64</code>,<code>NC_INT</code>,<code>NC_UINT</code>,<code>NC_SHORT</code>,<code>NC_USHORT</code>] to <code>NC_BYTE</code>&linebreak;
Types copied instead of packed: [<code>NC_CHAR</code>,<code>NC_BYTE</code>,<code>NC_UBYTE</code>]&linebreak;
<var>pck_map</var> key values: <samp>hgh_byt</samp>, <samp>pck_map_hgh_byt</samp>&linebreak;
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="dfn">Pack to Next Lesser Precision</itemformat></item>
</tableterm><tableitem><para>Definition: Pack each type to type of next lesser size&linebreak;
Map: Pack [<code>NC_DOUBLE</code>,<code>NC_INT64</code>,<code>NC_UINT64</code>], to <code>NC_INT</code>. 
Pack [<code>NC_FLOAT</code>,<code>NC_INT</code>,<code>NC_UINT</code>] to <code>NC_SHORT</code>.
Pack [<code>NC_SHORT</code>,<code>NC_USHORT</code>] to <code>NC_BYTE</code>.&linebreak;
Types copied instead of packed: [<code>NC_CHAR</code>,<code>NC_BYTE</code>,<code>NC_UBYTE</code>]&linebreak;
<var>pck_map</var> key values: <samp>nxt_lsr</samp>, <samp>pck_map_nxt_lsr</samp>&linebreak;
</para></tableitem></tableentry></table>
<noindent></noindent>
<para>The default <samp>all_new</samp> packing policy with the default
<samp>flt_sht</samp> packing map reduces the typical <code>NC_FLOAT</code>-dominated
file size by <w>about 50%.</w>
<samp>flt_byt</samp> packing reduces an <code>NC_DOUBLE</code>-dominated file by
<w>about 87%.</w> 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1462"><var>_FillValue</var></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1463"><code>_FillValue</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1464"><code>NUL</code></indexterm></cindex>
<para>The netCDF packing algorithm (<pxref label="Methods-and-functions"><xrefnodename>Methods and functions</xrefnodename></pxref>) is
lossy&textmdash;once packed, the exact original data cannot be recovered without
a full backup. 
Hence users should be aware of some packing caveats:
First, the interaction of packing and data equal to the
<var>_FillValue</var> is complex.
Test the <code>_FillValue</code> behavior by performing a pack/unpack cycle 
to ensure data that are missing <emph>stay</emph> missing and data that are
not misssing do not join the Air National Guard and go missing.
This may lead you to elect a new <var>_FillValue</var>.
Second, <code>ncpdq</code> actually allows packing into <code>NC_CHAR</code> (with,
e.g., <samp>flt_chr</samp>).
However, the intrinsic conversion of <code>signed char</code> to higher
precision types is tricky for values equal to zero, i.e., for
<code>NUL</code>.  
Hence packing to <code>NC_CHAR</code> is not documented or advertised.  
Pack into <code>NC_BYTE</code> (with, e.g., <samp>flt_byt</samp>) instead.
</para>
<html endspaces=" ">
&lt;a name=&quot;rvr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#rvr &textndash;&gt;
</html>
</unnumberedsubsec>
<unnumberedsubsec spaces=" "><sectiontitle>Dimension Permutation</sectiontitle>
<para><command>ncpdq</command> re-shapes variables in <var>input-file</var> by re-ordering
and/or reversing dimensions specified in the dimension list.
The dimension list is a whitespace-free, comma separated list of
dimension names, optionally prefixed by negative signs, that follows the 
<samp>-a</samp> (or long options <samp>--arrange</samp>, <samp>--permute</samp>,
<samp>--re-order</samp>, or <samp>--rdr</samp>) switch.  
To re-order variables by a subset of their dimensions, specify
these dimensions in a comma-separated list following <samp>-a</samp>, e.g., 
<samp>-a lon,lat</samp>. 
To reverse a dimension, prefix its name with a negative sign in the
dimension list, e.g., <samp>-a -lat</samp>. 
Re-ordering and reversal may be performed simultaneously, e.g.,
<samp>-a lon,-lat,time,-lev</samp>. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1465">record dimension</indexterm></cindex>
<para>Users may specify any permutation of dimensions, including
permutations which change the record dimension identity.
The record dimension is re-ordered like any other dimension.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1466">concatenation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1467">record dimension</indexterm></cindex>
This unique <command>ncpdq</command> capability makes it possible to concatenate
files along any dimension.
See <ref label="Concatenation"><xrefnodename>Concatenation</xrefnodename></ref> for a detailed example.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1468">record variable</indexterm></cindex>
The record dimension is always the most slowly varying dimension in a
record variable (<pxref label="C-and-Fortran-Index-Conventions"><xrefnodename>C and Fortran Index Conventions</xrefnodename></pxref>).
The specified re-ordering fails if it requires creating more than
one record dimension amongst all the output variables
<footnote><para>This limitation, imposed by the netCDF storage layer,
may be relaxed in the future with netCDF4.</para></footnote>.
</para>
<para>Two special cases of dimension re-ordering and reversal deserve special
mention. 
First, it may be desirable to completely reverse the storage order of a
variable. 
To do this, include all the variable&textrsquo;s dimensions in the dimension
re-order list in their original order, and prefix each dimension name
with the negative sign.  
<cindex index="cp" spaces=" "><indexterm index="cp" number="1469">transpose</indexterm></cindex>
Second, it may useful to transpose a variable&textrsquo;s storage order, e.g.,
<w>from C</w> to Fortran data storage order 
(<pxref label="C-and-Fortran-Index-Conventions"><xrefnodename>C and Fortran Index Conventions</xrefnodename></pxref>).
To do this, include all the variable&textrsquo;s dimensions in the dimension
re-order list in reversed order.
Explicit examples of these two techniques appear below.
</para>
<tex endspaces=" ">
NB: fxm ncpdq documentation will evolve through Fall 2004.
I will upload updates to documentation linked to by the NCO homepage.
ncpdq is a powerful operator, and 
I am unfamiliar with the terminology needed to describe what ncpdq does.
Sequences, sets, sheesh!
I just know that it does &quot;The right thing&quot; according to my gut feelings.
Now do you feel more comfortable using it?

Let $\dmnvct(\xxx)$ represent the dimensionality of the variable $\xxx$. 
Dimensionality describes the order and sizes of dimensions.
If $\xxx$ has rank $\dmnnbr$, then we may write $\dmnvct(\xxx)$ as the
$\dmnnbr$-element vector 
$$
\dmnvct(\xxx) = [ \dmn_{1}, \dmn_{2}, \dmn_{3}, \ldots, 
\dmn_{\dmnidx-1}, \dmn_{\dmnidx}, \dmn_{\dmnidx+1}, 
\ldots, \dmn_{\dmnnbr-2}, \dmn_{\dmnnbr-1}, \dmn_{\dmnnbr} ] 
$$
where $\dmn_{\dmnidx}$ is the size of the $\dmnidx$&textrsquo;th dimension.

The dimension re-order list specified with <samp>-a</samp> is the
$\rdrnbr$-element vector 
$$
\rdrvct = [ \rdr_{1}, \rdr_{2}, \rdr_{3}, \ldots, 
\rdr_{\rdridx-1}, \rdr_{\rdridx}, \rdr_{\rdridx+1}, 
\ldots, \rdr_{\rdrnbr-2}, \rdr_{\rdrnbr-1}, \rdr_{\rdrnbr} ] 
$$
There need be no relation between $\dmnnbr$ and $\rdrnbr$.
Let the $\shrnbr$-element vector $\shrvct$ be the intersection
(i.e., the ordered set of unique shared dimensions) of $\dmnvct$ and 
$\rdrvct$
Then
$$
\eqalign{{\shrvct} &amp;= \rdrvct \cap \dmnvct \cr
&amp;= [ \shr_{1}, \shr_{2}, \shr_{3}, \ldots, 
\shr_{\shridx-1}, \shr_{\shridx}, \shr_{\shridx+1}, 
\ldots, \shr_{\shrnbr-2}, \shr_{\shrnbr-1}, \shr_{\shrnbr} ]} 
$$
$\shrvct$ is empty if $\rdrvct \notin \dmnvct$.

Re-ordering (or re-shaping) a variable means mapping the input state
with dimensionality $\dmnvct(\xxx)$ to the output state with
dimensionality $\dmnvctprm(\xxxprm)$.  
In practice, mapping occurs in three logically distinct steps.
First, we tranlate the user input to a one-to-one mapping $\mpp$ 
between input and output dimensions, 
$\dmnvct \mapsto \dmnvctprm$.
This tentative map is final unless external constraints (typically
netCDF restrictions) impose themselves.   
Second, we check and, if necessary, refine the tentative mapping so that
the re-shaped variables will co-exist in the same file without violating 
netCDF-imposed storage restrictions. 
This refined map specifies the final (output) dimensionality.
Third, we translate the output dimensionality into one-dimensional  
memory offsets for each datum according to the <w>C language</w> convention 
for multi-dimensional array storage.
Dimension reversal changes the ordering of data, though not the
rank or dimensionality, and so is part of the third step. 

Dimensions $\rdr$ disjoint from $\dmnvct$ play no role in re-ordering.
The first step taken to re-order a variable is to determine $\shrvct$. 
$\rdrvct$ is constant for all variables, whereas $\dmnvct$, and hence
$\shrvct$, is variable-specific.
$\shrvct$ is empty if $\rdrvct \notin \dmnvct$.
This may be the case for some extracted variables.
The user may explicitly specify the one-to-one mapping of input
to output dimension order by supplying (with <samp>-a</samp>) a re-order list
$\rdrvct$ such that $\shrnbr = \dmnnbr$. 
In this case $\dmnsubnnnprm = \shrsubnnn$.  
The degenerate case occurs when $\dmnvct = \shrvct$.
This produces the identity mapping $\dmnsubnnnprm = \dmnsubnnn$.  

The mapping of input to output dimension order is more complex
when $\shrnbr \ne \dmnnbr$. 
In this case $\dmnsubnnnprm = \dmnsubnnn$ for the $\dmnnbr-\shrnbr$
dimensions $\dmnsubnnnprm \notin \shrvct$.
For the $\shrnbr$ dimensions $\dmnsubnnnprm \in \shrvct$, 
$\dmnsubnnnprm = \shrsubsss$.  
</tex>

<!-- c fxm: discuss netCDF-imposed constraints here -->

<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncpdq&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncpdq &textndash;&gt;
</html>
<para>EXAMPLES
</para>
<para>Pack and unpack all variables in file <file>in.nc</file> and store the results
in <file>out.nc</file>:  
</para><example endspaces=" ">
<pre xml:space="preserve">ncpdq in.nc out.nc # Same as ncpack in.nc out.nc
ncpdq -P all_new -M flt_sht in.nc out.nc # Defaults
ncpdq -P all_xst in.nc out.nc
ncpdq -P upk in.nc out.nc # Same as ncunpack in.nc out.nc
ncpdq -U in.nc out.nc # Same as ncunpack in.nc out.nc
</pre></example>
<para>The first two commands pack any unpacked variable in the input file.
They also unpack and then re-pack every packed variable.
The third command only packs unpacked variables in the input file.
If a variable is already packed, the third command copies it unchanged
to the output file. 
The fourth and fifth commands unpack any packed variables.
If a variable is not packed, the third command copies it unchanged.
</para>
<para>The previous examples all utilized the default packing map.
Suppose you wish to archive all data that are currently unpacked 
into a form which only preserves 256 distinct values.
Then you could specify the packing map <var>pck_map</var> as <samp>hgh_byt</samp>
and the packing policy <var>pck_plc</var> as <samp>all_xst</samp>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncpdq -P all_xst -M hgh_byt in.nc out.nc
</pre></example>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1470">appending variables</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1471"><samp>-A</samp></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1472"><samp>-v</samp></indexterm></cindex>
<para>Many different packing maps may be used to construct a given file 
by performing the packing on subsets of variables (e.g., with <samp>-v</samp>) 
and using the append feature with <samp>-A</samp> (<pxref label="Appending-Variables"><xrefnodename>Appending Variables</xrefnodename></pxref>).
</para>
<para>Users may wish to unpack data packed with the <acronym><acronymword>HDF</acronymword></acronym> convention,
and then re-pack it with the netCDF convention so that all their
datasets use the same packing convention prior to intercomparison.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1473"><command>ncl_convert2nc</command></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1474"><acronym><acronymword>NCL</acronymword></acronym></indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve"># One-step procedure: For NCO 4.4.0+, netCDF 4.3.1+
# 1. Convert, unpack, and repack HDF file into netCDF file
ncpdq --hdf_upk -P xst_new modis.hdf modis.nc # HDF4 files
ncpdq --hdf_upk -P xst_new modis.h5  modis.nc # HDF5 files

# One-step procedure: For NCO 4.3.7--4.3.9
# 1. Convert, unpack, and repack HDF file into netCDF file
ncpdq --hdf4 --hdf_upk -P xst_new modis.hdf modis.nc # HDF4
ncpdq        --hdf_upk -P xst_new modis.h5  modis.nc # HDF5

# Two-step procedure: For NCO 4.3.6 and earlier
# 1. Convert HDF file to netCDF file
ncl_convert2nc modis.hdf
# 2. Unpack using HDF convention and repack using netCDF convention
ncpdq --hdf_upk -P xst_new modis.nc modis.nc
</pre></example>
<para><acronym><acronymword>NCO</acronymword></acronym> now
<footnote spaces="\n"><para>Prior to <acronym><acronymword>NCO</acronymword></acronym> 4.4.0 and netCDF 4.3.1 (January, 2014),
<acronym><acronymword>NCO</acronymword></acronym> requires the <samp>--hdf4</samp> switch to correctly read
HDF4 input files.
For example, <samp>ncpdq --hdf4 --hdf_upk -P xst_new modis.hdf modis.nc</samp>.
That switch is now obsolete, though harmless for backwards compatibility.
Prior to version 4.3.7 (October, 2013), <acronym><acronymword>NCO</acronymword></acronym> lacked the
software necessary to workaround netCDF library flaws handling
<acronym><acronymword>HDF4</acronymword></acronym> files, and thus <acronym><acronymword>NCO</acronymword></acronym> failed to convert
<acronym><acronymword>HDF4</acronymword></acronym> files to netCDF files.
In those cases, use the <command>ncl_convert2nc</command> command distributed
with <acronym><acronymword>NCL</acronymword></acronym> to convert <acronym><acronymword>HDF4</acronymword></acronym> files to netCDF.</para></footnote>
automatically detects <acronym><acronymword>HDF4</acronymword></acronym> files.  
In this case it produces an output file <file>modis.nc</file> which preserves
the <acronym><acronymword>HDF</acronymword></acronym> packing used in the input file.
The <command>ncpdq</command> command first unpacks all packed variables using the
<acronym><acronymword>HDF</acronymword></acronym> unpacking algorithm (as specified by <samp>--hdf_upk</samp>), 
and then repacks those same variables using the netCDF algorithm
(because that is the only algorithm <acronym><acronymword>NCO</acronymword></acronym> packs with).
As described above the <samp>--P xst_new</samp> packing policy only repacks
variables that are already packed. 
Not-packed variables are copied directly without loss of precision
<footnote><para><command>ncpdq</command> does not support packing data using the
<acronym><acronymword>HDF</acronymword></acronym> convention.
Although it is now straightforward to support this, we think it might
sow more confusion than it reaps. 
Let us know if you disagree and would like <acronym><acronymword>NCO</acronymword></acronym> to support
packing data with <acronym><acronymword>HDF</acronymword></acronym> algorithm.</para></footnote>.
</para>
<para>Re-order file <file>in.nc</file> so that the dimension <code>lon</code> always
precedes the dimension <code>lat</code> and store the results in
<file>out.nc</file>:  
</para><example endspaces=" ">
<pre xml:space="preserve">ncpdq -a lon,lat in.nc out.nc
ncpdq -v three_dmn_var -a lon,lat in.nc out.nc
</pre></example>
<para>The first command re-orders every variable in the input file.
The second command extracts and re-orders only the variable
<code>three_dmn_var</code>. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1475">reverse dimensions</indexterm></cindex>
<para>Suppose the dimension <code>lat</code> represents latitude and monotonically 
increases increases from south to north. 
Reversing the <code>lat</code> dimension means re-ordering the data so that
latitude values decrease monotonically from north to south.
Accomplish this with
</para><example endspaces=" ">
<pre xml:space="preserve">% ncpdq -a -lat in.nc out.nc
% ncks -C -v lat in.nc
lat[0]=-90
lat[1]=90
% ncks -C -v lat out.nc
lat[0]=90
lat[1]=-90
</pre></example>
<para>This operation reversed the latitude dimension of all variables.
Whitespace immediately preceding the negative sign that specifies
dimension reversal may be dangerous.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1476">long options</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1477">quotes</indexterm></cindex>
Quotes and long options can help protect negative signs that should
indicate dimension reversal from being interpreted by the shell as
dashes that indicate new command line switches.
</para><example endspaces=" ">
<pre xml:space="preserve">ncpdq -a -lat in.nc out.nc # Dangerous? Whitespace before &quot;-lat&quot;
ncpdq -a '-lat' in.nc out.nc # OK. Quotes protect &quot;-&quot; in &quot;-lat&quot;
ncpdq -a lon,-lat in.nc out.nc # OK. No whitespace before &quot;-&quot;
ncpdq --rdr=-lat in.nc out.nc # Preferred. Uses &quot;=&quot; not whitespace
</pre></example>

<cindex index="cp" spaces=" "><indexterm index="cp" number="1478">reverse dimensions</indexterm></cindex>
<para>To create the mathematical transpose of a variable, place all its
dimensions in the dimension re-order list in reversed order.
This example creates the transpose of <code>three_dmn_var</code>: 
</para><example endspaces=" ">
<pre xml:space="preserve">% ncpdq -a lon,lev,lat -v three_dmn_var in.nc out.nc
% ncks -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0 
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1 
lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2 
...
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21 
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22 
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23 
% ncks -C -v three_dmn_var out.nc
lon[0]=0 lev[0]=100 lat[0]=-90 three_dmn_var[0]=0
lon[0]=0 lev[0]=100 lat[1]=90 three_dmn_var[1]=12
lon[0]=0 lev[1]=500 lat[0]=-90 three_dmn_var[2]=4
...
lon[3]=270 lev[1]=500 lat[1]=90 three_dmn_var[21]=19
lon[3]=270 lev[2]=1000 lat[0]=-90 three_dmn_var[22]=11
lon[3]=270 lev[2]=1000 lat[1]=90 three_dmn_var[23]=23
</pre></example>

<cindex index="cp" spaces=" "><indexterm index="cp" number="1479">reverse data</indexterm></cindex>
<para>To completely reverse the storage order of a variable, include
all its dimensions in the re-order list, each prefixed by a negative
sign. 
This example reverses the storage order of <code>three_dmn_var</code>: 
</para><example endspaces=" ">
<pre xml:space="preserve">% ncpdq -a -lat,-lev,-lon -v three_dmn_var in.nc out.nc
% ncks -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0 
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1 
lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2 
...
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21 
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22 
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23 
% ncks -C -v three_dmn_var out.nc
lat[0]=90 lev[0]=1000 lon[0]=270 three_dmn_var[0]=23
lat[0]=90 lev[0]=1000 lon[1]=180 three_dmn_var[1]=22
lat[0]=90 lev[0]=1000 lon[2]=90 three_dmn_var[2]=21
...
lat[1]=-90 lev[2]=100 lon[1]=180 three_dmn_var[21]=2
lat[1]=-90 lev[2]=100 lon[2]=90 three_dmn_var[22]=1
lat[1]=-90 lev[2]=100 lon[3]=0 three_dmn_var[23]=0
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;dmn_rcd_mk&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dmn_rcd_mk &textndash;&gt;
&lt;a name=&quot;mk_rcd_dmn&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mk_rcd_dmn &textndash;&gt;
</html>
<para>Creating a record dimension named, e.g., <code>time</code>, in a file which
has no existing record dimension is simple with <command>ncecat</command>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncecat -O -u time in.nc out.nc # Create degenerate record dimension named &quot;time&quot;
</pre></example>

<cindex index="cp" spaces=" "><indexterm index="cp" number="1480">record dimension</indexterm></cindex>
<para>Now consider a file with all dimensions, including <code>time</code>, fixed
(non-record).
Suppose the user wishes to convert <code>time</code> from a fixed dimension to  
a record dimension. 
This may be useful, for example, when the user wishes to append
additional time slices to the data.
As of <acronym><acronymword>NCO</acronymword></acronym> version 4.0.1 (April, 2010) the preferred method for
doing this is with <command>ncks</command>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncks -O --mk_rec_dmn time in.nc out.nc # Change &quot;time&quot; to record dimension
</pre></example>

<para>Prior to 4.0.1, the procedure to change an existing fixed dimension into
a record dimension required three separate commands,
<command>ncecat</command> followed by <command>ncpdq</command>, and then <command>ncwa</command>.
The recommended method is now to use <samp>ncks --fix_rec_dmn</samp>, yet it
is still instructive to present the original procedure, as it shows how
multiple operators can achieve the same ends by different means: 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1481">degenerate dimension</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">ncecat -O in.nc out.nc # Add degenerate record dimension named &quot;record&quot;
ncpdq -O -a time,record out.nc out.nc # Switch &quot;record&quot; and &quot;time&quot;
ncwa -O -a record out.nc out.nc # Remove (degenerate) &quot;record&quot;
</pre></example>
<noindent></noindent>
<para>The first step creates a degenerate (size equals one) record dimension
named (by default) <code>record</code>. 
The second step swaps the ordering of the dimensions named <code>time</code>
and <code>record</code>.
Since <code>time</code> now occupies the position of the first (least rapidly
varying) dimension, it becomes the record dimension.
The dimension named <code>record</code> is no longer a record dimension.
The third step averages over this degenerate <code>record</code> dimension.
Averaging over a degenerate dimension does not alter the data.
The ordering of other dimensions in the file (<code>lat</code>, <code>lon</code>,
etc.) is immaterial to this procedure. 
See <ref label="ncecat-netCDF-Ensemble-Concatenator"><xrefnodename>ncecat netCDF Ensemble Concatenator</xrefnodename></ref> and 
<ref label="ncks-netCDF-Kitchen-Sink"><xrefnodename>ncks netCDF Kitchen Sink</xrefnodename></ref> for other methods of
changing variable dimensionality, including the record dimension. 
</para>
<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncra&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncra &textndash;&gt;
</html>
</unnumberedsubsec>
</section>
<node name="ncra-netCDF-Record-Averager" spaces=" "><nodename>ncra netCDF Record Averager</nodename><nodenext spaces=" ">ncrcat netCDF Record Concatenator</nodenext><nodeprev spaces=" ">ncpdq netCDF Permute Dimensions Quickly</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncra</command> netCDF Record Averager</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1482">averaging data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1483">record average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1484">record dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1485">running average</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="38" mergedindex="cp">ncra</indexterm></findex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">ncra [-3] [-4] [-6] [-7] [-A] [-C] [-c]
[--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz]
[-D <var>dbg</var>] [-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>][,[<var>subcycle</var>]]]] [-F]
[-G <var>gpe_dsc</var>] [-g <var>grp</var>[,&dots;]] [-h] [--hdf] [--hdr_pad <var>nbr</var>] 
[-L <var>dfl_lvl</var>] [-l <var>path</var>] [--mro] [-n <var>loop</var>] [--no_tmp_fl]
[-O] [-o <var>output-file</var>] [-p <var>path</var>] [-R] [-r] [--ram_all] [--rec_apn] [--rth_dbl|flt]
[-t <var>thr_nbr</var>] [--unn] [-v <var>var</var>[,&dots;]] [-X ...] [-x] [-y <var>op_typ</var>]
[<var>input-files</var>] [<var>output-file</var>]
</pre></example>

<noindent></noindent>
<para>DESCRIPTION
</para>
<para><command>ncra</command> averages record variables across an arbitrary number of 
<var>input-files</var>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1486">degenerate dimension</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1487">record dimension</indexterm></cindex>
The record dimension is, by default, retained as a degenerate 
<w>(size 1)</w> dimension in the output variables.
<xref label="Statistics-vs_002e-Concatenation"><xrefnodename>Statistics vs. Concatenation</xrefnodename></xref>, for a description of the
distinctions between the various statistics tools and concatenators. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1488">multi-file operators</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1489">standard input</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1490"><code>stdin</code></indexterm></cindex>
As a multi-file operator, <command>ncra</command> will read the list of
<var>input-files</var> from <code>stdin</code> if they are not specified 
as positional arguments on the command line 
(<pxref label="Large-Numbers-of-Files"><xrefnodename>Large Numbers of Files</xrefnodename></pxref>).
</para>
<para>Input files may vary in size, but each must have a record dimension.
The record coordinate, if any, should be monotonic (or else non-fatal
warnings may be generated). 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1491">hyperslab</indexterm></cindex>
Hyperslabs of the record dimension which include more than one file 
work correctly.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1492">stride</indexterm></cindex>
<command>ncra</command> supports the <var>stride</var> argument to the <samp>-d</samp>
hyperslab option (<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>) for the record dimension only,
<var>stride</var> is not supported for non-record dimensions.
</para>
<para><command>ncra</command> weights each record (e.g., time slice) in the
<var>input-files</var> equally.
<command>ncra</command> does not attempt to see if, say, the <code>time</code>
coordinate is irregularly spaced and thus would require a weighted
average in order to be a true time average.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1493">operation types</indexterm></cindex>
<command>ncra</command> <emph>always averages</emph> coordinate variables regardless of 
the arithmetic operation type performed on the non-coordinate variables. 
(<pxref label="Operation-Types"><xrefnodename>Operation Types</xrefnodename></pxref>). 
</para>
<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncra&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncra &textndash;&gt;
</html>
<para>EXAMPLES
</para>
<para>Average files <file>85.nc</file>, <file>86.nc</file>, <w>&dots; <file>89.nc</file></w>
along the record dimension, and store the results in <file>8589.nc</file>: 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1494">globbing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1495"><code>NINTAP</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1496">Processor</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1497"><acronym><acronymword>CCM</acronymword></acronym> Processor</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">ncra 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra 8[56789].nc 8589.nc
ncra -n 5,2,1 85.nc 8589.nc
</pre></example>
<para>These three methods produce identical answers.
<xref label="Specifying-Input-Files"><xrefnodename>Specifying Input Files</xrefnodename></xref>, for an explanation of the distinctions
between these methods.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1498">Fortran</indexterm></cindex>
<para>Assume the files <file>85.nc</file>, <file>86.nc</file>, <w>&dots; <file>89.nc</file></w>
each contain a record coordinate <var>time</var> of length 12 defined such
that the third record in <file>86.nc</file> contains data from March 1986,
etc. 
<acronym><acronymword>NCO</acronymword></acronym> knows how to hyperslab the record dimension across files.
Thus, to average data from December, 1985 through February, 1986:
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -d time,11,13 85.nc 86.nc 87.nc 8512_8602.nc
ncra -F -d time,12,14 85.nc 86.nc 87.nc 8512_8602.nc
</pre></example>
<noindent></noindent>
<para>The file <file>87.nc</file> is superfluous, but does not cause an error.
The <samp>-F</samp> turns on the Fortran (1-based) indexing convention.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1499">stride</indexterm></cindex>
The following uses the <var>stride</var> option to average all the March
temperature data from multiple input files into a single output file
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -F -d time,3,,12 -v temperature 85.nc 86.nc 87.nc 858687_03.nc
</pre></example>
<para><xref label="Stride"><xrefnodename>Stride</xrefnodename></xref>, for a description of the <var>stride</var> argument.
</para>
<para>Assume the <var>time</var> coordinate is incrementally numbered such that
January, <w><math>1985 = 1</math></w> and December, <w><math>1989 = 60</math></w>.
Assuming <samp>??</samp> only expands to the five desired files, the following 
averages June, 1985&textndash;June, 1989: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -d time,6.,54. ??.nc 8506_8906.nc
</pre></example>

<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncrcat&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncrcat &textndash;&gt;
</html>
</section>
<node name="ncrcat-netCDF-Record-Concatenator" spaces=" "><nodename>ncrcat netCDF Record Concatenator</nodename><nodenext spaces=" ">ncrename netCDF Renamer</nodenext><nodeprev spaces=" ">ncra netCDF Record Averager</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncrcat</command> netCDF Record Concatenator</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1500">concatenation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1501">record concatenation</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="39" mergedindex="cp">ncrcat</indexterm></findex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">ncrcat [-3] [-4] [-6] [-7] [-A] [-C] [-c]
[--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz]
[-D <var>dbg</var>] [-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>][,[<var>subcycle</var>]]]] [-F]
[-G <var>gpe_dsc</var>] [-g <var>grp</var>[,&dots;]] [-h] [--hdr_pad <var>nbr</var>]
[-L <var>dfl_lvl</var>] [-l <var>path</var>] [--md5_digest] [-n <var>loop</var>] [--no_tmp_fl]
[-O] [-o <var>output-file</var>] [-p <var>path</var>] [-R] [-r] [--ram_all] [--rec_apn]
[-t <var>thr_nbr</var>] [--unn] [-v <var>var</var>[,&dots;]] [-X ...] [-x] 
[<var>input-files</var>] [<var>output-file</var>]
</pre></example>

<noindent></noindent>
<para>DESCRIPTION
</para>
<para><command>ncrcat</command> concatenates record variables across an arbitrary
number of <var>input-files</var>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1502">record dimension</indexterm></cindex>
The final record dimension is by default the sum of the lengths of the 
record dimensions in the input files.
<xref label="Statistics-vs_002e-Concatenation"><xrefnodename>Statistics vs. Concatenation</xrefnodename></xref>, for a description of the
distinctions between the various statistics tools and concatenators. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1503">multi-file operators</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1504">standard input</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1505"><code>stdin</code></indexterm></cindex>
As a multi-file operator, <command>ncrcat</command> will read the list of
<var>input-files</var> from <code>stdin</code> if they are not specified 
as positional arguments on the command line 
(<pxref label="Large-Numbers-of-Files"><xrefnodename>Large Numbers of Files</xrefnodename></pxref>).
</para>
<para>Input files may vary in size, but each must have a record dimension.
The record coordinate, if any, should be monotonic (or else non-fatal
warnings may be generated).
<cindex index="cp" spaces=" "><indexterm index="cp" number="1506">hyperslab</indexterm></cindex>
Hyperslabs along the record dimension that span more than one file are  
handled correctly.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1507">stride</indexterm></cindex>
<command>ncra</command> supports the <var>stride</var> argument to the <samp>-d</samp>
hyperslab option for the record dimension only, <var>stride</var> is not
supported for non-record dimensions.
</para>
<findex index="fn" spaces=" "><indexterm index="fn" number="40" mergedindex="cp">ncpdq</indexterm></findex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1508">packing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1509">unpacking</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1510"><code>add_offset</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1511"><code>scale_factor</code></indexterm></cindex>
<para>Concatenating a variable packed with different scales multiple datasets  
is beyond the capabilities of <command>ncrcat</command> (and <command>ncecat</command>,
the other concatenator (<ref label="Concatenation"><xrefnodename>Concatenation</xrefnodename></ref>).
<command>ncrcat</command> does not unpack data, it simply <emph>copies</emph> the data
from the <var>input-files</var>, and the metadata from the <emph>first</emph>
<var>input-file</var>, to the <var>output-file</var>. 
This means that data compressed with a packing convention must use
the identical packing parameters (e.g., <code>scale_factor</code> and
<code>add_offset</code>) for a given variable across <emph>all</emph> input files.
Otherwise the concatenated dataset will not unpack correctly.
The workaround for cases where the packing parameters differ across
<var>input-files</var> requires three steps:
First, unpack the data using <command>ncpdq</command>.
Second, concatenate the unpacked data using <command>ncrcat</command>, 
Third, re-pack the result with <command>ncpdq</command>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1512">ARM conventions</indexterm></cindex>
<para><command>ncrcat</command> applies special rules to <acronym><acronymword>ARM</acronymword></acronym> convention time
fields (e.g., <code>time_offset</code>).
See <ref label="ARM-Conventions"><xrefnodename>ARM Conventions</xrefnodename></ref> for a complete description.
</para>
<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncrcat&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncrcat &textndash;&gt;
</html>
<para>EXAMPLES
</para>
<para>Concatenate files <file>85.nc</file>, <file>86.nc</file>, <w>&dots; <file>89.nc</file></w>
along the record dimension, and store the results in <file>8589.nc</file>: 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1513">globbing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1514"><code>NINTAP</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1515">Processor</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1516"><acronym><acronymword>CCM</acronymword></acronym> Processor</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">ncrcat 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncrcat 8[56789].nc 8589.nc
ncrcat -n 5,2,1 85.nc 8589.nc
</pre></example>
<noindent></noindent>
<para>These three methods produce identical answers.
<xref label="Specifying-Input-Files"><xrefnodename>Specifying Input Files</xrefnodename></xref>, for an explanation of the distinctions
between these methods.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1517">Fortran</indexterm></cindex>
<para>Assume the files <file>85.nc</file>, <file>86.nc</file>, <w>&dots; <file>89.nc</file></w>
each contain a record coordinate <var>time</var> of <w>length 12</w> defined
such that the third record in <file>86.nc</file> contains data from March
1986, etc. 
<acronym><acronymword>NCO</acronymword></acronym> knows how to hyperslab the record dimension across files. 
Thus, to concatenate data from December, 1985&textndash;February, 1986:
</para><example endspaces=" ">
<pre xml:space="preserve">ncrcat -d time,11,13 85.nc 86.nc 87.nc 8512_8602.nc
ncrcat -F -d time,12,14 85.nc 86.nc 87.nc 8512_8602.nc
</pre></example>
<noindent></noindent>
<para>The file <file>87.nc</file> is superfluous, but does not cause an error.
When <command>ncra</command> and <command>ncrcat</command> encounter a file which does 
contain any records that meet the specified hyperslab criteria, they
disregard the file and proceed to the next file without failing.
The <samp>-F</samp> turns on the Fortran (1-based) indexing convention.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1518">stride</indexterm></cindex>
</para>
<para>The following uses the <var>stride</var> option to concatenate all the March 
temperature data from multiple input files into a single output file
</para><example endspaces=" ">
<pre xml:space="preserve">ncrcat -F -d time,3,,12 -v temperature 85.nc 86.nc 87.nc 858687_03.nc
</pre></example>
<para><xref label="Stride"><xrefnodename>Stride</xrefnodename></xref>, for a description of the <var>stride</var> argument.
</para>
<para>Assume the <var>time</var> coordinate is incrementally numbered such that
January, <w>1985 = 1</w> and December, <w>1989 = 60.</w>
Assuming <code>??</code> only expands to the five desired files, the following 
concatenates June, 1985&textndash;June, 1989: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncrcat -d time,6.,54. ??.nc 8506_8906.nc
</pre></example>

<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncrename&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncrename &textndash;&gt;
</html>
</section>
<node name="ncrename-netCDF-Renamer" spaces=" "><nodename>ncrename netCDF Renamer</nodename><nodenext spaces=" ">ncwa netCDF Weighted Averager</nodenext><nodeprev spaces=" ">ncrcat netCDF Record Concatenator</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncrename</command> netCDF Renamer</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1519">renaming variables</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1520">renaming groups</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1521">renaming dimensions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1522">renaming attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1523">variable names</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1524">dimension names</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1525">attribute names</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1526">group names</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="41" mergedindex="cp">ncrename</indexterm></findex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">ncrename [-a <var>old_name</var>,<var>new_name</var>] [-a &dots;] [-D <var>dbg</var>] 
[-d <var>old_name</var>,<var>new_name</var>] [-d &dots;] [-g <var>old_name</var>,<var>new_name</var>] [-g &dots;] 
[-h] [--hdf] [--hdr_pad <var>nbr</var>] [-l <var>path</var>] [-O] [-o <var>output-file</var>] [-p <var>path</var>] [-R] [-r] 
[-v <var>old_name</var>,<var>new_name</var>] [-v &dots;]
<var>input-file</var> [[<var>output-file</var>]]
</pre></example>
 
<noindent></noindent>
<para>DESCRIPTION
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1527"><kbd>.</kbd></indexterm></cindex>
<para><command>ncrename</command> renames netCDF dimensions, variables, attributes, and
groups. 
Each object that has a name in the list of old names is renamed using
the corresponding name in the list of new names. 
All the new names must be unique. 
Every old name must exist in the input file, unless the old name is
preceded by the period (or &textldquo;dot&textrdquo;) character <samp>.</samp>. 
The validity of <var>old_name</var> is not checked prior to the renaming. 
Thus, if <var>old_name</var> is specified without the the <samp>.</samp> prefix and
is not present in <var>input-file</var>, <command>ncrename</command> will abort. 
The <var>new_name</var> should never be prefixed by a <samp>.</samp> (or else the
period will be included as part of the new name).
The OPTIONS and EXAMPLES show how to select specific variables
whose attributes are to be renamed.
</para>
<para>Although <command>ncrename</command> supports full pathnames for both
<var>old_name</var> and <var>new_name</var>, this is really &textldquo;window dressing&textrdquo;.
The full-path to <var>new_name</var> must be identical to the full-path to 
<var>old_name</var> in all classes of objects (attributes, variables,
dimensions, or groups).
In other words, <command>ncrename</command> can change only the local names
of objects, it cannot change the location of the object in the group
hierarchy within the file.
Hence using a full-path in <var>new_name</var> is redundant. 
The object name is the terminal path component of <var>new_name</var> and
this object must already exist in the group specified by the 
<var>old_name</var> path.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1528">data safety</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1529">safeguards</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1530">temporary output files</indexterm></cindex>
<para><command>ncrename</command> is an exception to the normal <acronym><acronymword>NCO</acronymword></acronym> rule that
the user will be interactively prompted before an existing file is
changed, and that a temporary copy of an output file is constructed
during the operation. 
If only <var>input-file</var> is specified, then <command>ncrename</command> changes
the names of the <var>input-file</var> in place without prompting and without
creating a temporary copy of <code>input-file</code>.
This is because the renaming operation is considered reversible if the
user makes a mistake.
The <var>new_name</var> can easily be changed back to <var>old_name</var> by using 
<command>ncrename</command> one more time.
</para>
<para>Note that renaming a dimension to the name of a dependent variable can
be used to invert the relationship between an independent coordinate
variable and a dependent variable. 
In this case, the named dependent variable must be one-dimensional and
should have no missing values. 
Such a variable will become a coordinate variable.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1531">performance</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1532">operator speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1533">speed</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1534">execution time</indexterm></cindex>
<para>According to the <cite>netCDF User Guide</cite>, renaming properties in
netCDF files does not incur the penalty of recopying the entire file
when the <var>new_name</var> is shorter than the <var>old_name</var>.
</para>
<noindent></noindent>
<para>OPTIONS
</para>
<table commandarg="samp" spaces=" " endspaces=" ">
<tableentry><tableterm><item spaces=" "><itemformat command="samp">-a <var>old_name</var>,<var>new_name</var></itemformat></item>
</tableterm><tableitem><para>Attribute renaming. 
The old and new names of the attribute are specified with <samp>-a</samp>
(or <samp>--attribute</samp>) by the associated <var>old_name</var> and
<var>new_name</var> values.  
<cindex index="cp" spaces=" "><indexterm index="cp" number="1535"><code>global</code> attribute</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1536">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1537">attributes, global</indexterm></cindex>
Global attributes are treated no differently than variable attributes.
This option may be specified more than once.
As mentioned above, all occurrences of the attribute of a given name
will be renamed unless the <samp>.</samp> form is used, with one exception.
To change the attribute name for a particular variable, specify 
the <var>old_name</var> in the format <var>old_var_name&arobase;old_att_name</var>.
The <samp>&arobase;</samp> symbol delimits the variable from the attribute name.
If the attribute is uniquely named (no other variables contain the
attribute) then the <var>old_var_name&arobase;old_att_name</var> syntax is
redundant. 
The <var>var_nm</var> <code>global</code> has special significance&textmdash;it indicates
that <var>att_nm</var> refers to a global or group attribute, and not to a
variable named <code>global</code>.
In other words, a <var>var_nm</var> of <code>global</code> is syntactically
equivalent to a <var>var_nm</var> that is empty.
The <var>var_name&arobase;att_name</var> syntax is accepted, though not required,
for the <var>new_name</var>.
</para>
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-d <var>old_name</var>,<var>new_name</var></itemformat></item>
</tableterm><tableitem><para>Dimension renaming. 
The old and new names of the dimension are specified with <samp>-d</samp>
(or <samp>--dmn</samp>, <samp>--dimension</samp>) by the associated <var>old_name</var>
and <var>new_name</var> values.  
This option may be specified more than once.
</para> 
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-g <var>old_name</var>,<var>new_name</var></itemformat></item>
</tableterm><tableitem><para>Group renaming. 
The old and new names of the group are specified with <samp>-g</samp>
(or <samp>--grp</samp>, <samp>--group</samp>) by the associated <var>old_name</var>
and <var>new_name</var> values.  
This option may be specified more than once.
This functionality is only available in <acronym><acronymword>NCO</acronymword></acronym> version 4.3.7
(October, 2013) or later, and only when built on netCDF library version
4.3.1-rc1 (August, 2013) or later. 
</para> 
</tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="samp">-v <var>old_name</var>,<var>new_name</var></itemformat></item>
</tableterm><tableitem><para>Variable renaming. 
The old and new names of the variable are specified with <samp>-v</samp>
(or <samp>--variable</samp>) by the associated <var>old_name</var> and
<var>new_name</var> values.  
This option may be specified more than once.
</para></tableitem></tableentry></table>

<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncrename&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncrename &textndash;&gt;
</html>
<para>EXAMPLES
</para>
<para>Rename the variable <code>p</code> to <code>pressure</code> and <code>t</code> to
<code>temperature</code> in netCDF <file>in.nc</file>. 
In this case <code>p</code> must exist in the input file (or
<command>ncrename</command> will abort), but the presence of <code>t</code> is optional:
</para><example endspaces=" ">
<pre xml:space="preserve">ncrename -v p,pressure -v .t,temperature in.nc
</pre></example>

<para>Rename the attribute <code>long_name</code> to <code>largo_nombre</code> in the
variable <code>u</code>, and no other variables in netCDF <file>in.nc</file>. 
</para><example endspaces=" ">
<pre xml:space="preserve">ncrename -a u&arobase;long_name,largo_nombre in.nc
</pre></example>
 
<para>Rename the group <code>g8</code> to <code>g20</code> in netCDF4 file
<file>in_grp.nc</file>:   
</para><example endspaces=" ">
<pre xml:space="preserve">ncrename -g g8,g20 in_grp.nc
</pre></example>
 
<para>Rename the variable <code>/g1/lon</code> to <code>longitude</code> in netCDF4
<file>in_grp.nc</file>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncrename -v /g1/lon,longitude in_grp.nc
ncrename -v /g1/lon,/g1/longitude in_grp.nc # Alternate
</pre></example>
 
<html endspaces=" ">
&lt;a name=&quot;ncrename_crd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncrename_crd &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1538">coordinate variables</indexterm></cindex>
<para><command>ncrename</command> does not automatically attach dimensions to variables of
the same name.
This is done to make renaming an easy way to change whether a variable
is a coordinate.
If you want to rename a coordinate variable so that it remains a
coordinate variable, you must separately rename both the dimension and
the variable: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncrename -d lon,longitude -v lon,longitude in.nc
</pre></example>
<para>Unfortunately, the netCDF4 library has a longstanding bug (all versions
until 4.3.1-rc5 released in December, 2013) that causes <acronym><acronymword>NCO</acronymword></acronym> to
crash when performing this operation. 
Simultaneously renaming variables and dimensions in netCDF4 files with
earlier versions of netCDF is impossible; it must instead be done in two
separate <command>ncrename</command> invocations (e.g., first rename the
variable, then the dimension) to avoid triggering the libary bug.
</para>
<para>A related bug causes unintended side-effects with <command>ncrename</command> 
also built with all versions of the netCDF4 library until 4.3.1-rc5
released in December, 2013):
Renaming <emph>either</emph> a dimension <emph>or</emph> its assosiated
coordinate variable (not both, which would fail as above) in a netCDF4 
file inadvertently <emph>does</emph> rename both:
</para><example endspaces=" ">
<pre xml:space="preserve"># Demonstate bug in netCDF4/HDF5 library prior to netCDF-4.3.1-rc5
ncks -O -h -m -M -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -v lat_T42,lat ~/foo.nc ~/foo2.nc # Also renames dimension
ncrename -O -d lat_T42,lat ~/foo.nc ~/foo2.nc # Also renames variable
</pre></example>
<para>To avoid this faulty behavior, either build <acronym><acronymword>NCO</acronymword></acronym> with netCDF
version 4.3.1-rc5 or later, or convert the file to netCDF3 first,
then rename as intended, then convert back.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1539">global attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1540">attributes, global</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1541"><code>_FillValue</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1542"><code>missing_value</code></indexterm></cindex>
<para>Create netCDF <file>out.nc</file> identical to <file>in.nc</file> except the
attribute <code>_FillValue</code> is changed to <code>missing_value</code>, 
the attribute <code>units</code> is changed to <code>CGS_units</code> (but only in
those variables which possess it), the attribute <code>hieght</code> is
changed to <code>height</code> in the variable <code>tpt</code>, and in the
variable <code>prs_sfc</code>, if it exists.
</para><example endspaces=" ">
<pre xml:space="preserve">ncrename -a _FillValue,missing_value -a .units,CGS_units \
  -a tpt&arobase;hieght,height -a prs_sfc&arobase;.hieght,height in.nc out.nc 
</pre></example>
<para>The presence and absence of the <samp>.</samp> and <samp>&arobase;</samp> features
cause this command to execute successfully only if a number of 
conditions are met. 
All variables <emph>must</emph> have a <code>_FillValue</code> attribute <emph>and</emph> 
<code>_FillValue</code> must also be a global attribute.
The <code>units</code> attribute, on the other hand, will be renamed to
<code>CGS_units</code> wherever it is found but need not be present in
the file at all (either as a global or a variable attribute).
The variable <code>tpt</code> must contain the <code>hieght</code> attribute.
The variable <code>prs_sfc</code> need not exist, and need not contain the
<code>hieght</code> attribute.
</para>
<para>Rename the global or group attribute <code>Convention</code> to
<code>Conventions</code>
</para><example endspaces=" ">
<pre xml:space="preserve">ncrename -a Convention,Conventions  in.nc # Variable and global atts.
ncrename -a .Convention,Conventions in.nc # Variable and global atts.
ncrename -a &arobase;Convention,Conventions  in.nc # Global atts. only
ncrename -a &arobase;.Convention,Conventions in.nc # Global atts. only
ncrename -a global&arobase;Convention,Conventions   in.nc # Global atts. only
ncrename -a .global&arobase;.Convention,Conventions in.nc # Global atts. only
</pre></example>
<para>The examples without the <code>&arobase;</code> character attempt to change the
attribute name in both Global or Group and variable attributes.
The examples with the <code>&arobase;</code> character attempt to change only 
global and group <code>Convention</code> attributes, and leave unchanged any
<code>Convention</code> attributes attached directly to variables.
Attributes prefixed with a period (<code>.Convention</code>) need not be
present. 
Attributes not prefixed with a period (<code>Convention</code>) must be
present. 
Variables prefixed with a period (<code>.</code> or <code>.global</code>) need not 
be present.  
Variables not prefixed with a period (<code>global</code>) must be present.  
</para>
<page></page>
<html endspaces=" ">
&lt;a name=&quot;ncwa&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ncwa &textndash;&gt;
</html>
</section>
<node name="ncwa-netCDF-Weighted-Averager" spaces=" "><nodename>ncwa netCDF Weighted Averager</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">ncrename netCDF Renamer</nodeprev><nodeup spaces=" ">Operator Reference Manual</nodeup></node>
<section spaces=" "><sectiontitle><command>ncwa</command> netCDF Weighted Averager</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1543">averaging data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1544">weighted average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1545">masked average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1546">broadcasting variables</indexterm></cindex>
<findex index="fn" spaces=" "><indexterm index="fn" number="42" mergedindex="cp">ncwa</indexterm></findex>

<noindent></noindent>
<para>SYNTAX
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa [-3] [-4] [-6] [-7] [-A] [-a <var>dim</var>[,&dots;]] [-B <var>mask_cond</var>] [-b] [-C] [-c]
[--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz]
[-D <var>dbg</var>] [-d <var>dim</var>,[<var>min</var>][,[<var>max</var>][,[<var>stride</var>]]] [-F]
[-G <var>gpe_dsc</var>] [-g <var>grp</var>[,&dots;]] [-h] [--hdr_pad <var>nbr</var>] [-I]
[-L <var>dfl_lvl</var>] [-l <var>path</var>] [-M <var>mask_val</var>] [-m <var>mask_var</var>] [-N] [--no_tmp_fl]
[-O] [-o <var>output-file</var>] [-p <var>path</var>] [-R] [-r] [--ram_all] [--rth_dbl|flt]
[-T <var>mask_comp</var>] [-t <var>thr_nbr</var>] [--unn] [-v <var>var</var>[,&dots;]] [-w <var>weight</var>]
[-X ...] [-x] [-y <var>op_typ</var>]
<var>input-file</var> [<var>output-file</var>]
</pre></example>

<noindent></noindent>
<para>DESCRIPTION
</para>
<para><command>ncwa</command> averages variables in a single file over arbitrary
dimensions, with options to specify weights, masks, and normalization.   
<xref label="Statistics-vs_002e-Concatenation"><xrefnodename>Statistics vs. Concatenation</xrefnodename></xref>, for a description of the
distinctions between the various statistics tools and concatenators. 
The default behavior of <command>ncwa</command> is to arithmetically average
every numerical variable over all dimensions and to produce a scalar 
result for each. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1547">degenerate dimension</indexterm></cindex>
<para>Averaged dimensions are, by default, eliminated as dimensions.
Their corresponding coordinates, if any, are output as scalar
variables. 
The <samp>-b</samp> switch (and its long option equivalents <samp>--rdd</samp> and 
<samp>--retain-degenerate-dimensions</samp>) causes <command>ncwa</command> to retain
averaged dimensions as degenerate (<w>size 1</w>) dimensions.
This maintains the association between a dimension (or coordinate) and
variables after averaging and simplifies, for instance, later
concatenation along the degenerate dimension. 
</para>
<para>To average variables over only a subset of their dimensions, specify
these dimensions in a comma-separated list following <samp>-a</samp>, e.g.,
<samp>-a time,lat,lon</samp>. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1548">arithmetic operators</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1549">hyperslab</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1550"><code>-d <var>dim</var>,[<var>min</var>][,[<var>max</var>]]</code></indexterm></cindex>
As with all arithmetic operators, the operation may be restricted to
an arbitrary hypserslab by employing the <samp>-d</samp> option
(<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>). 
<command>ncwa</command> also handles values matching the variable&textrsquo;s
<code>_FillValue</code> attribute correctly. 
Moreover, <command>ncwa</command> understands how to manipulate user-specified
weights, masks, and normalization options.
With these options, <command>ncwa</command> can compute sophisticated averages
(and integrals) from the command line. 
</para>
<html endspaces=" ">
&lt;a name=&quot;-w&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-w &textndash;&gt;
&lt;a name=&quot;wgt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#wgt &textndash;&gt;
</html>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1551"><code>-w <var>weight</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1552"><code>--weight <var>weight</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1553"><code>--wgt_var <var>weight</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1554"><code>-m <var>mask_var</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1555"><code>--mask-variable <var>mask_var</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1556"><code>--mask_variable <var>mask_var</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1557"><code>--msk_nm <var>mask_var</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1558"><code>--msk_var <var>mask_var</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1559"><code>-B <var>mask_cond</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1560"><code>--msk_cnd <var>mask_cond</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1561"><code>--mask_condition <var>mask_cond</var></code></indexterm></cindex>
<para><var>mask_var</var> and <var>weight</var>, if specified, are broadcast to conform
to the variables being averaged. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1562">rank</indexterm></cindex>
The rank of variables is reduced by the number of dimensions which they
are averaged over.  
Thus arrays which are one dimensional in the <var>input-file</var> and are
averaged by <command>ncwa</command> appear in the <var>output-file</var> as scalars.
This allows the user to infer which dimensions may have been averaged.
Note that that it is impossible for <command>ncwa</command> to make make a
<var>weight</var> or <var>mask_var</var> of rank <var>W</var> conform to a <var>var</var> of
rank <var>V</var> if <var>W &gt; V</var>.
This situation often arises when coordinate variables (which, by
definition, are one dimensional) are weighted and averaged.
<command>ncwa</command> assumes you know this is impossible and so <command>ncwa</command>
does not attempt to broadcast <var>weight</var> or <var>mask_var</var> to conform
to <var>var</var> in this case, nor does <command>ncwa</command> print a warning
message telling you this, because it is so common.  
Specifying <var>dbg &gt; 2</var> does cause <command>ncwa</command> to emit warnings in
these situations, however.
</para>
<para>Non-coordinate variables are always masked and weighted if specified.
Coordinate variables, however, may be treated specially.
By default, an averaged coordinate variable, e.g., <code>latitude</code>,
appears in <var>output-file</var> averaged the same way as any other variable 
containing an averaged dimension.
In other words, by default <command>ncwa</command> weights and masks
coordinate variables like all other variables.  
This design decision was intended to be helpful but for some
applications it may be preferable not to weight or mask coordinate
variables just like all other variables.   
Consider the following arguments to <command>ncwa</command>: 
<code>-a latitude -w lat_wgt -d latitude,0.,90.</code> where <code>lat_wgt</code> is
a weight in the <code>latitude</code> dimension.
Since, by default <command>ncwa</command> weights coordinate variables, the
value of <code>latitude</code> in the <var>output-file</var> depends on the weights 
in <var>lat_wgt</var> and is not likely to <w>be 45.0</w>, the midpoint latitude
of the hyperslab.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1563">coordinate variable</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1564"><code>-I</code></indexterm></cindex>
Option <samp>-I</samp> overrides this default behavior and causes
<command>ncwa</command> not to weight or mask coordinate variables
<footnote><para>The default behavior of (<samp>-I</samp>) changed on
19981201&textmdash;before this date the default was not to weight or mask
coordinate variables.</para></footnote>.
In the above case, this causes the value of <code>latitude</code> in the
<var>output-file</var> to <w>be 45.0</w>, an appealing result.
Thus, <samp>-I</samp> specifies simple arithmetic averages for the coordinate
variables. 
In the case of latitude, <samp>-I</samp> specifies that you prefer to archive
the arithmetic mean latitude of the averaged hyperslabs rather than the 
area-weighted mean latitude.
<footnote><para>If <code>lat_wgt</code> contains Gaussian weights then the value of 
<code>latitude</code> in the <var>output-file</var> will be the area-weighted
centroid of the hyperslab. 
For the example given, this is about <w>30 degrees.</w></para></footnote>.  
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1565">average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1566">operation types</indexterm></cindex>
<para>As explained in <xref label="Operation-Types"><xrefnodename>Operation Types</xrefnodename></xref>, <command>ncwa</command> 
<emph>always averages</emph> coordinate variables regardless of the arithmetic
operation type performed on the non-coordinate variables. 
This is independent of the setting of the <samp>-I</samp> option.
The mathematical definition of operations involving rank reduction 
is given above (<pxref label="Operation-Types"><xrefnodename>Operation Types</xrefnodename></pxref>).
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Mask condition</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Normalization and Integration</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<html endspaces=" ">
&lt;a name=&quot;mask&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#mask &textndash;&gt;
&lt;a name=&quot;msk&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#msk &textndash;&gt;
&lt;a name=&quot;-m&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#-m &textndash;&gt;
</html>
<node name="Mask-condition" spaces=" "><nodename>Mask condition</nodename><nodenext spaces=" ">Normalization and Integration</nodenext><nodeprev spaces=" ">ncwa netCDF Weighted Averager</nodeprev><nodeup spaces=" ">ncwa netCDF Weighted Averager</nodeup></node>
<subsection spaces=" "><sectiontitle>Mask condition</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1567">mask condition</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1568">truth condition</indexterm></cindex>
<tex endspaces=" ">
Each $\xxx_{\idx}$ also has an associated masking
weight~$\mskflg_{\idx}$ whose value is~0 or~1 (false or true).
The value of~$\mskflg_{\idx}$ is always~1 unless a <var>mask\_var</var> is
specified (with 
<samp>-m</samp>).
As noted above, <var>mask\_var</var> is broadcast, if possible, to conform 
to the variable being averaged.  
In this case, the value of~$\mskflg_{\idx}$ depends on the 
<dfn>mask condition</dfn> also known as the <dfn>truth condition</dfn>.
As expected, $\mskflg_{\idx} = 1$ when the mask condition is
<dfn>true</dfn> and $\mskflg_{\idx} = 0$ otherwise.   
</tex>

<cindex index="cp" spaces=" "><indexterm index="cp" number="1569"><code>--op_rlt <var>mask_comp</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1570"><code>--mask_comparator <var>mask_comp</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1571"><code>--msk_cmp_typ <var>mask_comp</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1572"><code>--msk_cnd_sng <var>mask_cond</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1573"><code>--mask_condition <var>mask_cond</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1574"><code>-B <var>mask_cond</var></code></indexterm></cindex>
<para>The mask condition has the syntax <math><var>mask_var</var></math>
<math><var>mask_comp</var></math> <math><var>mask_val</var></math>. 
The preferred method to specify the mask condition is in one string with  
the <samp>-B</samp> or <samp>--mask_condition</samp> switches.
The older method is to use the three switches <samp>-m</samp>, <samp>-T</samp>, and
<samp>-M</samp> to specify the <var>mask_var</var>, <var>mask_comp</var>, and 
<var>mask_val</var>, respectively.  
<footnote><para>The three switches <samp>-m</samp>, <samp>-T</samp>, and <samp>-M</samp> are
maintained for backward compatibility and may be deprecated in the
future.
It is safest to write scripts using <samp>--mask_condition</samp>.</para></footnote>.
The <var>mask_condition</var> string is automatically parsed into its three
constituents <var>mask_var</var>, <var>mask_comp</var>, and <var>mask_val</var>.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1575">comparator</indexterm></cindex>
<para>Here <var>mask_var</var> is the name of the masking variable (specified with 
<samp>-m</samp>, <samp>--mask-variable</samp>, <samp>--mask_variable</samp>,
<samp>--msk_nm</samp>, or <samp>--msk_var</samp>).  
The truth <var>mask_comp</var> argument (specified with <samp>-T</samp>,
<samp>--mask_comparator</samp>, <samp>--msk_cmp_typ</samp>, or <samp>--op_rlt</samp> may 
be any one of the six arithmetic comparators: <kbd>eq</kbd>, <kbd>ne</kbd>,
<kbd>gt</kbd>, <kbd>lt</kbd>, <kbd>ge</kbd>, <kbd>le</kbd>. 
<set name="flg" line=" flg"></set>
<tex endspaces=" ">
These are the Fortran-style character abbreviations for the logical 
comparisons $=$, $\neq$, $&gt;$, $&lt;$, $\ge$, $\le$. 
<clear name="flg" line=" flg"></clear>
</tex>
These are the Fortran-style character abbreviations for the logical 
comparisons <math>==</math>, <math>!=</math>, <math>&gt;</math>, <math>&lt;</math>, <math>&gt;=</math>,
<math>&lt;=</math>. 
<clear name="flg" line=" flg"></clear>
The mask comparator defaults to <kbd>eq</kbd> (equality).
<cindex index="cp" spaces=" "><indexterm index="cp" number="1576"><code>--mask-value <var>mask_val</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1577"><code>--mask_value <var>mask_val</var></code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1578"><code>--msk_val <var>mask_val</var></code></indexterm></cindex>
The <var>mask_val</var> argument to <samp>-M</samp> (or <samp>--mask-value</samp>, or
<samp>--msk_val</samp>) is the right hand side of the
<dfn>mask condition</dfn>.
Thus for the <var>i</var>&textrsquo;th element of the hyperslab to be averaged,
the mask condition is 
<set name="flg" line=" flg"></set>
<tex endspaces=" ">
{\it mask$_{\idx}$ mask\_comp mask\_val}.
<clear name="flg" line=" flg"></clear>
</tex>
<math>mask(i)</math> <var>mask_comp</var> <var>mask_val</var>.
<clear name="flg" line=" flg"></clear>
</para>
<tex endspaces=" ">
Each~$\xxx_{\idx}$ is also associated with an additional
weight~$\wgt_{\idx}$ whose value may be user-specified.
The value of~$\wgt_{\idx}$ is identically~1 unless the user specifies a 
weighting variable <var>weight</var> (with <samp>-w</samp>, <samp>--weight</samp>, or
<samp>--wgt\_var</samp>). 
In this case, the value of~$\wgt_{\idx}$ is determined by the
<var>weight</var> variable in the <var>input-file</var>. 
As noted above, <var>weight</var> is broadcast, if possible, to conform 
to the variable being averaged.  

$\tllnbr$ is the number of input elements $\xxx_{\idx}$ which actually
contribute to output element $\xxx_{\jjj}$.
$\tllnbr$ is also known as the <dfn>tally</dfn> and is defined as 
$$
\tllnbr = \sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} 
$$
$\tllnbr$ is identical to the denominator of the generic averaging
expression except for the omission of the weight $\wgt_{\idx}$.
Thus $\tllnbr = \lmnnbr$ whenever no input points are missing values or
are masked.  
Whether an element contributes to the output, and thus increments
$\tllnbr$ by one, has more to do with the above two criteria (missing
value and masking) than with the numeric value of the element per se.
For example, $\xxx_{\idx}=0.0$ does contribute to $\xxx_{\jjj}$
(assuming the <code>_FillValue</code> attribute is not~0.0 and location
$\idx$ is not masked). 
The value $\xxx_{\idx}=0.0$ will not change the numerator of the generic 
averaging expression, but it will change the denominator (unless its
weight $\wgt_{\idx}=0.0$ as well).
</tex>

<html endspaces=" ">
&lt;a name=&quot;nrm&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#nrm &textndash;&gt;
&lt;a name=&quot;ntg&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ntg &textndash;&gt;
</html>
</subsection>
<node name="Normalization-and-Integration" spaces=" "><nodename>Normalization and Integration</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">Mask condition</nodeprev><nodeup spaces=" ">ncwa netCDF Weighted Averager</nodeup></node>
<subsection spaces=" "><sectiontitle>Normalization and Integration</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1579">normalization</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1580"><code>-N</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1581"><code>numerator</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1582">integration</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1583">dot product</indexterm></cindex>
<para><command>ncwa</command> has one switch which controls the normalization of the
averages appearing in the <var>output-file</var>.
Short option <samp>-N</samp> (or long options <samp>--nmr</samp> or
<samp>--numerator</samp>) prevents <command>ncwa</command> from dividing the weighted
sum of the variable (the numerator in the averaging expression) by the
weighted sum of the weights (the denominator in the averaging
expression).   
Thus <samp>-N</samp> tells <command>ncwa</command> to return just the numerator of the
arithmetic expression defining the operation (<pxref label="Operation-Types"><xrefnodename>Operation Types</xrefnodename></pxref>). 
</para>
<para>With this normalization option, <command>ncwa</command> can integrate variables.
Averages are first computed as sums, and then normalized to obtain the
average. 
The original sum (i.e., the numerator of the expression in
<ref label="Operation-Types"><xrefnodename>Operation Types</xrefnodename></ref>) is output if default normalization is turned off
(<w>with <samp>-N</samp></w>). 
This sum is the integral (not the average) over the specified 
(<w>with <samp>-a</samp></w>, or all, if none are specified) dimensions.
The weighting variable, if specified (<w>with <samp>-w</samp></w>), plays the
role of the differential increment and thus permits more sophisticated 
integrals (i.e., weighted sums) to be output.
For example, consider the variable 
<code>lev</code> where <math><var>lev</var> = [100,500,1000]</math> weighted by
the weight <code>lev_wgt</code> where <math><var>lev_wgt</var> = [10,2,1]</math>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1584">dot product</indexterm></cindex>
The vertical integral of <code>lev</code>, weighted by <code>lev_wgt</code>, 
is the dot product of <var>lev</var> and <var>lev_wgt</var>. 
That this is <w>is 3000.0</w> can be seen by inspection and verified with 
the integration command
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -N -a lev -v lev -w lev_wgt in.nc foo.nc;ncks foo.nc
</pre></example>

<ignore>
fxm TODO nco702
As explained in @xref{Operation Types}, @command{ncwa} 
@emph{always averages} coordinate variables regardless of the arithmetic 
operation type performed on the non-coordinate variables. 
The single exception is shown in the above example.
The @samp{-N} switch turns off normalization so variables which are to be
averaged, including coordinate variables, are not normalized.
This is equivalent to summation or integration.
</ignore>

<ignore>
@c NB: these masking features are deprecated
The second normalization option tells @command{ncwa} to multiply the
weighted average the variable (given by the averaging expression)
by the @w{tally, @var{M}}.
Thus this option is similar to integration---multiplying the mean value
of a quantity by the number of gridpoints to which it applies.

The third normalization option is equivalent to specifying the first two
options simultaneously. 
In other words this option causes @command{ncwa} to return 
@w{@var{M} times} the numerator of the generic averaging expression.  
With these normalization options, @command{ncwa} can compute
sophisticated averages (and integrals) from the command line.
</ignore>

<noindent></noindent>
<html endspaces=" ">
&lt;a name=&quot;xmp_ncwa&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#xmp_ncwa &textndash;&gt;
</html>
<para>EXAMPLES
</para>
<para>Given file <file>85_0112.nc</file>:
</para><example endspaces=" ">
<pre xml:space="preserve">netcdf 85_0112 &lbrace;
dimensions:
        lat = 64 ;
        lev = 18 ;
        lon = 128 ;
        time = UNLIMITED ; // (12 currently)
variables:
        float lat(lat) ;
        float lev(lev) ;
        float lon(lon) ;
        float time(time) ;
        float scalar_var ;
        float three_dmn_var(lat, lev, lon) ;
        float two_dmn_var(lat, lev) ;
        float mask(lat, lon) ;
        float gw(lat) ;
&rbrace; 
</pre></example>

<para>Average all variables in <file>in.nc</file> over all dimensions and store
results in <file>out.nc</file>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa in.nc out.nc
</pre></example>
<noindent></noindent>
<para>All variables in <file>in.nc</file> are reduced to scalars in <file>out.nc</file> 
since <command>ncwa</command> averages over all dimensions unless otherwise
specified (with <samp>-a</samp>).
</para>
<para>Store the zonal (longitudinal) mean of <file>in.nc</file> in <file>out.nc</file>:
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -a lon in.nc out1.nc
ncwa -a lon -b in.nc out2.nc
</pre></example>
<noindent></noindent>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1585">degenerate dimension</indexterm></cindex>
<para>The first command turns <code>lon</code> into a scalar and the second retains 
<code>lon</code> as a degenerate dimension in all variables.
</para><example endspaces=" ">
<pre xml:space="preserve">% ncks -C -H -v lon out1.nc
lon = 135
% ncks -C -H -v lon out2.nc
lon[0] = 135
</pre></example>
<para>In either case the tally is simply the size of <code>lon</code>, i.e., 180
for the <file>85_0112.nc</file> file described by the sample header above.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1586"><code>gw</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1587">Gaussian weights</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1588">climate model</indexterm></cindex>
<para>Compute the meridional (latitudinal) mean, with values weighted by
the corresponding element of <var>gw</var>
<footnote><para><code>gw</code> stands for <dfn>Gaussian weight</dfn> in many
climate models.</para></footnote>: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -w gw -a lat in.nc out.nc
</pre></example>
<noindent></noindent>
<para>Here the tally is simply the size of <code>lat</code>, <w>or 64.</w>
The sum of the Gaussian weights <w>is 2.0.</w>
</para>
<para>Compute the area mean over the tropical Pacific:
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -w gw -a lat,lon -d lat,-20.,20. -d lon,120.,270. in.nc out.nc
</pre></example>
<noindent></noindent>
<para>Here the tally is 
<set name="flg" line=" flg"></set>
<tex endspaces=" ">
$64 \times 128 = 8192$.
<clear name="flg" line=" flg"></clear>
</tex>
64 times 128 = 8192.
<clear name="flg" line=" flg"></clear>
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1589"><code>ORO</code></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1590">climate model</indexterm></cindex>
<para>Compute the area-mean over the globe using only points for which 
<set name="flg" line=" flg"></set>
<tex endspaces=" ">
$ORO &lt; 0.5$
<clear name="flg" line=" flg"></clear>
</tex>
<var>ORO</var> &lt; 0.5
<clear name="flg" line=" flg"></clear>
<footnote><para><code>ORO</code> stands for <dfn>Orography</dfn> in some climate models
and in those models <math><var>ORO</var> &lt; 0.5</math> selects ocean gridpoints.</para></footnote>: 
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -B 'ORO &lt; 0.5'      -w gw -a lat,lon in.nc out.nc
ncwa -m ORO -M 0.5 -T lt -w gw -a lat,lon in.nc out.nc
</pre></example>
<noindent></noindent>
<para>It is considerably simpler to specify the complete <var>mask_cond</var> with
the single string argument to <samp>-B</samp> than with the three separate
switches <samp>-m</samp>, <samp>-T</samp>, and <samp>-M</samp>
<footnote><para>Unfortunately the <samp>-B</samp> and <samp>--mask_condition</samp>
options are unsupported on Windows (with the <acronym><acronymword>MVS</acronymword></acronym> compiler),
which lacks a free, standard parser and lexer.</para></footnote>. 
If in doubt, enclose the <var>mask_cond</var> within quotes since some
of the comparators have special meanings to the shell.
</para>
<para>Assuming 70% of the gridpoints are maritime, then here the tally is
<set name="flg" line=" flg"></set>
<tex endspaces=" ">
$0.70 \times 8192 \approx 5734$.
<clear name="flg" line=" flg"></clear>
</tex>
0.70 times 8192 = 5734.
<clear name="flg" line=" flg"></clear>
</para>
<para>Compute the global annual mean over the maritime tropical Pacific:
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -B 'ORO &lt; 0.5'      -w gw -a lat,lon,time \
  -d lat,-20.0,20.0 -d lon,120.0,270.0 in.nc out.nc
ncwa -m ORO -M 0.5 -T lt -w gw -a lat,lon,time \
  -d lat,-20.0,20.0 -d lon,120.0,270.0 in.nc out.nc
</pre></example>
<para>Further examples will use the one-switch specification of
<var>mask_cond</var>.  
</para>
<para>Determine the total area of the maritime tropical Pacific, assuming
the variable <var>area</var> contains the area of each gridcell
</para><example endspaces=" ">
<pre xml:space="preserve">ncwa -N -v area -B 'ORO &lt; 0.5' -a lat,lon \
  -d lat,-20.0,20.0 -d lon,120.0,270.0 in.nc out.nc
</pre></example>
<para>Weighting <var>area</var> (e.g., by <var>gw</var>) is not appropriate because
<var>area</var> is <emph>already</emph> area-weighted by definition.
Thus the <samp>-N</samp> switch, or, equivalently, the <samp>-y ttl</samp> switch, 
correctly integrate the cell areas into a total regional area.
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1591">mask condition</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1592">truth condition</indexterm></cindex>
<para>Mask a file to contain <var>_FillValue</var> everywhere except where
<math><var>thr_min</var> &lt;= <var>msk_var</var> &lt;= <var>thr_max</var></math>:
</para><example endspaces=" ">
<pre xml:space="preserve"># Set masking variable and its scalar thresholds
export msk_var='three_dmn_var_dbl' # Masking variable
export thr_max='20' # Maximum allowed value
export thr_min='10' # Minimum allowed value
ncecat -O in.nc out.nc # Wrap out.nc in degenerate &quot;record&quot; dimension
ncwa -O -a record -B &quot;$&lbrace;msk_var&rbrace; &lt;= $&lbrace;thr_max&rbrace;&quot; out.nc out.nc
ncecat -O out.nc out.nc # Wrap out.nc in degenerate &quot;record&quot; dimension
ncwa -O -a record -B &quot;$&lbrace;msk_var&rbrace; &gt;= $&lbrace;thr_min&rbrace;&quot; out.nc out.nc
</pre></example>
<para>After the first use of <command>ncwa</command>, <file>out.nc</file> contains
<var>_FillValue</var> where <code>$&lbrace;msk_var&rbrace; &gt;= $&lbrace;thr_max&rbrace;</code>.
The process is then repeated on the remaining data to filter out 
points where <code>$&lbrace;msk_var&rbrace; &lt;= $&lbrace;thr_min&rbrace;</code>.
The resulting <file>out.nc</file> contains valid data only
where <math><var>thr_min</var> &lt;= <var>msk_var</var> &lt;= <var>thr_max</var></math>.
</para>
<html endspaces=" ">
&lt;a name=&quot;ctr&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ctr &textndash;&gt;
</html>
</subsection>
</section>
</chapter>
<node name="Contributing" spaces=" "><nodename>Contributing</nodename><nodenext spaces=" ">Quick Start</nodenext><nodeprev spaces=" ">Operator Reference Manual</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<chapter spaces=" "><sectiontitle>Contributing</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1593">contributing</indexterm></cindex>
<para>We welcome contributions from anyone.
The project homepage at <uref><urefurl>https://sf.net/projects/nco</urefurl></uref>
contains more information on how to contribute. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1594">PayPal</indexterm></cindex>
<para>Financial contributions to <acronym><acronymword>NCO</acronymword></acronym> development may be made through  
<uref><urefurl>https://www.paypal.com/xclick/business=zender%40uci.edu&amp;item_name=NCO+development&amp;item_number=nco_dnt_dvl&amp;no_note=1&amp;tax=0&amp;currency_code=USD</urefurl><urefdesc spaces=" ">PayPal</urefdesc></uref>.
<acronym><acronymword>NCO</acronymword></acronym> has been shared for over <w>10 years</w> yet only two 
users have contributed any money to the developers
<footnote spaces="\n"><cindex index="cp" spaces=" "><indexterm index="cp" number="1595">chocolate</indexterm></cindex>
<para>Happy users have sent me a few gifts, though.
This includes a box of imported chocolate.
Mmm.
Appreciation and gifts are definitely better than money.
Naturally, I&textrsquo;m too lazy to split and send gifts to the other developers.
However, unlike some <acronym><acronymword>NCO</acronymword></acronym> developers, I have a steady &quot;real job&quot;.
My intent is to split monetary donations among the active developers
and to send them their shares via PayPal.</para></footnote>. 
So you could be the third!
</para>
<html endspaces=" ">
&lt;a name=&quot;dvl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#dvl &textndash;&gt;
&lt;a name=&quot;cnt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cnt &textndash;&gt;
&lt;a name=&quot;ppl&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ppl &textndash;&gt;
</html>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Contributors</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Proposals for Institutional Funding</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<node name="Contributors" spaces=" "><nodename>Contributors</nodename><nodenext spaces=" ">Proposals for Institutional Funding</nodenext><nodeprev spaces=" ">Contributing</nodeprev><nodeup spaces=" ">Contributing</nodeup></node>
<section spaces=" "><sectiontitle>Contributors</sectiontitle>
<para><acronym><acronymword>NCO</acronymword></acronym> would not exist without the dedicated efforts of the
remarkable software engineers who conceive, develop, and
maintain netCDF, UDUnits, and OPeNDAP.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1596">Russ Rew</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1597">John Caron</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1598">Glenn Davis</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1599">Steve Emmerson</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1600">James Gallagher</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1601">Ed Hartnett</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1602">Dennis Heimbigner</indexterm></cindex>
Since 1995 <acronym><acronymword>NCO</acronymword></acronym> has received support from, I believe, the
entire staff of all these projects, including 
Russ Rew, 
John Caron,
Glenn Davis, 
Steve Emmerson, 
James Gallagher, 
Ed Hartnett, 
and Dennis Heimbigner.
In addition to their roles in maintaining the software stack on which
<acronym><acronymword>NCO</acronymword></acronym> perches, Yertl-like, some of these gentlemen have advised
or contributed to <acronym><acronymword>NCO</acronymword></acronym> specifically. That support is
acknowledged separately below. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1603">contributors</indexterm></cindex>
<para>The primary contributors to <acronym><acronymword>NCO</acronymword></acronym> development have been:
</para><table commandarg="asis" spaces=" " endspaces=" ">
<beforefirstitem><cindex index="cp" spaces=" "><indexterm index="cp" number="1604">Charlie Zender</indexterm></cindex>
</beforefirstitem><tableentry><tableterm><item spaces=" "><itemformat command="asis">Charlie Zender</itemformat></item>
</tableterm><tableitem><para>All concept, design and implementation from 1995-2000.
Since then autotools, bug-squashing, <acronym><acronymword>CDL</acronymword></acronym>, chunking,
documentation, anchoring, recursion, <acronym><acronymword>GPE</acronymword></acronym>, packing,
<acronym><acronymword>NCO</acronymword></acronym> library redesign, <command>ncap2</command> features,
<command>ncbo</command>, <command>ncpdq</command>, SMP threading and MPI parallelization,
netCDF4 integration, external funding, project management, science
research, releases. 
<cindex index="cp" spaces=" "><indexterm index="cp" number="1605">Henry Butowsky</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Henry Butowsky</itemformat></item>
</tableterm><tableitem><para>Non-linear operations and <code>min()</code>, <code>max()</code>, <code>total()</code>
support in <command>ncra</command> and <command>ncwa</command>. 
Type conversion for arithmetic.
Migration to netCDF3 API.
<command>ncap2</command> parser, lexer, <acronym><acronymword>GSL</acronymword></acronym>-support, <w>and I/O</w>.
Multislabbing algorithm.
Variable wildcarding.
Numerous hacks.
<command>ncap2</command> language.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1606">Rorik Peterson</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Rorik Peterson</itemformat></item>
</tableterm><tableitem><para>Original autotool build support. 
Long command-line options.
Original UDUnits support.
Debianization.
Numerous bug-fixes.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1607">Daniel Wang</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Daniel Wang</itemformat></item>
</tableterm><tableitem><para>Script Workflow Analysis for MultiProcessing (SWAMP).
RPM support.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1608">Harry Mangalam</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Harry Mangalam</itemformat></item>
</tableterm><tableitem><para>Benchmarking.
OPeNDAP configuration.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1609">Pedro Vicente</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Pedro Vicente</itemformat></item>
</tableterm><tableitem><para>Windows Visual Studio support.
netCDF4 groups.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1610">Russ Rew</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Russ Rew</itemformat></item>
</tableterm><tableitem><para>Advice on NCO structural algorithms
<cindex index="cp" spaces=" "><indexterm index="cp" number="1611">Brian Mays</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Brian Mays</itemformat></item>
</tableterm><tableitem><para>Original packaging for Debian <acronym><acronymword>GNU</acronymword></acronym>/Linux, <command>nroff</command> man pages.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1612">George Shapovalov</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">George Shapovalov</itemformat></item>
</tableterm><tableitem><para>Packaging for Gentoo <acronym><acronymword>GNU</acronymword></acronym>/Linux.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1613">Bill Kocik</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Bill Kocik</itemformat></item>
</tableterm><tableitem><para>Memory management.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1614">Len Makin</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Len Makin</itemformat></item>
</tableterm><tableitem><para>NEC SX architecture support.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1615">Jim Edwards</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Jim Edwards</itemformat></item>
</tableterm><tableitem><para>AIX architecture support.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1616">Juliana Rew</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Juliana Rew</itemformat></item>
</tableterm><tableitem><para>Compatibility with large <acronym><acronymword>PID</acronymword></acronym>s.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1617">Karen Schuchardt</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Karen Schuchardt</itemformat></item>
</tableterm><tableitem><para>Auxiliary coordinate support.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1618">Gayathri Venkitachalam</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Gayathri Venkitachalam</itemformat></item>
</tableterm><tableitem><para><acronym><acronymword>MPI</acronymword></acronym> implementation.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1619">Scott Capps</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Scott Capps</itemformat></item>
</tableterm><tableitem><para>Large work-load testing
<cindex index="cp" spaces=" "><indexterm index="cp" number="1620">Peter Campbell</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1621">Martin Dix</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1622">Mark Flanner</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1623">Markus Liebig</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1624">Keith Lindsay</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1625">Stu Muller</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1626">Mike Page</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1627">Martin Schmidt</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1628">Lori Sentman</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1629">Michael Schulz</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1630">Gary Strand</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1631">Andrew Wittenberg</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1632">George White</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1633">Remik Ziemlinski</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Peter Campbell, Martin Dix, Mark Flanner, Markus Liebig, Keith Lindsay, Mike Page, Martin Schmidt, Michael Schulz, Lori Sentman, Gary Strand, George White Andrew Wittenberg, Remik Ziemlinski</itemformat></item>
</tableterm><tableitem><para>Excellent bug reports and feature requests.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1634">Daniel Baumann</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1635">Nick Bower</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1636">Luk Claes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1637">Barry deFreese</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1638">Aleksandar Jelenak</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1639">Francesco Lovergine</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1640">Matej Vela</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Daniel Baumann, Nick Bower, Luk Claebs, Barry deFreese, Aleksandar Jelenak, Francesco Lovergine, Matej Vela</itemformat></item>
</tableterm><tableitem><para>Debian packaging
<cindex index="cp" spaces=" "><indexterm index="cp" number="1641">Patrice Dumas</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1642">Ed Hill</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1643">Orion Powlawski</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Patrice Dumas, Ed Hill, Orion Poplawski</itemformat></item>
</tableterm><tableitem><para>RedHat packaging
<cindex index="cp" spaces=" "><indexterm index="cp" number="1644">George Shapavalov</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1645">Patrick Kursawe</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">George Shapavalov, Patrick Kursawe</itemformat></item>
</tableterm><tableitem><para>Gentoo packaging
<cindex index="cp" spaces=" "><indexterm index="cp" number="1646">Filipe Fernandes</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Filipe Fernandes</itemformat></item>
</tableterm><tableitem><para>OpenSuse packaging
<cindex index="cp" spaces=" "><indexterm index="cp" number="1647">Alexander Hansen</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1648">Takeshi Enomoto</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Takeshi Enomoto, Alexander Hansen</itemformat></item>
</tableterm><tableitem><para>Mac OS packaging
<cindex index="cp" spaces=" "><indexterm index="cp" number="1649">Eric Blake</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Eric Blake</itemformat></item>
</tableterm><tableitem><para>Autoconf/M4 help
<cindex index="cp" spaces=" "><indexterm index="cp" number="1650">Gavin Burris</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1651">Kyle Wilcox</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Gavin Burris, Kyle Wilcox</itemformat></item>
</tableterm><tableitem><para>RHEL and CentOS build scripts and bug reports.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1652">Andrea Cimatoribus</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Andrea Cimatoribus</itemformat></item>
</tableterm><tableitem><para><acronym><acronymword>NCO</acronymword></acronym> Spiral Logo
<cindex index="cp" spaces=" "><indexterm index="cp" number="1653">Martin Otte</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1654">Etienne Tourigny</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Martin Otte, Etienne Tourigny</itemformat></item>
</tableterm><tableitem><para>Single bug reports and fixes
<cindex index="cp" spaces=" "><indexterm index="cp" number="1655">Wenshan Wang</indexterm></cindex>
</para></tableitem></tableentry><tableentry><tableterm><item spaces=" "><itemformat command="asis">Wenshan Wang</itemformat></item>
</tableterm><tableitem><para><acronym><acronymword>CMIP5</acronymword></acronym> and <acronym><acronymword>MODIS</acronymword></acronym> processing documentation.
</para></tableitem></tableentry></table>
<para>Please let me know if your name was omitted!
</para>
<html endspaces=" ">
&lt;a name=&quot;prp&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#prp &textndash;&gt;
&lt;a name=&quot;prp_sei&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#prp_sei &textndash;&gt;
&lt;a name=&quot;fnd&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#fnd &textndash;&gt;
</html>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::  ">Proposals for Institutional Funding</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

</section>
<node name="Proposals-for-Institutional-Funding" spaces=" "><nodename>Proposals for Institutional Funding</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">Contributors</nodeprev><nodeup spaces=" ">Contributing</nodeup></node>
<section spaces=" "><sectiontitle>Proposals for Institutional Funding</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1656">funding</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1657">proposals</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1658"><acronym><acronymword>NSF</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1659">server-side processing</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1660">Distributed Data Reduction &amp; Analysis</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1661">Scientific Data Operators</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1662"><acronym><acronymword>DDRA</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1663">Server-Side Distributed Data Reduction &amp; Analysis</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1664"><acronym><acronymword>SSDDRA</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1665"><acronym><acronymword>CCSM</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1666"><acronym><acronymword>IPCC</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1667"><acronym><acronymword>NSF</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1668"><acronym><acronymword>SDO</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1669"><acronym><acronymword>SEIII</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1670">OptIPuter</indexterm></cindex>
<para><acronym><acronymword>NSF</acronymword></acronym> has funded a
<uref><urefurl>http://nco.sf.net#prp_sei</urefurl><urefdesc spaces=" ">project</urefdesc></uref>
to improve Distributed Data Reduction &amp; Analysis (<acronym><acronymword>DDRA</acronymword></acronym>) by
evolving <acronym><acronymword>NCO</acronymword></acronym> into a suite of Scientific Data Operators called 
<acronym><acronymword>SDO</acronymword></acronym>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1671">parallelism</indexterm></cindex>
The two main components of this project are <acronym><acronymword>NCO</acronymword></acronym> parallelism
(OpenMP, <acronym><acronymword>MPI</acronymword></acronym>) and Server-Side <acronym><acronymword>DDRA</acronymword></acronym>
(<acronym><acronymword>SSDDRA</acronymword></acronym>) implemented through extensions to <acronym><acronymword>OPeNDAP</acronymword></acronym> 
and netCDF4. 
This project will dramatically reduce bandwidth usage for <acronym><acronymword>NCO</acronymword></acronym>
<acronym><acronymword>DDRA</acronymword></acronym>. 
</para>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1672"><acronym><acronymword>NASA</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1673"><acronym><acronymword>NRA</acronymword></acronym></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1674"><acronym><acronymword>HDF</acronymword></acronym></indexterm></cindex>
<para>With this first <acronym><acronymword>NCO</acronymword></acronym> proposal funded, the content of the
next <acronym><acronymword>NCO</acronymword></acronym> proposal is clear.
We are interested in obtaining <acronym><acronymword>NASA</acronymword></acronym> support for
<acronym><acronymword>HDF</acronymword></acronym>-specific enhancements to <acronym><acronymword>NCO</acronymword></acronym>. 
We plan to submit a proposal to the next suitable <acronym><acronymword>NASA</acronymword></acronym>
<acronym><acronymword>NRA</acronymword></acronym> or <acronym><acronymword>NSF</acronymword></acronym> opportunity. 
</para>
<para>We are considering other interesting ideas for still more proposals.
Please contact us if you wish to be involved with any future
<acronym><acronymword>NCO</acronymword></acronym>-related proposals.  
Comments on the proposals and letters of support are also very welcome.
</para>
<html endspaces=" ">
&lt;a name=&quot;quicksrt&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#quicksrt &textndash;&gt;
</html>
</section>
</chapter>
<node name="Quick-Start" spaces=" "><nodename>Quick Start</nodename><nodenext spaces=" ">CMIP5 Example</nodenext><nodeprev spaces=" ">Contributing</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<chapter spaces=" "><sectiontitle>Quick Start</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1675">Quick Start</indexterm></cindex>
<para>Simple examples in Bash shell scripts showing how to average data with
different file structures.  
Here we include monthly, seasonal and annual average with daily or
monthly data in either one file or multiple files. 
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Daily data in one file</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Monthly data in one file</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">One time point one file</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Multiple files with multiple time points</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<node name="Daily-data-in-one-file" spaces=" "><nodename>Daily data in one file</nodename><nodenext spaces=" ">Monthly data in one file</nodenext><nodeprev spaces=" ">Quick Start</nodeprev><nodeup spaces=" ">Quick Start</nodeup></node>
<section spaces=" "><sectiontitle>Daily data in one file</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1676">daily data</indexterm></cindex>
<para>Suppose we have daily data from Jan 1st, 1990 to Dec. 31, 2005 in the
file of <file>in.nc</file> with the record dimension as <code>time</code>.
</para>
<noindent></noindent>
<para><strong>Monthly average:</strong>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1677">monthly average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1678">average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1679">time-averaging</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">for yyyy in &lbrace;1990..2005&rbrace;; do      # Loop over years
  for moy in &lbrace;1..12&rbrace;; do          # Loop over months
    mm=$( printf &quot;%02d&quot; $&lbrace;moy&rbrace; )  # Change to 2-digit format

    # Average specific month yyyy-mm
    ncra -O -d time,&quot;$&lbrace;yyyy&rbrace;-$&lbrace;mm&rbrace;-01&quot;,&quot;$&lbrace;yyyy&rbrace;-$&lbrace;mm&rbrace;-31&quot; \
         in.nc in_$&lbrace;yyyy&rbrace;$&lbrace;mm&rbrace;.nc
  done
done

# Concatenate monthly files together
ncrcat -O in_??????.nc out.nc
</pre></example>

<noindent></noindent>
<para><strong>Annual average:</strong>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1680">annual average from daily data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1681">average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1682">time-averaging</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">for yyyy in &lbrace;1990..2005&rbrace;; do      # Loop over years
  ncra -O -d time,&quot;$&lbrace;yyyy&rbrace;-01-01&quot;,&quot;$&lbrace;yyyy&rbrace;-12-31&quot; in.nc in_$&lbrace;yyyy&rbrace;.nc
done

# Concatenate annual files together
ncrcat -O in_????.nc out.nc
</pre></example>
<para>The <option>-O</option> switch means to overwrite the pre-existing files (<pxref label="Batch-Mode"><xrefnodename>Batch Mode</xrefnodename></pxref>).
The <option>-d</option> option is to specify the range of hyperslabs (<pxref label="Hyperslabs"><xrefnodename>Hyperslabs</xrefnodename></pxref>).
There are detailed instructions on <command>ncra</command> (<pxref label="ncra-netCDF-Record-Averager"><xrefnodename>ncra netCDF Record Averager</xrefnodename></pxref> and <command>ncrcat</command> (<pxref label="ncrcat-netCDF-Record-Concatenator"><xrefnodename>ncrcat netCDF Record Concatenator</xrefnodename></pxref>).
<acronym><acronymword>NCO</acronymword></acronym> supports UDUnits so that we can use readable dates as time dimension (<pxref label="UDUnits-Support"><xrefnodename>UDUnits Support</xrefnodename></pxref>).
</para>
</section>
<node name="Monthly-data-in-one-file" spaces=" "><nodename>Monthly data in one file</nodename><nodenext spaces=" ">One time point one file</nodenext><nodeprev spaces=" ">Daily data in one file</nodeprev><nodeup spaces=" ">Quick Start</nodeup></node>
<section spaces=" "><sectiontitle>Monthly data in one file</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1683">monthly data</indexterm></cindex>
<para>Inside the input file <file>in.nc</file>, the record dimension <code>time</code> is from Jan 1990 to Dec 2005.
</para>
<noindent></noindent>
<para><strong>Seasonal average (e.g., DJF):</strong>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1684">seasonal average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1685">average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1686">time-averaging</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -O --mro -d time,&quot;1990-12-01&quot;,,12,3 in.nc out.nc
</pre></example>

<noindent></noindent>
<para><strong>Annual average:</strong>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1687">annual average from monthly data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1688">average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1689">time-averaging</indexterm></cindex>
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -O --mro -d time,,,12,12 in.nc out.nc
</pre></example>
<para>Here we use the subcycle feature (i.e., the number after the fourth comma: <samp>3</samp> in the seasonal example and the second <samp>12</samp> in the annual example) 
to retrieve groups of records separated by regular intervals (<pxref label="Subcycle"><xrefnodename>Subcycle</xrefnodename></pxref>).
The option <option>--mro</option> switches <command>ncra</command> to produce a Multi-Record Output instead of a single-record output.
For example, assume <var>snd</var> is a 3D array with dimensions <code>time</code> * <code>latitude</code> * <code>longitude</code> 
and <code>time</code> includes every month from Jan. 1990 to Dec. 2005, 192 months as total, which are 16 years.
Let&textrsquo;s look at the following two command lines.
</para><example endspaces=" ">
<pre xml:space="preserve">ncra --mro -v snd -d time,&quot;1990-12-01&quot;,,12,3 in.nc out_mro.nc
ncra -v snd -d time,&quot;1990-12-01&quot;,,12,3 in.nc out_sro.nc
</pre></example>
<para>In the first output file, <file>out_mro.nc</file>, <var>snd</var> is still a 3D array with dimensions <code>time</code> * <code>latitude</code> * <code>longitude</code>,
but the length of <code>time</code> now is 16, meaning 16 winters.
In the second output file, <file>out_sro.nc</file>, the length of <code>time</code> is <w>only 1</w>.
It is now the average of all the 16 winters.
</para>
<para>when using <samp>-d <var>dim</var>,min[,max]</samp> to specify the hyperslabs,
you can leave it blank if you want to include the minimum or the maximum of the data, like we did above.
</para>
</section>
<node name="One-time-point-one-file" spaces=" "><nodename>One time point one file</nodename><nodenext spaces=" ">Multiple files with multiple time points</nodenext><nodeprev spaces=" ">Monthly data in one file</nodeprev><nodeup spaces=" ">Quick Start</nodeup></node>
<section spaces=" "><sectiontitle>One time point one file</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1690">daily data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1691">monthly data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1692">average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1693">time-averaging</indexterm></cindex>
<para>This means if you have daily data of 30 days, there will be 30 data files.
Or if you have monthly data of 12 months, there will be 12 data files.
Dealing with this kind of files, you need to specify the file names in shell scripts and pass them to NCO operators.
For example, your daily data files may look like <file>snd_19900101.nc</file>, <file>snd_19900102.nc</file>, <file>snd_19900103.nc</file> ...
If you want to know the monthly average of Jan 1990, you can write like,
</para><example endspaces=" ">
<pre xml:space="preserve">ncra -O snd_199001??.nc out.nc
</pre></example>
<para>You might want to use loop if you need the average of each month.
</para><example endspaces=" ">
<pre xml:space="preserve">for moy in &lbrace;1..12&rbrace;; do          # Loop over months
  mm=$( printf &quot;%02d&quot; $&lbrace;moy&rbrace; )  # Change to 2-digit format

  ncra -O snd_????$&lbrace;mm&rbrace;??.nc out_$&lbrace;mm&rbrace;.nc
done
</pre></example>

</section>
<node name="Multiple-files-with-multiple-time-points" spaces=" "><nodename>Multiple files with multiple time points</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">One time point one file</nodeprev><nodeup spaces=" ">Quick Start</nodeup></node>
<section spaces=" "><sectiontitle>Multiple files with multiple time points</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1694">daily data</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1695">monthly data</indexterm></cindex>
<para>Similar as the last one, it&textrsquo;s more about shell scripts.
Suppose you have daily data with one month of them in one data file.
The monthly average is simply to apply <command>ncra</command> on the specific data file.
And for seasonal averages, you can specify the three months by shell scripts.
</para>
<html endspaces=" ">
&lt;a name=&quot;cmip5&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#cmip5 &textndash;&gt;
</html>
</section>
</chapter>
<node name="CMIP5-Example" spaces=" "><nodename>CMIP5 Example</nodename><nodenext spaces=" ">Parallel</nodenext><nodeprev spaces=" ">Quick Start</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<chapter spaces=" "><sectiontitle><acronym><acronymword>CMIP5</acronymword></acronym> Example</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1696"><acronym><acronymword>CMIP5</acronymword></acronym></indexterm></cindex>
<ignore>
This @uref{http://nco.sf.net/xmp_cesm.html,Wonderful CMIP5 Documentation}
shows complete processing of the @acronym{CMIP5} dataset.
</ignore>

<para>The fifth phase of the Coupled Model Intercomparison Project 
(<uref><urefurl>http://cmip-pcmdi.llnl.gov/cmip5/index.html?submenuheader=0</urefurl><urefdesc spaces=" "><acronym><acronymword>CMIP5</acronymword></acronym></urefdesc></uref>) 
provides a multi-model framework for comparing the mechanisms and
responses of climate models from around the world.   
However, it is a tremendous workload to retrieve a single climate
statistic from all these models, each of which includes several ensemble 
members.  
Not only that, it is too often a repetitive process which impedes new
research and hypothesis testing.  
Our <acronym><acronymword>NASA</acronymword></acronym> <acronym><acronymword>ACCESS</acronymword></acronym> project is designed to simplify and
accelerate this process.  
To begin, we document below a prototypical example of <acronym><acronymword>CMIP5</acronymword></acronym>
analysis and evaluation using traditional <acronym><acronymword>NCO</acronymword></acronym> commands on
netCDF3-format model and <acronym><acronymword>HDF-EOS</acronymword></acronym> format observational
(<acronym><acronymword>NASA</acronymword></acronym> <acronym><acronymword>MODIS</acronymword></acronym> satellite instrument) datasets.
These examples complement the <acronym><acronymword>NCO</acronymword></acronym> User Guide by detailing
in-depth data analysis in a frequently encountered &textldquo;real world&textrdquo;
context.   
Graphical representations of the results (<acronym><acronymword>NCL</acronymword></acronym> scripts
available upon request) are provided to illustrate physical meaning of
the analysis.
<ignore>
In 2013 we added scripts which make use of new @acronym{NCO} features
that combine all the loops in the analysis into single commands by
exploiting @acronym{NCO}'s new group aggregation and arithmetic
features.   
</ignore>
Since <acronym><acronymword>NCO</acronymword></acronym> can process hierarchical datasets, i.e., datasets
stored with netCDF4 groups, we present sample scripts illustrating
group-based processing as well. 
</para>
<menu endspaces=" ">
<menuentry leadingtext="* "><menunode separator="::">Combine Files</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Global Distribution of Long-term Average</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Annual Average over Regions</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Monthly Cycle</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Regrid MODIS Data</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Add Coordinates to MODIS Data</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry><menuentry leadingtext="* "><menunode separator="::">Permute MODIS Coordinates</menunode><menudescription><pre xml:space="preserve">
</pre></menudescription></menuentry></menu>

<node name="Combine-Files" spaces=" "><nodename>Combine Files</nodename><nodenext spaces=" ">Global Distribution of Long-term Average</nodenext><nodeprev spaces=" ">CMIP5 Example</nodeprev><nodeup spaces=" ">CMIP5 Example</nodeup></node>
<section spaces=" "><sectiontitle>Combine Files</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1697">file combination</indexterm></cindex>
<para>Sometimes, the data of one ensemble member will be stored in several
files to reduce single file size.
But it is not convenient to process in a batch mode.
The following script illustrates how to concatenate these files into one.
Key steps include: 
</para><enumerate first="1" endspaces=" ">
<listitem spaces=" "><para>Obtain number and names (or partial names) of files in a directory
</para></listitem><listitem spaces=" "><para>Concatenate files along the record dimension (usually time) using 
<command>ncrcat</command> (<pxref label="ncrcat-netCDF-Record-Concatenator"><xrefnodename>ncrcat netCDF Record Concatenator</xrefnodename></pxref>).
</para></listitem></enumerate>
<example endspaces=" ">
<verbatiminclude file="xmp/cmb_fl.sh" spaces=" ">xmp/cmb_fl.sh</verbatiminclude>
</example>

<para>Right now, <acronym><acronymword>CMIP5</acronymword></acronym> model data downloaded from Earth System Grid
Federation (<uref><urefurl>http://pcmdi9.llnl.gov/esgf-web-fe/</urefurl><urefdesc spaces=" "><acronym><acronymword>ESGF</acronymword></acronym></urefdesc></uref>) 
will not contain <key>group</key> features yet. 
Therefore users can aggregate the flat files into groups themselves.
The following script shows how to aggregate models to one file.
Each dataset becomes a group in the output file.
There can be several levels of groups.
In this example, we employ two experiments as the top-level.
The second-level comprises different models.
Some models have more than one ensemble member.
These ensemble members are on the third level.
In each sub-group of ensemble members, we appended two variables,
<var>snc</var> and <var>snd</var> (these stand for snow cover and snow depth, by
the way).
<cindex index="cp" spaces=" "><indexterm index="cp" number="1698"><option>--gag</option></indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1699">aggregation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1700">group aggregation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1701">groups, creating</indexterm></cindex>
</para><example endspaces=" ">
<verbatiminclude file="xmp/cmb_fl_grp.sh" spaces=" ">xmp/cmb_fl_grp.sh</verbatiminclude>
</example>

</section>
<node name="Global-Distribution-of-Long_002dterm-Average" spaces=" "><nodename>Global Distribution of Long-term Average</nodename><nodenext spaces=" ">Annual Average over Regions</nodenext><nodeprev spaces=" ">Combine Files</nodeprev><nodeup spaces=" ">CMIP5 Example</nodeup></node>
<section spaces=" "><sectiontitle>Global Distribution of Long-term Average</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1702">spatial distribution</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1703">long-term average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1704">average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1705">time-averaging</indexterm></cindex>
<float name="fgr_003aglb" type="Figure" number="7.1" spaces=" " endspaces=" "><floattype>Figure</floattype><floatname>fgr:glb</floatname>
<image><imagefile>xmp/fgr1</imagefile><imagewidth>3.5in</imagewidth></image> 
<caption><para>Global Distribution of Long-term Average.</para></caption>
</float>
<noindent></noindent>
<para>This section illustrates how to calculate the global distribution of
long-term average (<pxref label="fgr_003aglb"><xrefnodename>fgr:glb</xrefnodename></pxref>) with either flat files or 
<uref><urefurl>http://nco.sourceforge.net/nco.html#index-groups</urefurl><urefdesc spaces=" ">group file</urefdesc></uref>.
Key steps include: 
</para><enumerate first="1" endspaces=" ">
<listitem spaces=" "><para>Average ensemble members of each model using <command>nces</command> (<pxref label="nces-netCDF-Ensemble-Statistics"><xrefnodename>nces netCDF Ensemble Statistics</xrefnodename></pxref>)
</para></listitem><listitem spaces=" "><para>Average the record dimension using <command>ncra</command> (<pxref label="ncra-netCDF-Record-Averager"><xrefnodename>ncra netCDF Record Averager</xrefnodename></pxref>)
</para></listitem><listitem spaces=" "><para>Store results of each model as a distinct group in a single output file using <command>ncecat</command> (<pxref label="ncrcat-netCDF-Record-Concatenator"><xrefnodename>ncrcat netCDF Record Concatenator</xrefnodename></pxref>) with the <option>--gag</option> option
</para></listitem></enumerate>
<para>The first example shows how to process flat files.
</para><example endspaces=" ">
<verbatiminclude file="xmp/glb_avg.sh" spaces=" ">xmp/glb_avg.sh</verbatiminclude>
</example>

<para>With the use of <key>group</key>, the above script
will be shortened to just <w>TWO LINES</w>.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1706">groups, averaging</indexterm></cindex>
<ignore>
Isn't it amazing or what?!
</ignore>
</para><example endspaces=" ">
<pre xml:space="preserve"># Data from cmb_fl_grp.sh
# ensemble averaging
nces -O --nsm_grp --nsm_sfx='_avg' \
sn_LImon_all-mdl_all-xpt_all-nsm_200001-200512.nc \
  sn_LImon_all-mdl_all-xpt_nsm-avg.nc
ncra -O -g 'avg$' sn_LImon_all-mdl_all-xpt_nsm-avg.nc \
  sn_LImon_all-mdl_all-xpt_nsm-avg_tm-avg.nc
</pre></example>
<para>The input file, <file>sn_LImon_all-mdl_all-xpt_all-nsm_200001-200512.nc</file>, produced by <file>cmb_fl_grp.sh</file>, 
includes all the ensemble members as groups.
The option <samp>--nsm_grp</samp> denotes 
that we are using <uref><urefurl>http://nco.sf.net/nco.html#nsm_grp</urefurl><urefdesc spaces=" ">group ensembles mode</urefdesc></uref> of <command>nces</command>,
instead of <uref><urefurl>http://nco.sf.net/nco.html#nsm_fl</urefurl><urefdesc spaces=" ">file ensembles mode</urefdesc></uref>, <samp>--nsm_fl</samp>.
The option <samp>--nsm_sfx='_avg'</samp> instructs <command>nces</command> 
to store the output as a new child group <file>/[model]/[model name]_avg</file>,
instead of in the parent group <file>/[model]</file>. 
In the final output file, <file>sn_LImon_all-mdl_all-xpt_nsm-avg_tm-avg.nc</file>, 
sub-groups with a suffix of &textlsquo;avg&textrsquo; are the long-term averages of each model.
Here, <samp>-g 'avg$'</samp> is used to retrieve groups having &textrsquo;avg&textrsquo; at the end of their names,
i.e., the long-term average groups.
</para>
</section>
<node name="Annual-Average-over-Regions" spaces=" "><nodename>Annual Average over Regions</nodename><nodenext spaces=" ">Monthly Cycle</nodenext><nodeprev spaces=" ">Global Distribution of Long-term Average</nodeprev><nodeup spaces=" ">CMIP5 Example</nodeup></node>
<section spaces=" "><sectiontitle>Annual Average over Regions</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1707">annual average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1708">average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1709">time-averaging</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1710">area-averaging</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1711">dimension order</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1712">anomalies</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1713">standard deviation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1714">renaming variables</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1715">attributes, editing</indexterm></cindex>
<cindex index="cp" spaces="	"><indexterm index="cp" number="1716">attributes, modifying</indexterm></cindex>
<cindex index="cp" spaces="	"><indexterm index="cp" number="1717">attributes, overwriting</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1718">regression</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1719">nco script file</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1720">variables, appending</indexterm></cindex>
<float name="fgr_003aanl" type="Figure" number="7.2" spaces=" " endspaces=" "><floattype>Figure</floattype><floatname>fgr:anl</floatname>
<image><imagefile>xmp/fgr2</imagefile><imagewidth>4in</imagewidth></image>
<caption><para>Annual Average over Regions.</para></caption>
</float>
<noindent></noindent>
<para>This section illustrates how to calculate the annual average over
specific regions (<pxref label="fgr_003aanl"><xrefnodename>fgr:anl</xrefnodename></pxref>).
Key steps include: 
</para><enumerate first="1" endspaces=" ">
<listitem spaces=" "><para>Spatial average using <command>ncap2</command> (<pxref label="ncap2-netCDF-Arithmetic-Processor"><xrefnodename>ncap2 netCDF Arithmetic Processor</xrefnodename></pxref>) and <command>ncwa</command> (<pxref label="ncwa-netCDF-Weighted-Averager"><xrefnodename>ncwa netCDF Weighted Averager</xrefnodename></pxref>); 
</para></listitem><listitem spaces=" "><para>Change dimension order using <command>ncpdq</command> (<pxref label="ncpdq-netCDF-Permute-Dimensions-Quickly"><xrefnodename>ncpdq netCDF Permute Dimensions Quickly</xrefnodename></pxref>);
</para></listitem><listitem spaces=" "><para>Annual average using <command>ncra</command> (<pxref label="ncra-netCDF-Record-Averager"><xrefnodename>ncra netCDF Record Averager</xrefnodename></pxref>);
</para></listitem><listitem spaces=" "><para>Anomaly from long-term average using <command>ncbo</command> (<pxref label="ncbo-netCDF-Binary-Operator"><xrefnodename>ncbo netCDF Binary Operator</xrefnodename></pxref>);
</para></listitem><listitem spaces=" "><para>Standard deviation using <command>ncbo</command> (<pxref label="ncbo-netCDF-Binary-Operator"><xrefnodename>ncbo netCDF Binary Operator</xrefnodename></pxref>) and <command>nces</command> (<pxref label="nces-netCDF-Ensemble-Statistics"><xrefnodename>nces netCDF Ensemble Statistics</xrefnodename></pxref>);
</para></listitem><listitem spaces=" "><para>Rename variables using <command>ncrename</command> (<pxref label="ncrename-netCDF-Renamer"><xrefnodename>ncrename netCDF Renamer</xrefnodename></pxref>);
</para></listitem><listitem spaces=" "><para>Edit attributions using <command>ncatted</command> (<pxref label="ncatted-netCDF-Attribute-Editor"><xrefnodename>ncatted netCDF Attribute Editor</xrefnodename></pxref>);
</para></listitem><listitem spaces=" "><para>Linear regression using <command>ncap2</command> (<pxref label="ncap2-netCDF-Arithmetic-Processor"><xrefnodename>ncap2 netCDF Arithmetic Processor</xrefnodename></pxref>);
</para></listitem><listitem spaces=" "><para>Use <command>ncap2</command> (<pxref label="ncap2-netCDF-Arithmetic-Processor"><xrefnodename>ncap2 netCDF Arithmetic Processor</xrefnodename></pxref>) with nco script file (i.e., <file>.nco</file> file);
</para></listitem><listitem spaces=" "><para>Move variables around using <command>ncks</command> (<pxref label="ncks-netCDF-Kitchen-Sink"><xrefnodename>ncks netCDF Kitchen Sink</xrefnodename></pxref>).
</para></listitem></enumerate>
<para><strong>Flat files example</strong>
</para><example endspaces=" ">
<verbatiminclude file="xmp/ann_avg.sh" spaces=" ">xmp/ann_avg.sh</verbatiminclude>
</example>
<para><strong>gsl_rgr.nco</strong>
</para><example endspaces=" ">
<verbatiminclude file="xmp/gsl_rgr.nco" spaces=" ">xmp/gsl_rgr.nco</verbatiminclude>
</example>

<para>With the <key>group</key> feature, 
all the loops over experiments, models and ensemble members can be omitted.
As we are working on implementing <key>group</key> feature in all <acronym><acronymword>NCO</acronymword></acronym> operators,
some functions (e.g., regression and standard deviation over ensemble members) 
may have to wait until the new versions.
<cindex index="cp" spaces=" "><indexterm index="cp" number="1721">group, spatial averaging</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1722">group, temporal averaging</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1723">group, anomaly</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1724">group, standard deviation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1725">group, aggregation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1726">group, dimension permutation</indexterm></cindex>
</para><example endspaces=" ">
<verbatiminclude file="xmp/ann_avg_grp.sh" spaces=" ">xmp/ann_avg_grp.sh</verbatiminclude>
</example>

</section>
<node name="Monthly-Cycle" spaces=" "><nodename>Monthly Cycle</nodename><nodenext spaces=" ">Regrid MODIS Data</nodenext><nodeprev spaces=" ">Annual Average over Regions</nodeprev><nodeup spaces=" ">CMIP5 Example</nodeup></node>
<section spaces=" "><sectiontitle>Monthly Cycle</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1727">monthly average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1728">average</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1729">time-averaging</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1730">anomalies</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1731">geographical weight</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1732">weighted average</indexterm></cindex>
<float name="fgr_003amon" type="Figure" number="7.3" spaces=" " endspaces=" "><floattype>Figure</floattype><floatname>fgr:mon</floatname>
<image><imagefile>xmp/fgr3</imagefile><imagewidth>4in</imagewidth></image>
<caption><para>Monthly Cycle.</para></caption>
</float>
<noindent></noindent>
<para>This script illustrates how to calculate the monthly anomaly from the
annual average (<pxref label="fgr_003amon"><xrefnodename>fgr:mon</xrefnodename></pxref>). 
In order to keep only the monthly cycle,
we will subtract the annual average of each year from the monthly data,
instead of subtracting the long-term average.
This is a little more complicated in coding since we need to loop over years. 
</para>
<para><strong>Flat files example</strong>
</para><example endspaces=" ">
<verbatiminclude file="xmp/mcc.sh" spaces=" ">xmp/mcc.sh</verbatiminclude>
</example>
<para>Using <key>group</key> feature and <uref><urefurl>http://nco.sourceforge.net/nco.html#Hyperslabs</urefurl><urefdesc spaces=" ">hyperslabs</urefdesc></uref> of <command>ncbo</command>,
the script will be shortened.
</para><example endspaces=" ">
<verbatiminclude file="xmp/mcc_grp.sh" spaces=" ">xmp/mcc_grp.sh</verbatiminclude>
</example>

</section>
<node name="Regrid-MODIS-Data" spaces=" "><nodename>Regrid MODIS Data</nodename><nodenext spaces=" ">Add Coordinates to MODIS Data</nodenext><nodeprev spaces=" ">Monthly Cycle</nodeprev><nodeup spaces=" ">CMIP5 Example</nodeup></node>
<section spaces=" "><sectiontitle>Regrid <acronym><acronymword>MODIS</acronymword></acronym> Data</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1733">regrid</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1734">MODIS</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1735">bilinear interpolation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1736">interpolation</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1737">renaming variables</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1738">renaming attributes</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1739">renaming dimensions</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1740">attributes, editing</indexterm></cindex>
<cindex index="cp" spaces="	"><indexterm index="cp" number="1741">attributes, modifying</indexterm></cindex>
<cindex index="cp" spaces="	"><indexterm index="cp" number="1742">attributes, overwriting</indexterm></cindex>
<para>In order to compare the results between <acronym><acronymword>MODIS</acronymword></acronym> and
<acronym><acronymword>CMIP5</acronymword></acronym> models, one usually regrids one or both datasets so 
that the spatial resolutions match. 
Here, the script illustrates how to regrid <acronym><acronymword>MODIS</acronymword></acronym> data.
Key steps include:
</para><enumerate first="1" endspaces=" ">
<listitem spaces=" "><para>Regrid using bilinear interpolation (<pxref label="Bilinear-interpolation"><xrefnodename>Bilinear interpolation</xrefnodename></pxref>)
</para></listitem><listitem spaces=" "><para>Rename variables, dimensions and attributions using <command>ncrename</command> (<pxref label="ncrename-netCDF-Renamer"><xrefnodename>ncrename netCDF Renamer</xrefnodename></pxref>).
</para></listitem></enumerate>
<para><strong>Main Script</strong>
</para><example endspaces=" ">
<verbatiminclude file="xmp/rgr.sh" spaces=" ">xmp/rgr.sh</verbatiminclude>
</example>
<para><strong>bi_interp.nco</strong>
</para><example endspaces=" ">
<verbatiminclude file="xmp/bi_interp.nco" spaces=" ">xmp/bi_interp.nco</verbatiminclude>
</example>

</section>
<node name="Add-Coordinates-to-MODIS-Data" spaces=" "><nodename>Add Coordinates to MODIS Data</nodename><nodenext spaces=" ">Permute MODIS Coordinates</nodenext><nodeprev spaces=" ">Regrid MODIS Data</nodeprev><nodeup spaces=" ">CMIP5 Example</nodeup></node>
<section spaces=" "><sectiontitle>Add Coordinates to <acronym><acronymword>MODIS</acronymword></acronym> Data</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1743">MODIS</indexterm></cindex>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1744">coordinates</indexterm></cindex>
<para><strong>Main Script</strong>
</para><example endspaces=" ">
<verbatiminclude file="xmp/add_crd.sh" spaces=" ">xmp/add_crd.sh</verbatiminclude>
</example>
<para><strong>crd.nco</strong>
</para><example endspaces=" ">
<verbatiminclude file="xmp/crd.nco" spaces=" ">xmp/crd.nco</verbatiminclude>
</example>

</section>
<node name="Permute-MODIS-Coordinates" spaces=" "><nodename>Permute MODIS Coordinates</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">Add Coordinates to MODIS Data</nodeprev><nodeup spaces=" ">CMIP5 Example</nodeup></node>
<section spaces=" "><sectiontitle>Permute <acronym><acronymword>MODIS</acronymword></acronym> Coordinates</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1745">coordinates, modifying</indexterm></cindex>
<!-- c NB: 20140130: @textdegree is in TeXInfo 4.12 but Mac OS X 10.8 ships with 4.8 -->
<para><acronym><acronymword>MODIS</acronymword></acronym> orders latitude data from 90&deg;N to
-90&deg;N, and longitude from -180&deg;E to
180&deg;E.   
However, <acronym><acronymword>CMIP5</acronymword></acronym> orders latitude from -90&deg;N to
90&deg;N, and longitude from 0&deg;E to
360&deg;E.  
This script changes the <acronym><acronymword>MODIS</acronymword></acronym> coordinates to follow the
<acronym><acronymword>CMIP5</acronymword></acronym> convention.
</para><example endspaces=" ">
<verbatiminclude file="xmp/pmt_crd.sh" spaces=" ">xmp/pmt_crd.sh</verbatiminclude>
</example>

<ignore>
@node Hierarchical Data Files, Parallel, CMIP5 Example, Top
section Hierarchical Data Files
@cindex hierarchical data
@cindex groups
Hierarchical Data Files support arbitrarily nested groups.
The following @acronym{NCO} operators now can work recursively through all groups:
@itemize @bullet
@item @command{ncbo} (@pxref{ncbo netCDF Binary Operator})
@item @command{ncecat} (@pxref{ncecat netCDF Ensemble Concatenator})
@item @command{ncks} (@pxref{ncks netCDF Kitchen Sink})
@item @command{ncpdq} (@pxref{ncpdq netCDF Permute Dimensions Quickly})
@item @command{ncwa} (@pxref{ncwa netCDF Weighted Averager})
@end itemize
Here is an example showing:
@enumerate
@item How to create a hierarchical data file from multiple files using @command{ncecat} (@pxref{ncecat netCDF Ensemble Concatenator}) or @command{ncks} (@pxref{ncks netCDF Kitchen Sink});
@item Hyperslabs using @command{ncks} (@pxref{ncks netCDF Kitchen Sink});
@item Spatial average and time average using @command{ncwa} (@pxref{ncwa netCDF Weighted Averager});
@item Anomaly from average using @command{ncbo} (@pxref{ncbo netCDF Binary Operator}).
@end enumerate
@example
@verbatiminclude xmp/grp.sh
@end example
</ignore>

<html endspaces=" ">
&lt;a name=&quot;parallel&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#parallel &textndash;&gt;
</html>
</section>
</chapter>
<node name="Parallel" spaces=" "><nodename>Parallel</nodename><nodenext spaces=" ">CCSM Example</nodenext><nodeprev spaces=" ">CMIP5 Example</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<chapter spaces=" "><sectiontitle>Parallel</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1746"><command>parallel</command></indexterm></cindex>
<para>This section will describe scripting strategies, including the use of
<acronym><acronymword>GNU</acronymword></acronym> Parallel, to <acronym><acronymword>NCO</acronymword></acronym>.
</para><example endspaces=" ">
<pre xml:space="preserve">ls *historical*.nc | parallel ncks -O -d time,&quot;1950-01-01&quot;,&quot;2000-01-01&quot; &lbrace;&rbrace; 50y/&lbrace;&rbrace;
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;ccsm&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#ccsm &textndash;&gt;
</html>
</chapter>
<node name="CCSM-Example" spaces=" "><nodename>CCSM Example</nodename><nodenext spaces=" ">mybibnode</nodenext><nodeprev spaces=" ">Parallel</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<chapter spaces=" "><sectiontitle>CCSM Example</sectiontitle>
<cindex index="cp" spaces=" "><indexterm index="cp" number="1747">CCSM</indexterm></cindex>

<para>This chapter illustrates how to use <acronym><acronymword>NCO</acronymword></acronym> to
process and analyze the results of a <acronym><acronymword>CCSM</acronymword></acronym> climate simulation.
</para><example endspaces=" ">
<pre xml:space="preserve">************************************************************************
Task 0: Finding input files
************************************************************************
The CCSM model outputs files to a local directory like:

/ptmp/zender/archive/T42x1_40

Each component model has its own subdirectory, e.g., 

/ptmp/zender/archive/T42x1_40/atm
/ptmp/zender/archive/T42x1_40/cpl
/ptmp/zender/archive/T42x1_40/ice
/ptmp/zender/archive/T42x1_40/lnd
/ptmp/zender/archive/T42x1_40/ocn

within which model output is tagged with the particular model name

/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-01.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-02.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-03.nc
...
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-12.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0002-01.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0002-02.nc
...

or 

/ptmp/zender/archive/T42x1_40/lnd/T42x1_40.clm2.h0.0001-01.nc
/ptmp/zender/archive/T42x1_40/lnd/T42x1_40.clm2.h0.0001-02.nc
/ptmp/zender/archive/T42x1_40/lnd/T42x1_40.clm2.h0.0001-03.nc
...

************************************************************************
Task 1: Regional processing
************************************************************************
The first task in data processing is often creating seasonal cycles.
Imagine a 100-year simulation with its 1200 monthly mean files.
Our goal is to create a single file containing 12 months of data.
Each month in the output file is the mean of 100 input files.

Normally, we store the &quot;reduced&quot; data in a smaller, local directory.

caseid='T42x1_40'
#drc_in=&quot;$&lbrace;DATA&rbrace;/archive/$&lbrace;caseid&rbrace;/atm&quot;
drc_in=&quot;$&lbrace;DATA&rbrace;/$&lbrace;caseid&rbrace;&quot;
drc_out=&quot;$&lbrace;DATA&rbrace;/$&lbrace;caseid&rbrace;&quot;
mkdir -p $&lbrace;drc_out&rbrace;
cd $&lbrace;drc_out&rbrace;

Method 1: Assume all data in directory applies
for mth in &lbrace;1..12&rbrace;; do
  mm=`printf &quot;%02d&quot; $mth`
  ncra -O -D 1 -o $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm$&lbrace;mm&rbrace;.nc \
    $&lbrace;drc_in&rbrace;/$&lbrace;caseid&rbrace;.cam2.h0.*-$&lbrace;mm&rbrace;.nc 
done # end loop over mth

Method 2: Use shell 'globbing' to construct input filenames
for mth in &lbrace;1..12&rbrace;; do
  mm=`printf &quot;%02d&quot; $mth`
  ncra -O -D 1 -o $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm$&lbrace;mm&rbrace;.nc \
    $&lbrace;drc_in&rbrace;/$&lbrace;caseid&rbrace;.cam2.h0.00??-$&lbrace;mm&rbrace;.nc \
    $&lbrace;drc_in&rbrace;/$&lbrace;caseid&rbrace;.cam2.h0.0100-$&lbrace;mm&rbrace;.nc
done # end loop over mth

Method 3: Construct input filename list explicitly
for mth in &lbrace;1..12&rbrace;; do
  mm=`printf &quot;%02d&quot; $mth`
  fl_lst_in=''
  for yr in &lbrace;1..100&rbrace;; do
    yyyy=`printf &quot;%04d&quot; $yr`
    fl_in=$&lbrace;caseid&rbrace;.cam2.h0.$&lbrace;yyyy&rbrace;-$&lbrace;mm&rbrace;.nc
    fl_lst_in=&quot;$&lbrace;fl_lst_in&rbrace; $&lbrace;caseid&rbrace;.cam2.h0.$&lbrace;yyyy&rbrace;-$&lbrace;mm&rbrace;.nc&quot;
  done # end loop over yr
  ncra -O -D 1 -o $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm$&lbrace;mm&rbrace;.nc -p $&lbrace;drc_in&rbrace; \
    $&lbrace;fl_lst_in&rbrace;
done # end loop over mth

Make sure the output file averages correct input files!
ncks -M prints global metadata: 

  ncks -M $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm01.nc

The input files ncra used to create the climatological monthly mean
will appear in the global attribute named 'history'.

Use ncrcat to aggregate the climatological monthly means

  ncrcat -O -D 1 \
    $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm??.nc $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_0112.nc

Finally, create climatological means for reference.
The climatological time-mean:

  ncra -O -D 1 \
    $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_0112.nc $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm.nc

The climatological zonal-mean:

  ncwa -O -D 1 -a lon \
    $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm.nc $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_x.nc

The climatological time- and spatial-mean:

  ncwa -O -D 1 -a lon,lat,time -w gw \
    $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm.nc $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_xyt.nc

This file contains only scalars, e.g., &quot;global mean temperature&quot;,
used for summarizing global results of a climate experiment.

Climatological monthly anomalies = Annual Cycle: 
Subtract climatological mean from climatological monthly means. 
Result is annual cycle, i.e., climate-mean has been removed.

  ncbo -O -D 1 -o $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_0112_anm.nc \
    $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_0112.nc $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_xyt.nc

************************************************************************
Task 2: Correcting monthly averages
************************************************************************
The previous step appoximates all months as being equal, so, e.g.,
February weighs slightly too much in the climatological mean.
This approximation can be removed by weighting months appropriately.
We must add the number of days per month to the monthly mean files.
First, create a shell variable dpm:

unset dpm # Days per month
declare -a dpm
dpm=(0 31 28.25 31 30 31 30 31 31 30 31 30 31) # Allows 1-based indexing

Method 1: Create dpm directly in climatological monthly means
for mth in &lbrace;1..12&rbrace;; do
  mm=`printf &quot;%02d&quot; $&lbrace;mth&rbrace;`
  ncap2 -O -s &quot;dpm=0.0*date+$&lbrace;dpm[$&lbrace;mth&rbrace;]&rbrace;&quot; \
    $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm$&lbrace;mm&rbrace;.nc $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm$&lbrace;mm&rbrace;.nc
done # end loop over mth

Method 2: Create dpm by aggregating small files
for mth in &lbrace;1..12&rbrace;; do
  mm=`printf &quot;%02d&quot; $&lbrace;mth&rbrace;`
  ncap2 -O -v -s &quot;dpm=$&lbrace;dpm[$&lbrace;mth&rbrace;]&rbrace;&quot; ~/nco/data/in.nc \
    $&lbrace;drc_out&rbrace;/foo_$&lbrace;mm&rbrace;.nc
done # end loop over mth
ncecat -O -D 1 -p $&lbrace;drc_out&rbrace; -n 12,2,2 foo_$&lbrace;mm&rbrace;.nc foo.nc
ncrename -O -D 1 -d record,time $&lbrace;drc_out&rbrace;/foo.nc
ncatted -O -h \
  -a long_name,dpm,o,c,&quot;Days per month&quot; \
  -a units,dpm,o,c,&quot;days&quot; \
  $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_0112.nc
ncks -A -v dpm $&lbrace;drc_out&rbrace;/foo.nc $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_0112.nc

Method 3: Create small netCDF file using ncgen
cat &gt; foo.cdl &lt;&lt; 'EOF'
netcdf foo &lbrace; 
dimensions:
	time=unlimited;
variables:
	float dpm(time);
	dpm:long_name=&quot;Days per month&quot;;
	dpm:units=&quot;days&quot;;
data:
	dpm=31,28.25,31,30,31,30,31,31,30,31,30,31;
&rbrace;
EOF
ncgen -b -o foo.nc foo.cdl
ncks -A -v dpm $&lbrace;drc_out&rbrace;/foo.nc $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_0112.nc

Another way to get correct monthly weighting is to average daily
output files, if available.  

************************************************************************
Task 3: Regional processing
************************************************************************
Let's say you are interested in examining the California region.
Hyperslab your dataset to isolate the appropriate latitude/longitudes.

ncks -O -D 1 -d lat,30.0,37.0 -d lon,240.0,270.0 \ 
    $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_0112.nc \
    $&lbrace;drc_out&rbrace;/$&lbrace;caseid&rbrace;_clm_0112_Cal.nc

The dataset is now much smaller!
To examine particular metrics.

************************************************************************
Task 4: Accessing data stored remotely
************************************************************************
OPeNDAP server examples:

UCI DAP servers:
ncks -M -p http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata in.nc
ncrcat -O -C -D 3 \
  -p http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata \
  -l /tmp in.nc in.nc ~/foo.nc

Unidata DAP servers:
ncks -M -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
ncrcat -O -C -D 3 \
  -p http://thredds-test.ucar.edu/thredds/dodsC/testdods \
  -l /tmp in.nc in.nc ~/foo.nc

NOAA DAP servers:
ncwa -O -C -a lat,lon,time -d lon,-10.,10. -d lat,-10.,10. -l /tmp -p \
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.dailyavgs/surface \
pres.sfc.1969.nc ~/foo.nc

LLNL PCMDI IPCC OPeNDAP Data Portal: 
ncks -M -p http://username:password&arobase;esgcet.llnl.gov/cgi-bin/dap-cgi.py/ipcc4/sresa1b/ncar_ccsm3_0 pcmdi.ipcc4.ncar_ccsm3_0.sresa1b.run1.atm.mo.xml

Earth System Grid (ESG): http://www.earthsystemgrid.org

caseid='b30.025.ES01' 
CCSM3.0 1% increasing CO2 run, T42_gx1v3, 200 years starting in year 400
Atmospheric post-processed data, monthly averages, e.g.,
/data/zender/tmp/b30.025.ES01.cam2.h0.TREFHT.0400-01_cat_0449-12.nc
/data/zender/tmp/b30.025.ES01.cam2.h0.TREFHT.0400-01_cat_0599-12.nc

ESG supports password-protected FTP access by registered users
NCO uses the .netrc file, if present, for password-protected FTP access 
Syntax for accessing single file is, e.g.,
ncks -O -D 3 \
  -p ftp://climate.llnl.gov/sresa1b/atm/mo/tas/ncar_ccsm3_0/run1 \
  -l /tmp tas_A1.SRESA1B_1.CCSM.atmm.2000-01_cat_2099-12.nc ~/foo.nc 

# Average surface air temperature tas for SRESA1B scenario
# This loop is illustrative and will not work until NCO correctly
# translates '*' to FTP 'mget' all remote files
for var in 'tas'; do
for scn in 'sresa1b'; do
for mdl in 'cccma_cgcm3_1 cccma_cgcm3_1_t63 cnrm_cm3 csiro_mk3_0 \
gfdl_cm2_0 gfdl_cm2_1 giss_aom giss_model_e_h giss_model_e_r \
iap_fgoals1_0_g inmcm3_0 ipsl_cm4 miroc3_2_hires miroc3_2_medres \
miub_echo_g mpi_echam5 mri_cgcm2_3_2a ncar_ccsm3_0 ncar_pcm1 \
ukmo_hadcm3 ukmo_hadgem1'; do
for run in '1'; do
        ncks -R -O -D 3 -p ftp://climate.llnl.gov/$&lbrace;scn&rbrace;/atm/mo/$&lbrace;var&rbrace;/$&lbrace;mdl&rbrace;/run$&lbrace;run&rbrace; -l $&lbrace;DATA&rbrace;/$&lbrace;scn&rbrace;/atm/mo/$&lbrace;var&rbrace;/$&lbrace;mdl&rbrace;/run$&lbrace;run&rbrace; '*' $&lbrace;scn&rbrace;_$&lbrace;mdl&rbrace;_$&lbrace;run&rbrace;_$&lbrace;var&rbrace;_$&lbrace;yyyymm&rbrace;_$&lbrace;yyyymm&rbrace;.nc
done # end loop over run
done # end loop over mdl
done # end loop over scn
done # end loop over var

cd sresa1b/atm/mo/tas/ukmo_hadcm3/run1/
ncks -H -m -v lat,lon,lat_bnds,lon_bnds -M tas_A1.nc | m
bds -x 096 -y 073 -m 33 -o $&lbrace;DATA&rbrace;/data/dst_3.75x2.5.nc # ukmo_hadcm3
ncview $&lbrace;DATA&rbrace;/data/dst_3.75x2.5.nc

# msk_rgn is California mask on ukmo_hadcm3 grid
# area is correct area weight on ukmo_hadcm3 grid
ncks -A -v area,msk_rgn $&lbrace;DATA&rbrace;/data/dst_3.75x2.5.nc \
$&lbrace;DATA&rbrace;/sresa1b/atm/mo/tas/ukmo_hadcm3/run1/area_msk_ukmo_hadcm3.nc 

Template for standardized data:
$&lbrace;scn&rbrace;_$&lbrace;mdl&rbrace;_$&lbrace;run&rbrace;_$&lbrace;var&rbrace;_$&lbrace;yyyymm&rbrace;_$&lbrace;yyyymm&rbrace;.nc

e.g., raw data
$&lbrace;DATA&rbrace;/sresa1b/atm/mo/tas/ukmo_hadcm3/run1/tas_A1.nc
becomes standardized data

Level 0: raw from IPCC site--no changes except for name 
         Make symbolic link name match raw data
Template: $&lbrace;scn&rbrace;_$&lbrace;mdl&rbrace;_$&lbrace;run&rbrace;_$&lbrace;var&rbrace;_$&lbrace;yyyymm&rbrace;_$&lbrace;yyyymm&rbrace;.nc

ln -s -f tas_A1.nc sresa1b_ukmo_hadcm3_run1_tas_200101_209911.nc
area_msk_ukmo_hadcm3.nc

Level I: Add all variables (not standardized in time)
         to file containing msk_rgn and area
Template: $&lbrace;scn&rbrace;_$&lbrace;mdl&rbrace;_$&lbrace;run&rbrace;_$&lbrace;yyyymm&rbrace;_$&lbrace;yyyymm&rbrace;.nc

/bin/cp area_msk_ukmo_hadcm3.nc sresa1b_ukmo_hadcm3_run1_200101_209911.nc
ncks -A -v tas sresa1b_ukmo_hadcm3_run1_tas_200101_209911.nc \
               sresa1b_ukmo_hadcm3_run1_200101_209911.nc
ncks -A -v pr  sresa1b_ukmo_hadcm3_run1_pr_200101_209911.nc \
               sresa1b_ukmo_hadcm3_run1_200101_209911.nc

If already have file then:
mv sresa1b_ukmo_hadcm3_run1_200101_209911.nc foo.nc
/bin/cp area_msk_ukmo_hadcm3.nc sresa1b_ukmo_hadcm3_run1_200101_209911.nc
ncks -A -v tas,pr foo.nc sresa1b_ukmo_hadcm3_run1_200101_209911.nc

Level II: Correct # years, months
Template: $&lbrace;scn&rbrace;_$&lbrace;mdl&rbrace;_$&lbrace;run&rbrace;_$&lbrace;var&rbrace;_$&lbrace;yyyymm&rbrace;_$&lbrace;yyyymm&rbrace;.nc

ncks -d time,....... file1.nc file2.nc 
ncrcat file2.nc file3.nc sresa1b_ukmo_hadcm3_run1_200001_209912.nc

Level III: Many derived products from level II, e.g., 

      A. Global mean timeseries
      ncwa -w area -a lat,lon \
           sresa1b_ukmo_hadcm3_run1_200001_209912.nc \
	   sresa1b_ukmo_hadcm3_run1_200001_209912_xy.nc

      B. Califoria average timeseries
      ncwa -m msk_rgn -w area -a lat,lon \
           sresa1b_ukmo_hadcm3_run1_200001_209912.nc \
	   sresa1b_ukmo_hadcm3_run1_200001_209912_xy_Cal.nc
</pre></example>

<html endspaces=" ">
&lt;a name=&quot;bibliography&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#bibliography &textndash;&gt;
&lt;a name=&quot;bib&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#bib &textndash;&gt;
</html>
</chapter>
<node name="mybibnode" spaces=" "><nodename>mybibnode</nodename><nodenext spaces=" ">General Index</nodenext><nodeprev spaces=" ">CCSM Example</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<chapter spaces=" "><sectiontitle>References</sectiontitle>
<itemize commandarg="bullet" endspaces=" ">
<listitem><prepend>&bullet;</prepend>
<anchor name="ZeM07">ZeM07</anchor><para>[ZeM07]
<!-- c %**else if -->
 Zender, C. S., and H. J. Mangalam (2007), Scaling Properties of Common Statistical Operators for Gridded Datasets, Int. J. High Perform. Comput. Appl., 21(4), 485-498, doi:10.1177/1094342007083802.
</para></listitem><listitem><prepend>&bullet;</prepend>
<anchor name="Zen08">Zen08</anchor><para>[Zen08]
<!-- c %**else if -->
 Zender, C. S. (2008), Analysis of Self-describing Gridded Geoscience Data with netCDF Operators (NCO), Environ. Modell. Softw., 23(10), 1338-1342, doi:10.1016/j.envsoft.2008.03.004.
</para></listitem><listitem><prepend>&bullet;</prepend>
<anchor name="WZJ07">WZJ07</anchor><para>[WZJ07]
<!-- c %**else if -->
 Wang, D. L., C. S. Zender, and S. F. Jenks (2007), DAP-enabled Server-side Data Reduction and Analysis, Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, Paper 3B.2, January 14-18, San Antonio, TX. American Meteorological Society, AMS Press, Boston, MA.
</para></listitem><listitem><prepend>&bullet;</prepend>
<anchor name="ZMW06">ZMW06</anchor><para>[ZMW06]
<!-- c %**else if -->
 Zender, C. S., H. Mangalam, and D. L. Wang (2006), Improving Scaling Properties of Common Statistical Operators for Gridded Geoscience Datasets, Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract IN53B-0827.
</para></listitem><listitem><prepend>&bullet;</prepend>
<anchor name="ZeW07">ZeW07</anchor><para>[ZeW07]
<!-- c %**else if -->
 Zender, C. S., and D. L. Wang (2007), High performance distributed data reduction and analysis with the netCDF Operators (NCO), Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, Paper 3B.4, January 14-18, San Antonio, TX. American Meteorological Society, AMS Press, Boston, MA.
</para></listitem><listitem><prepend>&bullet;</prepend>
<anchor name="WZJ06">WZJ06</anchor><para>[WZJ06]
<!-- c %**else if -->
 Wang, D. L., C. S. Zender, and S. F. Jenks (2006), Server-side netCDF Data Reduction and Analysis, Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract IN53B-0826.
</para></listitem><listitem><prepend>&bullet;</prepend>
<anchor name="WZJ073">WZJ073</anchor><para>[WZJ073]
<!-- c %**else if -->
 Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side parallel data reduction and analysis, in Advances in Grid and Pervasive Computing, Second International Conference, GPC 2007, Paris, France, May 2-4, 2007, Proceedings. IEEE Lecture Notes in Computer Science, vol. 4459, edited by C. Cerin and K.-C. Li, pp. 744-750, Springer-Verlag, Berlin/Heidelberg, doi:10.1007/978-3-540-72360-8_67.
</para></listitem><listitem><prepend>&bullet;</prepend>
<anchor name="WZJ074">WZJ074</anchor><para>[WZJ074]
<!-- c %**else if -->
 Wang, D. L., C. S. Zender and S. F. Jenks (2007), A System for Scripted Data Analysis at Remote Data Centers, Eos Trans. AGU, 88(52), Fall Meet. Suppl., Abstract IN11B-0469.
</para></listitem><listitem><prepend>&bullet;</prepend>
<anchor name="WZJ081">WZJ081</anchor><para>[WZJ081]
<!-- c %**else if -->
 Wang, D. L., C. S. Zender and S. F. Jenks (2008), Cluster Workflow Execution of Retargeted Data Analysis Scripts, Proceedings of the 8th IEEE Int&textrsquo;l Symposium on Cluster Computing and the Grid (IEEE CCGRID &textrsquo;08), pp. 449-458, Lyon, France, May 2008.
</para></listitem><listitem><prepend>&bullet;</prepend>
<anchor name="WZJ091">WZJ091</anchor><para>[WZJ091]
<!-- c %**else if -->
 Wang, D. L., C. S. Zender, and S. F. Jenks (2009), Efficient Clustered Server-side Data Analysis Workflows using SWAMP, Earth Sci. Inform., 2(3), 141-155, doi:10.1007/s12145-009-0021-z.
</para></listitem><listitem><prepend>&bullet;</prepend>
<anchor name="PFT88">PFT88</anchor><para>[PFT88]
<!-- c %**else if -->
 Press, Flannery, Teukolsky, and Vetterling (1988), Numerical Recipes in C, Cambridge Univ. Press, New York, NY.
</para></listitem></itemize>

<!-- c @node Name Index, General Index, Operators, Top -->
<!-- c @unnumbered Function and Variable Index -->

<!-- c @printindex fn -->

<html endspaces=" ">
&lt;a name=&quot;index&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#index &textndash;&gt;
&lt;a name=&quot;idx&quot;&gt;&lt;/a&gt; &lt;!&textndash; http://nco.sf.net/nco.html#idx &textndash;&gt;
</html>
</chapter>
<node name="General-Index" spaces=" "><nodename>General Index</nodename><nodenext spaces="  "></nodenext><nodeprev spaces=" ">mybibnode</nodeprev><nodeup spaces=" ">Top</nodeup></node>
<unnumbered spaces=" "><sectiontitle>General Index</sectiontitle>

<syncodeindex from="fn" to="cp" line=" fn cp"></syncodeindex>
<printindex value="cp" line=" cp"></printindex>

<!-- c TTFN (Ta ta for now) -->
</unnumbered>
<bye></bye>
</texinfo>
