USF Home
Administrators
Hardware Status



Home
Getting Started
Support and Contacts
Software Portal
Documentation
Tutorial / Links
C Programming Linux MPI Programming
Get Account
Request Licensed Software
 

CMAQ v.4.6

  1. CMAQ v.4.6
    1. Description
    2. Version
    3. Plaforms
    4. Download
    5. Preliminaries
      1. 1. Edit Environment Settings
      2. 2. Create Directory Layout
      3. 3. Compiler Settings
      4. 4. Extract the packages
      5. 5. Errata and Bug Fixes
    6. Building Supporting Libraries and Tools
      1. 1. pario
      2. 2. build
      3. 3. stenex
    7. Building the Model Components
      1. 1. jproc
      2. 2. icon
      3. 3. bcon
      4. 4. cctm
    8. Tutorial: Day 1, 32 km Run
    9. Tutorial: Day 2, 32 km Run
    10. Tutorial: Nested 8k Run
      1. 1. icon
      2. 2. bcon
      3. 3. cctm
    11. Other Modules
      1. 1. pdm
      2. 4. procan

Last Modified: 07/13 14:12

Description

CMAQ is a community state-of-the-science modeling system for diagnosis and prediction of air pollution.

  • CMAQ version 4.6 (CMAQv4.6) is released to the user community accompanied by example scripts that invoke a specific configuration of the model. This configuration has been used by U.S. EPA in operational evaluation studies prior to release of the model, and the results of these evaluations accompany this release. There are other features and options within the CMAQ modeling system within this release, beyond this specific model configuration, that have not yet been fully evaluated and documented, that are also available to users.

From:  http://www.cmascenter.org/help/model_docs/cmaq/4.6/README.txt

Version

4.6

Plaforms

circe cluster

Download

You'll need to download the appropriate packages from the Community Modeling and Analysis System (CMAS) Center. You'll need to go to the Downloads page and create an account. Follow the directions on the CMAS web site for creating this account. Once it is created, you can log into the downloads section and make the following selections:

  1. Product: CMAQ v.4.6
  2. Type of Computer: Linux-PC
  3. Compiler: Linux Portland Group Fortran 90

Proceed to download all of the package files listed. This should include:

  1. M3DATA.CMAQv4.6.tar.gz
  2. M3DOCS.CMAQv4.6.tar.gz
  3. M3SCRIPTS.CMAQv4.6.tar.gz
  4. M3DATA_REF.CMAQv4.6.tar.gz
  5. M3MODELS.CMAQv4.6.tar.gz

Make the directory CMAQ/4.6/distfiles in your home directory on Circe like so: 

[user@host ~]$ mkdir -p ~/CMAQ/4.6/distfiles

Copy the package files that you downloaded into this new directory.

Preliminaries

We'll need to extract these packages into their appropriate directories and make a few directories and symbolic links to include system libraries. These instructions assume the following about your system:

  1. That you are using the Portland Group compilers and that the PGI executables pgcc and pgf90 are in your path
  2. That the MPI executable mpif90 is in your path
  3. That you are using Sun's GridEngine for resource management and job scheduling along with
    1. Our version of sge_mpirun which simplifies the calling of mpirun
    2. That your GridEngine installation is configured correctly to run parallel applications with MPI

If you are running on Circe, you do not have to worry about these conditions as the system is already configured accordingly.

1. Edit Environment Settings

We'll need to modify our default environment settings. You'll need to make the changes for C-Shell regardless of whether or not you use a different shell since the submit scripts do use C-Shell. You can add the following lines to your ~/.cshrc file: 

setenv M3HOME ~/CMAQ/4.6
setenv M3MODEL ${M3HOME}/models
setenv M3LIB ${M3HOME}/lib
setenv M3DATA ${M3HOME}/data

If you're a bash user, you can add the following to your ~/.bashrc file: 

export M3HOME=~/CMAQ/4.6
export M3MODEL=$M3HOME/models
export M3LIB=$M3HOME/lib
export M3DATA=$M3HOME/data

2. Create Directory Layout

You'll want to log out and log back into the system to see these changes (or, for the shell savvy, just source your ~/.(csh|bash)rc files). Now we can make some directories and link in our system libraries: 

[user@host ~]$ mkdir -p $M3LIB/build $M3LIB/netCDF $M3LIB/pario $M3LIB/stenex
[user@host ~]$ ln -s /opt/apps/ioapi $M3LIB/ioapi_3
[user@host ~]$ ln -s /opt/apps/netcdf/lib $M3LIB/netCDF/Linux

3. Compiler Settings

Load the following environment modules 

[user@host ~]$ module add compilers/pgi/7.2-5.x86_64 openmpi/1.3.0-x86_64-pgi-7.2-5
[user@host ~]$ module initadd compilers/pgi/7.2-5.x86_64 openmpi/1.3.0-x86_64-pgi-7.2-5

Do this whenever you build or re-build parts of the CMAQ package.

4. Extract the packages

Extract the packages into your $M3HOME directory like so: 

[user@host ~]$ cd $M3HOME
[user@host CMAQ]$ tar xvzf distfiles/M3DATA.CMAQv4.6.tar.gz
[user@host CMAQ]$ tar xvzf distfiles/M3DATA_REF.CMAQv4.6.tar.gz

[user@host CMAQ]$ tar xvzf distfiles/M3DOCS.CMAQv4.6.tar.gz
[user@host CMAQ]$ tar xvzf distfiles/M3MODELS.CMAQv4.6.tar.gz
[user@host CMAQ]$ tar xvzf distfiles/M3SCRIPTS.CMAQv4.6.tar.gz

5. Errata and Bug Fixes

We'll want to increase the argument buffer size so your jobs do not fail when running in the queue. In the file $M3MODEL/CCTM/src/par/par/distr_env.c,v, you'll want to make two changes using your favorite text editor:

  1. On lines 74-75, change the size of TEMP_BUF_SIZE and CURR_STR_SIZE to 
    #define TEMP_BUF_SIZE   20480
#define CURR_STR_SIZE   20480
  2. Save the file. You'll have to force the write to go through since the file is read-only (you can also temporarily modify the permissions with chmod).

Building Supporting Libraries and Tools

This section describes the process for building the supporting libraries, pario, build, and stenex.

1. pario

  1. Change directory to $M3HOME/scripts/pario
  2. Edit the file bldit.pario.pgf so that
    1. Line 41 reads 
      set MPI = .
    2. Line 45 reads 
      set FC = `which mpif90`
  3. Run the build script by executing 
    [user@host pario]$ ./bldit.pario.pgf &> bldit.pario.pgf.out
  4. Check the file bldit.pario.pgf.out to ensure that there are no errors and that the library compiled correctly. You should see the following files in $M3LIB/pario/Linux:
    1. alloc_data_module.mod
    2. libpario.a
    3. pinterpb_module.mod
    4. piomaps_module.mod

2. build

  1. Change directory to $M3HOME/scripts/build
  2. Run the build script by executing 
    [user@host build]$ ./bldit.m3bld &> bldit.m3bld.out
  3. Check the file bldit.m3bld.out to ensure that there are no errors and that the executable compiled correctly. You should see the executable m3bld in the directory $M3LIB/build/Linux

3. stenex

  1. Change directory to $M3HOME/scripts/stenex
  2. Edit the file bldit.se_noop.pgf so that
    1. Line 43 reads 
      set FC = `which pgf90`
  3. Edit the file bldit.se.pgf so that
    1. Line 43 reads 
      set INCL = .
    2. Line 46 reads 
      set FC = `which mpif90`
  4. Run the build scripts by executing 
    [user@host stenex]$ ./bldit.se_noop.pgf &> bldit.se_noop.pgf.out
[user@host stenex]$ ./bldit.se.pgf &> bldit.se.pgf.out
  5. Check the files bldit.se_noop.pgf.out and bldit.se.pgf.out to ensure that there are no errors and that the library compiled correctly. You should see the following files in the $M3LIB/stenex/Linux directory:
    1. libsef90_noop.a
    2. libse_snl.a

At this point, you have completed building the supporting libraries. These should not need to be recompiled in the future.

Building the Model Components

These particular sections may need to be recompiled from time to time, depending on parameter changes. For the sake of simplicity, we will go ahead and follow the tutorial instructions while building these modules. These modules include bcon, icon, jproc, and cctm.

1. jproc

  1. Change directory to $M3HOME/scripts/jproc
  2. Edit the file bldit.jproc.pgf so that
    1. Line 71 reads 
      set FC = `which pgf90`
  3. Run the build script by executing 
    [user@host jproc]$ ./bldit.jproc.pgf &> bldit.jproc.pgf.out
  4. Check the file bldit.jproc.pgf.out to ensure that there are no errors and that the executable compiled correctly. You should see the file JPROC_d1a in the current directory.
  5. Run the JPROC binary by running 
    [user@host jproc]$ ./run.jproc
  6. Data should be generated in the directory $M3DATA/jproc

2. icon

  1. Change directory to $M3HOME/scripts/icon
  2. Edit the file bldit.icon.pgf so that
    1. Line 89 reads 
      set FC = `which pgf90`
  3. Run the build script by executing 
    user@host icon]$ ./bldit.icon.pgf &> bldit.icon.pgf.out
  4. Check the file bldit.icon.pgf.out to ensure that there are no errors and that the executable compiled correctly. You should see the file ICON_e1a in the current directory.
  5. Run the ICON binary by running 
    [user@host icon]$ ./run.icon
  6. Data should be generated in the directory $M3DATA/icon.

3. bcon

a. Change directory to $M3HOME/scripts/bcon

  1. Edit the file bldit.bcon.pgf so that
    1. Line 84 reads 
      set FC = `which pgf90`
  2. Run the build script by executing 
    user@host bcon]$ ./bldit.bcon.pgf &> bldit.bcon.pgf.out
  3. Check the file bldit.bcon.pgf.out to ensure that there are no errors and that the executable compiled correctly. You should see the file BCON_e1a in the current directory.
  4. Run the BCON binary by running 
    [user@host bcon]$ ./run.bcon
  5. Data should be generated in the directory $M3DATA/bcon.

4. cctm

  1. Change directory to $M3HOME/scripts/cctm
  2. Patch the file bldit.cctm.pgf by doing the following (patch files can be viewed at the bottom of this document): 
    [user@host cctm]$ wget --no-check-certificate \
     https://rc.usf.edu/trac/doc/attachment/wiki/CMAQ-4.6/bldit.cctm.pgf.patch?format=raw \
     -O bldit.cctm.pgf.patch
[user@host cctm]$ cp bldit.cctm.pgf bldit.cctm.pgf.sge
[user@host cctm]$ patch -p0 < bldit.cctm.pgf.patch

  1. Run the build script by executing 
    [user@host cctm]$ ./bldit.cctm.pgf.sge &> bldit.cctm.pgf.sge.out
  2. Ensure that the binary CCTM_e3a was created in the current directory.
  3. Copy the file run.cctm to run.cctm.sge
  4. Patch the file run.cctm.sge by doing the following: 
    [user@host cctm]$ wget --no-check-certificate \
     https://rc.usf.edu/trac/doc/attachment/wiki/CMAQ-4.6/run.cctm.sge.patch?format=raw \
     -O run.cctm.sge.patch
[user@host cctm]$ cp run.cctm run.cctm.sge
[user@host cctm]$ patch -p0 < run.cctm.sge.patch

  1. Remove the file machines8. The scheduler creates this host list automatically and your run.cctm.par script is correctly modified to handle this change.

Tutorial: Day 1, 32 km Run

At this point, cctm should be ready to run the included 32 km tutorial. You can follow these instructions for running it if you wish to test your installation or get a feel for the program.

  1. Rename and submit the modified script to the queue by issuing 
    [user@host cctm]$ cp run.cctm.sge run.cctm.sge.tut_day1
[user@host cctm]$ qsub run.cctm.sge.tut_day1
  2. Verify that the job completed by checking the contents of output.$JOB_ID. Refer to the GridEngine User's Guide for getting the $JOB_ID number and checking the status of your job.
  • Note: Data from cctm will be output to /scratch/$USER/cctm rather than $M3DATA/cctm. This is because of a long-standing bug with automounted NFS volumes (like your home directory). The /scratch file system appears to handle the cctm I/O patterns correctly.

Tutorial: Day 2, 32 km Run

(This is directly from the CMAQ Tutorial documentation at  http://www.cmascenter.org/help/model_docs/cmaq/4.6/TUTORIAL_PROCEDURE.txt) You will need to make a few changes to the run script for the second day's run. Assuming that you have used the run script as supplied for the first day's run, the following is a "patch" style difference in run.cctm for the second day at 32 km.

  • Note: The line numbers in this patch format will not match with your run.cctm.sge file! Use the line's context to make the correct changes!
  • WARNING: Pay very close attention to the contents of this patch! There are subtle changes that you might miss which will cause your run to fail! I would make this easier on you, but when you get to know this program a little more, you'll see its for your own good! You must get used to using patches.
  • Remember: Left arrow at the beginning of the line means "before the change", right arrow means "after the change"!

Copy run.cctm.sge.tut_day1 to run.cctm.sge.tut_day2 and modify the new file as follows: 

7c7
< # Usage: run.cctm >&! cctm_e2a.log &                                  #
---
> # Usage: run.cctm >&! cctm_e2b.log &                                  #
22c22
<  set APPL     = e2a
---
>  set APPL     = e2b
42c42
<  set STDATE   = 1999183       # beginning date
---
>  set STDATE   = 1999184       # beginning date
106,107c106,107
<  set EMISfile  = e_32km_cb4.19990702.ncf
< #set EMISfile  = e_32km_cb4.19990703.ncf
---
> #set EMISfile  = e_32km_cb4.19990702.ncf
>  set EMISfile  = e_32km_cb4.19990703.ncf
114,117c114,117
< #set GC_ICpath = $OUTDIR
< #set GC_ICfile = CCTM_d1bCONC.d1b
<  set GC_ICpath = $M3DATA/icon
<  set GC_ICfile = ICON_cb4_M_32_99TUT02_profile
---
>  set GC_ICpath = $OUTDIR
>  set GC_ICfile = CCTM_e2aCONC.e2a
> #set GC_ICpath = $M3DATA/icon
> #set GC_ICfile = ICON_cb4_M_32_99TUT02_profile

Run the next day's run by submitting the script, run.cctm.sge.tut_day2, to the queue with qsub.

Tutorial: Nested 8k Run

After running both days of the 32 km, you are ready to run the nested 8 km. Again, this section is directly from  http://www.cmascenter.org/help/model_docs/cmaq/4.6/TUTORIAL_PROCEDURE.txt

  • Note: The same rules of thumb from the above patch file still apply!

1. icon

You will need to recompile icon to now use the first 32 km concentration file as input. The following is a "patch" style difference in the bldit.icon.pgf script: 

42c42
<  set APPL  = e1a
---
>  set APPL  = e3a
59,60c59,60
< #set ModInpt   = ( module m3conc            $Revision; )
<  set ModInpt   = ( module profile           $Revision; )
---
>  set ModInpt   = ( module m3conc            $Revision; )
> #set ModInpt   = ( module profile           $Revision; )
62c62
< #set ModMech   = ( module mc_noop           $Revision; )
---
>  set ModMech   = ( module mc_noop           $Revision; )
64c64
<  set ModMech   = ( module radm2_to_cb4      $Revision; )
---
> #set ModMech   = ( module radm2_to_cb4      $Revision; )

Likewise you will need to modify run.icon, both to point to the new executable and to use the m3conc switch rather than "profile." The following is a patch style difference in run.icon: 

7c7
< # Usage: run.icon >&! icon_e2a.log &                                  #
---
> # Usage: run.icon >&! icon_e3a.log &                                  #
23c23
<  set CFG      = e1a
---
>  set CFG      = e3a
35c35
<  setenv GRID_NAME M_32_99TUT02
---
>  setenv GRID_NAME M_08_99TUT02
39c39
<  setenv LAYER_FILE $M3DATA/mcip3/M_32_99NASH/METCRO3D_benchmark2005
---
>  setenv LAYER_FILE $M3DATA/mcip3/M_08_99NASH/METCRO3D_benchmark2005
54,55c54,55
<  set IC = profile      # use default profile file
< #set IC = m3conc       # use CMAQ CTM concentration files (nested runs)
---
> #set IC = profile      # use default profile file
>  set IC = m3conc       # use CMAQ CTM concentration files (nested runs)
115c115
<     setenv CTM_CONC_1 $M3DATA/cctm/CCTM_d1bCONC.d1b
---
>     setenv CTM_CONC_1 $M3DATA/cctm/CCTM_e2aCONC.e2a

2. bcon

You will also need to recompile bcon to now use the 32 km concentration file as input. The following is a "patch" style difference in the bldit.bcon.pgf script: 

42c42
<  set APPL  = e1a
---
>  set APPL  = e3a
57,58c57,58
< #set ModInpt   = ( module m3conc            $Revision; )
<  set ModInpt   = ( module profile           $Revision; )
---
>  set ModInpt   = ( module m3conc            $Revision; )
> #set ModInpt   = ( module profile           $Revision; )
60c60
< #set ModMech   = ( module mc_noop           $Revision; )
---
>  set ModMech   = ( module mc_noop           $Revision; )
62c62
<  set ModMech   = ( module radm2_to_cb4      $Revision; )
---
> #set ModMech   = ( module radm2_to_cb4      $Revision; )

The following are "patch" style differences in run.bcon, which must now be executed for every day of the nested, 8km runs. The first difference shows how to change the script from running the profile data to the first day of the nested data: 

7c7
< # Usage: run.bcon >&! bcon_e2a.log &                                  #
---
> # Usage: run.bcon >&! bcon_e3a.log &                                  #
23c23
<  set CFG      = e1a
---
>  set CFG      = e3a
34c34
<  setenv GRID_NAME M_32_99TUT02
---
>  setenv GRID_NAME M_08_99TUT02
38c38
<  setenv LAYER_FILE $M3DATA/mcip3/M_32_99NASH/METCRO3D_benchmark2005
---
>  setenv LAYER_FILE $M3DATA/mcip3/M_08_99NASH/METCRO3D_benchmark2005
56,57c56,57
<  set BC = profile      # use default profile file
< #set BC = m3conc       # use CMAQ CTM concentration files (nested runs)
---
> #set BC = profile      # use default profile file
>  set BC = m3conc       # use CMAQ CTM concentration files (nested runs)
119c119
<     setenv CTM_CONC_1 $M3DATA/cctm/CCTM_d1bCONC.d1b
---
>     setenv CTM_CONC_1 $M3DATA/cctm/CCTM_e2aCONC.e2a

The second difference shows how to change from the first day of the nest to running the second day: 

7c7
< # Usage: run.bcon >&! bcon_e3a.log &                                  #
---
> # Usage: run.bcon >&! bcon_e3b.log &                                  #
64,65c64,65
<     set DATE = 1999183  # July 02, 1999
< #   set DATE = 1999184  # July 03, 1999
---
> #   set DATE = 1999183  # July 02, 1999
>     set DATE = 1999184  # July 03, 1999
119c119
<     setenv CTM_CONC_1 $M3DATA/cctm/CCTM_e2aCONC.e2a
---
>     setenv CTM_CONC_1 $M3DATA/cctm/CCTM_e2aCONC.e2b

3. cctm

Just modify the run script. The first difference shows how to modify the run script run.cctm.sge.tut_day2 from running the second day of the 32 km to the first day of the 8 km nest. Copy run.cctm.sge.tut_day2 to run.cctm.sge.tut_8k_day1 and modify the new file as follows: 

7c7
< # Usage: run.cctm >&! cctm_e2b.log &                                  #
---
> # Usage: run.cctm >&! cctm_e3a.log &                                  #
22c22
<  set APPL     = e2b
---
>  set APPL     = e3a
42c42
<  set STDATE   = 1999184       # beginning date
---
>  set STDATE   = 1999183       # beginning date
92c92
<  setenv GRID_NAME M_32_99TUT02
---
>  setenv GRID_NAME M_08_99TUT02
103c103
<  set OCEANfile = SSMASK_tut32
---
>  set OCEANfile = SSMASK_tut08
107,108c107,108
<  set EMISfile  = e_32km_cb4.19990703.ncf
< #set EMISfile  = e_08km_cb4.19990702.ncf
---
> #set EMISfile  = e_32km_cb4.19990703.ncf
>  set EMISfile  = e_08km_cb4.19990702.ncf
114,117c114,117
<  set GC_ICpath = $OUTDIR
<  set GC_ICfile = CCTM_e2aCONC.e2a
< #set GC_ICpath = $M3DATA/icon
< #set GC_ICfile = ICON_cb4_M_32_99TUT02_profile
---
> #set GC_ICpath = $OUTDIR
> #set GC_ICfile = CCTM_e2aCONC.e2a
>  set GC_ICpath = $M3DATA/icon
>  set GC_ICfile = ICON_cb4_M_08_99TUT02_1999183
120c120
<  set GC_BCfile = BCON_cb4_M_32_99TUT02_profile
---
>  set GC_BCfile = BCON_cb4_M_08_99TUT02_1999183
122c122
<  set METpath   = $M3DATA/mcip3/M_32_99NASH
---
>  set METpath   = $M3DATA/mcip3/M_08_99NASH

You should submit this new job to the queue to run. Finally, the differences from the first to the second day of the 8 km runs. Copy run.cctm.sge.tut_8k_day1 to `run.cctm.sge.tut_8k_day2: 

7c7
< # Usage: run.cctm >&! cctm_e3a.log &                                  #
---
> # Usage: run.cctm >&! cctm_e3b.log &                                  #
22c22
<  set APPL     = e3a
---
>  set APPL     = e3b
42c42
<  set STDATE   = 1999183       # beginning date
---
>  set STDATE   = 1999184       # beginning date
108,109c108,109
<  set EMISfile  = e_08km_cb4.19990702.ncf
< #set EMISfile  = e_08km_cb4.19990703.ncf
---
> #set EMISfile  = e_08km_cb4.19990702.ncf
>  set EMISfile  = e_08km_cb4.19990703.ncf
114,117c114,117
< #set GC_ICpath = $OUTDIR
< #set GC_ICfile = CCTM_e2aCONC.e2a
<  set GC_ICpath = $M3DATA/icon
<  set GC_ICfile = ICON_cb4_M_08_99TUT02_1999183
---
>  set GC_ICpath = $OUTDIR
>  set GC_ICfile = CCTM_e2aCONC.e3a
> #set GC_ICpath = $M3DATA/icon
> #set GC_ICfile = ICON_cb4_M_08_99TUT02_1999183
120c120
<  set GC_BCfile = BCON_cb4_M_08_99TUT02_1999183
---
>  set GC_BCfile = BCON_cb4_M_08_99TUT02_1999184

You can run the second day of the 8 km run by submitting run.cctm.sge.tut_8k_day2 to the queue.

Other Modules

Two other modules, pdm and procan are included with the CMAQ distribution. Follow these instructions to build them if they are needed:

1. pdm

  1. Change directory to $M3HOME/scripts/pdm
  2. Edit the file bldit.pdm.pgf so that
    1. Line 84 reads 
      set FC = `which pgf90`
  3. Run the build script by executing 
    [user@host pdm]$ ./bldit.pdm.pgf &> bldit.pdm.pgf.out
  4. Check the file bldit.pdm.pgf.out to ensure that there are no errors and that the executable compiled correctly. You should see the file PDM_e2a in the current directory.

4. procan

  1. Change directory to $M3HOME/scripts/procan
  2. Edit the file bldit.pacp.pgf so that
    1. Line 73 reads 
      set FC = `which pgf90`
  3. Run the build script by executing 
    [user@host pdm]$ ./bldit.pacp.pgf &> bldit.pacp.pgf.out
  4. Check the file bldit.pacp.pgf.out to ensure that there are no errors and that the executable compiled correctly. You should see the file PACP_e2a in the current directory.

Attachments