1) git clone --recursive https://github.com/CoryMartin-NOAA/global-workflow.git
cd global-workflow
git checkout feature/snow-2dvar-ens
git checkout 10197d9 (this is the commit version that I used)
git submodule update --init --recursive

2) cd sorc

2a) ./build_all.sh -gu (For cycled (w/ data assimilation) use the -g option during build)

2b)./link_workflow.sh

3) Modify ci/cases/pr/C96C48_hybatmaerosnowDA.yaml
changed the app from ATMA to ATM
changed nens from 2 to 0

4) load run modeules
module use sorc/gdas.cd/modulefiles
module load GDAS/hera

5) ./coupled_snowDA_setup.sh

6) start a test run by following the instructions below:
https://global-workflow.readthedocs.io/en/latest/start.html

6a) cd EXPDIR/expName
6b) crontab expName.crontab

7) in EXPDIR/expName, modify config.base to change

(7a)export HOMEDIR="/scratch1/NCEPDEV/global/${USER}"
into
export HOMEDIR="/scratch1/NCEPDEV/da/${USER}"

(7b) # Project to use in mass store:
Change emc-global into emc-land for HPSS_PROJECT:
export HPSS_PROJECT="emc-land"

(7c) Change INTERVAL in config.base 
export INTERVAL_GFS=0 # for gdas only, 24 for both gdas and gfs, Frequency of GFS forecast

(7d) Change interval from 24 to 0 in  /scratch2/NCEPDEV/stmp3/Y
oulong.Xia/coulped_snowDA/global-workflow/ci/cases/pr/C384mx025_hybatmaerosnowDA
.yaml

(8) to rerun, need to delete expName.db and expName_lock.db files from EXPDIR/expName

10) to check if your run success, use the command below:
./rocoto_viewer.py -d EXPDIR/expName/expName.db -w EXPDIR/expName/expName.xml

(11) create restart file using UFS_UTILS for cold start
See README_notes_UFS_UTILS

(12) Use ncl scrip to replace land initials with offline spinup run
replace_soil_snow_C384_offlineSpinup_gdas.ncl in /scratch1/NCEPDEV/da/Youlong.Xia/Spinup_for_DA/replace_sfc

(13) cp repalced sfc files into ICSDIR directory
in /scratch1/NCEPDEV/da/Youlong.Xia/ICSDIR/C384mx025/gdas.20210901/00/model/atmos/input, do:

cp /scratch1/NCEPDEV/da/Youlong.Xia/ICSDIR/replace_sfc_data/gdas.20210901/sfc_data.tile* .


After the test, do continuous run:

cd EXPDIR/expName
edit expName.xml for
 <cycledef group="gdas">202109010600 202109011200 06:00:00</cycledef>

into 
 <cycledef group="gdas">202109010600 202112311800 06:00:00</cycledef>


rocotoboot -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname


in EXPDIR/snowvar1

rocotoboot -d snowvar1.db -w snowvar1.xml -c 202109011800 -t gdas_prep


This command can be used for job is running or the run is finished.

(14) Data in HPSS:

/NCEPDEV/emc-land/1year/Youlong.Xia/HERA/scratch/snowvar1

(15) When machine is maintained, generally gdas_arch will fail due to HPSS's maintaince, 
and you have to resubmit each of failed tasks independently in EXPDIR/snowvar1:

rocotorewind -d snowvar1.db -w snowvar1.xml -c 202110280000 -t gdas_arch
rocotorewind -d snowvar1.db -w snowvar1.xml -c 202110280600 -t gdas_arch
rocotorewind -d snowvar1.db -w snowvar1.xml -c 202110281200 -t gdas_arch

And then

The cron job just automatically launched the 18z cycle since it saw the 00z failed task is successfully fixed.
