single_column logic in buildnml is incorrect

[billsacks]

I noticed that the test SMS_Ln9_P1.T42_T42.2000_DATM%QIA_SLND_SICE_DOCN%DOM_SROF_SGLC_SWAV.derecho_intel.datm-scam is incorrectly (I think) keeping ATM_NX and ATM_NY set to their global (full domain) values rather than setting them to 1. That appears to be due to a bug in CDEPS (ESCOMP/CDEPS#419), but I think there's a similar bug in CMEPS:

The logic here for single_column appears incorrect:

CMEPS/cime_config/buildnml

Lines 96 to 106 in e976799

	# determine if need to set atm_domainfile
	scol_lon = float(case.get_value("PTS_LON"))
	scol_lat = float(case.get_value("PTS_LAT"))
	if (
	scol_lon > -999.0
	and scol_lat > -999.0
	and case.get_value("ATM_DOMAIN_FILE") != "UNSET"
	):
	config["single_column"] = "true"
	else:
	config["single_column"] = "false"

Specifically: there is no ATM_DOMAIN_FILE variable. Based on #164, I have a feeling this is supposed to refer to PTS_DOMAINFILE, but I'm not sure about this. @mvertens agrees that it looks like this is an issue.

I don't have enough familiarity with the subtleties of single-column mode to know what's right here, so I'm hoping that someone with more familiarity can look at this. This should include:

• Checking whether single-column mode is currently working correctly despite this issue
• Coming up with a fix that correctly determines single_column mode and differentiates it from a single point run (see notes in new CESM single/column single/point functionality #164)
• Confirming that the fixed code works correctly

[billsacks]

Note that, in contrast to the CDEPS logic, the logic in CMEPS does not have the initial if case.get_value('ATM_DOMAIN_FILE') check. So, whereas in CDEPS it seems like single_column is incorrectly being set to False when it should be True, here in CMEPS it seems like single_column is being set to True whenever PTS_LON and PTS_LAT are greater than -999. I think this may still be incorrect, but in a different way: based on the comments in #164 it sounds like there should be some situations where single_column is set to False despite PTS_LON and PTS_LAT being greater than -999.

[ekluzek]

Pinging @jtruesdal as he will want to know about this and can weigh in here.

Note that, in contrast to the CDEPS logic, the logic in CMEPS does not have the initial if case.get_value('ATM_DOMAIN_FILE') check. So, whereas in CDEPS it seems like single_column is incorrectly being set to False when it should be True, here in CMEPS it seems like single_column is being set to True whenever PTS_LON and PTS_LAT are greater than -999. I think this may still be incorrect, but in a different way: based on the comments in #164 it sounds like there should be some situations where single_column is set to False despite PTS_LON and PTS_LAT being greater than -999.

The way I'm reading the code and my understanding of it, is that for PTS_MODE to work you HAVE to set PTS_LON/PTS_LAT/PTS_DOMAINFILE to something sensible. PTS_DOMAINFILE will be set for you automatically from ccs_config for the: f09, f19, f45, T42, and ne3 grids. There's also PTS_MODE which can be set to TRUE, but I didn't find any logic that actually uses it anymore. So presumably it could be removed. Although maybe how it should be set is for the three above to be checked and PTS_MODE set to TRUE only if they all are. Then maybe the logic should be shifted to check it rather than the three of them. The logic is a little flakey right now since it doesn't universally check all three.

The mode where PTS_LON/PTS_LAT is set and single_column is FALSE are the single point cases such as NEON or PLUMBER2 sites which set those, but do NOT set PTS_DOMAINFILE and so are operating on input data that only contains a single point. So we do need to allow that mode, but really only PTS_DOMAINFILE needs to be checked to determine if it's in single_column mode.

Hopefully, that resolves everything you need? @jtruesdal has the most experience in running SCAM. I've been involved in all the work on the CLM side of it. We have it working as an option, but I don't get the impression that many people use it for "I compsets". But, it's a configuration we need to test to make sure it'll continue working for SCAM. So I think @jtruesdal and I are the right ones to ask about it.

[jtruesdal]

I'll take a look at this next week.

[billsacks]

Thanks a lot for your detailed reply, @ekluzek ! This is very helpful. My impression from this is that the buildnml code should be checking whether PTS_DOMAINFILE is set, in order to distinguish between single point cases and single-column mode. So I think your comment supports my initial guess that the reference to ATM_DOMAIN_FILE should be changed to PTS_DOMAINFILE.

However, my concern is that things may currently be working right for wrong / compensating reasons and that fixing the logic here might actually break things if they are currently giving the right final behavior.

So I think the first step is determining whether things are currently working correctly for both single-column mode and single-point mode, with a focus on variables that trigger off of single_column in the CDEPS and CMEPS namelist_definition files, and maybe in other components as well. This is the piece that's hard for me because I don't know what to look at to confirm this.

Then we should change references to ATM_DOMAIN_FILE to PTS_DOMAINFILE... I'm not sure if it's important to maintain the initial if case.get_value('PTS_DOMAINFILE') (similar to the initial if case.get_value('ATM_DOMAIN_FILE') that is currently in CDEPS but not in CMEPS), but if we do, we should probably put it in both CMEPS and CDEPS for consistency.

And then we'll want to redo the checks of whether things are working correctly for both single-column mode and single-point mode.

[mvertens]

@billsacks @ekluzek - thanks for this analysis. I'd like to add a few things:

• in cmeps - the nuopc.config variable mesh_mash is being set to $MASK_MESH rather than to null - and this is a bug
• the only routines that query this nuopc attribute are:
  • cice/src/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90
  • clm/src/cpl/nuopc/lnd_comp_nuopc.F90

CLM queries mesh_mask this OUTSIDE of a single_column if-block - so that fact that it is set incorrectly for single column is irrelevant

    if (scol_lon > scol_spval .and. scol_lat > scol_spval) then
       call lnd_set_decomp_and_domain_for_single_column(scol_lon, scol_lat, scol_mask, scol_frac)
       if (ChkErr(rc,__LINE__,u_FILE_u)) return
    else
       call NUOPC_CompAttributeGet(gcomp, name='mesh_lnd', value=model_meshfile, rc=rc)
       if (ChkErr(rc,__LINE__,u_FILE_u)) return
       call NUOPC_CompAttributeGet(gcomp, name='mesh_mask', value=meshfile_mask, rc=rc)
    .....

CICE also question mesh_mash OUTSIDE of a single_column if-block

    if (single_column .and. scol_valid) then
       call ice_mesh_init_tlon_tlat_area_hm()
    else
       ! Determine mesh input file
       call NUOPC_CompAttributeGet(gcomp, name='mesh_ice', value=ice_meshfile, rc=rc)
       if (ChkErr(rc,__LINE__,u_FILE_u)) return

       ! Determine mask input file
       call NUOPC_CompAttributeGet(gcomp, name='mesh_mask', value=cvalue, isPresent=isPresent, rc=rc)
       if (ChkErr(rc,__LINE__,u_FILE_u)) return

So again it does not matter if it is set incorrectly for single column.

I think this single_column logic in namelist_definition_drv.xml should remove the null query and buildnml should also not worry about setting it. I'm happy to have a call about this.

To add a few details as to where single_column is determined:
In CAM we have the following in atm_comp_nuopc.F90

! For single column mode in cam need to have a valid single_column_lnd_domainfile for the mask

    call NUOPC_CompAttributeGet(gcomp, name='single_column_lnd_domainfile', value=single_column_lnd_domainfile, rc=rc)
    if (ChkErr(rc,__LINE__,u_FILE_u)) return
    if (scol_lon > scol_spval .and. scol_lat > scol_spval) then
       if (trim(single_column_lnd_domainfile) /= 'UNSET') then
          single_column = .true.
       else
          call shr_sys_abort('single_column_lnd_domainfile cannot be null for single column mode')
       end if
    else
       single_column = .false.
    end if

So if you have single_column_lnd_domainfile set and PTS_LON and PTS_LAT are valid - then CAM sets single_column to .true.

The problem also seems to occur in CDEPS. As an example - if you look at datm/cime_config_buildnml

   if case.get_value('PTS_LON'):
        scol_lon = float(case.get_value('PTS_LON'))
    else:
        scol_lon = -999.
    if case.get_value('PTS_LAT'):
        scol_lat = float(case.get_value('PTS_LAT'))
    else:
        scol_lat = -999.
    if case.get_value('ATM_DOMAIN_FILE'):
        if scol_lon > -999. and scol_lat > -999. and case.get_value("ATM_DOMAIN_FILE") != "UNSET":
            config['single_column'] = 'true'
        else:
            config['single_column'] = 'false'
    ....

In namelist_definition_datm.xml - this sets the model_meshfile to be $ATM_DOMAIN_MESH rather than null.
However, this seems to be ignored in dshr_mod:dshr_mesh_init

   if (scol_lon > scol_spval .and. scol_lat > scol_spval) then
       ! This is simply a single point run
       call dshr_mesh_create_scol(gcomp, compname, scol_lon, scol_lat, model_mesh, model_mask, model_frac, rc=rc)
       if (ChkErr(rc,__LINE__,u_FILE_u)) return
    else
       ! check that model_meshfile and model_maskfile exists
       if (my_task == main_task) then
          inquire(file=trim(model_meshfile), exist=exists)
          .......
       endif
   endif

So with the data components - again if model_meshfile is set incorrectly to not be null - this is ignored if scol_lon and scol_lat are valid.

[billsacks]

in cmeps - the nuopc.config variable mesh_mash is being set to $MASK_MESH rather than to null - and this is a bug

My sense is actually the opposite for CMEPS: In contrast to CDEPS, I think in CMEPS single_column is getting set to True in some situations where it should be False (namely, single-point mode), and so I think the issue is that sometimes mesh_mask is set to null when it shouldn't be... but I'm not positive about that.

I haven't worked through what the implications of this might be.

[billsacks]

Based on discussion with @mvertens - it looks like mesh_mask is irrelevant if scol_lon and scol_lat are set - so this issue in CMEPS shouldn't impact anything. So I agree with @mvertens 's recommendation that we remove the single_column logic from CMEPS and always set mesh_mask.

Also, it seems weird that you'd set mesh_mask to null; why not set the mesh to null? Overall, we lean towards just removing this single-column logic from CMEPS since it doesn't seem important.

For CDEPS, we'll fix the logic to reference PTS_DOMAINFILE instead of ATM_DOMAIN_FILE so that variables still differ based on whether we're using single-column mode.

[ekluzek]

OK, it seems like there is a plan for what to do here. When the PR is made for this we should just make sure that the CTSM single point tests work and give the same answers. And the CAM SCAM tests as well.

Over all testlists there's a 43 tests for single-point/PTS_MODE cases for CTSM. You probably wouldn't need to run all of those, so here's a subset of 23 of them. The SUBSET data tests at the end require using the ctsm_pylib conda environment.

ERS_D_Ld5_Mmpi-serial.1x1_mexicocityMEX.I1PtClm60SpRs.derecho_gnu.clm-CLM1PTStartDate
ERS_D_Mmpi-serial_Ld5.1x1_brazil.I2000Clm50FatesRs.derecho_gnu.clm-FatesCold
ERS_D_Mmpi-serial_Ld5.1x1_brazil.I2000Clm50FatesRs.izumi_nag.clm-FatesCold
ERS_Ly3_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropQianRs.derecho_gnu.clm-cropMonthOutput
ERS_Ly5_Mmpi-serial.1x1_smallvilleIA.I1850Clm50BgcCrop.derecho_gnu.clm-ciso_monthly
ERS_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropQianRs.derecho_intel.clm-cropMonthOutput
SMS_D_Ld1_Mmpi-serial.ne3_ne3_mg37.I2000Clm50SpRs.derecho_gnu.clm-ptsRLA
SMS_D_Ld1_Mmpi-serial.ne3_ne3_mg37.I2000Clm50SpRs.derecho_gnu.clm-ptsROA
SMS_D_Ld1_Mmpi-serial.ne3_ne3_mg37.I2000Clm50SpRs.derecho_intel.clm-ptsRLA
SMS_D_Ld1_Mmpi-serial.ne3_ne3_mg37.I2000Clm50SpRs.izumi_nag.clm-ptsRLA
SMS_D_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm45BgcCropQianRs.derecho_intel.clm-cropMonthOutput
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Bgc.derecho_gnu.clm-default--clm-NEON-HARV
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Bgc.izumi_nag.clm-default--clm-NEON-HARV
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.derecho_intel.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.izumi_nag.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.derecho_intel.clm-default--clm-NEON-TOOL--clm-nofireemis
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.izumi_nag.clm-default--clm-NEON-TOOL--clm-nofireemis
SMS_Lm3_D_Mmpi-serial.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdHydro
SMS_Ly3_Mmpi-serial.1x1_numaIA.I2000Clm50BgcDvCropQianRs.derecho_gnu.clm-ignor_warn_cropMonthOutputColdStart
SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm60BgcCropQianRs.derecho_intel.clm-ciso_monthly
SUBSETDATAPOINT_Ld5_D_Mmpi-serial.CLM_USRDAT.I2000Clm60BgcCropCrujra.derecho_intel.clm-default
SUBSETDATAREGION_Ld5_D_Mmpi-serial.CLM_USRDAT.I2000Clm60BgcCropCrujra.derecho_intel.clm-default