DSM2 Real-Time Modeling
Frequently Asked Questions

General

What is the difference between real-time and real-tide?
Can the real-time procedure be used to run historical (real-tide) simulations?
Before I start, which files should I delete?

HYDRO

I have a channel drying up near a gate or barrier, how can I fix this?
What are the standard IEP DSS paths used in a DSM2 Real-time forecast?
What are the standard IEP DSS paths used in a DSM2 historical (real-tide) simulation?
What is the difference between an astronomical forecast for Martinez stage and the Martinez stage forecast by the real-time scripts?
OK, so why can't I just use the astronomical forecast?

QUAL

Should I fill in San Joaquin / Sacramento EC using the VPlotter scripts?
Some of the EC stations have missing data, what do I have to do to generate EC initial conditions?
The EC initial conditions created in my restart file look wrong, how can I fix this?

General

Q: What is the difference between real-time and real-tide?

Real-tide is a specific application of DSM2.  Real-tide DSM2 simulations use real tidal data for the downstream tidal boundary condition at Martinez instead of 19-year mean tidal data.  Real-tide simulations typically use 15-minute computational time steps.  The input data typically can be 15 minutes, 1 hour, or daily.

When we talk about "historical" simulations, we often mean real-tide simulations.

Real-time is used to describe either data or a simulation that is recent.  Real-time simulations are all real-tide simulations (though real-tide simulations that don't use recent data are simply historical simulations).  The same 15-minute computational time step is used.

Many locations in the Delta have real-time telemetry.  Hydrodynamic and water quality data from these stations is transmitted directly from the field into our data bases, including the IEP DSS database.  Often this data has not been screened, and is considered preliminary.

If a simulation makes use of this real-time data and it is fairly recent, it is generally considered real-time.

Q: Can the real-time procedure be used to run historical (real-tide) simulations?

Yes, but many of the DSM2 inputs and scripts involving the DSS data will need to be changed to point to historical instead of real-time DSS data.  These files include:

File	Directory Location
ECData.sty	/real-time/prepro
fillin.sty	/real-time/prepro
input-rim.inp	/real-time/input/hydro
pathlist.inp	/real-time/prepro/init
retriever.sty	/real-time/prepro

There are really just two major steps to changing the real-time setup:

First: Using the best available historical data.

The VPlotter scripts should be changed to use /HIST+CHAN/ IEP data instead of /RLTM+CHAN/ IEP data.  The historical data will be saved to the appropriate hydroraw, qualityraw, hydro, and quality DSS files.  Often the DSS paths for historical data will be very different than their real-time equivalents, because the data may easily come from another source.

Example:

Real-time Mallard Stage data typically is taken from:

/RLTM+CHAN/RSAC075/STAGE//1HOUR/CDEC/

But for historical studies DWR screened data is available:

/HIST+CHAN/RSAC075/STAGE//15MIN/DWR-ESO-D1485C/

Depending upon the time window, different historical data sets will need to be used because the data sets are not always complete.  You will want to go to http://wwwiep.water.ca.gov/dss/ and then use the Hydrodynamic and Water Quality Data Index Catalogs to determine which historical paths can be used.  It is recommended to use the most complete path available.

Example:

Several different paths for Martinez data are available:

/FILL+CHAN/RSAC054/STAGE//15MIN/UCB-ELI/
/HIST+CHAN/RSAC054/STAGE//15MIN/DWR-ESO-D1485C/
/HIST+CHAN/RSAC054/STAGE//15MIN/USGS/
/RLTM+CHAN/RSAC054/STAGE//1HOUR/CDEC/

In this case, the /FILL+CHAN/ data is the most complete data, but unfortunately it is only available from 1988 to 1998.  This /FILL+CHAN/ path is currently the only /FILL+CHAN/ data that is available for download from the IEP DSS database.  All other /FILL+CHAN/ data should be created by you (the modeler).  [NOTE: the filled portions of this path were created based on information from nearby stations.]

The USGS data runs from 1986 to 1992.  If the historical time window is 1987, then it is recommended to use and fill-in the USGS stage data exactly was normally would be done with the /RLTM+CHAN/ data.  But if the time window is 1997, then the /FILL+CHAN/ data should be used instead.  And in this special case, the /FILL+CHAN/ stage data can be directly saved to hydro.dss instead of hydroraw.dss.

Second: The input files used by HYDRO, QUAL, and for the warm start need to be changed.

Changes to the input-rim.inp files are simple.  Just point to the new paths that were saved in hydro.dss and quality.dss.

Changes to the pathlist.inp file used for the QUAL warm start are nearly as simple.  The real-time pathlist.inp file is available for downloading.  Follow these steps to change it:

1. Comment out the real-time 26 EC DSS paths listed in the first block.
2. Uncomment out the historical EC DSS paths listed in the second block.
3. Update the historical EC DSS path list to reflect the number of stations that are available during the time period of your study.  The file current lists 21 locations, but this number will change depending upon your time window.

The general format for entering a new path to the pathlist.inp file is:

<path>, <dsm2 translation name>, <scale for the error at that location>

/HIST+CHAN/RSAN058/EC//1HOUR/DWR-ESO-D1485C/,rsan058,100

Where,

• the path should be exactly as it appears in your qualityraw.dss file;
• the translation name normally will be the simple RKI translation; and
• the scale error should be a number between 100 and 2000 (higher scales are used for the Western Delta) that is used to weigh the EC variability of each location.

Q: Before I start, which files should I delete?

Before starting a new set of forecasts I like to zip all the unique files for the previous run in case I'm asked to rerun the forecast (which seems to happen regularly).  A few of these files need to be deleted before I begin the next forecast.

A more detailed description of all the files you should delete is located in the Concepts section.

HYDRO

Q: I have a channel drying up near a gate or barrier, how can I fix this?

There can be several reasons for DSM2 to report that a channel is drying up near a gate or barrier.  This is a problem I've commonly encountered with the South Delta temporary barriers.

First: Check your gates-dss.inp file.

The gates-dss.inp file is often changed between different scenarios and even different uses of the Real-time setup (forecast vs. historical simulations).  The gates-dss.inp file is very complex, thus it is easy to forget to update one of the DSS path names, priority levels, or even accidently comment out some header information.  For a complete example of the key blocks in the gates-dss.inp file look here.

Examples of common DSS gate path names:

/FORE+GATE/ROLD074/POS//IR-DECADE/20001023-14A/

/HIST+GATE/CHGRL009/NPIPES//IR-DECADE/DWR-ESO/

The most common errors come when changing: (1) the F_part, and (2) the A_part.  Double check to make sure all your DSS paths match values in your DSS files.  DSM2 will not always warn you if it is not reading irregular time series data!

Examples of correct priority levels:

As different DSS files and paths are used in alternate scenarios, the priority levels for the paths are changed.  I've often forgotten to change these priority levels back.  Remember that once you begin to have data in an irregular time series, DSM2 will not go to other priority paths.  Always use a higher priority for your forecast data.

For a forecast simulation:

Use 2 input blocks.
Set the forecast data to priority 1, the historical data to priority 2.

INPUTPATHS
NAME  A_PART    B_PART  C_PART INTERVAL  ID        FILLIN PRIORITY FILENAME
orhrb fore+gate ROLD074 npipes ir-decade 20001023-14A last   1   forecast.dss
END

INPUTPATHS
NAME  A_PART    B_PART  C_PART INTERVAL  ID        FILLIN PRIORITY FILENAME
orhrb hist+gate ROLD074 npipes ir-decade DWR-ESO   last    2       gates.dss
END

For a historical simulation:

Use 1 input block.
Set the historical data to priority 0.

INPUTPATHS
NAME  A_PART    B_PART  C_PART INTERVAL  ID        FILLIN PRIORITY FILENAME
orhrb hist+gate ROLD074 npipes ir-decade DWR-ESO   last    0       gates.dss
END

Example of gates header information:

INPUTPATHS
NAME  A_PART  B_PART  C_PART  INTERVAL  ID  FILLIN  PRIORITY  FILENAME

{gate information}

END

-or-

GATES
NAME   CHAN   LOC   OPER

{gate information}

END

These three lines should always enclose information blocks.  The most common errors I experience is when either the first two lines have been accidentally commented out, or when the END statement is commented out or completely missing.

Second: Check your gates.dss or forecast.dss data.

When stage is high enough that water flows both through barrier pipes as well as over the weir portion of the barrier, DSM2 may require a large number of numerical iterations.  Sometimes DSM2 crashes as a result of the numerical instability generated while trying to solve two different gate equations as the same time.

To see if this could be a problem, confirm that stage near the barrier in question is high enough that flow is passing both over the weir and through any pipes.  Check your gates.dss and/or forecast file(s) to confirm that you have the dimensions of your barrier set as you expect them to be.  Often one of the gate properties in your forecast.dss file will be incorrect.  If this is the case, rebuild your forecast.dss.  Similarly, if the historical file appears incorrect try rebuilding it.

If the gate parameters look correct, then it is possible you've modified the DSM2 geometry for your particular simulation (an example would be increasing the bottom depth of a channel for a dredging study) or that the hydrodynamics for your scenario have not been tested.  You can increase or decrease the weir coefficient for the problem barrier, but make sure to return the coefficient to its normal value for your next simulation.  Remember, any changes you make should be well documented, as you will now be using a non-calibrated geometry or set of coefficients! 

Q: What are the standard IEP DSS paths used in a DSM2 Real-time forecast?

For HYDRO:

All of the above real-time data is not verified.  The data sometimes is incomplete, but missing periods are filled in using the DSM2 real-time preprocessing scripts.  The scripts create new paths, all with new DSS path names.

For QUAL's Boundary Conditions:

For QUAL's Initial Conditions:

For QUAL, a linear interpolation is used to fill in both the San Joaquin and Sacramento boundaries.  Martinez EC is filled in using scripts.  The rest of the EC inputs should be left as is (the EC I.C. scripts will ignore missing values).  EC for Mallard and the raw Martinez EC are again used to calculate the optimal initial salinity concentrations.

NOTE: The information for computing the QUAL initial conditions is located in the pathlist.inp file.  If additional historical or real-time data is available, just add the complete path to the pathlist.inp file.

Q: What are the standard IEP DSS paths used in a DSM2 historical (real-tide) simulation?

Over time, different locations and different agencies have provided hydrodynamic and water quality data to the DSS database.  There is not really any one standard data set that is used for historical real-tide simulations.  The data used for a real-tide simulation from 1984 to 1988 will have different paths than data for similar locations if that same study was run from 1994 to 1998.

Anytime you are changing the DSS inputs to DSM2, it is your responsibility to make sure that your data paths are complete for the entire time period of your study.  Just because you may have data at the start of your study, does not mean that this DSS will continue.

The key differences between Real-time and historical real-tide DSS inputs will be:

1. A_part, which often changes from RLTM+CHAN to HIST+CHAN, and
2. F_part, where the agency name often will be completely different.

Q: What is the difference between an astronomical forecast for Martinez stage and the Martinez stage forecast by the real-time scripts?

Stage at Martinez is the downstream model boundary, and can not have missing values. A hybrid vector autoregressive and astronomical model is used to forecast and fill-in missing stage data at Martinez for real-time simulations.  The theory behind this model is completely described in Chapter 8 of the Delta Modeling Section 2000 Annual Report.

A pure astronomical model can be used to estimate stage at Martinez.  While there will not be any significant differences between the phase of the observed tide at Martinez and the phase of the astronomical based forecast, the difference in stage between the two values can be several tenths of a foot.

The hybrid vector autoregressive and astronomical model essentially uses only recent historical stage data from the NOAA station in San Francisco at the Golden Gate and the DWR data from Martinez and Mallard Island.  The model then computes the difference between each of these recently observed stage time series and subtracts out the astronomical prediction.  Then the model uses historical correlations between these three stations to forecast a future residual for Martinez.  Finally the astronomical component is added back into this residual time series.

For historical periods, the final stage estimate by the hybrid model matches the observed values very well, as is shown below in Figure 1.

Figure 1: Filled-in vs. Observed Stage at Martinez (RSAC054).

As is shown in Figure 2, the forecast stage not only exactly matches all the observed points, but since it is based on a 15-minute astronomical forecast, it provides a more accurate estimate of the Martinez stage for the times that don't fall exactly on the hour.  This is important because DSM2-HYDRO can be set either to use a linear interpolation when computing boundary parameters between the hour or it can just use the previous value.  Simply put, the DSM2 real-time forecast stage is actually better for use during historical periods than the hourly observed data.

Figure 2: Comparison of 15-Minute Filled-in vs. 1-Hour Observed Stage at Martinez (RSAC054).

The hybrid model that generated the filled-in stage shown in Figures 1 & 2 above, also generates complete stage forecasts.  The period at which observed data is no longer available is the time of forecast.  Figures 3 & 4 below show the comparison between the hybrid model's forecast stage and the observed stage.  As was shown in Figure 1, the filled-in stage data is always smoother.  The filled-in stage always has a slightly larger amplitude (simply because it is easier to fit 15-minute data to the natural oscillations than the coarser hourly data).

Figure 3 does not show this same trend.  For this particular example, the forecast (labeled filled in the graph) appears to be shifted slightly higher.  Unlike a historical fill-in, forecast stage can at times have a smaller amplitude than the observed.

Figure 3: Forecast vs. Observed Stage at Martinez (RSAC054).

Furthermore, the difference between the forecast stage and observed stage is greater, as is shown in Figure 4.

Figure 4: Comparison of 15-Minute Forecast vs. 1-Hour Observed Stage at Martinez (RSAC054).

As the time of the forecast value moves further and further away from the time of forecast, this error increases.  What really is happening is that the autoregressive component of the model is becoming less and less important the further in time we move, and the astronomical forecast begins to dominant.  This can also be seen when comparing the forecast stage with the astronomical stage.  In Figure 5, 3 months of Martinez stage are shown.  Initially the difference between the forecast and astronomical stage is large (NOTE: In Figure 5 the forecast start date was Oct. 10, 2000).  By early November this difference is barely noticeable.

RULE OF THUMB: Hybrid stage forecasts become pure astronomical forecasts within 14 days.

Figure 5: Comparison of Forecast vs. Astronomical Stage at Martinez (RSAC054).

Q: OK, so why can't I just use the astronomical forecast?

Good question.  It has always been very important to accurately model stage entering and exiting the Delta, as it represents a volume of water/salt.  While the astronomical forecast is good, real observations tend to be better.  [NOTE: The filled-in stage data created by the hybrid vector autoregressive / astronomical model should still be used instead of raw field observations.]

One might want to then switch from observed values to the astronomical at the time of forecast, instead of using the filled-in / forecast boundary stage for both the time before and after the forecast.  Doing this can create discontinuities in the stage when you move from one time series to another.  Several of these discontinuities are shown below in Figures 6-10.

In Figures 6 and 7, discontinuities created by jumping from observed stage data to astronomical forecast stage are shown at high tide.  It does not matter if this jump is positive or negative.  It is the discontinuity itself that should be avoided.

Figure 6: Poor Example of Observed Stage larger than Astronomical Stage at High Tide.

Figure 7: Poor Example of Observed Stage less than Astronomical Stage at High Tide.

Figure 8 shows another discontinuity, only this time it is for low tide.  Again, this type of situation should be avoided.

Figure 8: Poor Example of Observed Stage larger than Astronomical Stage at Low Tide.

When the discontinuity occurs in the middle of the tidal cycle, as is illustrated in Figure 9, the modeled stage will appear to have "steps".  Again, this is a situation that should be avoided.

Figure 9: Poor Example of Observed Stage vs. Astronomical Stage in the Middle of the Tidal Cycle.

Even though there are occasions where the discontinuity between the observed and astronomical stage is small (see Figure 10), it is difficult to predict when in the tidal cycle these discontinuities will occur.  It is also difficult to determine a working rule for when the magnitude of the discontinuities are acceptable.

Figure 10: Good Example of Observed Stage vs. Astronomical Stage in the Middle of the Tidal Cycle.

Another advantage of the hybrid stage model is that it gradually ramps stage from the observed values (which if fits nicely) to the astronomical forecast.  There are no discontinuities created by using the hybrid stage.

SHORT ANSWER: The astronomical stage is really just a tool that is used either when no other information is present or to fill-in existing stage data.  Be careful when using it.

QUAL

Q: Should I fill in San Joaquin / Sacramento EC using the VPlotter scripts?

No.  A specialized script called ECData.sty is used to fill in Martinez EC.  This script uses a modified G-Model and estimate of the Net Delta Outflow Index to predict Martinez EC.  Currently this technique can not be applied to the other DSM2 salinity boundary conditions.

When you look at San Joaquin and Sacramento EC you may notice that frequently the EC at both of these boundaries does not vary much day to day.  Often real-time runs just approximate forecast EC by "flat-lining" the EC at these two boundary conditions.

Q: Some of the EC stations have missing data, what do I have to do to generate EC initial conditions?

For real-time runs there is a maximum of 25 locations in the interior of the Delta and 3 boundary locations (Martinez, San Joaquin at Vernalis, and Sacramento at Hood) that are used by the scripts that create EC initial conditions.  The list of all these locations is in the chanlist.inp file.

The opstart.py script was designed to read all of the input paths, and if data is missing for either part or all of the initialization period, it will ignore these locations.

Make sure that between 20 and 30 stations appear to have EC data for at least half of the initialization period.  In other words, it is OK to have missing data in a path, just make sure that there is data for around 50% of the time in 20 to 30 stations.  If they don't, please make a note of the lack of data and consider starting your HYDRO and QUAL runs earlier during a time when there was more data.  

Q: The EC initial conditions created in my restart file look wrong, how can I fix this?

I always check the qual.rst file (not to be confused with the qual-test.rst file which is used for something else).  If the salinity gradient between Martinez and Jersey Point doesn't look right, then it is very likely that you forgot one of three things (I've done this myself several times) to:

1. Update the io.inp file,
2. Update the dsm2.inp file, or
3. Save your water quality data in qualraw.dss.

Check those three files!  Make sure use the latest-opt.inp (see the dsm2.inp file) when running optstart.py.

DSM2 Real-Time Index.

Questions?

Last revised: 2002-09-30