Endless sequence of assertion failures in Date_Time disaster [5623]
Bug Number: 5623
Release notes (y/n): Yes
For Release Nums: 6.7 and 6.6.4 (see "6.7 vs. 6.6.4" note at bottom).

Gnats 5623: Running CRSS and hitting an error gives many new assertion
messages ... If you run CRSS in MRM and it hits an error (e.g. Ruleset
versions is not correct), it then posts 13000 assertion errors of the form:
Assertion failed in file Date_Time.cpp at line 358: absm >= 0.

Summary:

    We've determined that this assertion in Date_Time::monthSeconds (int
    monthIndex) -- which fails when given a negative monthIndex -- is
    appropriate.  The higher level code allowing this condition to occur
    is not.  However, we need to prevent the low level assertion from
    generating this message so many times; once per run (or anytime
    after each run) is sufficient. (The major impact to the user of this
    particular problem was this error condition being reported 13000
    times).

    This, and related error conditions in the Date_Time class should
    never happen.  Any occurrence is a serious problem and needs to be
    addressed with RiverWare program changes, at a higher level in code,
    in a patch release.

    An initial analysis is posted here ...
    http://cadswes2.colorado.edu/~philw/2015/bugs/5623/Gnats5623.html
    ... we implemented a variation of the stated recommendation.

We are addressing this at two levels.

(1) Although the assertion check and failure at the low level (within the
Date_Time class) are appropriate and needed, we don't need it reported more
than once each run. Before applying the higher level fix (see the next
point), a mechanism for "throttling" the reporting of this sort of condition
in the Date_Time class (if it ever occurs for some other reason) was
implemented and tested such that it is reported only if it has not yet
occurred since prior "run start" or "model load" or "clear workspace".  That
is, it is reported at most once per run.  (The generated error message
explicitly states this behavior).  This "throttled" assertion mechanism was
also applied to the decrementing (or "incrementing" by a negative interval)
any Date_Time which currently has a zero value (zero absolute seconds).

The Date_Time class now maintains these three static counters for the
purpose of issuing an internal error message (and failing an assertion) only
once per run.  They are reset externally using the indicated public static
method:

   int Date_Time::_errorCnt_monthSecondsIndex (0);
   int Date_Time::_errorCnt_zeroDecrement (0);
   int Date_Time::_errorCnt_zeroIncrementNeg (0);
   void Date_Time::clearErrorCounts()

(2) The higher-level cause of this problem -- a run termination preventing a
particular Date_Time instance from being given a non-null value -- was
directly addressed.  This was done primarily in this method:

   MrmDmi::FileInfo::getRunParam()

... when computing the initDate from the startDate (i.e. one timestep before
the startDate), in the unusual circumstance of the startDate being ZERO, we
just (trivially) use that ZERO Date_Time as the initDate.

In the bug scenario, this is relevant after the run is aborted, and we are
in the process of "climbing out" from the (timestep-) run controller's
iterative processing.  In the course of confirming, in the debugger, that
the recovery from the method above results in "empty processing" within the
run controller code, it became apparent that the recovery clause's attempt
to set an empty interval with this method ...

   okstat RunParam::setDatesAndTimestep(
      const Date_Time* initDate,
      const Date_Time* endDate,
      const DeltaTime& timestep)

... failed because it rejected the setting of an empty run interval.  This
has been modified to accept the provided time data as long as at least the
initDate and endDate are correctly "aligned".  (It still returns an error
status in that circumstance, but at least the data is accepted).

Note: There is a slight difference between the changes being applied to
6.7 development and the 6.6.4 patch release.  In the latter, the Date_Time
class' new internal error reporting (which is now "throttled" -- basically
reported only once per run) is downgraded to a warning.  As in the bug
scenario, it's possible that any new occurrences of these errors may be
within temporary transitional sequences or otherwise unused computations. 
This far along in the "6.6" release sequence, it would be better to not
have such computations cause a run abort, and what not.  While we can't be
certain that such error conditions are, in fact, innocuous, catching that
during the 6.7 preparation testing and pre-release should be soon enough. 
(6.6.4 will at least show a warning).  In 6.6.4, the warning text is as 
follows (one of five similar warnings):

   Unexpected Error: A zero-value Date_Time is being decremented. 
   Note that this error is reported only once during each run.

---