CADSWES -- Release, Software Maintenance, and Development Environment Work Summary -- Fiscal Year 2017
Phil Weinstein, edit 10-19-2017, ready for review.

This document summarizes the maintenance work completed by CADSWES in these categories from October 1, 2016 through September 30, 2017.

  1. Releases, Patches and Snapshots
  2. Development Environment
  3. Software Maintenance
  4. Download, Installation, and Release Processes
  5. Licensing
  6. Regression Tests
  7. Bug Fixes

(1) Releases, Patches and Snapshots

Two major releases (7.0 and 7.1) and thirteen (13) patch releases of RiverWare were generated this year. The work involved in generating a patch release is very similar to a major release, except that a major release typically has an extended period dedicated to bug fixing prior to the release, and a more extensive set of release notes. Release notes may be found in the RiverWare online help.

Release Type Release Release Date
Patch 6.9.6 Oct 11, 2016
Patch 6.9.7 Oct 28, 2016
Release 7.0 Dec 16, 2016
Patch 7.0.1 Dec 30, 2016
Patch 7.0.2 Jan 25, 2017
Patch 7.0.3 Jan 31, 2017
Patch 7.0.4 Feb 14, 2017
Patch 7.0.5 Feb 24, 2017
Patch 7.0.6 Mar 23, 2017
Patch 7.0.7 May 11, 2017
Patch 7.0.8 Jun 02, 2017
Release 7.1 Jul 12, 2017
Patch 7.1.1 Jul 26, 2017
Patch 7.1.2 Sep 01, 2017
Patch 7.1.3 Sep 19, 2017

Six (6) snapshots of the build development area were made over the course of the fiscal year to allow users to test newly implemented features before they become part of an official RiverWare release.

RiverWare 7.0 was prereleased on December 01, 2016 and released on December 16, 2016.
RiverWare 7.1was prereleased twice, on June 20 and June 27, 2017 and released on July 12, 2017.

The tasks involved in generating releases include:

(2) Development Environment

The Development Environment category includes all of the associated software and tools used for developing RiverWare as well as their setup and interaction. Work completed this year in this category is presented below by major topic. These include:

  1. Adoption of OIT BitBucket Server (Git) repository hosting service for CADSWES source control
  2. New CADSWES Linux Server, Preparation
  3. Microsoft Visual Studio C++ Development Environment: Upgrade from VS 2010 to VS 2013
  4. Oracle Database Client and SQL Developer

Adoption of OIT BitBucket Server (Git) repository hosting service for CADSWES source control:

Git is the revision control system used by CADSWES for source code and model files. There are desktop clients as well as web interfaces for Git. The desktop clients provide read / write access to the Git repositories while the web interfaces most typically provide read-only access. The Git repositories had been hosted on the CADSWES Linux server. In FY 2017, CADWES transitioned to Git repositories in the CU cloud.

January 2017:

CADSWES staff (Edie Zagona, Bill Oakley, Jim Pasquotto, Patrick Lynn) met with Orrie Gartner and Vasa Dasan of CU’s Office of Information Technology (OIT) to share with them a description of the software development activities at CADSWES and to discuss with them how OIT services might help support those activities. As a result of that meeting, CADSWES have begun exploring the option to move the internally hosted Git repositories to the BitBucket Server hosting supported by OIT.

February 2017:

CADSWES staff met with Vasa Dasan of CU’s Office of Information Technology (OIT) to continue our discussion about how that office might support CADSWES's software development efforts. In addition, CADSWES continued to investigate the possibility of moving the internally hosted Git repositories to the BitBucket Server hosting supported by OIT.

March 2017:

We continued to investigate the possibility of moving the internally hosted Git repositories to the BitBucket Server hosting supported by OIT. The primary RiverWare source code repository was cloned to the OIT site and then developers explored the functionality available via the web for accessing that repository. 

Software developers evaluated working with the cloud-base repositories. To a large degree the transition is transparent; the desktop client currently in use, Git Extensions, works equally well with the locally hosted repositories and the cloud-based repositories. However, the web interfaces would not be available for the cloud-based repositories -- they require a locally hosted repository and the effort to mirror the repositories on the local host would be too burdensome. (It should be noted that the web interfaces provide capabilities not available in Git Extensions, for example searching the repository by committer.) To address this, alternative desktop clients were evaluated, with Atlassian SourceTree being the "winner". It provides a clean, intuitive interface and the features provided by the web interfaces.

May 2017:

CADSWES moved the internally hosted Git repository for the RiverWare source code to the BitBucket Server hosting supported by OIT.

New CADSWES Linux Server, Preparation

The current Linux server, Alamosa, is being replaced by a new Linux server, Animas. The CU Office of Information Technology (OIT) is doing much of the work, but there are tools we use which OIT isn't familiar with and therefore can't "port" from Alamosa to Animas. A good example is our bug tracking system; we access the open source Gnats software from web browsers by running a suite of home grown Perl scripts from the Apache web server.

In September 2017, we verified Gnats is working on Animas. (Gnats has a command line interface which is cumbersome to use but adequate to confirm it's working.) We started to verify the Perl scripts are working but ran into problems with the web server configuration, which OIT is resolving. Although unable to run the Perl scripts from the web server, we began to edit the scripts as we anticipate they will need to change with many, but mostly straightforward changes (for example, file and directory paths).

Microsoft Visual Studio C++ Development Environment: Upgrade from VS 2010 to VS 2013

Periodically CADSWES upgrades the version of Microsoft Visual Studio being used for RiverWare development. In FY2017, we upgraded from VS 2010 to VS 2013. This involved several major steps, including:

There are eighteen (18) 3rd party libraries used by RiverWare -- including Qt, ICU, CPLEX, FlexLM, RepriseLM, GDAL / MrSID, NetCDF / HDF5, Oracle Client, Google protocol buffers, QuaZIP, Qwt and Tcl. There are dependencies between these libraries, Visual Studio and Windows which can sometimes make the path forward difficult, if not impossible. For example, we would prefer to use VS 2015 (or perhaps VS 2017) but Oracle Client requires VS 2013, so Oracle Client is preventing us from moving beyond VS 2013. Furthermore, VS 2013 isn't supported on Windows 10, so Oracle Client is preventing us from moving beyond Windows 8.1, and Microsoft has stated that PCs based on new CPUs will require Windows 10, so Oracle Client may constrain our ability to buy new PCs. This is but one example of the types of problems which arise with RiverWare's complex development environment. Navigating the dependencies is difficult but necessary to ensure RiverWare's future as a supportable product.

The "Building RiverWare Related Software" document describes for each library whether it's downloaded or built from source, and the instructions for doing so. The document is describing a moving target -- websites change, procedures for building libraries from source change, and so on. For example, the instructions for building GDAL / MrSID, NetCDF / HDF5 and Qwt are substantially different now than they were when the document was last updated. This document was updated throughout this Visual Studio migration process.

February 2017: Visual Studio Upgrade Research (from VS 2010 to VS 2013 or VS 2015):

  1. Installed Visual Studio 2013 and Visual Studio 2015 on Granby
  2. Installed VisualAssist and GitExt plug-ins for 2013 and 2015 on Granby
  3. Acquired version 12.7 of CPLEX needed for VisualStudio 2013 & 2015
  4. Began dependency analysis and compilation of code.

March 2017: RiverWare Visual Studio Upgrade (from VS 2010 to VS 2013):

In March the 3rd party libraries used by RiverWare were either downloaded or built from source (upgrading to new versions when appropriate) and the document was updated to reflect the new instructions. This document represent "best practices" at the time it is updated and can provide a road map the next time though the process. Specific related decisions and tasks completed in March include:

  1. Settled on VS2013 (rather than VS2015) due to a compatibility issue with Oracle Client.
  2. Upgraded project files and solution file to the VS2013 toolset and pushed to the VS2013 Git branch.
  3. Applied minor code fixes to DbDmi, Q3Gui, and Sim to address breaking changes and pushed to branch. (to revisit).
  4. Compiled projects with VS2013 using existing include directories and library header files.
  5. Built all 3rd party libraries (except netcdf - in progress) with VS2013.
  6. Installed new 3rd party builds in c:/riverware/tools
  7. Updated documentation with detailed instructions for each 3rd party build.
  8. Updated .pro files INCLUDEPATH with new version locations and pushed to branch.
  9. Ran qmake and encountered an issue: qmake mysteriously reverted the ToosVersion in the.vcxproj file to 4.0 (VS2003)
  10. Updated include directories in project files and pushed to branch.
  11. Updated PATH on granby to reflect new location of Quazip, and UIC
  12. Recompiled all projects with new library header files and include paths.

April 2017: RiverWare Visual Studio Upgrade (from VS 2010 to VS 2013):

  1. Resolved release build "strictstrings" compile errors by opting-out of the "strictstrings" compiler flag provided by qmake. 
  2. Finalized contents of the 64-bit c:/riverware/tools directory
  3. Compiled, linked, and ran RiverWare debug-64, and release-64 from Visual Studio 2013.
  4. Upgraded TK and Perl modules in the 64-bit environment.
  5. Updated the Visual Studio 2013 build with the latest code enhancements.
  6. Ran 64-bit debug regression tests.  Looking into 3 issues.
  7. Set-up of 32-bit Visual Studio environment in-process.

May 2017: RiverWare Visual Studio Upgrade (from VS 2010 to VS 2013):

In May, focus turned to the 32-bit development environment. The 32-bit 3rd party libraries were either downloaded as precompiled binaries or built from source. Also in May a few "straggler" issues were resolved. By the end of May the Visual Studio 2013 port was for the most part complete – there were working 64-bit and 32-bit release and debug executables available to begin testing.

May accomplishments include:

  1. A 32-bit Visual Studio 2013 environment was created and configured in addition to the existing 64-bit environment.
  2. Visual Studio 2013-compatible versions of all required 3rd party libraries were built from source code or acquired in binary form.
  3. Regression tests passed fully for the 64-bit version.
  4. A 64-bit snapshot install file was generated.
  5. Manual testing of the 64-bit by the RiverWare development team began, using an assigned-task test plan. This testing continued into June.
  6. The install procedure for the Visual Studio 2013 software was developed and tested for use on in-house development machines.  Both the old and the new version will operate simultaneously.
  7. The RiverWare library install process for in-house development machines was upgraded to support Visual Studio 2013, documented, and made available for immediate use.
  8. Debugged optimization "infeasibility" in the Visual Studio 2013 build with a new CPLEX library version.

June 2017: RiverWare Visual Studio Upgrade (from VS 2010 to VS 2013):

In June 2017, CADSWES staff completed the Visual Studio Upgrade from VS2010 to VS2013 and pre-released RiverWare 7.1 build with Visual Studio 2013. Specific June tasks included:

June 2017: RiverSMART Visual Studio Upgrade (from VS 2010 to VS 2013):

During the first few months of 2017 RiverWare was ported to Visual Studio 2013, including upgrading the 3rd party libraries RiverWare uses. RiverSMART is also built using Visual Studio, although it uses a small subset of the eighteen 3rd party library RiverWare uses. Usually this means that once RiverWare is ported it's a straightforward matter to port RiverSMART. In June RiverSMART was ported to Visual Studio 2013 and the upgraded 3rd party libraries, although the port wasn't as straightforward as expected, and the reason illustrate why the full scope of the necessary effort is hard to anticipate.

RiverSMART uses the zlib library to read compressed RiverWare model files (to "mine" the model files for metadata describing aspects of the model). RiverWare uses the zlib library for the same purpose. After its port, the RiverWare code (which was taken directly from RiverSMART) worked correctly. However, after its port the RiverSMART code was crashing in a call to the zlib gzopen() function. Same code, same library yet one application worked correctly and the other application crashed. (I say "same code," yet there were differences. For example, RiverWare calls gzopen() from the executable whereas RiverSMART calls it from a DLL. Is this a significant difference? It can be difficult to find the answer, and searching the Internet can consume a lot of time without yielding a definitive answer.) The time spent debugging the crash didn't reveal the problem (much less a solution), so attention turned to using a different zlib library. (There wasn't a good reason to think the zlib library was the problem, but sometimes something is tried for lack of something better to try.) Searching the Internet identified the zlib implementation bundled with Qt 5.5.1 as a candidate, although there are those who say it shouldn't be used. (Their rationale is that Qt doesn't document that it uses zlib and therefore one should view zlib as a Qt implementation detail and not rely on it.) But having already spent more time than anticipated on the RiverSMART port, it was decided to use the zlib bundled with Qt. Should a future Qt release not provide zlib the decision will be revisited.

Oracle Database Client:

In February 2017, Oracle 12c Client was installed on two computers for future work on the HDB DMI. The installer was a rookie installer so there was a learning curve, but the knowledge acquired will be beneficial in the future.

SQL Developer:

SQL Developer is a tool which enables developers to examine and modify SQL databases such as Oracle using a modern user interface.

In March 2017, SQL Developer was installed on two computers. To use SQL Developer effectively a user must be logged into the database as an administrator which creates a "high reward / high risk" situation. A novice (new to SQL databases and SQL Developer) could easily do something "not good". To reduce the odds of this, the novice in question took a tutorial on SQL Developer.

General Hardware maintenance

In January 2017, the disk drive of a software development machine began to show signs of failure and CADSWES staff began the process of backing up and replacing the drive.

(3) Software Maintenance

The Software Maintenance category includes work required to maintain the RiverWare code that is apart from work funded to implement specific new features and new functionality. This can include, for example, removing obsolete code, porting code to use newer versions of libraries (like Qt) or to use alternate libraries, addressing compiler warnings and memory leaks, non-project improvement to online help, or changing existing code to improve maintainability and efficiency.

  1. Water Engineering Enhancements
    1. Inline Power Specify Units Generating Method
    2. Reservoir Diversion Power, Tailwater and Power Bypass Methods
    3. Water Engineering Code Refactoring
  2. Numeric and Unit Enhancements
    1. Changes to RiverWare model file numeric precision
    2. Added BCM (Billion Cubic Meters) and related flow units
  3. Software Engineering and I/O Enhancements
    1. RiverWare Qt4 to Qt5 Port
    2. Video File Generation / FFmpeg utility deployment
    3. Significant size reduction of generated image files
  4. Diagnostics Enhancements
    1. Rules Diagnostics Enhancements
    2. Modification to begin and end run diagnostic messages
  5. Dialog an GUI Enhancements
    1. New RPL editor "Rename variable/argument" action
    2. "About RiverWare" Dialog Redesign / 3rd Party Library citations
    3. Simplification of the treatment of the SCT's "Default" font
    4. Uniform Snapshot Camera-Themed Icons
    5. Complete OK/Cancel/Apply button order on remaining dialogs
  6. Documentation Enhancements
    1. Optimization Documentation
    2. RPL Predefined Function Documentation

(A1) Inline Power Specify Units Generating Method:

In July 2017, a new method, Specify Units Generating, was added to the Inline Power category on the Inline Power object. In the original version of this method implemented for RiverWare 7.1, the user specifies the generating capacity for each unit in the Unit Capacity slot (table slot) and the fraction of capacity at which each unit is generating in the Unit Generation Fraction slot (agg series slot). The method then calculates the Unit Power and Unit Energy (agg series slots) as well as the total plant Power and Energy. For the RiverWare 7.1.1 patch release, a further enhancement was added to the method to allow the user to optionally specify the unit turbine capacity in the Unit Capacity table slot. The user can then optionally specify the Unit Turbine Release (agg series slot) and let the method calculate the Unit Generation Fraction and Unit Power or specify the Unit Generation Fraction and let the method calculate Unit Turbine Release and Unit Power. If neither Unit Turbine Release nor Unit Generation Fraction is specified, then the Unit Generation Fraction defaults to 1, and the unit operates as full capacity. The method now also calculates total plant Turbine Release and Bypass in addition to total plant Power and Energy. Documentation for the new method was added to the RiverWare Help for release 7.1 and patch release 7.1.1.

(A2) Reservoir Diversion Power, Tailwater and Power Bypass Methods:

In July 2017, three new categories were added to the Reservoir objects to allow for the modeling of power on the reservoir Diversion. One new method was added in each category. These new methods and categories (described below) were implemented for the RiverWare 7.1.1 patch release.

The Diversion Power Efficiency Curve method in the Diversion Power category is analogous to the Plant Efficiency Curve method on the primary power plant. It uses a 3-D table to calculate Diversion Power as a function of Diversion Turbine Flow and Diversion Operating Head. Diversion Operating Head is calculated as the difference between the average Pool Elevation and the Diversion Tailwater Elevation. The Diversion Power Efficiency Curve method is more limited than the existing Plant Efficiency Curve method. It does not allow Energy or Power to be input or set by rules. It does not include Plant Power Limit or Plant Failure methods.

The Diversion Base Value Plus Lookup method in the Diversion Tailwater category is analogous to Base Value Plus Lookup Table method for the standard Tailwater category. Diversion Tailwater is a function of Diversion (from the Diversion Tailwater Table slot) added to the Diversion Tailwater Base Value, which can be linked or defaults to zero if not input or set by rules.

The Diversion Power Bypass Capacity Table method in the Diversion Power Bypass category is analogous to the Regulated Spill method in the standard Spill category. If not input or set by rules, Diversion Power Bypass is set to Diversion minus Diversion Turbine Flow. It is limited to the max bypass interpolated from the Diversion Power Bypass Table (analogous to the Regulated Spill Table). It is not permitted for both Diversion Turbine Flow and Diversion Power Bypass to be specified (input or set by rules).

The method in the new Diversion Power category is called from each of the reservoir dispatch methods, at the end of the dispatch method. When the Diversion Power method is called, Diversion is already known. The new methods do nothing to affect mass balance or dispatching. They are calculated "after the fact." The Diversion Power method calls the Diversion Tailwater and Diversion Power Bypass methods.

Documentation for the new Diversion Power, Tailwater and Power Bypass categories and methods was added to the RiverWare Help for the 7.1.1 patch release.

(A3) Water Engineering Code Refactoring:

In May 2017, the Slope Power Reservoir slope storage calculation was refactored for improved maintenance, readability, and understanding.

(B1) Changes to RiverWare model file numeric precision:

As of June 2017, for RiverWare 7.2, in order to mitigate the risk of loss of information when floating point values are written to a model file, most values will be written with full precision (17 decimal digits, in standard units). Through RiverWare 7.1 the default precision was 12 digits, so in order to balance the potential increase in file size due to the general increase in precision, users will be able to choose to use a lower precision for simulation results (defined series slot values with the Output or Rule flag). As part of this work, the Model Save Confirmation dialog was modified to present the new options.

In July 2017, further testing was conducted and addition refinements to the Model Save Confirmation dialog were designed.

(B2) Added BCM (Billion Cubic Meters) and related flow units:

Similar to TCM (thousand cubic meters) and MCM (million cubic meters), the following units have been added for RiverWare 7.1 in May 2017. This applies to both conventional units (e.g. used on slots) and RPL units. These definitions are expressed in the "units" and "RplUnits" text files deployed with the RiverWare executable.

(C1) RiverWare Qt4 to Qt5 Port

Most of the work for the upgrade of RiverWare from Qt 4.8.5 to Qt 5.5.1 was completed in the prior year (FY 2016). The resolution of some ensuing problems was completed in the subsequent months (early FY 2017) for the release of RiverWare 7.0 in December 2016. Major FY 2017 RiverWare / Qt5 porting tasks are enumerated below.

October 2016: RiverWare Qt4 to Qt5 Port:

In October, the RiverWare 7.0 master development branch was switched from Qt 4.8.5 to Qt 5.5.1. Two new Windows build servers, "aspen" (64-bit) and "kickappo" (32-bit) are now running builds and regression tests nightly for RiverWare 7.0, using Qt 5.5.1.

It was discovered that, on most Windows 8.1 development machines, a lot of unnecessary recompilation was occurring each time RiverWare was built. Two distinct causes contributing to this problem were discovered, and were, at least partially, addressed. We hope to find more fundamental fixes for these issues when we move beyond Visual Studio 2010, to Visual Studio 2013 or 2015.

  1. Absolute file paths into the local development area are appearing in Visual Studio "make" files (.vcxproj) generated using the Qt 5.5.1 version of the Qt tool, qmake.exe. These paths (apparently playing a role in file dependency calculations) are invalid when used in a differently-named development area. This has been provisionally addressed by devising and applying a "sed" (unix tool) command to translate those undesirable absolute paths to relative paths. 
      
  2. It was discovered that four (4) RiverWare C++ header files (of over 1100 such files) were redundantly included in both the standard and Qt-generated Qt project (.pro) files. Removing these redundancies has mostly solved the unexpected rebuild problem. (There is still a substantial rebuild when switching between "debug" and "release" build modes; this had not generally been the case with the RiverWare build using Qt4).

Earlier RiverWare testing with Qt 5.5.1 had revealed visual problems with tree views on Windows 8.1 (but not on Windows 7) when using the standard "Windows Vista" widget style. Our initial solution had been to change to the new "Fusion" widget style. But in October, we decided that that was not an acceptable solution, as that would be inconsistent with the current RiverWare user documentation -- and reconciling that would be very time consuming. There were also a few visual properties of "Fusion" which we would have needed to address.

We learned that the Windows Vista widget style problems we were seeing had been discovered during Qt 5.6 development, and were eventually fully resolved in Qt 5.6.1 [QTBUG-51799 and QTBUG-52230]. Unfortunately, it is difficult for us to move from Qt 5.5 to Qt 5.6 because QtWebKit (used by RiverWare model reports) has been removed. (QtWebEngine is a partial replacement of QtWebKit in Qt; we will probably port our use of QtWebKit to QtWebEngine in the near future). An earlier Qt version -- Qt 5.4.2 -- also exhibited the Windows Vista widget style problem. We then attempted to build Qt 5.6.2 and a version of QtWebKit as two distinct libraries. But before achieving full success with a Qt 5.6.2 solution, after some unsuccessful initial attempts to instead apply the Qt 5.6.1 fixes to Qt 5.5.1, we enlisted some help from Digia/The Qt Company support engineers. They were able to provide us with a custom patch to Qt 5.5.1 source code which fixed this "Windows Vista" widget style problem.

A problem (in RiverWare with Qt 5.5.1) with menus not always appearing was found to be due to a work-around fix applied in 2010 for Gnats 4737. That original problem we had experienced with Qt4 was fixed in Qt 5.3, and the work-around we had introduced was actually causing a new problem with Qt 5.5.1. This was easily resolved by removing that 2010 work-around fix.

November 2016: RiverWare Qt4 to Qt5 Port:

Additional problems resulting from upgrading RiverWare from Qt 4.8.5 to Qt 5.5.1 were addressed. These include:

(C2) Video File Generation / FFmpeg utility deployment:

RiverWare 7.0 invokes an external program, FFmpeg, to combine a sequence of generated video frame image files into single a video file. Potentially relevant to 3rd party proprietary concerns, it's significant that the FFmpeg library isn't actually built into RiverWare -- not even as an externally (dynamically) linked library.

In December 2016, to avoid any appearance of such an integration, we switched from using the "shared" distribution of FFmpeg (which includes several DLL files) to the "static" distribution (consisting of only the unified "ffmpeg.exe" executable file).

(C3) Simplification of the treatment of the SCT's "Default" font:

The SCT configuration supports three font choices: "Default", "Font A", and "Font B". The latter two are user configurable fonts retained within the SCT configuration. The effective font used in the Default case had previously depended on the workspace font at the time the SCT was created.

As of January 2017, in RiverWare 7.1, the Default font selection is the font currently in use in the Workspace; changes to the Workspace font are immediately propagated to the SCT. (The Workspace font is set from the Workspace menubar: Utilities >> Windows >> "Set Font...", and, as currently implemented, applies to only the current RiverWare session).

This change is described further in this document:

(D1) Rules Diagnostics Enhancements:

In December 2016, CADSWES improved diagnostics output by adding the priority to the Rule diagnostics context, and adding the Rule index to the Initialization Rule diagnostics context.  This allows unambiguous identification of a goal or rule, even when there are multiple goals or rules having the same name.

(D2) Modification to begin and end run diagnostic messages:

RiverWare posts diagnostic messages at the beginning and end of each simulation which include the current "wall clock" time, as hours and minutes. In June 2016, the format of these times was modified to include also seconds.

(E1) New RPL editor "Rename variable/argument" action:

In December 2016, CADSWES staff added a new "Rename..." operation that provides an easy way to rename all occurrences of a variable or argument within a RPL editor. The "Rename…" operation is available in the right click context menu when the current RPL selection is a variable or argument name, whether that name is part of the declaration of that name or is a reference to the value associated with the name. Note that names can be appear within statements (FOR and WITH variables names), expressions (FOR and WITH variable names), and functions (argument names). When the new operation is invoked, RiverWare first presents a simple dialog that displays the number of occurrences of the name to be replaced and allows the user to type in the new name. Once the user has entered a new variable name and clicked the OK button, the old name is replaced with the new name throughout the scope of the name's declaration.

Activities in January 2017 include: software refactoring, bug fixing, extension of functionality to function constraints, and simplification of the Rename dialog.

(E2) "About RiverWare" Dialog Redesign / 3rd Party Library citations

The "About RiverWare" used to provide information about 3rd party libraries used within RiverWare in separate popup dialogs accessible via distinct push buttons in a separate "System Info" dialog.

As of November 2016, all of this information, and other general version and environment information is shown in a scrollable window right in the "About RiverWare" dialog from which formatted text can be selected, copied and pasted (e.g. into an e-mail message or Word document).

(E3) Significant size reduction of generated image files:

In the course of providing an option to generate RiverWare Model Report HTML files with all images embedded (see COE SWD 5, part C), it was discovered that the image files we were generating for model reports -- and also for frame images for the new chart animation video file generation feature -- were much larger than they needed to be (e.g. by a factor of 30). This often resulted in Model Report files (either with embedded images, or image files bundled with the HTML file) too large to attach to an e-mail message or convey in some other way.

In January 2017, this major size inefficiency was fixed for images generated from RiverWare graphics (e.g. plots, charts, RPL code) in RiverWare 7.0.2 and 7.1 development.

For chart animation video file generation, this frame image file size reduction doesn't seem to reduce the size of the generated video file. But the size reduction should improve the performance of the video file generation process, as image files for every frame of the video needs to be written out to the hard disk; those image files are now significantly smaller than they had been.

Note: This optimization was not applied to Model Report Image items (originally created by loading an external image file) when generating Model Reports with external image files (rather than using the new "embedded" image capability). Instead of just recreating the image file with the captured image file data, the image file is regenerated using the old algorithm without compression. Such image files are still unnecessarily large -- larger than the original image file from which the Model Report Image item had been created. This could be addressed in the future.

(E4) Uniform Snapshot Camera-Themed Icons:

RiverWare 7.0 and prior versions had inconsistent graphical "motifs" to refer to Snapshot-object-related functions and objects, making use of essentially two, unrelated "camera" graphics. In May 2017, a new 24x24 pixel "Snapshot Manager" camera icon, e.g. for the Workspace toolbar, is now a graphic component within the 40x40 pixel Snapshot-Object (special Data Object) simulation object icons. This was done for RiverWare 7.1.

(E5) Complete OK/Cancel/Apply button order on remaining dialogs:

We had recently standardized the order of common buttons along the bottom of most RiverWare dialogs. In July, we finished four remaining dialogs (or panels) which had some tricky issues in making this change:

(F1) Optimization Documentation:

In October 2016, CADSWES continued to enhance the RiverWare Help section on Optimization. New sections were added under the heading "Preemptive Linear Goal Programming: The Mathematics." These included new, simple examples to illustrate basics of preemptive goal programming, including figures, a section on soft constraints and derived objectives, and a section on freezing constraints and variables.

In November 2016, the new section on "Preemptive Linear Goal Programming: The Mathematics" was completed in the Optimization section of the RiverWare Help. A subsection on shrinking constraints to earlier constraints was added, editing was completed to improve technical details and accuracy for sections added in earlier months. The revised Optimization documentation was incorporated into the master version to be included in the RiverWare 7.0 release.

(F2) RPL Predefined Function Documentation:

In November 2016, for all RPL Predefined Functions that include a mass balance calculation, the RiverWare Help documentation was improved to explicitly list all components that are automatically included in the mass balance.

(4) Download, Installation, and Release Processes

Flexera InstallShield is the software used to create RiverWare installation files. Many of the tasks described in the following sections are related to InstallShield maintenance and support.

(a) InstallShield Install File Preparation for RiverWare Releases
(b) Other Release Support Tasks
(c) License and Download User Instructions
(e) Addressing InstallShield Problems

(a) InstallShield Install File Preparation for RiverWare Releases 

(b) Other Release and Install Support Tasks:

October 2016:

December 2016:

January 2017:

May 2017:

June 2017:

July/Aug/Sep 2017:

 

(c) License and Download User Instructions

November 2016:

March 2017:

May/June 2017:

July/Aug/Sep 2017:

(d) Addressing InstallShield Problems:

October 2016:

December 2016:

January 2017:

February 2017:

March 2017:

April 2017:

May 2017:

(5) Licensing

(a) Ongoing Licensing Work
(b) Viewer License Support
(c) Special User License Support
(d) License Related Internal and User Documentation
(e) Other Reprise License Manager Tasks

(a) Ongoing Licensing Work:

(b) Viewer License Support

July / Aug / Sept 2017:

(c) Special User License Support

October 2016:

February 2017:

(d) License Related Internal and User Documentation

October 2016:

December 2016:

April 2017:

 

(e) Other Reprise License Manager Tasks:

October 2016:

November 2016:

December 2016:

February 2017:

March 2017:

May 2017:

July / Aug / Sept 2017:

 

(6) Regression Tests

Regression tests are a suite of RiverWare models that were automatically run every night to look for changes in results or performance from new code that was checked in to the development area. Results of the tests were evaluated on a daily basis to identify unexpected changes in model results. The model comparisons performed in the nightly regression tests can show expected differences (for example, because a new method category may have been added). When this occurred, the regression tests had to be updated to reflect the current state of the development area so model comparisons did not fail going forward. In addition, every week, the daily history of each regression test was analyzed to determine if the run time or model size had significantly changed because of new development.

In addition to the regular regression test work, the following regression test items were addressed over the course of the year:

Nov 2016:  

With the approaching 7.0 prerelease, the regression tests were update more frequently and had to be fixed more often as the new release branch was activated.  

Dec 2016:   With the 7.0 release, the regression tests were update more frequently and had addressed more often as the new 7.0 and future 7.1 release branches were activated.
Jan 2017   The 32-bit builds on and regression tests stopped working because of configuration conflicts. These were fixed.
Feb 2017:   The regression tests and build machines were fixed when conflicts occurred due to changes to the Git ignore file.
Mar 2017:   The 32 bit build machines had to be updated regularly due to conflicts in the merge. This prevented the regression tests from running each night.
Apr 2017:   The 32 bit build machines had to be updated regularly due to conflicts in the merge. This prevented the regression tests from running each night.
Jun 2017:  
  1. The tvaOptRPL test exhibited an apparent slow down due to the new CPLEX version. CADSWES investigated and found that it was likely due to the number of processors used, but there is nothing to do to address it in the short term.
     
  2. The regression tests were updated from version 7.0.8 to 7.1 in the prerelease and from 7.1 to 7.2 in the development area. This required a lot of attention to get the overnight builds and test operating correctly on the correct development area.
     
  3. An optimization regression test was observed to have increased its run time by 30%. It was determined that this was due to the upgrade of CPLEX from version 12.5.1 to 12.6, which was required by the upgrade to the Visual Studio 2013 compiler, and further analysis led to the conclusion that this slowdown could be mitigated by adjusting the number of threads made available to CPLEX, and that no further changes were required.
Jul 2017:   The regression tests were watched closely during the 7.1 release period. They were updated with new version numbers and histories were monitored closely.
Sep 2016:   The regression tests were monitored and updated as necessary. In particular, the 32 bit development machine failed to pull when project files changed. These were pulled by hand and then the reconfigured to run.  In addition, in September, the regression tests were run with different RPL units by changing the argument used in –rplslotvalunits from the default "mixed" to "std" and "user". There were differences in both cases which were reported to the RiverWare development team via email.

(7) Bug Fixes

228 Total closed in Oct 2016 through Sep 2017.

193 Fixed RiverWare Bugs

Bug Description Closed
3976 RPL Palette needs common timestep buttons or new functions Nov 2016
5203 bug 5171 does not seem to be resolved Sep 2017
5478 Cant configure columns individually for Auto Max Turbine Q table Dec 2016
5488 Importing tables and periodic slots with Excel DMI Dec 2016
5535 RiverWare gets confused between similar Rule names Dec 2016
5565 Non visable input slots are propagating values Dec 2016
5604 RPL copy paste with break points causes assertion failure Nov 2016
5610 Copying a DMI Database. The copy is connected to the copied. Dec 2016
5624 Warning Statement causes assertion error when showing statements in set editor Nov 2016
5649 Crash opening dataset dialog on modified dataset Dec 2016
5650 Clearing WS after resetting database DMI edit dialog crashes Dec 2016
5652 Debug only database DMI manager diagnostics crash Sep 2017
5653 IntegerToString does not deal well with units Dec 2016
5654 Export Import of Database DMIs do not change references to datasets and namemaps. Dec 2016
5678 SolveTurbineRelGivenEnergyInflow exceeds the specified energy when near Hydro Capacity Feb 2017
5693 File chooser for workspace object export gives the current directory as the suggested file name Nov 2016
5713 Energy is being increased to MEL Feb 2017
5716 Geospatial view config dialog pops up error dialog when no background image path Nov 2016
5718 Problem with HypLimitSimWithStatus built-in function Dec 2016
5720 crash when linking a diversion object passthrough account to a reservoir storage account Oct 2016
5743 RiverWare can't find the help if RiverWare is started from a different directory Dec 2016
5750 Pressing "Stop" in the debugger does not disable the "Stop" Button Nov 2016
5751 Opt goal processing takes significantly longer on some goals with no apparent reason Nov 2016
5755 Reservoir Segmented 2 Layer Salt method gives no error message for missing initial Bank Storage Salt Conc Oct 2016
5756 Excessive memory issues formulating a goal Nov 2016
5766 Environmment variables do not work in Tabular Series Slot Reports Dec 2016
5772 DMI import of data onto expression slot should not be permissible Nov 2016
5778 Geospatial view is blank and there is no way to recover it Oct 2016
5784 Crash closing and loading large model file Nov 2016
5795 Hourly Model:Failed interpolation with Cheatham EV table using Storage/PlantPowerTable Oct 2016
5796 Ampersand (&) in Script name results in error upon import into model from a saved script file Oct 2016
5797 Internal Error dialog hidden by Run Aborted dialog; Cannot close any dialogs Oct 2016
5798 Unit Generator Power reports internal error when Generators Available slot not populated Nov 2016
5799 RPL Policy can be enabled or disabled within a run Oct 2016
5800 Accounting slots aggregated incorrectly Nov 2016
5801 No way to search and replace or rename within a function or rule Jan 2017
5802 Chart timestep slider does not scale properly when timestep is different from model timestep Dec 2016
5803 Links remain to slots that are no longer in use after changing methods on a reach object Dec 2016
5804 Crash adding Utility Group to Global Functions Set Oct 2016
5805 Assignment of a FullDateTime to a slot is made incorrectly Oct 2016
5807 Mass Balance slot values not always correct when combining flow and volume slots Nov 2016
5808 Mass Balance Summary slot flow units always default to cfs instead of unit scheme rule Dec 2016
5809 Importing doesn't overwrite existing rules. Nov 2016
5810 Can't export or copy and paste report groups. Nov 2016
5811 RPL set editor is not deleted / "show statements" setting not saved reliably Nov 2016
5812 Edit Links dialog does not allow link creation Nov 2016
5814 Applying slot filter to all objects can add slots to filter list that do not exist on other objects Nov 2016
5815 Crash starting model run Nov 2016
5816 Exceptions for unit type NONE do not get applied correctly Nov 2016
5818 Model Report stripped of contents after save and reload of model Nov 2016
5819 RPL "Common Values" menu is disabled when it shouldn't be Nov 2016
5821 Set Dimensions for a Table Slot does not resize the table immediately Dec 2016
5822 Delete Column from a Table Slot does not present a confirmation dialog Dec 2016
5823 Expression Slots always reduce the evaluation range by one timestep Dec 2016
5824 RiverWare crashes when opening function from RPL Analysis, deleting that function, and then saving the ruleset Oct 2016
5825 RiverWare stopped working when using RPL Set Comparison Tool Nov 2016
5827 Series Slot Notes bookmarks are repeating entries when notes column is shown Nov 2016
5828 In the Excel Dataset dialog, both radio buttons can be disabled Nov 2016
5830 Script Set Scalar Slot Value action does not always use the default value Nov 2016
5832 Adding an empty Periodic Slot as the first slot in an empty plot fails an assertion. Dec 2016
5833 Changing grid line color in plot preferences results in no grid lines until RiverWare is restarted Nov 2016
5834 Excel DMI with 2 column table crashes Nov 2016
5835 Memory leak when a RootSelection is created from text Dec 2016
5836 Plotting - Show Curve Tool Tips menu is not functionall Nov 2016
5838 Changing type of WITH variable crashes, QT5? Nov 2016
5839 Bad error message when Energy and Turbine Release are both input Feb 2017
5840 When the Water Rights Solver aborts, there is a long delay before the interface is usable Nov 2016
5842 RPL Set Comparison Read Set from Model always reads in RBS Ruleset Nov 2016
5843 Plot Editor/Viewer: Hide curve, legend button synchronization problem Nov 2016
5844 Plot shift-click on scale to time range does not bring all plots to front Nov 2016
5845 Accounting slot flag set to input Dec 2016
5846 Plot Membership selection Nov 2016
5847 plot editor legend area height is too small to show text when scrolling is required Dec 2016
5848 Evaluation of scalar slot expression fails unnecessarily when it involves a symbolic date/time Nov 2016
5849 Excel DMI can leave TableSlot in broken state Dec 2016
5850 Crash when re-running a water rights accounting model after an abort Nov 2016
5851 Crash opening series slot w/ expression RPL set Nov 2016
5854 Turning on Dispatch Management Slot diagnostics leads to crash Dec 2016
5855 Periodic Table Slot dialog has Delete Column menu enabled when no are columns selected Dec 2016
5856 Open Table Slot Dialog: column deletion operations create a phantom GUI column Jan 2017
5857 Disabled DMI datasets and slot selections are not saved with model Jul 2017
5858 Periodic Slot: Delete column doesn't adjust numeric column headers Mar 2017
5860 Bad data-entry behavior in selected, off-screen cell within an open SCT Dec 2016
5861 Locator window reference rectangle is too subtle, light gray. Dec 2016
5862 SCT font problem in RW7.0 Dec 2016
5863 core dump when trying to DMI Periodic slot to Excel Dec 2016
5864 Patterned legend and plot lines appear different if thickness is greater than 1 Apr 2017
5865 Plots in model reports are not scaled correctly, clipped. Dec 2016
5867 Automatically generated post-optimization ruleset is invalid Jan 2017
5868 Setting an integer Scalar slot with "Slider" Script action writes a float to slot Dec 2016
5869 Adding Comment to Item in RPL list disables right-click "Enabled" option; deleting comment deletes list item Jul 2017
5870 SCT's Default Font selection is not working well; Needs simplification. Jan 2017
5871 SCT pre-run divider is incorrect when run aborts Jan 2017
5872 Turbine Release Extrapolation during Sim results in RBS interpolation error Feb 2017
5874 Value for Script Slider for Scalar slot does not display minimum value Apr 2017
5875 Still possible to open the old Plot Page editor Jan 2017
5876 Crash loading new model file with database DMI dialog open Mar 2017
5877 SCT display issues when cutting (removing) a slot Jan 2017
5878 Time Agg Series Slot unit column label incorrect after config change, until slot re-opened. Feb 2017
5879 RPL Search and Replace of names does not work if Rule or Function editor is open Jul 2017
5880 Printed SCT has extra partial rows printed Jan 2017
5881 Script Manager creates new script even if user cancels the operation. Apr 2017
5884 Object Account Summary has phantom columns when accounts are removed from the list Jan 2017
5885 Muskingum Cunge routing does not get set up correctly Jan 2017
5886 Crash saving a model after deleting an object Jan 2017
5890 Right click in empty Model Report layout causes crash. Jan 2017
5891 Deleting a column on custom table slot on a simulation object blows away column labels Jan 2017
5894 RDF to Excel does not handle slash in environment variable correctly Jul 2017
5896 'RPL Sets to Compare' dialog spans two monitors and size cannot be reduced Apr 2017
5897 rpl set comparison shows only first difference within a rule Jun 2017
5899 RiverWare Crashes when using Ctl-Z to undo in initialization rule Jan 2017
5900 RiverWare shifts marker X-position back 1 day when editing Jan 2017
5901 Marker Label does not display unless a Line Style is selected Apr 2017
5902 Plot Pages created in the Output Manager are not saved in the model file Jan 2017
5903 Set Table Slot Value Script Action clears Value if Allow Editing is set and does not use the Value (for dates) set for the Action Jan 2017
5905 AggSeries Slot Add/Remove Column Operations Missing Feb 2017
5908 After switching to a different sheet, SCT scroll position can become out-of-synch. Feb 2017
5911 SCT: When 'Sheets' are used, the 'Go To' function doesn't always scroll correctly to the picked slot divider. Feb 2017
5912 Inserting columns in table slots with different units messes up the units on the existing columns Mar 2017
5913 SCT, switching Sheets, vertical time scrolling out of sync Mar 2017
5914 Model crashes when running in MRM mode Feb 2017
5915 Excel DMI reading table written by range crashes Jul 2017
5917 Convergence error with drift and large changes in headwater Mar 2017
5920 Note group manager dialog doesn't update when DMI creates groups / notes Jul 2017
5922 Incompatible units error message uses bad display units Mar 2017
5923 Alignment of plot grid with right axis is not working. Mar 2017
5924 Numeric display incorrect when Scale equals scale of unit with respect to standard unit (e.g. 1000 ML). Mar 2017
5925 Charts are too large, resizing makes them disappear Apr 2017
5926 Crash when running a model after synchronizing slots to new run period Mar 2017
5930 Optimization fails for some policies that look reasonable Mar 2017
5931 minor documentation error in OffsetDate Jun 2017
5932 Water user shows efficiency slot even when not a part of currently selected method Sep 2017
5933 Crash when evaluating expression slot that calls invalid function Mar 2017
5934 DMI check mark color Jul 2017
5935 ambiguous Column Labels radio-button state in Tabular Series Slot report dialog Apr 2017
5936 prepending a blank to name of Tabular Series Slot report hangs RiverWare Jul 2017
5938 delete script then run script from dashboard, RiverWare crashes Apr 2017
5939 script editor retains changes even after Cancel button is clicked Apr 2017
5942 RPL Set Comparison Tool - highlighting problems May 2017
5943 run-script controls in dialogs get stuck as being disabled Apr 2017
5945 RPL Comment bad wrapping behavior when comment has multiple paragraphs Apr 2017
5946 GetMinSpillGivenInflowRelease function has side effects for slope storage reservoirs May 2017
5947 GetMinSpillGivenInflowRelease handles flashboards differently than solveMB_GivenInflowRelease May 2017
5948 In Edit RPL Comment dialog, if click Cancel, edits are applied anyway. Apr 2017
5949 Loading model removes new unit type's standard unit rule Sep 2017
5950 Crash when copying and pasting RPL statements after showing statements in the set editor May 2017
5951 Crash when deleting a RPL statement after showing statements in the set editor Apr 2017
5952 RiverWare crash on close May 2017
5953 Plotting: Patterned lines sometimes look solid, esp. with thickness greater than one Apr 2017
5954 Line types and styles appear different in legend than in plot Apr 2017
5955 RiverWare crashes when applying changes in DMI editor Jul 2017
5956 POSAT crash when viewing solution info for a goal that is modified after the run May 2017
5957 Crash when exiting RiverWare after making RPL edits Apr 2017
5958 Creating SCT for Integer Indexed Series Slots does not always work May 2017
5959 Interpolation issue with power table with units May 2017
5961 WaterQuality methods stop solving May 2017
5962 keyboard shortcut (Alt+E) fails for series slot with expression May 2017
5963 Always sync with run period fails excludes initial timestep for series slots with expressions Jun 2017
5965 URGWOM AOP model run not executing successfully with Windows 10 OS Sep 2017
5966 Slot selection in database DMI not staying "off" after model save and reload Jul 2017
5967 Crash on Copy/Paste of empty <statement></statement> Jun 2017
5968 RPL Analysis for Global Functions Set shows cumulative values fom successive runs Jun 2017
5969 RPL Set Comparison Tool: function name drawing problem Jun 2017
5970 Crash in RPL Set Comparison Tool when a set goes away Jun 2017
5972 Optimization model aborts with arithmetic error Jun 2017
5973 Showing a tooltip raises inactive window obscuring other dialogs Jul 2017
5974 Incorrect reservoir solution during HypSim Jun 2017
5975 Rapid number enty on SCT can cause model crash Jul 2017
5976 Excel DMI crash reading TableSlot by range Jul 2017
5978 Assertion Failure in 3D Table Interpolation Jun 2017
5979 Editing plot axis changes Y axis range Jul 2017
5980 Importing plot causes axis to show different units Jul 2017
5981 Plot time axis settings lost after saving and reloading 'twice' Aug 2017
5982 Opening Agg element from slot gives pink internal error about Object Viewer tab Jul 2017
5983 Printing SCTs with Sheets: Garbage printed for slots on hidden sheets. (Similar to Gnats 5880). Jul 2017
5986 Distributed MRM isn't removing partial RDF files Jul 2017
5987 RPL Notes on opt goals are called Rule Notes Aug 2017
5989 Object viewer drag object tab cursor icon can be difficult to see Aug 2017
5990 SCT does not refresh correctly when paused in RPL Debugger and slot range changes Aug 2017
5991 Entering data while paused in the RPL Debugger crashes RiverWare Aug 2017
5992 Name Map Manager File->Close does nothing Aug 2017
5995 DMI Lock icon behavior is backward Aug 2017
5996 Output Canvas background image not displayed Sep 2017
5997 SCT priority column width not correctly sized when paused in RPL Debugger Aug 2017
5998 RPL Element Numbers: the 1 has a different font than the rest Sep 2017
6000 Run time is displayed incorrectly after stopping a run after Initialization only Sep 2017
6002 In SCT showing only a single flag in the flag legend prevents SCT from opening Aug 2017
6003 Copying and pasting changes values in cells Sep 2017
6005 WQ model fails when "calculating a non zero value for outflow salt mass but outflow is zero" which I think is a tolerance issue Sep 2017
6008 RootSelection serialization doesn't cope with double quotes in names Sep 2017
6009 Cannot add text to rule notes that already have text saved in them. Sep 2017
6010 NaN in Excel imported as integer to non-text slot in RiverWare Sep 2017
6012 Slot viewer allows text series slots to be configured Sep 2017

32 RiverWare Bugs Analized and Closed without source code changes in FY 2017

Bug Description Closed
3962 Assertion failed with Database DMI slot list Sep 2017
4067 DMI's dates do not update with run control Sep 2017
4328 RPL search and replace annoyance Jul 2017
5516 Likely memory corruption with repeated rules runs Dec 2016
5529 User defined reach account slot inflow method not refiring after dependents change Dec 2016
5549 Program unexpectedly shut down Dec 2016
5561 Distributed run MRM using different trace than normal MRM Dec 2016
5570 Output in accounting slots not cleared at run time if they are after the run end date Dec 2016
5571 Core dump from cut/copy/paste of RPL Dec 2016
5588 Java DSS server times out Dec 2016
5617 RiverWare crashes - on RBS with diagnostics turned on Dec 2016
5635 Model crash if initialized and then run Dec 2016
5659 6.7 skipping breakpoints Dec 2016
5677 Begin accrual date change on timestep change Dec 2016
5691 Storage being calculated as negative in groundwater objects in RW 6.7.3 Dec 2016
5699 DSS DMI Problem. Error says: "DSS read series failed: Error sending header "garbage here": Socket ins not connected" Sep 2017
5710 Unable to create supply between Aggregate Water User and Reservoir Passthrough Dec 2016
5725 Unexpected ruleset behavior Dec 2016
5785 RCL Diagnostic Logs lost when window resizes Oct 2016
5813 Loading a problem into POSAT doesn't work if active goals change Feb 2017
5829 WRS crash Nov 2016
5837 Model Report Crash Jul 2017
5852 Reach Object with Time Lag routing method doesn't solve following script clearing of inputs Nov 2016
5859 Water Rights Solver not working correctly with shared priorities Dec 2016
5887 Exporting and then importing an Excel DMI reorders the slot list Sep 2017
5888 Plot pages not formatting correctly in model reports. Jan 2017
5892 Program Freezes Jan 2017
5916 Inputs changed when double click on row of SCT sheet with custom summary rows Feb 2017
5921 Series slot dialog dialog doesn't fully update when DMI creates groups / notes Jul 2017
5927 some script errors are not reported when executed from dashboard or script editor Apr 2017
5971 RPL expressions with comments can overlap other expressions Jun 2017
5985 Crash when attempting to edit RPL from FEWS server instance Jul 2017

3 Fixed RiverSMART Bugs


Bug Description Closed
5563  RiverSMART not creating the scenario output folder  Sep 2017
5586  RiverSMART can't simulate non-distributed MRM  Sep 2017
5765  Some (seemingly) random slots are not saved to rdf files for different scenarios.  Dec 2016

--- (end) ---