CADSWES -- Release, Software Maintenance, and Development Environment Work Summary -- Fiscal Year 2016
Phil Weinstein, edit 1-10-2017 (a)

This document summarizes the maintenance work completed by CADSWES in these categories from October 1, 2015 through September 30, 2016.

  1. Releases, Patches and Snapshots
  2. Development Environment
  3. Software Maintenance
  4. Download, Installation, and Release Processes
  5. Licensing
  6. Regression Tests
  7. Bug Fixes

(1) Releases, Patches and Snapshots

Two major releases (6.8 and 6.9) and nine (9) patch releases of RiverWare were generated this year. The work involved in generating a patch release is very similar to a major release, except that a major release typically has an extended period dedicated to bug fixing prior to the release, and a more extensive set of release notes. Release notes may be found in the RiverWare online help.

Release Type Release Release Date
Patch 6.7.2 Oct 23, 2015
Patch 6.7.3 Nov 13, 2015
Release 6.8 Jan 28, 2016
Patch 6.8.1 Mar 1, 2016
Patch 6.8.2 Apr 13, 2016
Release 6.9 May 11, 2016
Patch 6.9.1 Jun 1, 2016
Patch 6.9.2 Jul 26, 2016
Patch 6.9.3 Aug 9, 2016
Patch 6.9.4 Aug 22, 2016
Patch 6.9.5 Sep 20, 2016

Sixteen (16) snapshots of the build development area were made over the course of the fiscal year to allow users to test newly implemented features before they become part of an official RiverWare release.

RiverWare 6.8: was prereleased on December 22, 2015 and released on January 28, 2016.
RiverWare 6.9 was prereleased on April 27, 2016 and released on May 11, 2016.

The tasks involved in generating releases include:

(2) Development Environment

The Development Environment category includes all of the associated software and tools used for developing RiverWare as well as their setup and interaction. Work completed this year in this category is presented below by major topic.

Microsoft Visual Studio C++ Development Environment

In January 2016, CADSWES purchased twelve (12) Visual Studio 2015 site licenses (ten copies of Professional Edition and two copies of Enterprise with MSDN). We will be able to "downgrade" our licenses to use Visual Studio 2013 (ten copies of Professional Edition and two copies of Ultimate Edition). We are currently using Visual Studio 2010 -- this was sufficient for our port from Qt 4.8.5 to Qt 5.5.1. Subsequent to that, we will want to move to Visual Studio 2013 for Qt 5.6 (or later) with QtWebEngine.

DevPartner

A trial copy of DevPartner was installed on a development machine to see if it would be useful in testing for memory problems during a crash on model load (Bug 5687). This was run with several configurations, but either did not detect any problem or else crashed before the problem line involved in this bug was reached. The one-week trial period was allowed to expire without purchasing a permanent copy of the software.

GIT source control software:

CADSWES Incoming FTP area cleanup:

The "FTP" area where CADSWES receives RiverWare models and other large files from users, generally for the purpose of diagnosing problems discovered by RiverWare users, was culled by CADSWES staff.

(3) Software Maintenance

The Software Maintenance category includes work required to maintain the RiverWare code that is apart from work funded to implement specific new features and new functionality. This can include, for example, removing obsolete code, porting code to use newer versions of libraries (like Qt) or to use alternate libraries, addressing compiler warnings and memory leaks, non-project improvement to online help, or changing existing code to improve maintainability and efficiency.

  1. Qwt Plotting Library Upgrade
  2. Plotting Appearance Improvements
  3. Plotting and Output Architectural Proposal
  4. RiverWare Optimization Documentation
  5. TCL Upgrade to solve Model Loading Problem
  6. RiverWare Qt4 to Qt5 Port

(a) Qwt Plotting Library Upgrade

Qwt is the open source C++/Qt library used in RiverWare to implement plot dialogs. RiverWare was upgraded to Qwt 5.2.3 to Qwt 6.1.2 (released Dec 2014) in October 2015 for RiverWare 6.8, and subsequently to Qwt 6.1.3 (released June 2016) in August 2016 for RiverWare 7.0. Several aspects of the newer Qwt (6.1) version had significant architectural differences, notably a completely revised legend API which accommodates the possibility of a custom implementation. No notable functional or graphical enhancements were introduced with this upgrade, but being current in our Qwt library version positions us better for the development of future plotting enhancements. This was also a prerequisite for the Qt 4 to Qt 5 port of RiverWare which was started in October 2015. (As of January 2017, Qwt 6.1.3 is still the latest available Qwt version).

Major porting tasks completed in October 2015 include:

  1. Building Qwt 6.1.2 from source code, and deploying it within our Windows development build system, including modifications of our win-config.pl tool. Both 32-bit and 64-bit Windows MSVC builds were required.
  2. Modifying RiverWare build source files, EngrObjs/riverware.pro and Makefiles/riverwarebase.pro.
  3. Creation of temporary stub classes for eliminated Qwt classes, primarily QwtPlotPrintFilter and QwtScaleTransformation.
  4. Various data type changes, including:
    1. Basic global data type substitutions. The new Qwt version relies more on basic Qt data types.
    2. QwtScaleDiv is now passed by value instead of by pointer.
    3. QwtSymbol is now passed by pointers to dynamically allocated instances.
  5. Modifications of printing and plot image export using QwtPlotRenderer.
  6. Modifications for changes to the QwtScaleTransformation class hierarchy. This is used only for our custom "probability scale" which is applied only to percentage series values.
  7. Extensive recoding of plot legends, many aspects, including special handling for Markers conditionally shown in the legend. The new Qwt library has a more extensible architecture for custom legend implementations.
  8. Adaptation of our custom code to support series having NaN values. NaNs are handled as gaps in the drawn slot curve.
  9. Recoding of special mouse operations in the plot canvas: rubber-band zoom and re-centering.

Although, as mentioned above, while no feature improvements were introduced due to this Qwt upgrade, we did enable "anti-aliasing" (smoothing) in RiverWare 6.8 for the drawing of slot curves within plots.

Additionally, an analysis was done on other simple ways of improving the appearance of plots. Some experimentation was done, using the new Qwt library, to eliminate the gap between the plot and its axes.

(b) Plotting Appearance Improvements

The border and spacing around and inside RiverWare plot panels, and in between the plot panel and the axes, have been removed.  This simplifies the appearance of plots. This was a followup from updating our Qwt plotting library from Qwt 5.2.3 to Qwt 6.1.2, also being done for RiverWare 6.8.

(c) Plotting and Output Architectural Proposal

Users have expressed the desire to have more user-friendly and better looking plotting in RiverWare. Sponsors have allocated funds for improvements, but before these are implemented, it seemed prudent to revisit the way that users create, save, and edit plots and other output devices. For example, there are multiple dialogs where the user can select slots to plot and configure the layout (E.g. 2X1 curves) of the plot. In both places, the user can edit the plot and the interaction between them is not very intuitive. In addition, the plots use a "Save" paradigm which has never been easy to explain or use.

(d) RiverWare Optimization Documentation

The RiverWare Help section on Optimization was expanded in the following sections:

Additionally, the following revisions were made:

CADSWES also drafted a revised outline of the Optimization Help. The purpose is to restructure the Optimization Help to make it more user-friendly and to make it easier to find essential information. The current draft is available here:

(e) TCL Upgrade to solve Model Loading Problem

TCL is a third-party software utilized in loading and interpreting RiverWare model files. In investigating bug 5687 where a crash was occurring on model load, it was found that the problem was occurring within TCL after a call to interpret a particular model file line. Trying to debug within TCL was not successful, so the version of the software was updated from 8.5.1 to 8.5.18 (the current 8.5 release). This update fixed the crash and allowed the model to load successfully.

Updating the TCL version required building the new source code with Visual Studio 2010 on both 32 and 64 bit platforms. Include paths and library paths in RiverWare were updated to point to the new version, and a new compilation symbol had to be introduced to allow RiverWare source code to compile successfully with TCL 8.5.18. Packages for 32 and 64 bit were created and incorporated into the win-config process so that development machines could be updated with the new TCL libraries and dlls.

With TCL 8.5.18, a slowdown of up to 10% was noticed in some of the regression tests that are run nightly for RiverWare. A potential fix for this was tried to eliminate the writing of continued separate lines for the data in long time series so that these lines do not need to be rejoined during loading. This did not improve the slowdown in regression tests.

A second attempt at a fix was successful, accomplished by upgrading TCL to the most recent 8.6 version (8.6.4). This fix required minor changes to some TCL method calls made from the RiverWare code, but resulted in restoring regression test run times to prior levels. The new version of TCL was incorporated into the RiverWare build process. This involved:

(f) RiverWare Qt4 to Qt5 Port

Most of the work for the upgrade of RiverWare from Qt 4.8.5 to Qt 5.5.1 was finalized in FY 2016. The resolution of some ensuing GUI problems was completed in the subsequent months (early FY 2017) for the release of RiverWare 7.0 in December 2016. Major FY 2016 RiverWare / Qt5 porting milestones and tasks are enumerated below.

October 2015:

November 2015:

The following document describes in detail this RiverWare porting and building work done in November.

January 2016:

February-April 2016:

May 2016:

June 2016:

July / August 2016:

(4) Download, Installation, and Release Processes

Flexera InstallShield is the software used to create RiverWare installation files. Many of the tasks described in the followng sections are related to InstallShield maintenance and support.

(a) InstallShield Intall File Preparation for RiverWare Releases
(b) Other Release Support Tasks
(c) License and Download User Instructions
(e) Addressing InstallShield Problems

(a) InstallShield Intall File Preparation for RiverWare Releases

(b) Other Release Support Tasks:

December 2015:

January 2016:

February 2016:

March 2016:

April 2016:

June 2016:

July 2016:

Aug: 2016:

September 2016:

(c) License and Download User Instructions

The Viewer License User Guide was completed in October 2015. This is the complete user instruction document for how to request a Viewer license activation key and how to activate the activation key to create the Viewer license file on the user’s computer.

Created also an instructions webpage (a PDF file) with a link to the Viewer License Guide and simplified instructions for users who are already familiar with the Viewer license request process.

Created a new online document "Which RiverWare license type to choose". This document describes the differences in details among the various types of RiverWare licenses available to users, so users can decide which type of license to choose from when purchasing a license. [Oct 2015 .. Jan 2016]

March 2016 updates:

(d) Addressing InstallShield Problems:

November 2015:

Provisionally fixed a problem with the Print button in the License Agreement window during installation process. The button is for printing a paper copy of the legal license agreement (or to PDF if a PDF print driver is available on the user's system). Applied the fix to all six snapshot (6.8), prerel (6.7), and release (6.7.X) InstallShield project files. However further testing revealed an error of not able to print when doing an actual installing from a generated setup file. Printing from inside of the IS program while running the "Test User Interface" step does work. This only happens on my development computer, not on other two computers tested.

March 2016:

May 2016:

July 2016:

Aug: 2016:

September 2016:

(5) Licensing

(a) Ongoing Licensing Work
(b) Viewer License Support
(c) Reprise Library /RiverWare Linking Method Change
(d) License Related Internal and User Documentation
(e) Other Reprise License Manager Tasks

(a) Ongoing Licensing Work:

(b) Viewer License Support:

October/November 2015:

(c) Reprise Library / RiverWare Linking Method Change

November 2015:

Reprise is the licensing software that is used by RiverWare. To avoid using the Reprise dynamically linked library file, which is susceptible to hacking to bypass the licensing, RiverWare was configured to link with static Reprise libraries. This means that the licensing code is compiled directly into the RiverWare binary executable file and does not rely on the separate Reprise dll file. This statically linked executable is being tested. If all aspects of license functionality work successfully, the static linking will be incorporated into the RiverWare build process.

December 2015:

Reprise is the licensing software that is used by RiverWare. The RiverWare executable developed to use static linking to Reprise was tested and found to work correctly. However, when a statically linked 32 bit executable was attempted, there were errors due to multiply defined symbols in the dongle libraries of Reprise and FlexLM (FlexLM is the prior license software used by RiverWare, which is still used for various reasons by some customers on 32 bit). In consultation with Reprise customer support, several approaches were tried to fix these conflicts, but none were successful. It was decided that the 32 bit build of RiverWare would continue to use dynamic linking to Reprise, but the resulting 32 bit executable would be placed on a password protected area of the RiverWare web pages and be available to customers by request only. This is due to security concerns from hacking of the Reprise dynamically linked library (DLL) file.

RiverWare configuration files were modified to link statically on 64 bit and dynamically on 32 bit. These changes were incorporated into the RiverWare build process by creating 32 and 64 bit package files, modifying the window configuration process for development machines, committing the necessary RiverWare source code changes, and updating all development machines with the Reprise changes.

Subtasks for these changes include:

  1. Tested the RiverWare executable which has RLM library statically linked.
  2. Created a testing InstallShield project file for generating the install file that does not include the RLM DLL file in the release.
  3. Revised the internal document Build Generation Guide.

February/March 2016:

Even after we statically linking the RLM library file into RiverWare in place of use of the RLM DLL file, it was discovered that license hacking continued to be possible. We worked with Reprise support on hacked RLM license server program files found on internet, reported by Jim P. Reprise recommended that we to implement rlm_auth_check() in our application to detect this hack. It is used by the application to verify the integrity of the license server. This call is not required by RLM, and is used only as an additional check that the license server has not been modified. Before we make this update, the current password protected download page seems to have been effective.

(d) License Related Internal and User Documentation:

January 2016:

February 2016:

March 2016:

April 2016:

June 2016:

Aug 2016:

 

(e) Other Reprise License Manager Tasks:

November 2015:

December 2015:

January 2016:

February 2016:

April 2016:

A new version of the Reprise License Manager software (Version 12.0) was incorporated into RiverWare. This involved revising RiverWare project files to include a new header file and to link in new Reprise libraries. New versions of RiverWare were built and tested for both the 32 bit and 64 bit Windows platforms. RiverWare code changes were committed to the code repository and the new Reprise libraries were incorporated into the configuration process for setting up and updating development computers. All development machines were updated with the new Reprise libraries. Individual tasks included:

May 2016:

June 2016:

July 2016:

Aug 2016:

Sept 2016:

(6) Regression Tests

Regression tests are a suite of RiverWare models that were automatically run every night to look for changes in results or performance from new code that was checked in to the development area. Results of the tests were evaluated on a daily basis to identify unexpected changes in model results. The model comparisons performed in the nightly regression tests can show expected differences (for example, because a new method category may have been added). When this occurred, the regression tests had to be updated to reflect the current state of the development area so model comparisons did not fail going forward. In addition, every week, the daily history of each regression test was analyzed to determine if the run time or model size had significantly changed because of new development.

In addition to the regular regression test work, the following regression test items were addressed over the course of the year:

Nov 2015:  
  • A slowdown of certain tests was noticed. After tracing through the logs, it was found that the updated version of TCL was the cause of this slow down. This will be investigated in December.
Dec 2015:  
  • The slowdown of certain tests that occurred in previous months was further monitored and found to be improved after the TCL version was updated to the current version.  In addition, the tests were updated and or restarted after the version number was increased for the pre-release.
Apr 2016:  
  • The regression tests were investigated as there were a number of failures due to optimization development. One test was failing only on a certain computer for a release executable when run overnight, but no other times. Time was spent investigating the regression test environment for a possible cause. Work will continue on this test in the future.
May 2016:  
  • The regression tests required additional attention due to the release and new optimization tests (see following) that were failing under certain conditions.
  • Two new regression tests were added based on BPA Big10 models, one based on a Real Time RBS model, and one based on a Short Term Planning Optimization model: BPA_RTBig10_RBS and BPA_STBig10Opt. These tests exercise the following functionality not previously included in other regression tests:
    • Seeded Optimization run using a Slot Cache
    • CreateSlotCach Rcl command
    • Optimization Seed diagnostics
    • Power Coefficient method in the Optimization Power category
    • Coefficients Table method in the Tailwater category
    • Opt Coefficients Table method in the Optimization Tailwater category
    • Constraint Based Single Timestep method in the Optimization Reserves category
    • With Reward Table statement with a Summation derived objective
    • Extensive use of Summation derived objectives
    • Barrier Method with Crossover solution for Optimization
    • Use of multiple threads (4) for Optimization
    • Export of Optimization Solution Analysis to CSV files
    • ExportOptAnalysisAsCSV Rcl command
    • Reordering of Optimization Goal set: statements, goals and policy groups
    • Export of CSV Output Device.
Aug 2016:  
  • Overnight Builds and Regression Tests for RiverWare migration from Qt 4.8.5 to Qt 5.5.1:
    Modified the overnight build Perl scripts to support Qt 5.5.1 as well as the new Active Directory users buil9080 and prel6811; set up the overnight builds and regression tests on kickapoo (new 32-bit machine); started to set up the overnight builds and regression tests on aspen (new 64-bit machine).

 

(7) Bug Fixes

The following tables enumerate the eighty-seven (87( bugs fixed in FY 2016 (Oct 2015 through Sep 2016).

Bug Description Closed
5589 RPL Search and Replace Dialog now shows statement names instead of rule names. 2015-10-21
5608 DMI Error For AggSeries Slots 2016-01-28
5627 Plot Auto-scale Scales to hidden slots 2016-09-19
5656 Output data in table series slots is not cleared when saving a model with no output 2015-12-28
5675 Monthly timestep issue with reservoir spilling 2015-10-22
5676 Reprioritize Opt Set - reordering a Goal relative to a With Statement is not supported 2015-11-17
5679 Build Supply Attributes not saved 2015-10-20
5681 Model Reports: heading text is always bold, even when 'normal' font style (weight) is specified 2015-10-05
5683 Output Canvas Flowlines bug 2015-10-09
5684 New RPL objects use default precision (8 digits) instead of RPL Set's precision 2015-10-13
5685 Opt PSA method selection in 6.7 can cause crash when running in RBS in 6.8 2015-11-17
5686 Head Based Percolation will continue if zero groundwater object storage 2015-11-09
5687 Model saved in 32bit RiverWare does not open in 64bit RiverWare 2015-12-01
5688 RiverWare shuts down when trying to import a .csv file 2015-11-03
5689 Output Canvas: Multiple Teacup Groups not honoring Maximum Bar Height 2015-11-19
5690 Unable to add columns to table slot with numeric headers 2015-12-07
5692 File chooser dialogs for Saving (Creating) and Unpacking RiverWare packages does not remember last location 2015-12-10
5694 Open Object - sort by Slot Type no longer works 2015-12-21
5695 Riverware hangs on subsequent debug runs 2015-12-30
5696 Tabular Series Slot Report not omitting slots properly 2016-01-05
5698 RPL search and replace dialog has a couple of minor issues 2016-08-05
5700 When running a script , if I click on the dashboard stop button, I get a crash (similar to bug 5551) 2016-01-25
5701 Crash: Loading Model having Scripts after having shown a Script Dashboard for a Script 2016-01-14
5702 SCT Custom Cell Colors: Can result in non-contrasting text color for flag letters 2016-01-19
5703 USACE SWD Flood Control Rule Share using incorrect reservoir 2016-01-22
5704 Computed Routing Coefficients resized and stuck at one row 2016-04-13
5705 Deleting multiple objects leads to crash 2016-02-11
5706 Deleting object display group doesn't update objects on workspace 2016-02-03
5707 Model runs in 6.6.6, aborts in 6.7.3 2016-02-05
5708 Output Canvas loses Text Type reference of Value and goes to Plain Text 2016-02-24
5709 Scripting error when creating a snapshot 2016-02-23
5711 Fail to unload initialization rules from model after import 2016-02-25
5712 Global time scroll gets lost after run 2016-02-25
5715 Mass Balance Summary Slot creates new Agg Series Slots when model run and saved 2016-03-10
5717 Database DMI Editor dialog changes expanded column width every time a tree view triangle is clicked 2016-04-07
5719 Looking up an invalid Turbine Release 2016-04-06
5721 NextDate function returns incorrect value when crossing new year 2016-03-18
5722 Locked SCT fails to open as locked 2016-07-07
5723 Series Slot with Periodic Input does not expand and populate after reloading model file 2016-03-22
5724 Accrual on Water User 2016-05-04
5726 Supply Links Visually Disappear/Reappear 2016-04-27
5728 Assertion Failures when Auto-Scaling Plot having Periodic Slot and Empty Series Slot 2016-04-07
5729 URGWOM initialization rule 'Set Crop ET Rates' singularity 2016-04-13
5730 Plot Legend Marker Items: Not always removed when hiding marker items; Can subsequently CRASH. 2016-04-15
5731 Model save as truncates a global function and then unloadable 2016-04-20
5732 Accounting Workspace Supply lines are not showing up 2016-04-27
5733 Plot Zoom/Pan broken in plots having only Periodic Slots and Empty Series Slots 2016-07-27
5734 Max Request slot not displayed on Edit Account... 2016-05-04
5736 Attempting to remove slots from Plot Page Output fails and causes crash 2016-05-05
5737 Plotting: Step curves are clipped misleadingly at edge of plot 2016-07-29
5738 Mass Balance Slots: Possible crash deleting multiple slots. 2016-07-11
5739 MRM with Ensembles from HDB does not work with Version 6.8 2016-06-02
5740 Open Account dialog Value column weirdly aligned 2016-06-10
5741 Evaporation and Precipitation Volume not calculating if Pool Elevation and Outflow are inputs 2016-05-31
5742 MRM Concurrent Run Trace and DMI Repeat Mismatch 2016-06-02
5744 Can't add RPL Utility Group when a Policy Group Item is selected 2016-07-28
5747 Copy/move of mass balance summary slot retains reference to original slot 2016-07-08
5748 Periodic Slot Plot Labels Changing Back to Default 2016-07-07
5752 Copy and paste of a slot does not copy description or notes 2016-07-05
5753 Infinite RPL 'While' loop decrementing date/time generates error and locks up. 2016-07-05
5754 The order of slots in .rdf output does not match order in output control file 2016-09-19
5757 Name Maps don't work for all columns of Aggregate Series slot 2016-09-19
5758 DSS server doesn't handle errors well 2016-07-18
5759 Accounting View not showing cross hatching the same at certain zoom levels 2016-07-29
5760 Closing an Open Account dialog using the red X is not the same as using the Close button 2016-07-28
5762 Tabular Series Slot Report configuration erased when model saved with configuration open 2016-09-16
5767 Generating Tabular Series Slot Report as a Script action using ExecuteScript RCL command causes error 2016-08-05
5768 Plot Y axis bounds do not persist if plot has no data 2016-08-05
5769 DMI import sluggish when DMIs are executed via a script action with SCTs open 2016-08-04
5770 RPL Search and Replace deletes "Search for:" when changing the search location 2016-08-05
5771 RPL Search and Replace reverts "case" if search performed 2016-08-05
5773 No preferred opt units for PowerPerFlow 2016-08-08
5774 Compute aggregate routing coefficients on control point without variable routing upstream not resulting in an error 2016-09-26
5775 Account displays are overlapping in new release 2016-08-08
5776 Replacing Initialization Ruleset from a file can cause a crash 2016-08-12
5777 bug in Save Model script function 2016-08-10
5779 Supply Links no longer show up in Accounting View 2016-08-18
5782 Receiving error message "cwMessageMgr Invalid message identifer: 701" during script execution 2016-08-31
5783 Reach No Local Inflow Solve Outflow does not always maintain mass balance 2016-09-23
5786 Timestep error when switching from daily to monthly timestep. 2016-09-19
5787 Diagnostics "copy message line" doesn't always work 2016-09-16
5788 Loading a model twice fails on the second load 2016-09-08
5789 Inflow Stage Table Look Up won't work with linked Tailwater Base Elevation 2016-09-20
5790 Synchronizing a multi-slot before the subslot 2016-09-19
5791 Model run fails at high flows and Max Outflow Flag at Kentucky 2016-09-20
5793 Printing SCT Slot Selection: Detail Rows for the Last Selected Slot are Omitted 2016-09-15
5794 RPL Reopen 'Recent Directory' menus mistakenly include environment symbol name 2016-09-15

--- (end) ---