Autocalibration¶

Hydraulic models need to be calibrated and verified based on the measured data, typically flows and pressures. Model calibration and verification is necessary in order to understand limitations of the model and ensure proper use of the model. The model can be calibrated in different ways including semi-automatic methods.

The purpose of this autocalibration-based tool is to support finding model settings, such as pipe friction, demands, isolating valves, that improve the match between the hydraulic model and observed data. The autocalibration is based on optimization algorithms that can run with any extended period analysis model.

These two algorithms are available:

SCE: The Shuffled Complex Evolution (SCE) method is a global optimization algorithm that combines various search strategies, including (i) competitive evolution, (ii) controlled random search, (iii) the simplex method, and (iv) complex shuffling.
DDS: The Dynamically Dimensioned Search algorithm (DDS) is an optimization algorithm designed for large set of parameters. It automatically scales the search to find good solution within the maximum number of model calls. The algorithm starts to search globally, and then becomes more local as the number of models call approaches the maximum allowable number. Candidate solutions are created by perturbing the current values in the randomly selected dimensions only. The dimension of search is automatically adjusted.

The autocalibration provides the following controls:

Pipe friction autocalibration
Water demand autocalibration (demand adjustment)
Finding closed links
Leakage adjustment.

These controls can be combined into the same autocalibration run even though it is unlikely doing different adjustments at the same time.

Each calibration process consists of several steps:

Definition of controls, e.g., pipe roughness groups and pipe group assignment
Definition of targeted pressure and/or flow values
Automated calibration of control parameters e.g., pipe roughness coefficients by autocalibration methods
Assignment of computed parameters to the model, e.g., calibrated pipe roughness values to the pipes.

Identification¶

A list of the Autocalibration parameters and buttons follows with a short description given for each of them. Note, that it is possible to insert multiple Autocalibration runs, each with its own settings, and then run the selected Autocalibration run by selecting it from the list. This is convenient when you e.g. need to calibrate different zones individually.

ID: The ID of the autocalibration run.
Simulation ID: This is the ID of the associated simulation (from the 'Simulation setup' editor), which is used as the base of the autocalibration.
Insert: This button is used to insert a new autocalibration into the list.
Delete: This button is used to delete an autocalibration from the list.
Run: This button is used to run the autocalibration currently active in the list.
Stop: This button is used to stop (cancel) the autocalibration that is currently running.
Report: This button is used to report the autocalibration results.

Methods¶

DDS Optimization parameters¶

This section describes settings for the DDS optimization method.

Figure: DDS Optimization settings

Maximum number of call: This is one of the stop criteria and it is the maximum number of model call during the optimization process. If the maximum number of model calls is reached, the optimization process will stop, and it will report the best solution that was found.
Optimizer seed: This is used to initiate random number generator. There is no need to change this value except in special cases.
Target objective: This defines another stop criteria which compares the actual value of the objective function and it stops when its value is below the specified target objective.
Use maximum run time per simulation: This can be used in special cases when the hydraulic simulation due to the "random" choice of control variables by the optimizer might take exceptionally long time to finish. In such case, if this criterion is used, the program will cancel the hydraulic simulation and it will use a high penalty for its settings for the optimizer.
Use maximum number of invalid solutions per simulation: This can be used in special cases when the hydraulic simulation due to the "random" choice of control variables by the optimizer results in unbalanced or hydraulically unstable conditions during the extended period simulation run. In such case, if this criterion is used, the program will cancel the hydraulic simulation and it will use a high penalty for its settings for the optimizer.

SCE Optimization parameters¶

This section describes settings for the SCE optimization method.

Figure: SCE Optimization settings

Maximum number of calls: This is one of the stop criteria and it is the maximum number of model call during the optimization process. If the maximum number of model calls is reached, the optimization process will stop, and it will report the best solution that was found.
Number of complexes: This defines the number of complexes (p) used in the optimization process. It is used to calculate the initial population (number of different model setups to run). The population size is s = p x m.
Number of points in each complex: This data entry defines the number of points in each complex (m) used in the optimization process. It is used to calculate the initial population (number of different model setups to run). The population size is s = p x m.
Minimum number of complexes: The minimum number of complexes is required when the number of complexes is reduced during the optimization.
Number of evolution steps: This defines the number of evolution steps taken by each complex before complexes are shuffled.
Number of points in each sub-complex: This defines the number of points in each sub-complex.
Optimizer seed: This is used to initiate random number generator. There is no need to change this value except in special cases.
Target objective: This defines another stop criteria which compares the actual value of the objective function and it stops when its value is below the specified target objective.
Maximum loops of convergence: This defines the maximum number of loops used in the optimization.
Minimum relative change in objective function: This is another stop criteria where solutions are compared in terms of a change in the objective function and the optimization stops when there is no improvement.
Use maximum run time per simulation: This can be used in special cases when the hydraulic simulation due to the "random" choice of control variables by the optimizer might take exceptionally long time to finish. In such case, if this criterion is used, the program will cancel the hydraulic simulation and it will use a high penalty for its settings for the optimizer.
Use hotstart file: This can be used to start the new autocalibration run from the results of a previous run. Select the file with the extension .dat that contains the previous results.
Use maximum number of invalid solutions per simulation: This can be used in special cases when the hydraulic simulation due to the "random" choice of control variables by the optimizer results in unbalanced or hydraulically unstable conditions during the extended period simulation run. In such case, if this criterion is used, the program will cancel the hydraulic simulation and it will use a high penalty for its settings for the optimizer.

Pipe frictions¶

In this tab, it is possible to define settings for calibration of friction in pipes (parameter 'Roughness' in the pipes editor).

One of the common calibration parameters in the pipe network hydrodynamic model is the roughness coefficients. Pipe roughness values may be estimated in two ways: using values from literature or directly from field measurements. To obtain initial estimates of pipe roughness through field testing, it is a good practice to divide the water distribution system into composite zones that contain pipes of similar material and age. Additionally, several pipes of different diameters should be tested in each zone to obtain individual pipe roughness estimates. The process of calibration ideally requires simulation over an extended period, such as a time range for the maximum day - not just the maximum and minimum hour for the maximum day.

The pipe roughness can be autocalibrated no matter which head loss formulation is applied in the simulation: Hazen-Williams, Darcy-Weisbach or Manning.

Each record in the table corresponds to a geographical location to calibrate (defined by a list of pipes). Each list / location defined in this table will be calibrated with a uniform friction, therefore different locations should be defined in the table whenever different frictions are expected, depending on the known physical characteristics of the pipes such as material, age, and diameter.

After running the autocalibration, it is possible to review the calibrated value and adjust it, before applying the desired value to the network.

Pipe friction will be calibrated only if some pipes groups are defined in the table.

Figure: The pipe friction calibration settings

Calibration of pipe friction involves the settings described below:

Control ID: This is identification name for the active group of pipes to be calibrated with its uniform friction value.
Is active: The active pipes group will be calibrated only when its 'Is active' option is ticked.
Selection ID: This is the ID of the selection list, containing the group of pipes to be calibrated. This list can be reviewed and edited from the 'Selection manager'. The '…' button opens a list showing the valid list of selection IDs.
Minimum friction: This is the minimum value of the friction allowed during the autocalibration.
Maximum friction: This is the maximum value of the friction allowed during the autocalibration.
Calibrated friction: This is the calibrated friction value returned at the end of the autocalibration process. This value is computed during the simulation and cannot be edited manually.
Approved friction: This is the friction value approved, which will be applied to the pipes network. At the end of the autocalibration process, this value matches by default the calibrated value. The modeler can then review the calibrated values and adjust the approved values, if necessary. The approved friction values will be actually applied to the pipes network only when pressing the 'Apply all frictions' button.
Description: This is an optional description of the pipes group.
Apply all frictions: This button updates the friction value for the various groups of pipes defined in the table, with their approved values. A group of pipes is updated only when its 'Is active' option is ticked.

Node demands¶

In this tab, it is possible to define settings for calibration of node demands' patterns.

Each record in the table corresponds to a specific pattern to calibrate. During the autocalibration, the pattern will be adjusted by a multiplier that will increase or decrease the water consumption for nodes that refer to the updated pattern.

For water distribution systems with multiple zones, such as pressures zones, distribution or DMA (district metered area) zones, it is recommended to develop individual patterns that are unique for the respective zone.

After running the autocalibration, it is possible to review the calibrated factor and modify it, before applying the desired value to the pattern.

Node demands will be calibrated only if some patterns to calibrate are added to the table.

Figure: The node demands calibration settings

Calibration of node demands involves the settings described below:

Pattern ID: This is the ID of the selected pattern to be calibrated.
Is active: The pattern will be calibrated only when its 'Is active' option is ticked.
Minimum factor: This is the minimum factor for correction of the pattern.
Maximum factor: This is the maximum factor for correction of the pattern.
Calibrated factor: This is the calibrated factor returned at the end of the autocalibration process. This value is computed during the simulation and cannot be edited manually.
Approved factor: This is the approved factor, which will be applied to the pipes network. At the end of the autocalibration process, this value matches by default the calibrated factor. The modeler can then review the calibrated factors and adjust the approved values, if necessary. The approved factors will be actually applied to the patterns only when pressing the 'Apply all factors' button.
Description: This is an optional description of the node demand pattern calibration.
Apply all factors: This button updates the patterns selected in the table, with their approved factor. A pattern is updated only when its 'Is active' option is ticked.

Closed links¶

In this tab, it is possible to optimize the Open / Closed status of selected links (pipes or TCV valves).

Unknown closed section or isolation valves cause often serious problems when calibrating the hydraulic model. This autocalibration option allows to pick multiple pipes or valves and test them for being potentially closed.

Each record in the table corresponds to a specific link. The autocalibration will return a proposed status, which can either be approved or rejected, before applying the desired value to the pattern.

Links statuses will be calibrated only if some links are added to the table.

Figure: The closed links calibration settings

Calibration of links statuses involves the settings described below:

Link type: This controls the type of link to be selected and calibrated. It can either be a pipe or a TCV valve.
Link ID: This is the ID of the selected pipe or valve to be calibrated, depending on the selected link type. The '…' button opens a list showing the valid list of links. The arrow button can be used to select the link by clicking on the map.
Is active: The link will be calibrated only when its 'Is active' option is ticked.
Current status: This is the current status of the link, before starting the autocalibration. This value is shown for information only, and cannot be edited manually.
Calibrated status: This is the calibrated status returned at the end of the autocalibration process. This value is computed during the simulation and cannot be edited manually.
Approved status: This is the approved status, which will be applied to the link. At the end of the autocalibration process, this value matches by default the calibrated status. The modeler can then review the calibrated statuses and adjust them, if necessary. The approved statuses will be actually applied to the links only when pressing the 'Apply all statuses' button.
Description: This is an optional description of the link status calibration.
Apply all statuses: This button updates the links selected in the table, with their approved status. A link is updated only when its 'Is active' option is ticked.

Leaks¶

In this tab, it is possible to optimize the node demands using the pressure dependent emitter coefficient of selected junctions ('Flow coefficient' parameter from the 'Junctions' editor).

The purpose of this control is not to simulate small background leakages that occur practically everywhere within the water distribution system but to identify major leaks that might be responsible for pressure loss that is measured but cannot be represented in the model using water usage data.

Each record in the table corresponds to a specific junction. After running the autocalibration, it is possible to review the calibrated emitter coefficient and adjust it, before applying the desired value to the network.

Emitter coefficients will be calibrated only if some junctions are added to the table.

Figure: The leaks calibration settings

Calibration of emitter coefficients involves the settings described below:

Junction ID: This is the ID of the selected junction to be calibrated. The '…' button opens a list showing the valid list of junctions. The arrow button can be used to select the junction by clicking on the map.
Is active: The junction will be calibrated only when its 'Is active' option is ticked.
Maximum emitter coeff: This is the maximum value of the emitter coefficient allowed during the autocalibration.
Calibrated emitter coeff: This is the calibrated coefficient returned at the end of the autocalibration process. This value is computed during the simulation and cannot be edited manually.
Calibrated leakage: This is the calibrated total leakage at the junction, returned at the end of the autocalibration process. This value is computed during the simulation and cannot be edited manually.
Approved emitter coeff: This is the approved emitter coefficient, which will be applied to the junction. At the end of the autocalibration process, this value matches by default the calibrated emitter coefficient. The modeler can then review the calibrated coefficients and adjust them, if necessary. The approved coefficients will be actually applied to the junctions only when pressing the 'Apply all coefficients' button.
Description: This is an optional description of the junction coefficient calibration.
Apply all coefficients: This button updates the junctions selected in the table, with their approved emitter coefficient / flow coefficient. All updated junctions are also changed to node type 'Emitter'. A junction is updated only when its 'Is active' option is ticked.

Targets¶

This tab contains the calibration objectives, which the simulation will aim to meet during the autocalibration process.

Each record in the table corresponds to a target, which is usually a location with measured data to match.

Figure: The calibration targets settings

Definition of targets involves the settings described below:

Target ID: This is the ID of the target.
Is active: The target will be used in the calibration only if its 'Is active' option is ticked.
Target data type: This list defines the type of the target. The following options are available:
- Tank water depth
- Pressure
- Flow
- Head
Objective weight: This is used to specify the weight of this target. Use different number if you want to prioritize one target over another.
Location type: This is the type of network element where the target applies. Depending on the target data type, it can either be a junction, tank, pipe, pump or valve.
Location ID: This is the ID of the junction, tank, pipe, pump or valve where the target applies. The '…' button opens a list showing the valid list of elements. The arrow button can be used to select the element by clicking on the map.
Measured data type: The target measured data can either be defined as a time series or a constant value.
Plot ID: When the target measured data is a time series, this time series must be specified in a plot in the 'Plots and statistics' editor, and the plot ID must be selected here. The '…' button opens the list of existing plots to select from. The 'Create/Edit' button will open the 'Plots and statistics' editor to either edit the measured data time series, or to review the time series comparison after the autocalibration. The selected time series will be compared to the results computed over the extended period simulation.
Evaluation type: When the target measured data is a time series, this controls how the goodness-of-fit between the measured and computed time series is evaluated:
- Root mean square error
- Standard deviation
- Pearson correlation
- Average
Constant value: This is the constant target value, when the target measured data type is a constant value. It will be compared to the steady state results during the autocalibration.

Outputs¶

This tab provides a list of computed controls and their optimized settings and a summary of targets with requested and computed values.

Report¶

The 'Report' button provides a XML version of the outputs, i.e. a list of computed controls and their optimized settings and a summary of targets with requested and computed values.

Figure: The autocalibration report

Examples¶

Several examples are provided here to illustrate how the autocalibration can be used.

Example 1 - Pipe friction steady state simulation¶

The treated water system contains a pump station, a long water supply pipeline consisting of multiple pipelines and a storage tank that is receiving the pumped water. The steady state model with assumed friction factor, entered as 0.1 mm (rugosity for Darcy-Weisbach friction formulation), gives the discharge of about 2500 l/s. Measured data show 2350 l/s.

To use the autocalibration for pipe friction, open the 'Autocalibration' editor and insert a new record, use the SCE method, for example, and accept all default settings but change the maximum number of calls to 5000.

Next, create a selection list called e.g. "CWP" (Clean Water Pipeline) and assign all pipes in the water supply pipeline from the treated water pump station pump station to the downstream reservoir into this selection.

Next, enter a new control into the 'Pipe frictions' tab:

Selection ID: "CWP"
Minimum friction: 0.1 mm (*)
Maximum friction: 1.0 mm (*)

(*) We know that with 0.1 mm the discharge in the pipeline is 2500 l/s and so it is apparent that the friction factor (rugosity) must be bigger than 0.1mm in order to give smaller discharge.

Figure: Example 1 - Pipe friction

Apply the following settings in the 'Targets' tab:

Target data type: select 'Flow'
Objective weight: 100%
Link type: select 'Pipe'
Link ID: enter a pipeline where the discharge of 2350 l/s is measured, it can be any pipeline along the water supply water main unless there are any turnouts (demands) in-between the pump station and the downstream storage tank
Measured data type: select 'Constant value' for steady state discharge
Constant value: enter the measured discharge of 2350 l/s.

Figure: Example 1 - Targets

Next, click 'Run' and the program will start optimizing the friction factor.

Figure: Example of simulation output in display

The report from the simulation shows that the simulation was completed and that the target flow was satisfied. Results are visible in the 'Outputs' tab.

Figure: Example 1 - Outputs

The target discharge was 2350 l/s, and the computed discharge is 2349.91 l/s. The calibrated friction factor is 0.766mm.

Finally, go to the 'Pipe frictions' tab, and review the calibrated friction value. You can either accept or adjust the approved friction, and then click 'Apply all frictions' to apply the approved friction factor to all pipes within the "CWP" selection.

Example 2 - Pipe friction extended period simulation¶

The treated water system contains a pump station, a long water supply pipeline consisting of multiple pipelines and a storage tank that is receiving the pumped water. The extended period (24-hours) model was developed with assumed friction factor, entered as C values (Hazen-William's friction coefficient). Time series of measured pressure are available at 3 locations within the distribution system.

To use the autocalibration for pipe friction, open the 'Autocalibration' editor and insert a new record, use the SCE method, for example, and accept all default settings but change the maximum number of calls to 5000.

Next, create selection lists to group pipes (e.g. called Group_A, Group_B, Group_C) within the pipe network into groups based on similar material and installation date. We will assume 3 groups in this example.

Next, enter 3 new controls into the 'Pipe friction' tab: one for each pipe group. Define the minimum and maximum values of C-friction factor as illustrated below for each group.

Selection ID: "Group_A"
Minimum friction: 10
Maximum friction: 140

Figure: Example 2 - Pipe friction

Next, enter 3 new records in the 'Targets' tab (one for each of the 3 measurements):

Target data type: select 'Pressure'
Objective weight: 100%
Junction ID: enter the junction node that corresponds to the pressure sensor location
Measured data type: select "Time series" for extended period simulations with time series of measured data
Plot ID: select the name of the Plot ID, defined in the 'Plots and statistics' editor, which contains the time series of measured data
Evaluation type: select one of the available options.

Figure: Example 2 - Targets

When 'Pipe frictions' and 'Targets' are defined, click 'Run' to run the autocalibration.

Figure: Example of simulation output in display

The report from the simulation shows that the simulation was completed and that the target flow was satisfied. Results are visible in the 'Outputs' tab.

Figure: Example 2 - Outputs

The program also generates a XML report with autocalibration results, which is opened with the 'Report' button.

Figure: Example 2 - Reports

To review the match between computed and measured pressure time series, go to the 'Plots and statistics' editor, where the measured data have been specified. In this dialog, select the results file from autocalibration run and select 'Pressure' from the item list. Once defined, the time series plot will superimpose computed and measured data.

Figure: Example 2 - Plots and statistics with autocalibration results

Finally, go to the 'Pipe frictions' tab of the Autocalibration editor, and review the calibrated friction values. You can either accept or adjust the approved frictions, and then click 'Apply all frictions' to apply the approved friction factors to all pipes within the respective selections.