Appendices

A.1. Introduction

In Chapter 5, the SDANP models for assessing the dynamics of social, technical, economic, environmental and political (STEEP) risks in transportation megaprojects were developed and major observations made from their simulated behaviour modes which replicate the existing problem entities (risks of project time and cost overruns and project quality deficiency) of the Edinburgh Tram Network (ETN) project at the construction phase in Scotland, the United Kingdom.

This appendix is about the major aspects of the model validation. It presents the final process that is to be carried out using the system dynamics methodology to address the objectives stated in Chapter 1. It is organized around six major sections, namely Introduction; Model Validation Process; Validation Scheme for the Integrated System Models; Data Validity; Policy Analysis and Design and finally, a brief Summary.

A.2. Philosophical Aspects of Model Validity

Validation of dynamic simulation models is one of the most vital phases in the process of modelling real systems. However, as is true for scientific theories in general, dynamic model validation also faces the problem that ‘correctness’ of a model cannot be proven. That means validation and verification of models is impossible (Sterman, 2010). The word ‘verify’ derived from the Latin word ‘verus’ means ‘truth’ and is defined by the Webster dictionary as ‘to establish the truth, accuracy, or reality of’. While ‘valid’ is defined as ‘well-grounded or justifiable’.

By these definitions, it can be said that a model can only deliver correct results in a specific setting (reproduce the behaviour of the original) and cannot constitute proof that it will work correctly in all or even other circumstances. As Forrester (1961, p. 123) states: ‘The validity (or significance) of a model should be judged by its suitability for a particular purpose. A model is sound and defendable if it accomplishes what is expected of it …. Validity, as an abstract concept divorced from purpose, has no useful meaning’. With regards to objective criteria for model validity, Forrester further states that: ‘Any “objective” model-validation procedure rests eventually at some lower level on a judgement or faith that either the procedure or its goals are acceptable without objective proof’. Greenberger, Crenson, and Crissey (1976, pp. 70–71) emphasized that: “No model has ever been or ever will be thoroughly validated. Instead, “Useful”, illuminating,” “convincing,” or “inspiring confidence” are more apt descriptors applying to models than “valid.” Sterman (2010, p. 846) confirmed this to conclude that: ‘Some modellers have long recognized the impossibility of validation in the sense of establishing the truth’.

The author, therefore, does not speak of the ‘correctness’ of the dynamic STEEP system models for transportation megaprojects in this book but only of their validity relative to their purposes in risk descriptions and assessment. This validity can be established by extensive scenario trials, but it is only true until evidence to the contrary appears.

A.3. Model Validation Process

Figure A1 illustrates a simpler form for the model validation process. The ‘problem entity’ is the system (real or proposed — e.g. for this research, a Dynamic Systems Approach to Risk Assessment in Megaprojects is considered as a problem entity) to be modelled. The ‘conceptual model’ is the mathematical–verbal representation (influence or causal loop diagram) of the problem entity developed for a particular study, and the ‘computerised model’ is the conceptual model implemented on a computer (dynamic simulation model). The inferences about the problem entity are obtained by conducting simulations on the computerized model in the experimentation phase.

Opens in a new window.

Figure A1

Model validation process.

There are three steps in deciding if a simulation is an accurate representation of the actual system considered, namely verification, validation and credibility (Garzia & Garzia, 1990; Law, 2003). Sargent (2003) defines ‘Conceptual model validation’ as the process of determining that the theories and assumptions underlying the conceptual model are correct and that the model representation of the problem entity is ‘reasonable’ for the intended purpose of the model. ‘Computerised model verification’ is the process of determining that the model implementation accurately depicts the developers’ conceptual description of the model and the solution to the model (AIAA, 1998). ‘Operational validation’ is defined as determining that the model’s output behaviour has adequate exactness for the model’s intended purpose over the domain of the model’s intended applicability (Sargent, 2003). Operational validity determines the model credibility. ‘Data validity’ ensures that the data necessary for model building, model evaluation and conducting the model experiments to solve the problem are adequate and correct (Love & Back, 2000).

A.4. Methods for Testing and Validating the Integrated System Models

In order to show that the STEEP risk system models represent the original system well enough for the model purpose, validity was demonstrated with respect to a wide variety of specific system dynamics tests promoted by Forrester and Senge (1980) which are adopted and modified from Sterman (2010, esp. pp. 859–861) to uncover flaws and improve the models. Table A1 summarizes the main tests used to build confidence in the models and the question addressed by each test.

It is necessary to distinguish three systems (real, model and hypothesized) that are mentioned in Table A1. The real system includes existing components, interactions, causal linkages between these components and the resulting behaviour of the system in reality. However, in most cases limited knowledge about the real system is known. The model system is the abstract system built by the modeller to simulate the real system, which can help megaproject managers, engineers and consultants in decision-making processes. The hypothesized system is the counterpart of the real system, which is constructed from the dynamic hypotheses models for the purpose of validation. The hypothesized system is created by and from the available knowledge of experts and/or the experiences of the stakeholders with the real system through the process of observation and reasoning.

A.5. Importance of the Integrated System Model Objective

The objective of the STEEP system models is to assess the dynamics of risk in transportation megaprojects and its impact on project performance with respect to time, cost and quality at the construction phase overtime. The risks considered are social risks (P_R1), technical risks (P_R2), economic risks (P_R3), environmental risks (P_R4) and political risks (P_R5).

A.6. Validating the Model Structure

All the tests listed in Table A1 have been applied to evaluate the structural validity of the STEEP risk system models. The models were constructed to understand the dynamics of STEEP risks on transportation megaprojects in the construction phase. These tests by no means are exhaustive but constitute the core of tests for the structural validity of the integrated SDANP simulation models. The purpose of these models is to describe and assess the impact of risks on project objectives of a transportation megaproject in Edinburgh (the United Kingdom) at the construction phase over time (the simulations runs from 2008 to 2015). The STEEP models are generally dynamic disequilibrium representation of STEEP risks identified in the ETN project. Although illustration of the applicability of structural validity tests being demonstrated in this book is on risks assessment in megaprojects, it can also be applicable to any simulation model built to support policy decision-making in similar complex dynamic systems with uncertain data.

A.7. Tests of Suitability

A.7.2. Parameter Verification

The values assigned to the parameters of STEEP models are sourced from the existing knowledge (literature and project documents) and questionnaire survey conducted on the ETN project.

Thereafter, data estimation was performed to derive numerical values for each parameter using Weighted Quantitative Scores (WQS) and the analytic network process (ANP) pairwise calculations. The estimated values for the parameters using the WQS and the ANP are the Risk Priority Indexes (RPIs). For illustration purposes, Table A2 lists the stock and exogenous parameters and their RPIs used in constructing the STEEP model.

A.7.3. Boundary Adequacy

The purpose of this test is to determine the important concepts for addressing the problems that are endogenous to the STEEP risk system models and to check for significant changes when boundary assumptions are relaxed (Sterman, 2000). As indicated in Chapter 3, the system boundary for risks (STEEP) models present system elements consistent with the purpose of models which were developed based on relevant sources such as literature review, interviews, expert opinions and questionnaire survey conducted on the ETN project.

As indicated in the integrated stock diagrams, all elements relating to risks of project time overrun, risks of project cost overrun and project quality deficiency are represented endogenously. Only elements such as social issues for the social risks model (Figure 5.27), ambiguity of project scope, unforeseen modification to project, cost estimation problems, engineering and design changes, supply chain breakdown and inadequate site investigation for the technical risks model (Figure 5.28), government discontinuity, economic recession, catastrophic environmental effects and project technical difficulties for the economic risks model (Figure 5.29), environmental issues from works and unfavourable climatic conditions for the environmental risks model (Figure 5.30), and political opposition, government discontinuity, protectionism and delay in obtaining temporary Traffic Regulation Orders (TROs) for the political risks model (Figure 5.31) are indicated as exogenous system. During simulation, the boundary adequacy was checked by modifying the endogenous risk element to exogenous and then to excluded. The reason was to observe the dynamic changes of the model outputs over time when the system boundary is extended so that policies can be analysed and recommended.

A.7.4. Dimensional Consistency

Mathematical equations involving dimensional quantities are correct only if the operations presented on both sides of the equation agree not only in terms of the numerical value of the quantities but also in terms of their units of measurement (dimensions). In formulating the equations for the STEEP models, the requirements of the dimensional consistency were used to:

• –

  Check the validity of the model equations.

• –

  Determine correct conversion factors.

• –

  Formulate model equations.

In checking the equations, a built-in function in the Vensim software was used to ensure that:

1. The mathematical expression used is legitimate.

2. Units on both sides of the equations agree after performing the mathematical operations. Where there was no agreement, the two possibilities considered were:

   1. The expression may be correct except for a conversion variable.

   2. The expression may be completely illegitimate.

After the dimensional analysis, it was noted that, not only are the values of the elements in the models based on the existing knowledge of the real system, but they are also dimensionally consistent.

A.7.5. Extreme Conditions

Individual STEEP risk system models have been tested against extreme values. For example, the actual construction period for the case study megaproject is 78 months (between December 2008 and July 2014). However, simulation for the models varied from 0 to 84 months (between 2008 and 2015). Beyond 78 months, there was no significant change of behaviour. Also, the structures of the models and outputs were plausible for extreme and unlikely combinations of levels of variables in the system. In the integrated sub models, exogenous variables for each model were set at a high and low of ±50% to test their robustness to extreme conditions. Outputs of individual models showed realistic trends and hence indicated no significant change in behaviours beyond the normal trend.

A.8. Validating the Model Behaviour

A.8.1. Behaviour Reproduction Test

As stated by Sterman (2010), the purpose of the behaviour reproduction test, are to:

• Produce the model behaviour of interest in the system qualitatively and quantitatively.

• Generate endogenously the symptoms of difficulty motivating the study.

• Generate the various modes of the model behaviour observed in the real system.

Figure A2 shows the baseline (current run) output from the system with all variables at their baseline levels. Since this chapter contains a number of these figures, the forthcoming discussion will explain the dynamic behaviour of the STEEP risks and how they are organized. At the bottom of Figure A2, there are a number of system variable names: (1) social risks; (2) technical risks; (3) economic risks, (4) environmental risks and (5) political risks. Each of the traces of these risks on the graph, labelled 1 through 5, represents each of their respective displayed variables. The scale on the left side of the graph (Y-axis) shows the scales for each of the traces. The X-axis presents the time scale in years. The time scope for the simulation is between year 2008 and year 2015, so the X-axis ends at seven years. In the baseline (current run) condition shown in Figure A2, the various patterns represent the desired level of impacts these risks have on the project performance of the case study megaproject at the construction phase with respect to time, cost and quality and are in tune with the real life situations.

Opens in a new window.

Figure A2

Behaviour reproduction test for the level of STEEP risks impacts on the system (all variables at baseline levels).

A.8.2. Sensitivity Analysis

System dynamics models are generally not sensitive to changes in parameters and behavioural relationships. Sensitivity analysis is performed to test the robustness of the model by ensuring that uncertainties and estimating errors do not significantly affect the overall behaviour of the model. Sensitivity analysis is to test the limits of the STEEP models and their ability to adjust to changes. According to Sterman (2010), a model is considered robust if its behaviour does not change drastically when a parameter or behavioural relationship is altered. In this research, the extensive tests conducted on the models revealed that, the models are not sensitive behaviourally. Visual inspection of the dynamic graphs showed that the patterns generated were similar to those generated by the current (actual) runs. However, the magnitude and value of the system variables changed when the values (RPIs) of the parameters were altered. There are three types of sensitivity: numerical, behaviour mode and policy sensitivity.

A.8.3. Numerical Sensitivity Analysis

The numerical sensitivity analysis is carried out by changing the parameters in the models such as the initial value of stocks and the value of the exogenous system variables. For each parameter, the numerical sensitivity test is conducted by reducing and increasing the value of the parameter by 25% (±25%). Parameters for stock and exogenous system entities of the STEEP models and their distribution functions used in screening the analysis of the entire MegaDS model are given in Table A3. To ascertain the sensitivity of each parameter, the dynamic simulation results of the system variables are compared with the base run (current run) results. Summaries of these results are presented in Tables A4–A8.

A.8.4. Behaviour Mode Sensitivity Analysis

After the numerical sensitivity, behaviour mode sensitivity analysis was performed to check model behaviour and to gain more confidence in the models. Using the changes in the parameter (stock and exogenous system variables) values, the modes of individual models were experimented with to see the resulting changes in behaviour under different parameter settings.

For example, in the social risks model, three parameters and their initial ANP’s RPIs were used to explore the sensitivity of the model. The three parameters are ‘Social Issues’, ‘Social Risks’ and ‘Social Grievances’. The comparative run for the social risks and the social grievances are shown in Figure A3. Although the three curves do not look exactly the same, changes in the parameters do not affect the general mode of behaviour of the system. All three curves for the social risks show a small decrease in the stock right after the step increase and then continue to behave in similar manner until year 2015. On the other hand, the three behaviour patterns for the social grievances show increase in stock and then a slow approach to equilibrium. The curves indicate that the faster project managers adopt proactive risk mitigation techniques, the faster the stock of social grievances will approach equilibrium.

Opens in a new window.

Figure A3

Behaviour mode sensitivity graphs for social risks and social grievances.

‘Technical risks’; ‘Economic risks’; ‘Environmental risks’ and ‘Political risks’ are other major parameters about whose level of impacts on the performance of the ETN project are uncertain. Figures A4–A7 show the comparative runs of these parameters.

Opens in a new window.

Figure A4

Behaviour mode sensitivity graphs for technical risks.

Opens in a new window.

Figure A5

Behaviour mode sensitivity graphs for economic risks.

Opens in a new window.

Figure A6

Behaviour mode sensitivity graphs for environmental risks.

Opens in a new window.

Figure A7

Behaviour mode sensitivity graphs for political risks.

It is noticed again that although the behaviour modes of these risks look different from one another and from the social risks and social grievances, the general behaviours have not changed. Even when each of these risks starts out with a larger or smaller amount of parameter values, the behaviours of the stocks of the models will not change greatly.

As expected, changing the values of parameters in the model produces certain differences in the behaviours observed. Also, the sensitivity tests indicate that some parameter changes result in ‘greater’, or more significant, changes than others. For example, compare Figures A6 and A7. In Figure A6, the changes in ‘Environmental Issues from Works’ and ‘Unfavourable Climatic Conditions’ produce little difference in the behaviours, while in Figure A7, the curves show the same behaviours, but at different values of the stocks. This measure of more significant changes is studied through sensitivity analysis. In all cases, however, it is the structure of the system that primarily determines the behaviour mode. In general, but with exceptions, parameter values, when altered individually, only have a small influence on behaviour.

Now what should be expected if works on the ETN project are not completed as per the revised completion dates in summer 2014? That means there is a need for simulation to continue for another few months or even years. If so then will the uncertainties and risks continue to grow larger with time or not within the extended time? The situations represented in Figures A4–A7 are not the most likely outcomes. With a limited time available for works, one would expect the STEEP models to exhibit various shapes of growth over time. Monte Carlo simulation helps to generate most likely outcomes with dynamic confidence intervals for the trajectories of the variables in the STEEP models using the ranges of the probability distributions for the parameters represented in Table A3. The results of the Monte Carlo simulation are represented in Figures A8–A12. The figures show the 50%, 75%, 95% and 100% levels for social grievances, technical risks, economic risks, environmental risks and political risks in a sample of 500 simulations.

Opens in a new window.

Figure A8

Dynamic confidence bounds sensitivity graph for social grievances.

Opens in a new window.

Figure A9

Dynamic confidence bounds sensitivity graph for technical risks.

Opens in a new window.

Figure A10

Dynamic confidence bounds sensitivity graph for economic risks.

Opens in a new window.

Figure A11

Dynamic confidence bounds sensitivity graph for environmental risks.

Opens in a new window.

Figure A12

Dynamic confidence bounds sensitivity graph for political risks.

Figure A8 shows the sensitivity analysis of the social grievances. There are 0.06 (6%) of initial grievances level at the time of simulation, and the base case simulation (actual run) shows the grievance level growing to around 19% after two years. The confidence bounds show the same general pattern as in the actual base run. There is a narrow band of uncertainty in the first quarter of year 2008 when the project commenced but the width of the interval grows in an equilibrium form over time. By the year 2010, the 95% confidence bounds suggest that the level of social grievances as a result of the construction activities range from a low of 18% to as high as 22%. The eventual equilibrium is found when the positive and negative loops come into balance.

Similarly, in Figure A9, the analysis reveals that the width of the simulation intervals continue to grow larger over time. Narrow range of the technical risks in the early years of project development is typical to systems that are dominated by negative feedback loops. Differences in the input parameters are eventually overridden by the actions of the negative feedback loops and hence the technical uncertainties may shrink over time.

In the case of Figure A10, the narrowing in the range of economic risks between year 2008 and year 2010 seems similar to that of the technical risks but much more dominated by negative feedback loops in the systems. By the first quarter in year 2010, the width of the interval started to grow larger. However, the graph declines steadily over time. A similar result is seen in Figure A11.

Finally, Figure A12 shows a sensitivity analysis for the political risks model. The analysis reveals that the width of the simulation interval continues to grow larger over time. Until the 7th year (2015) of the simulation, there is a 50% chance that the level of political risks will be between 15% and 55%. By the same year, the 75% and the 95% confidence bounds suggest that the level of political risks could range from 10% to 65% and 5% to 80%, respectively.

A.9. Other Tests

The remainder of the tests under the model behaviour test, namely behaviour Anomaly; family member; surprise behaviour; extreme policy; behavioural boundary adequacy and statistical character are all interrelated, so the STEEP risk system models were tested concurrently with their respective questions indicated in Table A1 in mind. The main goal for these tests is to determine if the model responds as expected under abnormal conditions and if the model responds (or does not respond) when system variables of each sub model are changed from their baseline values. Therefore, tests for effects from extreme policies were performed on each model. During these tests, the integration system variables were the sole focus, since this is the focus of the research. Comparative traces of baseline, minimum and maximum values on each associated graph were also performed. Vensim software allows the modeller to simulate the same variable multiple times under differing conditions for each simulation run to allow comparison. Testing consisted of multiple simulation runs (and, as a result, multiple traces) of the same variable and graphed under different model conditions. Hence, each of the simulation runs and traces were represented by different behaviour based on the extreme points for the system variables being studied. In these cases, the variable scale was the same throughout each figure.

A.9.1. Policy Analysis, Design and Improvement

Once the model is fully tested and its properties understood, the final step is to test alternative new policies for system improvement. The system improvement tests ask whether the modelling process helps to change the system for better. To pass the test, the modelling process must identify policies that lead to improvement; those policies must be implemented for improved performance of the system. A policy is a decision rule, a general way of making decisions. In practice, assessing the impact of a model is extremely difficult. It is hard to assess the extent to which the modelling process will change people’s mental models and beliefs. It is rare for clients to adopt the recommendations of any model promptly or without modification.

In this last step, alternative policies are designed and tested by simulation runs to minimize risks at the construction phase of transportation megaprojects. It must be noted that many other variables and conditions may change at the same time the new policies are implemented, confounding attempts to attribute any results to the policies. Performance improvement following a study does not mean the model-based policies were responsible; the system may have improved for reasons unrelated to the modelling process. Likewise, deteriorating performance after policy implementation does not mean the models failed since the outcome could have been even worse without the new policies.

To improve the system, the policy analysis and design are performed by altering one or more characteristics of the STEEP models and examining the resulting behaviours. Like sensitivity analysis, policy analysis can also be numerical or pattern oriented. Pattern-oriented policy analysis is naturally much more important, since the purpose of system dynamics studies is to improve undesirable dynamic behaviour patterns. Policy design is determining what changes in the model structure and parameters would lead to improved model behaviour. While choosing the policies, practicality and usefulness have been checked with the experts and industrial stakeholders working on mega transportation projects.

With regard to the STEEP models, it is argued here that four central characteristics make these models well-suited for learning about and designing effective policies:

1. the feedback approach and emphasis on endogenous explanations of behaviour

2. the disaggregation approach

3. the simulation approach

4. the fact that the models are manageable enough such that their structures are clear and the links between structure and behaviour can be easily discovered through experimentation. Each of these four characteristics is explored in turn.

A.9.2. Feedback Approach

First, the STEEP models share feedback loop approaches to modelling endogenous sources of behaviour. The models illustrate how megaprojects under construction can be affected by risks and can endogenously create the conditions for time and cost overruns and quality deficiency once social, technical, economic, environmental and political uncertainties become high, causing excessive impact on project performance. By emphasizing feedback and an endogenous perspective, these system models will help policy-makers understand how policy resistance can arise. The models challenge common beliefs about how systems work by revealing feedback loops that can exacerbate the situation, thereby facilitating learning for even the most overconfident users.

A.9.3. Disaggregation Approach

Second, the MegaDS model takes a disaggregation approach to modelling. This implies that the STEEP system models are heterogeneous and do not track each individual model in the group, but instead are grouped in disaggregation. In keeping with the system dynamics modelling tradition, the building blocks of the model structure are stocks and flows rather than individual agents. However, these models are statistically estimated from data based on individual STEEP risk characteristics and level of impacts on a transportation megaproject. As such, a more efficient analysis, involving a better set of explanatory variables, can be carried out directly using disaggregated (i.e. individual risk level) data and model relationships. These reasons led to the development of the disaggregated models into five STEEP risk system models. The MegaDS model has five stocks each for the social, technical, economic and political sub models and four stocks for the environmental sub models (see Figure A13). All the models have common detailed implications of time and cost overruns and quality deficiency that arise from the interrelationship of variables within each sub system.

Opens in a new window.

Figure A13

Disaggregation of the dynamic simulation models for transportation megaprojects. Note: The full figure can be found at: https://heriotwatt-my.sharepoint.com/personal/soo13_hw_ac_uk/_layouts/15/guestaccess.aspx?docid=0974bb3a6894a4271b31430b22fb7cac7&authkey=AZRBWHeTMYq4NSRcrwV4IJE

While there is much interest among modellers in an aggregated approach to the modelling of social problems, Rahmandad and Sterman (2008) argued that differential equation-based models — of which the models here are examples — are easier to understand and usually have similar policy implications when disaggregated. In addition, disaggregation reduces the models into manageable sizes, thereby decreasing the cost of developing and running complex models and allowing for easy but clear experimentation. Given limitations in individuals’ cognitive capacity, disaggregation also allows users to focus on feedback ahead of agent level detail and therefore develop a more holistic and endogenous perspective to the problem.

Further, recent research has shown that individuals often fail to understand the dynamics of accumulation (Sterman, 2008), with huge implications for the policies that they will then support. By focusing on stocks and flows as the building blocks of model structure, the STEEP models can directly help policy-makers build intuition regarding the dynamics of accumulation and thereby overcome one potential source of policy error.

A.9.4. Simulation Approach

Third, the reviewed models are running mathematical simulations that provide the opportunity to conduct experiments. While many lessons can be learned from a causal loop diagram, other more substantial insights require the development and testing of a simulation model. In both cases, simulation helps to illustrate why deliberate rational policies lead to policy resistance. In addition, the simulation models provide learning environments where modellers, policy-makers and other industrial stakeholders can design and test policies. Given the complexity of many policy environments, experimentation is essential for the design of effective policies. Simulations provide a helpful environment where policy-makers can experiment and learn about the effects of different policies without any significant social and economic cost for policy-makers.

Finally, simulations can help to build consensus surrounding difficult policy problems. By communicating the counter-intuitive nature of policy problems to policy-makers, simulations can encourage dialogue and lead to the development of shared interpretations regarding the source of problem behaviour. Even when different goals and value systems persist, simulation can help to focus the discussion on specific variables and outcomes that are the sources of divergence.

A.9.5. Manageable Model Size

Finally, the STEEP models are ‘manageable’. Here, we define ‘manageable’ to mean that the models consist of few significant stocks and feedback loops. There are two main benefits to these types of model sizes. First, they are small in size and allow for exhaustive experimentation through parameter changes. With these types of models, it is much easier to learn from sensitivity analysis (as shown in Figures A8–A12) and examine the interactions among different parameters. Thus, important leverage points in the system can more be easily identified.

Second, the manageable size ensures that the results of experiments can be fully and easily understood by policy-makers. Short exposition makes a holistic view possible. Due to the small size, individuals can see the feedback structure as a whole and not be frustrated by the need to track many variables and links at once. In addition, short exposition facilitates presentation of lessons to others, and helps bring the dynamic lessons to the meetings of stakeholders. Our emphasis on small models reflects that of Repenning (2003), who argues that in an academic context as well, small models are necessary to build the intuition of readers who are not accustomed to a dynamic or holistic view of systems.

In conclusion, manageable but small system dynamics models offer numerous benefits to the policy-making process during megaproject development. Table A9 summarizes the above discussion by depicting how each of the characteristics of the MegaDS models can help address the challenges inherent in policy-making during megaproject development.

A.10. Policy Implementation

Having studied the influences of critical system variables on project performance for various simulation scenarios overtime, the megaproject managers can now implement appropriate policies that best suit the situation at hand to assess risks effectively. For best and worst case simulation scenarios that can inform project managers to design effective risk mitigation policies, the behaviour mode sensitivity graphs represented in Figures A3–A7 for the ETN project are typical examples.

Other recommendations to contractors involved in mega construction projects for policy implementation are:

1. Megaproject contractors must obtain assurances from the relevant government departments in the host country, especially as regards the availability of consents and permits.

2. The central bank of the host government may be persuaded to guarantee the availability of hard currency for export in connection with the project.

3. As a last resort, but an exercise which should be undertaken in any event, by a thorough review of the legal and regulatory regime in the country where the project is to be executed to ensure that all laws and regulations are strictly complied with and all the correct procedures are followed with a view to reducing the scope for challenges at a future date.

A.11. Summary

This penultimate section has addressed the validation of the model developed in this research. Unfortunately, there is no set of specific tests that can easily be applied to determine the ‘correctness’ of the MegaDS models. Furthermore, no algorithm exists to determine what techniques or procedures to use because every new simulation project presents a new and unique challenge.

In this study, two major groups of tests (empirical and rational) were carried out and described with and without field data. The empirical tests were conducted to examine the ability of the STEEP models to match the historical data of the ETN project. The findings of these tests from the simulated results on the level of risks impact on project cost and time and quality compared to the real system suggest that the models reflect reasonable predictive fit and could therefore be generalized.

On the other hand, the hypothesized system and the model system are used to conduct a series of rational tests, such as: parameter-verification, structure-verification, extreme policy tests and sensitivity tests. Throughout this process, the concepts, methodology and the findings of the research have been found to be reasonably supported by the extensive use of the Vensim software tools in support of the study. It is therefore contended that the developed model has the potential for subsequent development and use by practitioners.

Finally, it can be said that validation is both an art and a science, requiring creativity and insight. But validation is difficult to comprehend and has diverse procedures, and is unavoidable as it is the evidence for the steadfastness and legitimacy of the model. This chapter has provided an insight on the widely approved schemes of model validation and techniques in practice. The validation schemes can be applicable to quantitative (mathematical/computerized) as well as qualitative (conceptual) models. But reliability of the model can only be ascertained as the model passes more and more tests. Also, the decision of accepting a model as valid cannot be left to the modeller alone and inclusion of the industrial practitioners involved in megaprojects development in the validation procedure should be obligatory. Researchers and practitioners may find this chapter quite useful as the procedures for validation discussed are quite generic, and hence may be applied to other dynamic models as well. The next chapter therefore concludes the research by providing a summary of the work done, drawing the main conclusions arising from the study and making recommendations for future research.

Structured Interview Questionnaire

A Dynamic System Approach to Risk Assessment in Megaprojects

Profile/Demography of Interviewee

Type of Organisation: ________________ Date: __________ Time: _______

Type of Megaproject: ________________________________

Size of Megaproject: _______________________

Designation: ___________________

• 1.

  Role/Responsibility of Interviewee

  1. What was your role on the Project?

  2. How long were you involved in mega construction projects?

• 2.

  Project Goal/Scope

  1. What were the main goals and objectives?

  2. How did the project scope change over time?

• 3.

  Generic Risk Events:

  1. What were the generic risk events inherit in the project?

  2. How did the generic risk events affect the project schedule overtime?

  3. How did the generic risk events affect the project cost?

• 4.

  Funding

  1. Was the project funding source a dedicated fund source?

  2. How were additional funds obtained as project costs increased?

  3. Was the funding source stable over time?

• 5.

  How can the qualitative risk effects on project performance be quantified and analyzed to reduce under performances in mega construction projects?

• 6.

  How effective were the risks assessment practices used in managing/modelling risk interrelationships in megaprojects during construction?