Embed Size (px)
Citation preview
Consensus: An Intelligent Platform for Decision Support
Evangelos Psomakelis1,2, Konstantinos Tserpes1,2, Angeliki Kopsacheili3, Moran Gavish4, Nikos Dimakopoulos5, Dimosthenis Anagnostopoulos1, George Yannis6 and Theodora Varvarigou2
1Dept. of Informatics and Telematics, Harokopio University of Athens, Omirou 9, Tavros, Greece2Dept. of Electrical and Computer Engineering, National Technical University of Athens, Iroon Polytecniou 9,
Zografou, Greece3European Union Road Federation
4Outbrain Inc. (at the time of writing, IBM Haifa Research Lab)5Athens Technology Center,Rizariou 10, Chalandri, Greece
6 Dept of Transportation Planning and Engineering, School of Civil Engineering, National Technical University of Athens, Iroon Polytechniou 5, Zografou, Greece
{vpsomak, tserpes}@hua.gr, [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
KEYWORDS
policy making, decision support, consensus, multi-criteria decision, machine learning, public opinion, social media, text mining, data analysis, european policies, sentiment analysis
ABSTRACT
Every real-world planning problem possesses several objectives that are typically subject to inherent conflicts with underlying tradeoffs. Policy makers are in need of tools that will utilize an overall analytical process, assisting in modeling the real-world planning process by obtaining the best attainable tradeoffs and facilitating efficient exploration. Consensus models existing real-world use-cases within the relevant policy-making context and employs measurable quantifiers in order to investigate how and whether preferable tradeoffs can be identified. Those quantifiers are being sought in multiple realms – such as analytical models, numerical simulations, statistical tools and even public opinion evaluators – in order to link the domain data to the set of objectives, and by that to reflect the expected success-rate of policies and their implementation. Furthermore, Consensus is investigating the balance shift of the objectives, in scenarios where certain resources are being deployed to primarily address one of them through EU- or international-level policies, covering two important real-world use-cases: EU Renewable Energy Directive and trans-European transport network guidelines. Consensus also seeks the citizens’ involvement in policy making, from formulating public opinion as one of the objectives in the model, to eventually playing the role of exploring the attained tradeoffs and contributing to their weighing.
1 IntroductionWhen dealing with complex decisions we have to consider a plethora of targets and an even greater quantity of factors that affect these targets. For example if the EU takes a decision to have a fossil fuel to biofuel balance of 40% - 60% by 2020 in the European Union countries, they have to consider how they could achieve that scenario. The methodology by which someone achieves such a high level goal is usually called a policy. This policy contains a series of smaller goals that can be modelled as achieving a set of objectives. These objectives can be anything that affects, or is affected by, the greater goal, directly or even indirectly. For example, when considering how to reduce the fossil fuel usage, one would say that they need to consider the price of imported biofuels, the change in greenhouse gasses emissions, the amount of cropland and forestland converted to biofuel farms, the change in price of food due to less croplands being used for crops and a plethora of other objectives.
These objectives form complex relations between them that are near impossible for someone to track and predict. This makes the work of policy analysts and policy makers a very difficult task with a lot of effort being wasted on predictions that are inaccurate due to miscalculated, missing or even erroneously
ignored data. Consensus tries to solve this problem by creating an easy to use, interactive platform that enables the policy analysts, or even the policy makers themselves, to explore the connections between all these objectives and watch predictions about how each choice in one objective affects the others. This process makes the different tradeoffs between the objectives obvious by employing advanced visualization techniques and mathematical models.
The complexity of relationships between the policy objectives is only half the problem. The other half concerns the people affected by each policy. For example, even if, in theory, a policy is predicted to work wonders in the natural environment and the economic situation of a country it may require the policy maker to increase the tax on fossil fuels. This small change in the whole policy structure could mean its own failure if the people deem it too aggressive and start resisting, actively or even passively, to the other objectives set by the policy. In fact, this “rebellion of the people” could lead to a complete reverse result from the one predicted in the models created by the analysts. That is the reason why the Consensus also takes into consideration the public opinion for each objective in a policy. By predicting the response of the public to a change in any objective we can create an aggregate response to the policy as a whole and predict if the citizens will accept or reject this new policy.
This paper is contributing to the machine aided decision support field by a number of factors:
Demonstrating that a multi-criteria decision support tool that operates on top of model-generated data can comprise a trustful and useful source of information for the decision makers.
Proposing an interactive way for the policy analysts to explore the tradeoffs in a policy model. Proposing an effective way of integrating the public opinion, as an objective measurement,
into any policy scenario. Creating a clear distinction between assumptions and objectives. Providing a novel policy model for decision support in the field of transportation policies.
The rest of the paper is structured as follows:
In chapter 2 we present a short reference to the current state of the art of the domains relevant to our work. We will mention the current research state of three basic fields; the multi-objective decision support, the policy modeling and the public acceptability modeling.
In chapter 3 we provide a description of our approach to policy modeling. We explain the proposed methodology in detail, allowing the reader to get accustomed to the Consensus framework, the models that are working in the background and the data they will be using.
In chapter 4 we present one of the use cases that Consensus was tested on, the Transportation policy scenario. In this chapter the implementation of the methodology described in chapter 3 is made clear, with realistic data and actual objectives defined by the experts in the field.
Finally, in chapter 5 we provide information on the evaluation of the Consensus project. We focus on rating the usability of the platform and the trust that our model inspires to the policy analysts and the policy makers. This evaluation is done according to international standards, with objective KPIs in order for it to be as objective as possible.
2 State of the Art2.1 Multi-Objective Decision SupportThe Multiple-criteria decision-making - also known as multiple-criteria decision analysis is a sub-discipline of operations research that explicitly considers multiple criteria in decision-making environments. For example: selecting public policy that maximizing efficiency in achieving its goals while minimizing tax-payers expenditures and minimizing negative environmental effects. For a nontrivial multi-objective optimization problem, there is no single solution that simultaneously optimizes all the objectives at once. In that case, the objective functions are said to be conflicting, and there exists a (possibly infinite number of) Pareto optimal solutions. A solution is called non-dominated, Pareto optimal (“Pareto efficiency,” 2016), Pareto efficient or non-inferior, if none of the
2
objective functions can be improved in value without degrading some of the other objective values. Without additional subjective preference information, all Pareto optimal solutions are considered equally good (as vectors cannot be ordered completely).
2.1.1 Multi-Objective OptimizationMulti-Objective optimization aims at simultaneously optimizing a number of conflicting objectives, and thereby revealing the Pareto optimal set or a region of interest in the trade-off surface between the objectives. This framework is unlike traditional optimization approaches that consider multi-objective problems by posing a weighted sum of its objectives and employ single-objective optimization to solve it (Das and Dennis, 1997).
Let a vector of objective functions in Rm , f ( x )=( f 1 ( x ) , f 1 ( x ) ,⋯ , f m ( x ) )T , be subject to
minimization, and let a partial order be defined in the following manner. Given any f (1)∈Rm, and
f (2)∈Rm, we state that f (1) strictly Pareto dominates f (2), which is denoted as f (1)≺ f (2), if and only if
∀ i∈ {1 , …, m }: f i(1 )≤ f i
( 2) Λ∃ i∈ {1, …, m }: f i(1 )< f i
(2). The individual Pareto-ranking of a given candidate solution is defined as the number of other solutions dominating it. The crucial claim is that for any compact subset of Rm, there exists a non-empty set of minimal elements with respect to the partial order ≼ (see, e.g., (Ehrgott, 2006)). Non-dominated points are then defined as the set of minimal elements with respect to the partial order ≼, and by definition their Pareto-ranking is zero. The goal of Pareto optimization is thus to obtain the non-dominated set and its pre-image in the design space, the so-called Pareto optimal set, also referred to as the efficient set.
The Efficient (Pareto) Frontier f is defined as the set of all points in the objective space that correspond to the solutions in the Pareto optimal set. The set that is jointly dominated by f but is not dominated by any other solution has Pareto-ranking 1, and so goes the ranking for subsequently dominated sets; following this notion the ranking of each solution can be defined (see, e.g., (Bader and Zitzler, 2011)).
The computational problem of attaining the Pareto Frontier of a multi-objective optimization problem (Papadimitriou and Yannakakis, 2000) can be treated by means of algorithms utilizing mathematical programming solvers (e.g., the so-called Diversity Maximization Approach (Masin and Bukchin, 2008)employing IBM's ILOG-CPLEX), or alternatively, approximated by population-based heuristics. The wide applicability of Pareto-driven optimization is evident in the vast number of published work - see, e.g., (Knowles, 2009; Knowles et al., 2007). The crucial claim is that many real-world problems are inherently multi-objective in nature. This concept ranges from Combustion Processes (Buche et al., 2002), Yeast Fermentations (O’Hagan et al., 2005) and Photoinduced Processes (Bonacina et al., 2007) to potentially as far as to Theory Choice (see (Kuhn, 1996) for the broad overview, and (Kuhn, 1977) for the explicit multi-criterion perspective).
2.2 Policy Modeling2.2.1 IntroductionThe creation of a model for complex and multi-objective policy scenarios is a field described by many as an art. The researcher often has to make assumptions based only on the field experience of experts or on her own experience of the policy world. In computer science though, we do not like this “gut feeling” aspect of policy making, so we are trying to emulate this unconscious process using complex models, that go all the way from the source information, be it textual data or expert interviews relevant to the policy scenarios, to the extrapolation of a policy option and its results on society.
2.2.2 Software Aided Policy MakingThis field has already been explored before in a large degree as we will see in the rest of this section. Firstly, we have to mention the most impactful deficiencies in the traditional policy making system, without, or with limited usage of smart aiding tools. In their work, Wimmer et. al. (Wimmer, 2011), identified six key weaknesses of this system:
a. Complex socioeconomic environments create a great volume of interconnected variables, unmanageable without the usage of proper tools
3
b. There are few to none appropriate ICT platforms that can be used to aid this process in a long-term policy planning case
c. Complex social or macroeconomic models provide little to no collaboration and transparency, prohibiting policy analysts from exploring alternative solutions and their impacts
d. Stakeholders are largely uninformed about the need for ICT solutions in order to collaborate efficiently with other stakeholders in the same domain
e. There are few to none ICT solutions that support both the policy modeling process and the collaboration among policy analysts and the general public
f. Research is not focused on developing and visualizing policy models in order to better understand the tradeoffs between each variable in them and have a more clear picture of the effects that each part of the model has if the policy option is implemented
The engagement of the public in addition to the stakeholders is actually one of the main goals in the work of Professor Pahl-Wostl Claudia (Pahl-Wostl, 2002). She is proposing an Agent Based Model (ABM) that simulates the process of a multi-opinion congress. Each agent in this model is an individual software component, simulating the opinion and actions of a group of people. These agents are based on actual behaviors, belonging to certain individuals, organizations or other groups of people, observed in realistic scenarios. Professor Pahl-Wostl (Pahl-Wostl, 2002) is mentioning five distinct difficulties, encountered when trying to integrate the participation of the public into a policy model:
a. There is no consensus on the importance of cooperative over individualistic strategiesb. Policy analysts often neglect the difference between subjective and objective knowledgec. The impact of emotions and the unconscious is largely underestimatedd. The ability of people to learn and change their point of view is in questione. The gap between realistic and expected human behavior is not always clear
2.3 Public Acceptability Modeling2.3.1 Basic PrinciplesWhen researching Sentiment analysis problems, the common procedure found in the literature starts with the discovery of the right data source for the researched problem, continues with the usage of Natural Language Processing ( NLP ) methods on the collected data and finalizes the process with one or more machine learning algorithms. The NLP methods are converting the natural language text that is collected from the different data sources into a vector of numerical attributes. These attributes are then fed to the machine learning algorithms in order to train them according to the sentiment polarity of each vector.
As sentiment polarity the literature defines a value that can show the researcher if the vector is connected with a positive, a neutral or a negative view on the subject of the text it came from. The point is to categorize the author’s opinion on the subject. A positive opinion could be happy or proud about the subject and a negative opinion could be angry or hateful against the subject. This polarity is most of the times either negative or positive (Godbole et al., 2007) but sometimes it can be neutral too (Aisopos et al., 2012). On less cases the polarity is represented by a numerical value that shows the weight of the polarity, how much positive or negative it is. Either way it converts the vector of extracted attributes into a single value that shows the author’s opinion.
2.3.2 Machine Learning AlgorithmsThe machine learning algorithms are necessary in tackling the sentiment analysis problems because they can categorize objectively an attribute vector. They can be trained to distinguish a positive, a neutral or a negative vector and make the decision faster than any human. This categorization can be split on more than one level for increased accuracy, as it will be presented at a later point. The most commonly used algorithms in the literature are the Naïve Bayesian Networks, C4.5, Hidden Markov Models and Support Vector Machines (Aisopos et al., 2012; Baum and Petrie, 1966; Cortes and Vapnik, 1995; Morency et al., 2011; Mullen and Collier, 2004; Psomakelis et al., 2014; Quinlan, 1996).
4
3 Proposed Policy Modeling3.1 IntroductionConsensus is meant to aid the policy analysts and the policy makers in their work and in no case replace a policy analyst. It creates a framework for quick comparison between different policy options and their effects on the social, economic and ecological domains. As such, the Consensus provides the end users with two browser based tools. The first tool is called ConsensusMOOViz and it is an interactive graphical user interface (GUI) in which the policy makers and policy analysts can easily make assumptions and set values to objectives and explore how each set of assumptions and values affects the rest of the objectives and their values. The second tool is called ConsensusGame and it is also an interactive web based GUI that allows everyday citizens to “play” with the different choices and tradeoffs that a policy maker has to make in order to form a policy. Each of these tools will be explained in depth in the following paragraphs.
3.1.1 ConsensusMOOVizMOOViz stands for Multi-Objective Optimization and Visualization Prototype and it is a framework for decision makers to better understand policy context, objectives, alternatives, trade-offs and decision consequences in order to make conscious choices that take into account the wide range of aspects and considerations that are effecting and being affected from the underlined decision context. Specifically, the ConsensusMOOViz tool is targeted to decision makers in the fields of transport, and more specifically road pricing, policies and biofuel policies, which are represented by the end user partners within the Consensus project. Nevertheless, the principles and approach aimed to be developed within Consensus, is generic and expandable into many other public domains involving of complex policy context decisions. The analysis is done in five discreet steps as presented here:
Policy context definition: the definition of policy assumptions and the objectives to be considered in order to form a context for the model.
Simulation and Optimization: the model explores all of the possible combinations of values set for each objective and creates a set of optimal combinations to be presented to the user.
Exploration: during this step, the user is presented with the optimal policy options (pareto frontier) as created in the previous step and she has the chance to explore the different options and see the predicted tradeoffs they entail.
Analysis: the systems shall provide means to drill down, assess, compare and analyse the different policy options in order for the end user to analyse in depth her preferred options.
Decision: in this, final step, the end user can eliminate options or just chose one or more of the options and present them as a valid choice.
3.1.2 ConsensusGameThe ConsensusGame targets everyday citizens in order to achieve two of the primary goals set for the Consensus project: to educate citizens about the hardships and difficult choices of policy making and to explore the preferences of the public in order to entail the public opinion in the policy modelling. This is done by using gamification technologies, which involve disguising scientific software as a game in order to more easily involve the citizens. In this paper we will not present gamification theories in depth but we will mention that its main focus is making a mundane work more fun in order to give incentive to a large number of people to do it.
Through the collected data, primarily the selections of the users while playing the ConsensusGame, we are able to derive conclusions about citizen preferences and relay them in a usable way back to the policy maker. Also by using a rating system, badges, achievements and tutorials we are interactively educating the “players” about the trade-offs that each policy option entails.
3.2 Policy AssumptionsAs assumptions we define all the required parameters in order to form a complete context for a policy option. These parameters, also called concrete context parameters in parts of the documentation, are defined before the modeling procedure and they form the context in which the trade-off analysis is performed. For the transportation scenario, for example, some assumptions include the population density of the area around the road, the length of the road, the owner of the road (public or private) and
5
others. These parameters have predefined values that stay unchanged during the modeling procedure, so the Consensus tools do not present comparison options for the trade-offs between them.
3.3 Policy ObjectivesThe objectives differ greatly from the assumptions. These parameters are defined before the modeling process starts, like the assumptions, but in the case of objectives we do not have predefined values. Their values are the target of the modeling process. The models assign all possible values to each objective, taking into consideration the restrictions imposed by the trade-offs and the context of each policy scenario (as a policy scenario we mean the set of defined objectives and assumptions). A policy option is defined as a vector containing one value for each objective defined in the related policy context. These parameters do not have predefined values but they usually have predefined ranges of possible values, so the Consensus tools can study the correlations between their values and present them to the end user for comparison.
3.4 Model InputsThe model uses input from the users as well as from databases, containing historical data or data provided by experts. The data contained in the databases form a set of basic scenarios on which the model will be building its predictions. These data are compiled from a wide set of sources, depending on the domain that we need to analyze. A common data source for many cases is Wikipedia, blogs and fora and of course data provided by experts on the field, either in the form of statistical data or in textual form during interviews. These data form the basis of the model.
Right before and during the execution of the model, the user needs to provide some additional data, again depending on the domain tested, in order to customize the model to fit into her preferences and context. The user can input her data in a number of forms and formats, such as textual, csv files, dropdown lists, check buttons, xml files e.t.c., depending on the domain and the models used. These data are:
the assumption values which assumptions are active the objectives that the model needs to analyze out of all the defined objectives in the targeted
domain
The user can also input her preferences between objectives, again in a variety of forms such as free text, drag and drop ordering, numerical ordering, weight assignment e.t.c., in order for the model to know which objectives are more important to the user. This is useful because each pareto-frontier policy option is achieving different objectives than the others so the model needs a set of criteria in order to provide a ranking between them. Without this ranking all pareto-frontier options are considered equally optimized solutions.
3.5 IntegrationThe Integrated Consensus Platform (ICP) is a web application that integrates all of the components created in the context of this project. It uses a loosely coupled architecture called MVC. Model-View-Controller (MVC) is actually a design pattern adapted by most of the web applications as the base architecture or the framework. MVC architecture is widely used as an architectural pattern for contemporary systems development and integration. This architecture is defined under different components that include the pattern specification, language specification and the implementation in the specific framework. The main feature of this architecture is the definition of three main components in a clean connection form. These three components are defined independently and with complete separation while providing an integrated connection between them and they are called Model, View and Controller.
6
Business Layer (Controllers)
External Services
Data Layer (Models)
Data Models
Presentation Layer (Web View)CONSENSUS Game Front-end
CONSENSUS Platform Controllers
Access CONSENSUS
Policy MakersGame Playes
Transport
DB
Game
Transport Scenario Controller
Biofuels Scenario Controller
Results
User Profile
Solutions
Biofuel User Profile
CONSENSUS Game Controllers
Transport Controller Biofuels Controller
Social Analytics
MOOViz Back-end
Scoring Mechanism
CONSENSUS platform Front-end
MOOViz Front-end Project Scenarios
RESTful API
RESTful API
Visualization Front-end
RESTful API
Visualization
Figure 3.1.1: MVC architecture for ICP
4 Transportation Scenario4.1 IntroductionUp to today most of EU transport infrastructure, have been developed under national policy premises and the lion’s share of investment, operation/management and maintenance costs comes from public budgets, through taxpayers contribution (national or European) (European Union Road Federation, 2006). Public budgets though can no longer fully sustain this financial effort. Due to the economic crisis, EU governments had to adopt severe austerity measures and national taxpayers are already suffering to be further burdened with road infrastructure related costs. In theory, when public money is scarce, the only available tool is relying to a greater extent on users to finance the infrastructure. In practice infrastructure charging is widely used across transport modes. In the railway sector, it applies for the use of the whole network. Charges are also levied for the use of airports, air navigation services, maritime and river ports. Road transport is, together with inland waterways, the only sector where charges apply still only to a small part of the network.
7
To this end the EU is promoting “user pays” principle, important to maintain and develop the European road network, and the “polluter pays” principle, as enshrined in the treaties, to encourage Member States to consider road pricing as a most effective and fair manner in order to:
pursuit a more efficient use of road transport infrastructures currently affected by congestion and consequent problems, and
adopt a much fairer, compared to taxes, approach to generate new sources of revenue to help develop and/or maintain Europe's vital road infrastructure; and in cases cross-finance cleaner and less energy consuming modes of transport.
Therefore, one of the basic challenges in all governmental levels (national, regional or local) of the EU countries, concerning road transport sector, is to seek a “balanced way” to include road pricing in the political agenda in order not only to overcome the present financing gap in road investments but also promote a new culture for sustainable development and operation of road transport network, without compromising individual mobility, environmental and social conditions. The aforementioned “balanced way” basically implies the development of a coherent decision-making evaluation framework to support the assessment of various types of road pricing and (hopefully) the identification of the most optimal one.
Figure 4.2: Transportation model structure
As we can see in Figure 4.1, the transportation model is using two kinds of data, input from the policy maker and stored data, in order to calculate the policy options and present them to the end user, be it a policy maker or a policy analyst. These data and the inner working of the excel based model will be detailed in the rest of this section.
4.2 Policy AssumptionsEach possible application scenario, else road pricing policy alternatives for a road project, is basically a (different) combination of the following components:
i. The project type: entirely new project or upgrade of an existing one.
ii. The project scale: corridor, facility or spot (bridge/tunnel) further differentiated by
Length: five possible strata (<20km, 20-45km, 45-75km, 75-100km and > 100km) Typical cross-section (lanes number/ direction): six possible cross-sections (1 to 6
lanes/directions)
iii. The application area: urban or interurban. Urban area is further differentiated by
Population size: small (population < 2.000.000) or large (population > 2.000.000)
8
4.3 Policy ObjectivesAs presented in Schwaab’s and Thielman’s work (Schwaab and Thielmann, 2001), funding of infrastructure and regulation of demand entails three dimensions: (a) economic sustainability (economic efficiency), (b) environmental sustainability (ecological stability) and (c) social sustainability (distributional/social equity). Valentin and Spangenberg (Valentin and Spangenberg, 2000), presented an advanced approach of sustainability; the so-called “prism of sustainability” (see Figure 4.1.1). This prism was based on a four-dimensional approach firstly introduced by the UNCSD in 1996 (Coordination and Development, 1996).
Figure 4.3.1: The prism of sustainability (Source: Valentin and Spangenberg, 2000 - pg. 383)
Still, this general approach had to be adapted to the specific needs of the transport sector. This was done by Omann (Omann, 2004) who adapted the prism of sustainability to the transport sector by changing ‘Institutional’ dimension into ‘Mobility’ dimension. The four corners thus became the economic, the social, the environmental and the mobility dimension. The mobility dimension represents the transport system and can be seen as the institution that is above the three usual dimensions or the apex to which all other dimensions are primarily related. The final assessment parameters, else the policy objectives, against which the policy alternatives will be comparatively assessed, are summarized (with their metrics) in Table 4.2.1.
Table 4.2.1: Objectives and their metrics for road pricing schemes evaluation
Objectives Metrics Metrics Measurement (type; units)
RP economic feasibility Relative investment cost Qualitative; Verbal Scale:
5- Very Low, 4- Low, 3- Medium, 2- High, 1- Very High
RP financial viability Relative Operational Cost Quantitative; in % of gross revenues
Reduce traffic congestion
Increase in Level of Service Quantitative; % decrease of ratio Traffic Flow/Capacity
Improve safety Reduction of accidents costs Quantitative; in % decrease of accident costs
Improve air quality Reduction of air pollution external costs
Quantitative; in % decrease of air pollution external costs
Reduce noise annoyance
Reduction of noise external costs Quantitative; in % decrease of noise external costs
Ensure user convenience
User convenience level in using the RP system
Qualitative, Verbal Scale: 5- Very High, 4- High,
9
3- Medium, 2- Low, 1- Very Low
4.4 Model inputThe reliability of model results depends on the accuracy and quality of data used. Data requirements and assumptions depend on the modeling technique and level of analysis in each stage. For instance, simplified modeling and strategic/quick response analysis requires less data (in term of both quantity and quality) compared to conventional models and operational analysis which are performed at a higher degree of detail and accuracy. Nonetheless even simplified models have some data requirements and each stage of the modeling process (Prototyping: Conceptual and Computing stages; Calibration stage; Validation stage) has specific data requirements.
Apart from the Perceptual stage, where no data requirements exist –in the strict sense- a brief summary of the data requirements is presented right next, per stage.
Conceptual stage:
(a) Socio-economic data, to be used as default values and/or coefficients of parametric equations of the conceptual stage, to ultimately provide estimates of traffic changes and traffic-related impacts changes resulting from road pricing policy alternatives.
(b) General Policy Option/s Information. Those directly describe the policies under evaluation. Such data are necessary to create the set of the alternative road pricing policy scenarios to be tested on the proposed road project. Such data include: road pricing policy, toll collection techniques, the authority responsible for road operation, etc.
Computing Stage:
(c) General Project Information. Description and characteristics of (i) the current situation (“base-case” scenario) that generates the need of a specific road project and (ii) the proposed road project on which the various road pricing policy alternative scenarios will be tested.
(d) Traffic Data. Current (ideally measured) traffic data for the “base-case” scenario road link (either to be replaced or to be upgraded). Such data are necessary to set the basis for estimating/forecasting the traffic changes -and the traffic-related impacts changes- resulting from road pricing policy alternatives.
During execution of the model a small number of extra data are required by the end user in order to set the parameters for each different execution, according to the user’s needs for each individual project.
4.5 Modeling ProcedureTransport models are predominantly used to predict transport demand under specific conditions’ changes (i.e. infrastructure provision, management measures implementation, pricing instruments enforcement). Two main model types are commonly used, namely:
Conventional, four-step transport models; the most well-known and commonly used in practice.
Simplified models; usually applied to make rapid progress in particular circumstances.
Regardless of the modeling procedure used, a common underlying assumption exists; the one that travellers make economically rational choices in deciding where to go (destination choice), what means of transportation to use (mode choice), and what route to take (route choice). In other words, all modeling processes assume that travellers choose among a set of alternatives and select those having the lowest generalized cost (a combination of monetary and non-monetary costs of a journey) (Spear, 2005). The generalized cost is equivalent to the price of the good in supply and demand theory, and so demand for journeys can be related to the generalized cost of those journeys using the price elasticity of demand. Supply is equivalent to capacity (and, for roads, road quality) of the network.
10
The basic equations of this stage include:
Demand drivers’ estimation; mainly generalised cost (incl. travel time cost and vehicle operating costs),
Demand changes estimation; as a function of specific drivers/factors affecting it (generalised cost and roadway capacity) as well as respective elasticity of demand,
Demand-related impacts (safety and environmental) estimation, per vehicle category and network type, based on demand changes and impacts’ unit values.
Based on the above equations the demand-related objectives mentioned earlier in this section are assessed. For the estimation of the non-demand related objectives’, direct valuation on an artificial scale takes place using simple utility functions and readily available data and/or opinions of domain experts.
5 Evaluation5.1 IntroductionConsensus is a complex system which has a vital dependence on user trust and acceptance. As such, we cannot apply standard evaluation techniques that target only the technical aspects of the platform; we must also consider the opinion of the end users. For that purpose we created a customized evaluation framework based on a combination of technical and behavioral evaluation elements, the basic principles of which we will present in this chapter.
5.1.1 Technical Assessment MethodsOver the last years numerous ICT projects [indicatively: eMPOWER, CITADIS, OASIS, OCOPOMO, Policy Compass, IMPACT, VIDI, OpenCUBE] had developed their technical evaluation frameworks on the ISO/IEC 25010:2011 “Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models” standard. The ISO/IEC 25010:2011 standard has replaced the previous standard on software quality, which was the ISO/IEC 9126-1, and provides a new view on how software (and thus software platforms) should be assessed.
5.1.2 Behavioral Assessment MethodsICT and IS literature provides several definitions and measures of success. As DeLone and McLean state, there are nearly as many measures as there are studies; obviously, there is no ultimate definition of IS success (DeLone and McLean, 1992). In particular this work considered two main research streams: Technology Adoption and IS Success. As for the first research stream, the reference theory is the so called technology acceptance model (TAM), proposed by Davis (Davis, 1989) and widely developed in the following years. The second research stream is about information systems success and was based on the DeLone and McLean model (DeLone and McLean, 1992) or Information System Success Model (ISS).
For the purpose of evaluating the user experience related to Consensus technology and services, the two noteworthy models, TAM and DeLone and McLean IS success model, were used, although not in the same manner. Obviously, this differentiation reflects the different character of each tool in terms of purpose and targeted-users.
5.2 Model ViabilityDuring the Consensus evaluation we tested the integrated platform against a variety of KPIs with various rates of success. Many of the KPIs were focusing on the viability of the models behind the platform, giving us an overview of the model viability. The results are presented in table 5.1.
Table 5.1: KPI Achievement Status
Functional Requirements Description Achievement StatusSetup and InputCM.SI.1 Context The tool should be capable of supporting the
agenda of the user in terms of the: project Achieved
11
description, alternative policy measures, policy objectives, evaluation criteria, indicators for measurement of criteria, priorities (weights) or visualization of preferences and public participation needs. According to the pre-defined lists of schemes, objectives, criteria, indicators, weights and visualization possibilities, the user shall be able to select/define her own. Nonetheless, especially for criteria there will be an option for the user to add/remove by her own.
Decision Making/AssessmentCM.DM.1
Generation of Alternatives
The tool will automatically generate or extract alternatives taking into account the scope and constraints defined by the user.
Achieved
CM.DM.2
Multi-Objective Evaluation of alternatives
Criteria priorities should be allowed (if the user wishes to) by introducing –and be able to experiment with- weights or by visual means to perform exploration of the alternative space in order to identify the best alternatives.
Achieved
CM.DM.3
Interactive Filtering Criteria
There should be a scale to interactively filtering out un-desirable alternatives through quantify or qualitative criteria.
Achieved
CM.DM.6
Knowledge/History database
The tool should take into consideration former decisions taken under similar circumstances. Therefore, the knowledge/history database should allow to store and consult available and suitable previous examples or role models for the case.
Achieved
5.3 Model CredibilityDuring the evaluation phases of the project, users from public sector authorities, from private companies acting as consultants and academia showed a positive impression and willingness to employ the tool, at least in the early stages of policy decision-making. Road operators’ representatives though, who deal directly with toll-charging/toll-collection, had a very cautious reaction. For them the tool definitely needs more sophisticated choices, in terms of inputs and policies, before considering using it. Obviously this difference in the “intention to use the tool” is explained by the difference in users’ scope of work but also from the fact that tool’s purpose, at least in the transportation scenario, was “the support of public authorities, relevant to the development, planning and management of roads, to comparatively evaluate and identify optimal Road Pricing schemes and understand the trade-offs between them -for further/more detailed assessment..”
We have a few KPIs that are involved in building trust: Reliability, Adaptability, Trust, Completeness, Understandability and Accuracy. Their scores are presented later in this section. As a comment on their scores though, we can extract the conclusion that this tool has all the requirements to start gaining trust. Of course trust gaining is a slow process, it takes many hours of work with the tool to build it, but as we will see later in the section, we have the amount of reliability, accuracy and understandability in order to start building a solid trust base.
5.4 KPI EvaluationThe Integrated Consensus Platform went through two evaluation phases with mostly slight differences in the KPIs. In this paper we will be focusing on the second and final evaluation phase, although we may mention data from the first as well.
The complete list of KPIs for the ICP and their scores are presented in table 5.4.1 bellow.
Table 5.4.1: KPI Evaluation
Dimensions & KPIs
Statement/ Question Evaluation Phases
Diff.
1st 2nd
12
User Assessment 3,80 4,39 0,59
System quality 3,88 4,44 0,56
Reliability The tool does not display redundant information and provides a good-response time to my requests 4,27 4,56 0,29
Adaptability The tool is flexible; it adapts the outputs when I change my input parameters and/or requests 4,00 4,63 0,63
Trust Tool’s concept and design gives me control over my tasks 3,36 4,15 0,79
Information Quality 3,64 4,32 0,68
CompletenessThe tool prevents me from taking unnecessary actions; I’m able to easily provide/reset input and set/reset my requests
3,55 4,48 0,93
Understandability It is easy to understand the tool’s operation and results 3,91 4,52 0,61
Accuracy The output information of the tool seems realistic. 3,45 3,96 0,51
Perceived Usefulness 3,82 4,24 0,42
Performance Using the tool could improve my performance at work 3,55 4,00 0,45
Relevancy The tool is relevant to my work 4,09 4,48 0,39
Perceived Ease of Use 3,86 4,57 0,71
Easy to use & manage
Interacting with the tool does not require a lot of my mental effort 4,00 4,70 0,7
Compatibility The tool provides sufficient information/help for using it 3,73 4,44 0,71
Behavioural intention 3,77 4,28 0,51
Effectiveness Using the tool could improve the effectiveness of my work 3,55 4,00 0,45
Response time Scrolling through the tool’s menu/s is kept to a minimum 4,00 4,56 0,56
User satisfaction 3,95 4,50 0,55
Contentment Overall, I am satisfied with my experience with the tool 4,00 4,48 0,48
Enjoyment Overall, I am pleased with the experience of using the tool 3,91 4,52 0,61
Impact Assessment 3,70 4,18 0,48
Individual Impact 3,64 4,44 0,8
Perceived improvement in
The tool provides the basic foundations for policy analysis –at early stages- (summarize input data,
3,82 4,48 0,66
13
policy analysis
chose policy objectives and identify policy options)
The use of the tool has reduced the time and resources it takes to create the basic foundation for policy analysis –at early stages- (summarize input data, chose policy objectives and identify policy options )
3,45 4,56 1,11
Clear –and early- identification of optimal or nearly optimal solutions (policy options) is supported by the tool; trade-offs between different policy objectives are identified.
3,55 4,52 0,97
Overall I rate the tool as an important support in the early stages of policy analysis 3,73 4,22 0,49
Organizational Impact 3,68 3,92 0,24
Perceived improvement in policy making
Could employment of the tool improve efficiency of road pricing policy making? i.e. reducing the time and money between alternative road pricing schemes analysis and implementation
4,20 4,00 -0,2
Could employment of the tool improve effectiveness of road pricing policy making? i.e. support sustainable development; achieve balance between costs and benefits.
3,80 4,00 0,2
Could employment of the tool improve transparency of policy making? i.e. clearness of decision-making process; choice of a road pricing scheme clearly justified based on its various impacts’ balance
3,60 4,06 0,46
Could employment of the tool improve inclusiveness of all related actors into policy making? i.e. from decision makers to citizens; citizens opinion could be included and as such support a specific decision
3,60 3,81 0,21
Could employment of the tool improve overall quality in the results of policy making? i.e. optimal or near-optimal results; what optimal means for a decision maker like you (less costly, more beneficial, most popular, a mixed balance?)
3,20 3,72 0,52
In an attempt to explain the differences in various KPI’s scores –that ultimately led in differences in the overall assessment score-, having in mind also on the comments received through the interviews, we can assume the following:
(a) System’s Quality scored much better than the previous time. This was mainly a result of the great improvement in “Trust” KPI’s score. MOOViz tool version in the 1st evaluation phase was criticized since it didn’t allow much user control in terms of the allowed choices (i.e. no. of lanes/direction) or the allowed range of values in traffic input parameters (AADT, speeds and considerable toll min/max values) but the major drawback was in the “exploration” of the results, in MOOViz page, where users could not experiment directly with objectives’ priorities. In the version of the MOOViz tool -that was evaluated during the 2nd evaluation phase- these problems have been mostly solved. More pre-defined choices and wider range of values were included in specific input parameters but the most important additions included ranking functionality, “weights” sliders and the “analyse me” feature that made it easily possible for the user to experiment with his/her own objectives’ priorities.
14
(b) Information’s Quality improvement was a result of much improvement into all KPIs’ scores. This was mainly a result of various improvements in the tool as well as in users’ engagement process. Concerning tool’s improvements, informatory text boxes added in User Interface (UI), the additional Explanation page informing the user about the road pricing policies to be examined, buttons explaining in short the difference between the Map and Lines view in the MOOViz page and finally the tutorial in MOOViz page explaining how to examine and interpret tool’s results upgraded users’ perception in terms of tool’s “Completeness” and “Understandability”. Finally, the rephrasing of the statement measuring “Accuracy” KPI, probably made it clear to the users that we need to evaluate output information in terms of being realistic –not necessarily accurate- and as such able to support a first comparative analysis of road pricing policies against specific policy objectives. The interviewed pilot users –especially those with a specialization in transport modeling- had also the chance to take a look of the respective domain (transport) model, which according to their short review might have certain limitations in terms of producing accurate results of road pricing policies impacts; nonetheless it can at least provide approximate yet comparatively reliable- estimates of their impacts.
(c) Perceived Usefulness and Perceived Ease of Use both scored better than in the 1st evaluation phase. According to the Integrated (TAM and IS Success Model) Behavioral Assessment Model for Consensus MOOViz this was expectable since they are both affected by improvements in the previous dimensions (System Quality and Information Quality). Especially the dimension Perceived Ease of Use, which is mostly affected by tool’s features (compared to Perceived Usefulness that is also affected by users’ specialization and/or exact line of work), showed the greater improvement due to improvements in MOOViz tool. As already mentioned above these improvements included informatory text boxes in User Interface (UI), the additional Explanation page informing the user about the road pricing policies to be examined, buttons explaining in short the difference between the Map and Lines view in the MOOViz page, the tutorial in MOOViz page explaining how to examine and interpret tool’s results and finally ranking functionality, “weights” sliders and the “analyse me” feature that made it easily possible for the user to experiment with his/her own objectives’ priorities. The aforementioned improvements in the tool, upgraded users’ perception in terms of easiness in using the tool including quality of the provided information/help of how to use it.
(d) Behavioral intention of using the tool scored also higher compared to the 1st evaluation phase. According to the Integrated (TAM and IS Success Model) Behavioral Assessment Model for Consensus MOOViz this was expectable since Behavioral Intention of using the tool is basically affected by the (improved) previous dimensions (Perceived Usefulness and Perceived Ease of Use). The exact same logic explains the improvement in User Satisfaction that is basically driven by the improvement in Behavioral intention.
(e) Individual impact’s better evaluation seems “statistically” normal considering that User Satisfaction, which is the decisive parameter for individual impact of an ICT tool/IS, scored also higher. It should be also mentioned that rephrasing of all statements measuring “Perceived improvement in policy analysis” KPIs, probably made it clear to the users that tool’s role should be limited to early stages of policy analysis; it should not replace i.e. a detailed modeling exercise or a full feasibility study or a Cost-benefit analysis.
(f) Organisational Impact’s evaluation was slightly better; it could be even better if the tool had scored better (compared to the previous time) in “improving the efficiency of road pricing policy making”. This can be explained by the fact that (a couple of users), with a strict toll-charging/toll-collection related background, do not believe much in high-level policy analysis but prefer direct and thorough cost-benefit analysis of options. And since interviewed users were only eight (8), two (2) of them is basically the 25% and that can affect the mean values.
15
ConclusionsAs we mentioned, the Consensus Integrated Platform was tested on two domains for the pilot evaluation of the project. These two domains were the Biofuel policies and Road Pricing policies. In this work we are mainly dealing with the road pricing domain. As mentioned in chapter 5, during the evaluation of the project, we measured a set of KPIs and we interviewed a group of end users in order to estimate the degree on which the Consensus platform achieved its purpose.
In detail we have two phases of evaluation, split into two parts each. These parts were the technical assessment, which rated the model viability and the behavioral assessment, which rated the project credibility. The model viability was tested by a set of functional requirements, the achievement status of which was considered our KPIs. The model credibility was tested by a set of individual KPIs, forming a complete assessment framework. Both of these parts of the framework are described in detail in chapter 5 of this paper.
The results showed that not only it started gaining their trust but they also recognized most of its results as accurate, they were seeing what their experience has taught them to look for. This fact is considered a success for every prediction model, so it proves that the Consensus platform and its models are an effective tool for decision support, thus providing a proof of concept for this project.
AcknowledgementsThis work has been supported by the Consensus project (http://www.consensus-project.eu) and has been partly funded by the EU Seventh Framework Programme, theme ICT-2013.5.4: ICT for Governance and Policy Modelling under Contract No. 611688.
ReferencesAisopos, F., 0001, G.P., Tserpes, K., Varvarigou, T.A., 2012. Content vs. context for sentiment
analysis: a comparative analysis over microblogs., in: Munson, E.V., Strohmaier, M. (Eds.), HT. ACM, pp. 187–196.
Arup, Oxera, 2010. How should the revised elasticity-based framework be implemented? Guidance report, Prepared for the DfT Transport Scotland and the Passenger Demand Forecasting Council.
Bader, J., Zitzler, E., 2011. HypE: An algorithm for fast hypervolume-based many-objective optimization. Evol. Comput. 19, 45–76.
Balducci, P., 2011. Costs of Alternative Revenue-Generation Systems. Transportation Research Board.Baum, L.E., Petrie, T., 1966. Statistical inference for probabilistic functions of finite state Markov
chains. Ann. Math. Stat. 37, 1554–1563.Bonacina, L., Extermann, J., Rondi, A., Boutou, V., Wolf, J.-P., 2007. Multiobjective genetic approach
for optimal control of photoinduced processes. Phys. Rev. A 76, 23408.Buche, D., Stoll, P., Dornberger, R., Koumoutsakos, P., 2002. Multiobjective evolutionary algorithm
for the optimization of noisy combustion processes. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 32, 460–473.
Change, I.P.O.C., 2007. Climate change 2007: The physical science basis. Agenda 6, 333.Coordination, U.N.D. for P., Development, S., 1996. Indicators of sustainable development:
Framework and methodologies. United Nations.Cortes, C., Vapnik, V., 1995. Support-vector networks. Mach. Learn. 20, 273–297.Danna, S., Mori, K., Vela, J., Ward, M., 2012. A benefit-cost analysis of road pricing in Downtown
Seattle. Evans Sch. Rev. 2, 26–46.Das, I., Dennis, J.E., 1997. A closer look at drawbacks of minimizing weighted sums of objectives for
Pareto set generation in multicriteria optimization problems. Struct. Optim. 14, 63–69.Davis, F.D., 1989. Perceived usefulness, perceived ease of use, and user acceptance of information
technology. MIS Q. 319–340.De Borger, B., Wouters, S., 1998. Transport externalities and optimal pricing and supply decisions in
urban transportation: a simulation analysis for Belgium. Reg. Sci. Urban Econ. 28, 163–197.De Jong, G.C., Tegge, O., Dohmen, R., Etterna, D.F., Hamer, R.N., Massiani, J., Van Vuren, T., 1998.
Review of existing evidence on time and cost elasticities of travel demand and on value of travel time (Deliverable 1) TRACE Costs of private road travel and their effects on demand, including short and long term elasticities. European Commission Directorate-General for Transport.
16
DeLone, W.H., McLean, E.R., 1992. Information systems success: The quest for the dependent variable. Inf. Syst. Res. 3, 60–95.
Department for Transport, 2004. Transport of Goods by Road in Great Britain 2003, Transport Statistics Bulletin. London.
Department for Transport, 2001. Transport Economics Note (TEN). London.Ehrgott, M., 2006. Multicriteria optimization. Springer Science & Business Media, Berlin.European Union Road Federation, 2006. The Socio-Economic Benefits of Roads in Europe, 2nd
Edition. ed. ERF Publication.Fleming, D.S., McDaniel, T.L., Grijalva, R.L., Sanchez-Ruiz, L.A., 2012. Dispelling the myths: Toll
and fuel tax collection costs in the 21st century.Friedman, D.A., Waldfogel, J., 1995. The administrative and compliance cost of manual highway toll
collection: evidence from Massachusetts and New Jersey. Natl. Tax J. 217–228.Godbole, N., Srinivasaiah, M., Skiena, S., 2007. Large-Scale Sentiment Analysis for News and Blogs,
in: Proceedings of the International Conference on Weblogs and Social Media (ICWSM).Greene, D.L., 1992. Vehicle Use and Fuel Economy: How Big is the“ Rebound” Effect? Energy J.
117–143.Gunn, H., 1998. ANTONIN: A FORECASTING MODEL FOR TRVEL DEMAND IN THE ILE-DE-
FRANCE, in: TRANSPORTATION PLANNING METHODS. PROCEEDINGS OF SEMINAR E HELD AT AET EUROPEAN TRANSPORT CONFERENCE, LOUGHBOROUGH UNIVERSITY, UK, 14-18 SEPTEMBER 1998. VOLUME P424.
Hansen, M., Huang, Y., 1997. Road supply and traffic in California urban areas. Transp. Res. Part Policy Pract. 31, 205–218.
HDR, 2009. Comparing Administrative Costs of Collecting Highway Revenues: Fuel Tax vs. Direct User Charge, prepared for the Office of Economic and Strategic Analysis. U.S. Department of Transportation.
HEATCO, 2006. Deliverable 5: Proposal for Harmonised Guidelines, Project: Developing Harmonised European Approaches for Transport Costing and Project Assessment, prepared for the European Commission.
IBI Group, 2007. Comparative Analysis of Toll Facility Operational Costs, Report for the Washington State Department of Transportation.
Jaako, P.I., HB-Verkehrsconsult, 2004. State of the Art Analysis of European Toll Collection Systems. IBTTA Fall Technology Workshop, Madrid, Spain.
JV SDG-NAMA-SALFO, 2003. Provision of Services of Specialized Traffic Advisor for Motorway Concession Contracts in Greece: Data Processing and Traffic Model Development, prepared for the Hellenic Ministry of Environment, Physical Planning and Public Works.
Knowles, J., 2009. Closed-loop evolutionary multiobjective optimization. IEEE Comput. Intell. Mag. 4, 77–91.
Knowles, J., Corne, D., Deb, K., 2007. Multiobjective problem solving from nature: from concepts to applications. Springer Science & Business Media.
Kuhn, T., 1996. The structure of science revolutions. Chicago and London: The University of Chicago Press.
Kuhn, T.S., 1977. Objectivity, value judgment, and theory choice. Arguing Sci. 74–86.Litman, T., 2012. Generated traffic and induced travel. Implic. Transp. Plan.Maibach, M., Schreyer, C., Sutter, D., Van Essen, H., Boon, B., Smokers, R., Schroten, A., Doll, C.,
Pawlowska, B., Bak, M., 2008. Handbook on estimation of external costs in the transport sector. CE Delft.
Masin, M., Bukchin, Y., 2008. Diversity Maximization Approach for Multiobjective Optimization. Oper. Res. 56, 411–424.
Morency, L.-P., Mihalcea, R., Doshi, P., 2011. Towards multimodal sentiment analysis: harvesting opinions from the web., in: Bourlard, H., Huang, T.S., Vidal, E., Gatica-Perez, D., Morency, L.-P., Sebe, N. (Eds.), ICMI. ACM, pp. 169–176.
Mullen, T., Collier, N., 2004. Sentiment Analysis Using Support Vector Machines with Diverse Information Sources. Barcelona, Spain, pp. 412–418.
Noland, R.B., Lem, L.L., 2002. A review of the evidence for induced travel and changes in transportation and environmental policy in the US and the UK. Transp. Res. Part Transp. Environ. 7, 1–26.
O’Hagan, S., Dunn, W.B., Brown, M., Knowles, J.D., Kell, D.B., 2005. Closed-loop, multiobjective optimization of analytical instrumentation: gas chromatography/time-of-flight mass spectrometry of the metabolomes of human serum and of yeast fermentations. Anal. Chem. 77, 290–303.
17
Omann, I., 2004. Multi-criteria decision aid as an approach for sustainable development analysis and implementation. na.
Pahl-Wostl, C., 2002. Participative and stakeholder-based policy design, evaluation and modeling processes. Integr. Assess. 3, 3–14.
Papadimitriou, C.H., Yannakakis, M., 2000. On the approximability of trade-offs and optimal access of web sources, in: Foundations of Computer Science, 2000. Proceedings. 41st Annual Symposium on. IEEE, pp. 86–92.
Pareto efficiency, 2016. . Wikipedia.PBS&J, 2009. WSDOT Statewide Tolls Program: Toll Operations Benchmarks and Incentives,
prepared for the Washington State Department of Transportation (WSDOT).Poriotis, N., Vakirtzidis, N., 2001. Study on Vehicle Operating Costs Estimation, prepared for the
Directorate of Road Construction Works Studies/Research, Physical Planning and Public Works. Hellenic Ministry of the Environment.
Psomakelis, E., Tserpes, K., Anagnostopoulos, D., Theodora, V., 2014. Comparing Methods for Twitter Sentiment Analysis. Proc. 6th Int. Conf. Knowl. Discov. Inf. Retr. - KDIR 2014 225–232.
Quinlan, J.R., 1996. Improved use of continuous attributes in C4.5. J. Artif. Intell. Res. 4, 77–90.Rothengatter, W., Doll, K., 2001. Anforderungen an eine umweltorientierte Schwerverkehrsabgabe für
den Stra\s sengüterverkehr. Forschungsbericht 200 96 130 UBA-FB 000217 Texte Umweltbundesamtes 5701 Berl.
Ruiter, E.R., Loudon, W., Kern, C., Bell, D., 1980. THE VEHICLE-MILES OF TRAVEL–URBAN HIGHWAY SUPPLY RELATIONSHIP. NCHRP Res. Results Dig.
Santos, G., 2000. On the economic, technological and political aspects of road pricing as a tool for traffic demand management, in: PROCEEDINGS OF SEMINAR C OF THE EUROPEAN TRANSPORT CONFERENCE 2000, HELD HOMERTON COLLEGE, CAMBRIDGE, UK, 11-13 SEPTEMBER 2000-DISCOVERING LOCAL TRANSPORT PLANS AND ROAD TRAFFIC REDUCTION. VOLUME P438.
Santos, G., Shaffer, B., 2004. Preliminary results of the London congestion charging scheme. Public Works Manag. Policy 9, 164–181.
Schnier, J., 1999. Stadtvignetten als Steuerungsinstrument im Strassenverkehr-Beurteilung und Wirkungsprognose am Beispiel der Stadt Koeln. SCHRIFTENREIHE FUER Verk. Tech.
Schwaab, J., Thielmann, S., 2001. Economic Instruments for Sustainable Road Transport. Overv. Policy Mak. Dev. Ctries. Dtsch. Ges. Für Tech. Zusammenarbeit GTZ GmbH.
Small, K., Winston, C., others, 1998. The demand for transportation: models and applications.Spear, B.D., 2005. A Summary of the Current State of the Practice in Modeling Road Pricing, in:
Expert Forum on Road Pricing and Travel Demand Modeling.Troyer, S., 2002. Verteilungswirkungen der Mauten im Alpenraum. na.Valentin, A., Spangenberg, J.H., 2000. A guide to community sustainability indicators. Environ.
Impact Assess. Rev. 20, 381–392.Victoria Transport Policy Institute, 2013. Transport Demand Management Encyclopedia - Road
Pricing. http://www.vtpi.org/tdm/tdm35.htm.Vovsha, P., Davidson, W., Donnelly, R., 2006. Making the state of the art the state of the practice:
advanced modeling techniques for road pricing, in: Expert Forum on Road Pricing and Travel Demand Modeling.
Wimmer, M.A., 2011. Open government in policy development: from collaborative scenario texts to formal policy models, in: International Conference on Distributed Computing and Internet Technology. Springer, pp. 76–91.
Winston, C., 1985. Conceptual developments in the economics of transportation: an interpretive survey. J. Econ. Lit. 23, 57–94.
Winter, M., Hirschhausen, C. von, 2005. Multi Functional Urban HDV Road Pricing–An Empirical Analysis of Berlin. Final Rep. LUS WP 6.℡
18
