Follow up phase NISP follow-up, monitoring, control and adaptation Evaluation Illustration 19. Follow-up components Milestone This distance may result from the intervention of random elements and/or the governments or chosen organization’s handling of determined obstacles. In general, the monitoring and evaluation processes measure the distance between the orchestrated policy and the initial plan, and the economic effects generated by the executed policy.
Illustration 20. Inputs of the follow-up phase 2.6.1. Monitoring According to Phil Bartle (2007), monitoring is the regular observation and recording of activities taking place in a project or programme. It is a process of routinely gathering information on all aspects of the project. In this case, to monitor is to check on how NISP’s activities are progressing. Monitoring also involves giving feedback about the progress of the NISP to the stakeholders, implementers and beneficiaries of the project. Reporting enables the gathered information to be used in making decisions for improving the NISP’s performance.
It is important to consider that generally there is no data available to consider the long term effects of the NISP. Therefore, rather than the accurate evaluation of the NISP’s implementation results, a complete analysis or monitoring during several years is necessary.
Monitoring provides information that will be useful in:
Analysing the situation in the country or community;
Determining whether the inputs in the NISP are well utilized;
Identifying problems facing the NISP’s implementation and finding solutions;
Ensuring all activities are carried out properly by the right people and in time;
Using lessons from the experience to update the NISP, its strategies and tactics;
Determining whether the way the NISP implementation was planned is the most appropriate way of achieving the goals.
2.6.2. Evaluation Evaluation is a key phase, measuring and analyzing the impact of actions taken, to judge whether goals have been attained. In order to achieve effective evaluations, the departing situations or diagnostics have to be taken into account, in order to verify the changes that have been triggered by the NISP and its successive phases. Evaluation is not limited to the NISP’s application: it should take place in all the phases of the NISP. As a result of this process, it may prove necessary to establish corrective measures demanding the formulation of new policy guidelines and implementation of new strategic actions, taking situational shifts into account. The policy can thus be updated. It should also be updated after some years.
Evaluation is a process of placing value on what an NISP has achieved particularly in relation to activities planned and overall objectives. It involves value judgment and hence it is different from monitoring (which is observation and reporting of observations). It is important to identify the constraints or bottlenecks that hinder the NISP implementation in achieving its goals. Solutions to the constraints can then be identified and implemented.
Evaluation should provide a clear picture of the extent to which the intended objectives of the NISP’s actions and policies have been realized. Evaluation can and should be done during and after implementation.
Before implementing the NISP, evaluation is needed in order to:
Assess the possible consequences of the planned NISP to the country over a period of time;
Assist in making decisions on how the project will be implemented.
During the NISP’s implementation:
Evaluation should be a continuous process and should take place in all the implementation activities. This enables the organization in charge to progressively review the strategies according to the changing circumstances in order to attain the desired activity and objectives.
After projects’ implementation:
Evaluation should be used to retrace the NISP’s planning and implementation process and results after its implementation.
Due to the time inbetween the layout or planning and the effective instrumentation, the evaluation of technological and organizational policies becomes an additional tool to understand the faults in the process, from the elaboration of the NISP to its application.
Evaluating an NISP and studying its limitations can help formulate a new suitable policy which contemplates the real necessities of the country. In many cases, it is verified that the implementation difficulties are due to lack of coordination between the agents who act in the innovation system (companies, research centers, universities, NGOs) and financing institutions.
The second aspect of the evaluation is centered on the axis that links the policy with its economic effects. In this case, the evaluation aims to understand the way in which the implemented NISP directly and indirectly affected the performance of the participating agents, as well as other spheres of the economy. The first methods of evaluation were created decades ago in developed countries and were based, mainly, on quantitative analysis, using two tools: the “administrative information” of the companies to catch the policy’s impact on the sales, and the “cost-benefit” analysis to understand the relation between the financial gains and losses of the companies favored by the program.
However, it is considered that those two evaluation tools are limited because they summarize the impacts of the policy with a unique financial variable and do not grasp all dimensions of the process.
The difficulties in measuring the effects of innovation policies, such as NISPs, are due to the fact that the innovation factor is the result of a dynamic process that supposes both short and long-term articulations among diverse stakeholders. In addition this process deals with the establishment of an innovative institutional environment, as well as new regulatory policies; both effects that are not easily measurable by means of traditional cost-benefit analysis.
In addition to quantitative methods (surveys, questionnaires), it may be useful to employ qualitative evaluation methods, including interviews to key informers, questionnaires, surveys, and case studies.
Illustration 21. Processes of the follow-up phase Example 22. The Macedonian Strategy The Macedonian Strategy On September 21, 2005, the Parliament of the Republic of Macedonia adopted the National Information Society Development Strategy 1 (hereinafter “the Strategy”). The Strategy represents the result of numerous efforts and processes in which various entities took place from the domestic political scene, the civil sector, international organizations, as well as from the political processes. The National Information Society Policy of the Republic of Macedonia States the “Development of a process of permanent monitoring and evaluation of the achieved results in the development of the Information society, with an emphasis of mandatory usage of the feedback (indicators) to create the future policies, strategies and plans in the Republic of Macedonia”.
Source: Republic of Macedonia, 2.6.3. The use of indicators An indicator provides evidence that certain condition exists or certain results have or have not been achieved. Indicators enable decision-makers to assess progress towards the achievement of intended outputs, outcomes, goals, and objectives. As such, indicators are an integral part of a results-based accountability system16.
Indicators can measure inputs, processes, outputs, and outcomes. Input indicators measure resources, both human and financial, devoted to a particular program or intervention (i.e., number of case workers). Input indicators can also include measures of characteristics of target populations (i.e., number of clients eligible for a program). Process indicators measure ways in which program services and goods are provided (i.e., error rates). Output indicators measure the quantity of goods and services produced and the efficiency of production (i.e., number of people served, speed of response to reports of abuse). These indicators can be identified for programs, sub-programs, agencies, and multi-unit/agency initiatives. Outcome indicators measure the broader results achieved through the provision of goods and services. These indicators can exist at various levels:
population, agency, and program.
As for the criteria for selecting indicators, Horsch (2007) admits that choosing the most appropriate indicators can be difficult. Development of a successful accountability system requires that several people be involved in identifying indicators, including those who will collect the data, those who will use the data, and those who have the technical expertise to understand the strengths and limitations of specific measures.
Some questions that may guide the selection of indicators are:
Does this indicator enable one to know about the expected result or condition Is the indicator defined in the same way over time Are data for the indicator collected in the same way over time Will data be available for an indicator Are data currently being collected If not, can cost effective instruments for data collection be developed Is this indicator important to most people Will this indicator provide sufficient information about a condition or result to convince both supporters and skeptics Is the indicator quantitative Horsch, Karen: Indicators: Definition and Use in a Results-Based Accountability System, Harvard Family Research Project, 1997, http://www.hfrp.org/publications-resources/browse-our-publications/indicatorsdefinition-and-use-in-a-results-based-accountability-system As stated by Horsch, it is important to note that indicators serve as a red flag; good indicators merely provide a sense of whether expected results are being achieved. They do not answer questions about why results are or are not achieved, unintended results, the linkages existing between interventions and outcomes, or actions that should be taken to improve results. As such, data on indicators need to be interpreted with caution. They are best used to point to results that need further exploration, rather than as definitive assessments of program success or failure.
Some indicators systems developed by international organizations, and national and regional governments, are the following: OECDs Guide to Measuring the Information Society (OECD, 2009); the ICT Development Index (IDI) of the International Communication Union - ITU (ITU, 2009b) and UNCTADs "The Global Information Society: a Statistical View” (UNCTAD, 2008).
Illustration 22. Outcomes of the follow-up phase National system for evaluation and Output monitoring the NISP 2.7. Permanent evaluation: a key element in the whole process Working on an NISP does not finish with the final report or action plan. As a matter of fact, an NISP’s work continues through monitoring and permanent evaluation. The main criteria of evaluation should be the verification of the achievement of goals and objectives laid down in an NISP. These criteria should be relevant to each of the goals and objectives.
There are many methodologies to carry out assessments and evaluations. One of them is the outcome mapping, a methodology endorsed by International Development Research centre, IDRC, Canada. Outcome mapping provides not only a guide to essential evaluation map-making, but also a guide to learning and increased effectiveness, and affirmation that being attentive along the journey is as important as, and critical to, arriving at a destination.
It will help a program be specific about the actors it targets, the changes it expects to see, and the strategies it employs and, as a result, be more effective in terms of the results it achieves17.
Evaluation of an NISP also provides an assessment of the NISPs relevance, effectiveness and impact, efficiency and utility. An key aim of the evaluation is to assess the country’s added value of these initiatives; their impacts at national level and lessons to be learned that may inform work-programme development of the agreed time line.
The process of monitoring and evaluating progress in achieving the goals of an Information Society policy is decisive in actually implementing the chosen goals. Without some indication, signals, even warnings of how all elements of society are adapting to the installation and application of the NISP, there can be no way of understanding whether the shift towards the construction of an Information Society or its permanent updating is actually taking place or working in positive ways. Moreover, there can be no understanding of future policy steps without reference to the current status of the NISP implementation and application procedures.
A multistakeholder commission may be designated in order to periodically monitor and assess the NISPs efficiency and impacts.
Example 23. eEurope 2005 Final Evaluation eEurope 2005 Final Evaluation This evaluation contains the eEurope 2005 Action Plan, complementing the evaluation of the multi-annual programme of MODINIS (2003-2005). Its assessment includes three See Outcome Mapping: Building Learning And Reflection Into Development Programs, 2002, by Sarah Earl, Fred Carden and Terry Smutylo. This publication explains the various steps in the outcome mapping approach and provides detailed information on workshop design and facilitation. It includes numerous worksheets and examples.
different evaluation criteria:
1. Relevance and utility: whether the objectives of that programme corresponded to the needs, opportunities and challenges of society 2. Efficiency: examining the level of resource use (inputs) required to produce outputs and generate results 3. Impact: whether the intervention has created the intended effects Within each of these criteria a set of evaluation questions have been formulated to make the scope of the evaluation operational. The methodological approach is based on four types of analysis conducted in consecutive phases and makes use of multiple data sources; programme analysis, peer group analysis, country analysis and an impact analysis – developing an impact model.
Материалы этого сайта размещены для ознакомления, все права принадлежат их авторам.
Если Вы не согласны с тем, что Ваш материал размещён на этом сайте, пожалуйста, напишите нам, мы в течении 1-2 рабочих дней удалим его.