Monitoring is an instrument for systematic collection of data on specific indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives.

Evaluation is the systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability (OECD DAC (2002)).

Controlling is often been limited to controlling of the budget, schedule and resources. However, in a policy context controlling also includes the monitoring and evaluation of policy impacts and outputs results in the desired quality. Another relevant aspect is to ensure the the internal and external information flow is provided and updated (communication management).

Monitoring and evaluation takes place at two levels:

• The first level is assessing the impact of the e-business policy on the regional level and more concrete on the business processes of SMEs. In short, does the e-business policy efficiently and effectively address regional gaps and needs and to which extent?
• The second level is the internal management of e-business support measures.

1. Quality Management Plan

The Quality Management Plan is a guideline for reporting quality control and assurance problems. This plan will help to monitor and control the level of quality produced by the e-business support measure. The quality management plan will ensure that

• Deliverables are designed to meet agreed-upon standards and requirements
• Work processes are performed efficiently and as documented
• Non-conformances found are identified and appropriate corrective action is taken

The Quality Management Plan consists of

• Quality objectives
• Key project deliverables and processes to be reviewed for satisfactory quality level
• Quality standards
• Quality control and assurance activities
• Quality roles and responsibilities

Three good reasons for establishing a quality management plan:

• Enable accountability for the initiative’s spending
• Quality assurance of e-business support measure
• Possibility for improvement during the policy implementation

2. Impact assessment of regional e-business policies

The regional needs analysis covers the current state of play concerning e-business uptake in SMEs at regional level and is often the basis for further undertakings. In addition, existing support schemes are being screened and evaluated with regard to their current efficiency, effectively and impact for the region.

Methods of evaluation and monitoring are

• Uptake of measures and activities (uptake)
• Recipients’ opinion of with the service provided (satisfaction)
• Comparison of the performance of SMEs with and without assistance (outcome for SMEs)

An interesting instrument for data collection is designing useful, quick and tailored ICT tools that allow obtain update and cross data (control panel) at any time. A good practice can be found in Valencia.

3. Difficulties measuring the impact

However, assessing the impact of "soft measures" such as information activities or awareness raising activities in comparison to "hard measures" such as grants and loans is limited. Often it remains on the level of measuring the uptake of an e-business measure and/or the satisfaction with the services (e.g. no of participants in awareness raising workshop). Some do also assess the difference made by the support service (e.g. through questionnaires). But only very few evaluate the performance of an e-business policy (e.g. comparison of performance of assisted SMEs with SMEs without any assistance).

Furthermore, measuring the difference is often impossible until a certain amount of time has passed e.g. in order to claim that one concrete measure is the result of a certain e-business support action.

4. Indicators

The following table is taken out of the Final Impact Assessment Report published by the European Commission. We have revised with regard to a more practical approach. This table is not exhaustive but should give the opportunity for a quick self-assessment and an overview which indicators might be useful for monitoring and evaluation.

In the two left columns you will find the indicators for evaluating e-business support measures. The attributes in the red and green boxes illustrate the two ends of the scale.

 

Impacts of e-business policies for SMEs

 

Reference: EU (2005), own presentation