Method Method = FORMAT methodology
The main method of evaluation guidelines depends on the evaluation metrics developed in the project [1]. These evaluation metrics depends on information extracted from users and beneficiaries of the project. Interviews and/or questionnaires can be used.
The main aim of the assessment is to give as much constructive feedback as possible about the case studies: the practical use of the methodology, the detected gaps, the value of the outcomes and their reliability. Two study groups are chosen: users and beneficiaries. After this primary assessment is finished, a scoring card is used to receivefeedback from the beneficiaries about their satisfaction of the outcomes of the case study
Ingredients
Knowledge
Interviewing skills.
Questionnaire building skills.
Time
30 minutes meeting per participant and 3 sessions for collecting data,building questionnaires and analyzing data (1h to 2h each) within 4 to 7 working days depending on the time of participants.
Material
Deliverable 4.1.
People
1-3 analysts to carry out the assessment.
As possible, all users and beneficiaries to be interviewed.
Tools
Structured interviews.
Questionnaire.
Final report and presentation.
Software
Office software available.
Instructions
- Collecting information:
- Case study sessions
- Reports
- Evaluate time spent on the case study by users and summarize the outcome
- Interviews: use evaluation metrics (annex in Deliverable 4.1) to interview the participants. Add extra questions to ask about feedback for developing the case study
- Send questionnaires to beneficiaries to ask them about the questions in the scoring card (annex in Deliverable 4.1)
- Collect information and classify it into a SWOT analysis (strengths, weaknesses, opportunities and threats)
- Qualitatively evaluate interviews and questionnaires with respect to the following criteria:
- the practical use of the methodology,
- the detected gaps,
- the value of the outcomes and
- the reliability of the methodology.
- Evaluate theanswers of thequestionnaires qualitatively following the annex of Deliverable 4.1 (the scoring card):
- 1 = definitely no
- 2 = more no
- 3 = more yes
- 4 = definitely yes
- Calculate the average of each answer and the standard deviation
- Represent the results of team experience (the interviews with users) and beneficiaries satisfaction (the questionnaire and the interviews with beneficiaries)
Some basic definitions: (for more details, check the FORMAT Glossary)
The beneficiaries of the Technology Forecast:
The person(s) to whom a Technology Forecast may be beneficial, e.g. by supporting them in decision-making, by helping them gain knowledge on a technological system… The beneficiaries are mainly the “end-users” of the forecast, usually the people in the company that are involved in decision-making.
The users of the Technology Forecast:
The person(s) who are using results of or Technology Forecast study in practice. For instance, Users consider results of forecast when developing new technologies or preparing specifications for new equipment acquisition.
The analyst = the user of the FORMAT methodology:
The person(s) that follows the stage-gate FORMAT methodology to prepare and execute a Technology Forecast, using the indicated methods in each stage to get a required output for each gate. Using the methodology includes data and information collection, analysis, synthesis, and the formulation of the presentation and report of the results.
Tips
➔ During the interviews, aim at a feedback that can support your later analysis into strengths, weaknesses, opportunities and threats. However, avoid mentioning these terms.
➔ Carry out interviews individually
➔ Questionnaires should be anonymous among participants.
➔ Record sessions if possible.
➔ Take notes and make sure information is explained clearly to participants.
➔ Identify who is doing what during the case study and ask more questions about his/her personal evaluation of this part.
➔ Always ask for suggestions for improvement.
Suggested reading
Deliverable 4.1, FORMAT Project, Deliverables (2014) ,:http://www.format-project.eu/deliverables/public-reports-and-white-papers/evaluation-metrics/at_download/file
Deliverable 4.5, FORMAT Project, Deliverables (to be finalized by the end of November 2014)
Example
Deliverable 4.3 of the FORMAT methodology represents the assessment of the case study of vacuum forming in the FORMAT project based on investigating the team experience through interviews, a questionnaire and a preliminary monitoring of similar/related non-FORMAT studies. The FORMAT builders and user participated in an online questionnaire and individual online-interviews which were a direct application of the evaluation metrics and scoring card previously published in deliverable 4.1. Each participant extended his/her answers to give recommendations for the improvement of the methodology, case-studies and deliverables. These participations were clustered by the researcher into main four categories according to SWOT analysis to represent the strengths, weaknesses, opportunities and threats. Although in swot analysis opportunities and threats are external due to the environment, most of the participant recommendations and warnings were directly related to the case-study activities. Therefore, other two categories of external opportunities and threats were added to recommendations based on preliminary study of similarly related studies in thermoforming, plastic machinery and appliances manufacturing.