Identifying user needs for the EFFORTI-tool box

From February to March 2017 the EFFORTI-team conducted five national workshops with policy makers and evaluation practitioners to learn about their needs for a further development of the EFFORTI-gender evaluation framework. The workshops took place in Austria, France, Germany, Hungary and Spain while interviews with stakeholders were conducted in Denmark.

The national workshops and interviews had two goals: the first goal was to discuss the main results from the national reports the EFFORTI-team had written to identify the relevant contexts of gender equality measures in RTDI. The feedback provided by the participants allowed the EFFORTI team to verify and sharpen their argumentation and was therefore included into the country reports. The second goal was a needs assessment for the further development of the EFFORTI evaluation tool. The participants were asked to imagine potential use-cases, to describe their national evaluation culture and to explain their expectations and demands for the EFFORTI-toolbox regarding content and form.

The needs assessment revealed a broad variety of demands and possible applications for the planned online tool. The need for a tool assisting in the evaluation of various organisational or legislative gender equality measures was repeatedly expressed, for example, by enabling the writing of tenders free from gender bias and providing a methodological box for companies. It was confirmed that the evaluation culture significantly varies between countries. According to the workshop participants in Austria and Germany, programme evaluations are quite common. However, because a comparative perspective and contextual considerations often are missing, evaluators often struggle with the correct interpretation of the results. Therefore the discourse within the national evaluation communities is characterised by advanced methodological questions as which context factors have to be considered or how to measure organizational culture and long-term effects. In Denmark, France and Hungary, the participants stated that the gender dimension is not frequently considered in evaluations. In Spain, the participants reported, evaluations were common and supported by public policies, but implementation of evaluations differed between regions and there was often a lack of political commitment.

Regarding content, the participants generally required a database of indicators and instructions on what an evaluation report should look like. Also smart practices should be integrated describing examples of methodological sound and reliable evaluations of gender equality measures. Some participants pointed to the need of a strong theoretical grounding of the tool which would allow a clear operationalization of evaluative questions and related indicators. The users want to be be supported in selecting indicators that are meaningful in measuring progress with their goals. The participants adverted to various research gaps in the development of indicators, especially concerning the measurement of cultural change, long-term effects and effects that go beyond the implementing organisation.

In line with the variety of use cases, one particular concern was the integration of various stakeholders and purposes. How can a generic tool be both comparable and allow a deeper analysis of very specific organisational contexts with varying purposes? This fundamental challenge was also expressed by questions on the scope and the level of detail the tool can provide, the extent to which specific methods for collecting and analysing data should be integrated and the relationship between the tool and the expert knowledge that will be required for its application. From the user perspective, the participants expressed diverging needs, for instance, the demand for a quick evaluation guide to support a sound and extensive evaluation.

The next step in stakeholder involvement will be the EFFORTI conference in Brussels on October 19th. The participants will get an update on the current development of the EFFORTI tool box and have the chance for an international exchange about the state of the art in gender evaluation.