Management Response

: Policy, Programme and Intergovernmental Division
: 2016 - 2017 , Policy, Programme and Intergovernmental Division (HQ)
: Evaluation of UN Women's flagship report Progress of the World's Women
:
: Policy, Programme and Intergovernmental Division
:

The Research and Data team thanks Paige Arthur and her team for their work to conduct the first evaluation of UN Women’s Flagship Report, Progress of the World’s Women. We acknowledge that there are few precedents for evaluating similar flagship publications or knowledge products, and therefore particularly appreciate the insights and learning that the process and the final evaluation report have provided. The evaluation demonstrates that Progress is widely used and valued, is generally regarded as high quality, and helps to position UN Women as a global knowledge hub on gender equality. It also provides concrete lessons on how we can ensure that the report has even broader outreach, greater uptake by our key stakeholders and can inform policy processes to a larger extent. Another important lesson is about the critical necessity of tracking the use of the report in a timely way, so that specific examples of the report’s use and influence on policy processes are not lost due to short personal and institutional memories. The evaluation reinforces the utility and value of having long form flagship reports for an organization like UN Women, while also pointing to the need to make better use of a range of shorter formats for different audiences, and throughout the report cycle rather than only at the time of the launch. The evaluation recommends that we publish Progress more often, but recognizes that in order to do so, and to improve our outreach, we will require greater human and financial capacity. Compared to the Human Development Report, Progress has only one sixth the amount of staff resources and one seventh of the annual budget available to it. The evaluation recommends that we work to foster greater buy-in and support for the report from senior management and ensure that it is recognized as a key resource informing UN Women’s advocacy in global policy debates. The final evaluation report and recommendations came at a time when we had already started the 2019-2020 edition of Progress, so we will implement as many of the recommendations as possible within this report cycle, and aim to implement the others in the following cycle. Since our team is responsible for other flagship reports and research publications, we will apply these recommendations to all of our reports, as relevant.
: Submitted
Recommendation: Recommendations to strengthen the relevance and effectiveness of the contents of the Progress reports
Management Response: The team notes these recommendations in the context of the overall finding of the evaluation that quality of the reports was generally judged to be high. However, we welcome these recommendations for improvement. Although we have not conducted a full literature review on the theme of the 2019-2020 report, focusing on families as such, we produced a detailed concept note for the report, and our research strategy has been to commission papers that scan the literature on some topics, and to conduct our own in-house literature reviews on other topics. For the next report, we will consider commissioning a comprehensive literature review upfront and publishing it as a separate resource. The feasibility of this will depend on selecting a narrower theme (than eg. economic and social rights, or families), but this is something that we want to do anyway, as part of efforts to publish the reports more regularly. We have revised our peer review process to include one anonymous peer reviewer per chapter. As well as peer reviewers for the individual chapters, we selected four people to do a global read of the report and asked them to focus on the key aspects of quality identified in the evaluation. In the 2019-2020 edition of the report, we included a statistical note which included key methodological information on the data elements of the report. We will consider expanding that out to produce a broader methodological annex to the report. We always strive to commission original and innovative research and data analysis and will continue to do so. For the 2019-2020 report, we developed data partnerships with both UN DESA and ILO to generate new statistics on household composition and labour force participation by marital status, which are a rich source of previously unavailable data on families. As with the 2011 report, we made the data annexes available to download as excel spreadsheets. While the statistics we use typically do not get updated as frequently as every year, we have plans to push out the statistics and other elements of the report more effectively after the launch, through blog posts, infographics etc. In addition, the R&D team has launched a new Women Count data portal to provide access to statistics on gender and the SDGs. The hub includes a dashboard with Progress data and features blogs and stories related to the report, giving our readers another way to access and use the report. Based on our experience of working with the LAC region to produce a companion report for the 2015 report, we agree that national/regional reports have a lot of potential for greater outreach and influence of the Progress series. For the 2019-2020 report, there are several country/regional products planned/in progress – in Mexico, Northern Africa, Eastern Europe and Central Asia. We plan to reflect on these products and produce a guidance note for colleagues to support future efforts.
More detail
Description: R&D team should continue to improve the quality of the Progress reports. Possible steps include: conduct a systematic literature review at the start of every Progress to assess the quality of existing evidence (not just an annotated bibliography), which could be a publishable research product in itself; conduct anonymous, paid peer review of chapters and a sample of the statistics to ensure quality; focus on evidence basis for key claims and practical examples in boxes; put a methodological annex in the back of the book or make it available online, for those who want it; continue to ensure adequate framing and conceptualization—Progress should hold together. R&D team should Identify low-resource ways to deliver the original research and messages that many readers want to see more of, even if it is a matter of just a few statistics; it should also make data downloadable. Moreover, it should choose a set of statistics to refresh each year and launch as an infographic. Finally, UN Women regional and country offices, in coordination with R&D team, should pursue opportunities to translate substantive contents to the regional/national level through the development of regional/national Progress reports. Such reports should be encouraged by senior management and included in strategic planning documents.
Management Response Category: Accepted
Thematic Area: Not applicable
Operating Principles: Not applicable
Organizational Priorities: Not applicable
UNEG Criteria: Effectiveness
Key Actions
Responsible Deadline Status Comments
Revise peer review process to include one anonymous peer reviewer per chapter; and ensure all peer reviewers focus on the evidence basis for key claims and boxed examples, and clear framing and conceptualization R&D team, working with our expert advisory group 2018/12 Completed
Include an anonymous peer review of a sample of the data in the report. R&D team 2018/12 No Longer Applicable The data in the report was extensively peer reviewed, although it was not anonymous, since we did not consider this to be necessary to get critical feedback.
Depending on the theme of the next report, consider commissioning a full literature review of the theme, to be published as a separate resource R&D team 2020/06 Overdue-Initiated
Ensure that our outreach strategy includes ongoing release of interesting stats (updated where possible) after the launch R&D team, working with the comms team 2019/12 Ongoing Although deadline entered is December 2019, this action should be completed for Progress reports going forward.
Scope the possibility of including Progress data in a new data portal on SDGs R&D team 2019/06 Completed
Reach out to regional/country offices to scope the possibility of one or two regional companion pieces for the Families report R&D team, in partnership with field colleagues 2018/06 Completed
Recommendation: Recommendations to strengthen and innovate the communication and outreach strategy of the Progress reports
Management Response: We welcome these recommendations and will endeavor to take them on board. We agree that we need to be more transparent about our approach to choosing themes for the reports. For the 2019-2020 report, we spent considerable time consulting with different internal stakeholders on the theme, both at HQ and the field, presenting a number of options and proposing our preferred theme. In the next cycle, we will consider a more open-ended process in which we survey colleagues for their most pressing information needs. However, we are also keen to retain the innovative, horizon-scanning element of Progress reports, and we note that taking this approach may mean that we end up with a theme that reflects UN Women’s current themes, rather than more cutting edge and emerging issues. We acknowledge the need to develop a long term, strategic, outreach strategy for Progress (beyond the global launch plan), and implemented this recommendation for our flagship report on gender equality and the SDGs, which was launched in early 2018. In terms of engaging end users early, we took steps to do this by announcing the theme of the next Progress report on the International Day of Families on 15 May 2017 and putting out a short animation, and blog posts by Shahra Razavi on why the theme of families posted on the UN Women website. We will also ask the Communications Team to better integrate our flagship reports into their strategies and ask them to take responsibility for periodically mining the reports and making better use of their rich contents. More generally, we agree that we need greater support and buy-in for outreach and advocacy work from other parts of UN Women. As part of our work to position the 2019-2020, we worked with an internal working group made up of colleagues from the Executive Director’s office, the communications team, civil society team, intergovernmental team and regional offices. Senior management were very positive about this approach to getting organization-wide buy-in. We note that the evaluation found that take-up of the 2011 report was greater among those constituencies that were either involved in the research process (eg. field colleagues who were asked to contribute information for boxes etc) or who were on the advisory group (eg. CEDAW committee members). We had already selected the advisory group for the next report before the evaluation was finalized, and in this case, it is made up mainly of leading feminist scholar/activists on different aspects of families. However, we supplemented their expert advice with a number of other steps. We held a series of pre-briefings before the report was launched with key stakeholders, including member state and key civil society stakeholders, which helped to generate buy-in for the report and its findings before the report was launched. The proposal for a research blog is well taken and would help to ensure that Progress stays on our audiences’ radars in between editions, but would require a devoted outreach specialist to ensure quality and sustainability (see below). We have instead launched a Progress News newsletter, which includes highlights and updates on the report, and we also have a new twitter hashtag to highlight the publication of UN Women research papers and briefs, including background papers for the report. We accept and will implement the recommendations on engagement with the traditional and new media, recognizing that some aspects of this approach have been part of the process that we have followed in previous years. We accept the recommendation to develop and implement a robust monitoring plan. We have already taken steps in this regard: we are now able to track downloads of our publications from the UN Women website and related microsites; and we are working on using DOIs on our publications, via a partnership with the UN Publications’ iLibrary. A more expansive monitoring plan is in process, but we will need to be modest in our aims, because we lack staff capacity to implement it on a sustained basis.
More detail
Description: R&D team should develop a written engagement strategy for UN Women staff in deciding on the theme; consider them as the primary end-users; develop a process for analyzing their most important information needs in their work, rather than asking them to take part in the research itself (which they may not have time or capacity to do). R&D team should identify external target audiences more clearly for each report; start to engage them before the research is finished. Map early on the UN Women staff and other relevant stakeholders who are likely to be potential end-users; invite high profile actors in target groups to be on the Advisory Group; develop a research blog that updates potential end-users on the process: choice of theme; composition of the Advisory Group; selection of background paper topics; striking findings from initial research; problems and challenges; etc. R&D team should identify early on the strikingly original statements the report will make, in order to start building these into an outreach strategy. Senior management should appoint a small team or at least one senior staff member to lead the task of identifying policy windows and target audiences, as well as translating the report in to advocacy actions in the global policy space. R&D team, working with Communications and the team appointed by senior management, should develop a written, medium- to long-term, sustainable outreach strategy that identifies relevant global forums and debates as well as entry points for advocacy and influential people to act as mediators to translate and advocate for Progress ’s messages. This strategy should: ensure early engagement with media, including providing copies of the report and access to the web site well in advance; identify new formats and continued strong engagement with social media; include an agreement that Communications should mine the report on a periodic basis as relevant issues emerge in the news, providing journalists with other opportunities to cite the report throughout the year; and ensure that Communications associates Digital Object Identifiers (DOIs) with all publications in order to enable Altmetrics. R&D team should develop a robust monitoring strategy to track key performance indicators, qualitative information on uptake at the national, regional, and global levels; and content analysis of important citations of the report.
Management Response Category: Accepted
Thematic Area: Not applicable
Operating Principles: Not applicable
Organizational Priorities: Not applicable
UNEG Criteria: Impact
Key Actions
Responsible Deadline Status Comments
Develop a note on the process for deciding the next report theme and include the most pressing information needs of UN Women staff as one of the criteria for selecting a theme for Progress and other flagship reports. R&D 2020/06 Overdue-Not Initiated
Develop a long term, strategic, outreach strategy for Progress, with inputs from a senior political advisor; and inputs, buy-in and responsibility to deliver shared by senior management and the communications team R&D, EDO and Communications team 2019/06 Completed
Consider options for ongoing social media/online communications work related to the report and research process, eg. a blog or twitter account R&D 2019/12 Completed
Develop and implement a robust monitoring plan for Progress R&D 2019/09 Overdue-Initiated
Recommendation: Recommendations to clarify and improve the positioning of Progress within UN Women
Management Response: We agree with these recommendations. The R&D team came up with a theory of change for the 2019-2020 Progress, and we note that each report will require a new one tailored to the issues it tackles. We will present the other recommendations to senior management, including the Chief of Staff, the Chief of Communications and Advocacy, as well as the Executive Director’s Senior Communications Advisor.
More detail
Description: R&D team should develop a new theory of change, based on findings in this evaluation. Senior management should develop a document that clearly explains the role and position of Progress in relation to the strategic goals of the institution, including its normative mandate. This document should include an indication of whether senior management supports Progress continuing to take a distinctive rights-based approach; the extent to which linkages and alignment with Policy, Program, and other relevant units should be expected (or not expected); and it should clearly outline the expected roles and responsibilities for aligning Progress’s messages across the institution. It should identify specifically what level of financial and human resource R&D team requires to adequately perform the functions expected of it. Communications should develop a strategy for a more sustained engagement with the flagship beyond the launch period.
Management Response Category: Accepted
Thematic Area: Not applicable
Operating Principles: Internal coordination and communication
Organizational Priorities: Not applicable
UNEG Criteria: Sustainability
Key Actions
Responsible Deadline Status Comments
Draft a new theory of change R&D 2019/03 Completed
Meeting with Senior Management to discuss strategic positioning of Progress; its approach; roles and responsibilities; human and financial resources available to fulfill its function within the organization R&D and EDO 2020/03 Overdue-Initiated
Recommendation: Recommendations to improve resource allocation and efficiency
Management Response: As noted above, we accept the recommendation to have a structured discussion with senior management about Progress. We accept the need to improve the web presence of the report, but have decided it would be more cost effective to link up with the new UN Women data hub. We commissioned fewer background papers for the 2019-2020 edition of the report and will reflect further on the research process for the next edition. We always strive to ensure that background research and data analysis deliver striking and original findings. Equally important is to ensure that those striking findings are communicated effectively, not only in the report itself, but as part of efforts to increase the dissemination of the papers including as part of our discussion paper series.
More detail
Description: R&D team and senior management should have a formal, facilitated discussion about the resource commitment needed to improve the outreach and production functions for Progress, alongside its current commitment for a periodic, high quality research publication; they should also reach a formal agreement about the timetable for sign off, production, and launch for each Progress report. R&D team should develop a resource plan (including staff costs) for creating a more dynamic web site, including provisions for updating content with blog posts, interesting new data or research, etc. This plan should be shared with senior management. R&D team should reduce the number of background papers commissioned; it should consider combining resources to commission a smaller number of papers than might deliver the kind of strikingly original research and messages that many readers wish to see more of.
Management Response Category: Accepted
Thematic Area: Not applicable
Operating Principles: Not applicable
Organizational Priorities: Not applicable
UNEG Criteria: Efficiency
Key Actions
Key Action not added.