We sat down with Dr. Mónica Ruiz-Casares to discuss her experience as PI for an evaluation she is leading in West Java, Indonesia.
The Evaluation Fund was established in 2011 to support high-quality evaluations of programs that are designed to prevent violence against children. It focuses its work in low- and middle-income countries to build the evidence base for what works to reduce violence against children in these settings.
Conducting research in these places can be challenging, but also extremely rewarding, as our recent conversation with researcher Dr. Mónica Ruiz-Casares shows.
Dr. Ruiz-Casares is the Primary Investigator for a research project that the Evaluation Fund has been supporting over the last few years. The project examines a program in West Java, Indonesia, that aims to prevent violence that young children experience in the home by providing parents with training on child development, positive discipline practices, and support to connect families with community resources and government social protection schemes.
The research project examined the effectiveness of Save the Children’s Families First with Home Visitation Program which adapted a program called the Positive Discipline in Everyday Parenting (PDEP) for use in West Java. This project was led by McGill University in partnership with an independent Indonesian research and policy organization, SMERU Research Institute.
We sat down with Dr. Ruiz-Casares, a seasoned researcher at McGill University’s Department of Psychiatry, to learn more about her experience leading this project and her insights on what made this evaluation possible and meaningful despite the challenges that come with doing research in low- and middle-income settings.
Flexibility
Throughout this interview, Dr. Ruiz-Casares highlighted the need for evaluators to remain flexible and adaptable, adjusting their professional, and sometimes personal, timelines according to the progress of the project and to be prepared for unforeseen obstacles that are likely to arise. “So, you have to have the flexibility and openness to adjust, while maintaining very high standards,” Dr. Ruiz Casares noted.
For example, though the evaluation was planned for 18 months, it has taken five years to come to a close. A key delay began with the cultural adaption of the PDEP program to the Indonesian context, which required Save the Children and the University of Manitoba – who co-own the program – to agree on its adaptation. Dr. Ruiz-Casares remained supportive of this process, recognizing that a proper adaptation with buy-in from all stakeholders may require additional flexibility from the research team, noting that the implementing partners “couldn’t do an independent evaluation without us, but we certainly couldn’t have done it without them.”
“Don’t take English for granted.”
There was also the obstacle of translation, requiring forward and backward translation between English, Bahasa Indonesian, and Sundanese to ensure consistency and quality of measures. Dr. Ruiz-Casares warned, “Don’t take English for granted,” adding that translations can be challenging and time-consuming, and must therefore be factored into the timeline.
Nevertheless, Dr. Ruiz-Casares noted, “Every delay has ripple effects.” Challenges encountered with the programmatic adaptation set back the process significantly. This also caused the project to lose its co-Primary Investigator in the first six months, requiring McGill to restart the search process for a qualified research partner to fill the spot.
“Usually you say it takes two to tango, but this time, it took three.”
Flexibility was also required from the funders’ side. “We couldn’t have done this if there had been very strict funding cycles,” Dr. Ruiz-Casares explained, “It is difficult to find a funder with this kind of flexibility. Usually you say it takes two to tango, but this time, it took three.”
Communication
Dr. Ruiz-Casares explained that consistent, open, and honest communication is another key component of a successful evaluation. Teams were created to collaborate during the data collection and analysis process, using WhatsApp and Skype to keep communications among partners open and frank.
“this was a safe space”
One of these teams was the evaluation coordinator team, which included representatives from each organization involved in the project: Save the Children UK, Save the Children Indonesia, SMERU Research Institute, and McGill University. At least one representative from each organization needed to be present in order for a meeting to be held, which was especially challenging as these representatives spanned three continents with as much as twelve hours of time difference between them. “Sometimes we met once a week, sometimes we met every other day,” explained Dr. Ruiz-Casares. “During these meetings, we would share our concerns and at times it was not easy… but this was a safe space.”
“Everybody was learning; it was very frank and open communication.”
Two other WhatsApp groups were set up to coordinate the ‘quantitative’ and ‘qualitative’ field teams. Dr. Ruiz-Casares would join calls and chats with the research leads and other staff to work through any issues regarding data collection. “The researchers have a particular set of skills and they don’t get to talk to each other much, so both were a part of the discussion to learn from each other and support each other… everybody was learning; it was very frank and open communication.”
Dedication to Quality
In addition to communication and flexibility, as Dr. Ruiz-Casares told us, “Who is part of your team makes a huge difference – people matter. When things don’t work out as planned, that’s when our quality as professionals comes out,” and it was this team’s commitment to quality and ethical conduct that helped to make this project successful.
“Everyone felt like they owned the project. It was everyone’s solution as much as everyone’s problem.”
“Everyone had agreed to be transparent,” Dr. Ruiz-Casares said when describing the fieldwork. “It would have been very easy to swipe things under the carpet and not inform each other – there were twelve hours of time difference and we are on the other side of the world.” Instead, communication between all research teams remained open and honest, she noted. “Everyone felt like they owned the project. It was everyone’s solution as much as everyone’s problem.”
Dr. Ruiz-Casares acknowledged how valuable it was to work with these researchers and in this setting. “One common piece for everyone is a joy of learning. Everyone felt alive and everything was falling into place. How else could we stay [on a call] for three hours that late or that early? They have learned so much and I have learned so much,” she concluded.
“One common piece for everyone is a joy of learning. Everyone felt alive and everything was falling into place.”
The research team is currently working on the final report of this study and public presentations of findings for participating communities and local decision-makers in Indonesia.
You can learn more about the intervention and evaluation here.
If you would like to be informed when results from this study become available, please sign up for our newsletter.