Thought Leader Q&A: Exploring The Importance Of Tracking eLearning ROI For Nonprofits With Raneem Mourad

With over 10 years of experience in marketing and advertising, Raneem Mourad transitioned to the EdTech sector in 2013 and became the Managing Partner at Kashida in 2016. Kashida designs and produces bespoke impactful eLearning content that meets the unique needs of its clients, ranging from corporates to NGOs to educational institutions. As the Managing Partner, Raneem leads a team of talented designers, developers, and Learning Experience Designers, as well as oversees the business development, market research, and client relations aspects of the company. Today, she speaks with us about measuring the true impact and ROI of learning initiatives for the nonprofit sector and why it’s so essential for these organizations to choose highly adaptable L&D solutions.

Given that many not-for-profit organizations are looking for ways to stretch limited resources, how can they make a strong business case for L&D?

Especially for not-for-profit organizations, it is very important that any project they undertake is in line with their general mandate and can create measurable impact, whether it is humanitarian or development work. The same should be the case for any L&D strategy they implement, especially when beneficiaries can be quite diverse, including internal beneficiaries and the external audiences they might want to reach. But how can one ensure that beyond just alignment with the mandate, which is quite straightforward, the organization can also create measurable impact? Here, I believe from our experience, we have seen not-for-profit organizations successfully do the following:

1. Instill appropriate policies to ensure project leaders plan for sustainable implementation of their L&D initiatives with impact extending beyond project completion.
2. Plan the evaluation criteria from the start of the project and arrange for it during production and after. Utilizing models such as the Kirkpatrick model, for example, can really help if relevant to your project.

One can assume that the corporate world would be better suited for this because they have a very clear “bottom line” that they can look at and can measure the impact of L&D based on their “profitability” and “efficiency.” However, I would disagree here and find that the not-for-profit sector can benefit much more from a robust L&D strategy that can encourage impact on a scale greater than the organization itself. Here, we refer to impact on society, behavioral change, long-term economic impact, and many more aspects.

Based on your experience, what is one of the most common faux pas that nonprofits make when measuring the ROI of learning initiatives?

One of the main issues we’ve seen recurring in many of the projects we’ve worked on is the short-term vision when it comes to ROI, and this is mainly led by a short-term outlook on the KPIs.

When projects are led by either a country-level strategy or a target audience-level strategy that usually spans the course of one project manager’s assignment duration, the project ends up being abandoned once that initial objective is achieved. And this makes sense in the not-for-profit sector, particularly because projects rely heavily on grants and donors, which come with limitations around duration and scope; this often necessitates having short-term goals that are achievable within the lifespan of that grant.

The positive outcome of this is we end up working with incredible individuals who are typically amazing Subject Matter Experts in their field and are fully invested in the success of the project. On the downside, though, the project’s “limited” duration becomes a parameter that restrains the potential for more long-term ROI objectives. While we typically recommend that any project have a sustainability plan that ensures more long-term objectives, this can be difficult to achieve when there is no funding for proper follow-up and surveying of beneficiaries months or years after completion of a project. Hence, the project’s KPIs become limited to criteria such as the number of beneficiaries reached throughout its duration only, completion rates, minimum passing score, and other parameters that do not lend themselves to more sustainable, long-term ROI targets such as behavioral change and societal impact (related to GDP, the SDGs, etc.), which I believe we talked about in more depth in our first question.

One more thing I would add is we’ve seen many cases where several NPOs are working on similar initiatives and would benefit from collaborating, whereby I believe they can magnify their impact. So, perhaps a sort of space where such collaborations can be encouraged would be a great initiative.

Why is it so essential for NGO, development, or government organizations to choose an eLearning services provider that offers highly adaptable solutions?

It is absolutely essential in most cases, and this is because of many aspects. Below, I’ve mentioned some of the reasons why this sector would require custom and highly adaptable solutions.

• In this sector, the subject area is often incredibly specialized and specific to a target market/audience, which means it is very difficult to find ready-made or off-the-shelf solutions.
• In our experience, we’ve also noted that in many cases, they also need to reach multiple regions in many languages and are particularly attuned to cultural and social diversity, so they need a team that can bring in that same diversity and customized solutions.
• They often have limited budgets or other very specific constraints, which means we need to really customize the services and the solution to meet their needs.
• In terms of platform, they often want to avoid paying a per-user fee because their beneficiaries can be from a wide range of audiences and in some cases want to reach millions, which we achieved with the riyali.com platform for the Riyali Foundation in Saudi Arabia; a platform that can scale with their needs.
• To ensure their investment results in outputs that can be reused, repurposed, or reallocated, the right service provider will be able to implement such adaptable solutions and also advise organizations on a strategy to ensure greater and longer-lasting impact.

What is one of your biggest nonprofit client success stories?

One of the biggest success stories in terms of impact and sustainability would have to be our Riyali success story. Riyali is a foundation focused on financial awareness and financial literacy in Saudi Arabia. We have so far been able to reach over 2 million beneficiaries through several programs targeting diverse age groups. While the program initially relied on in-person classes and online courses, when the pandemic hit in 2020, we had to work with the foundation to build a robust solution that does not disrupt their goals of reaching their target beneficiaries.

Several steps were taken, but the most impactful one was the creation of the ambassador loyalty program (ALP). The program was custom-built and integrated into their Open edX LMS, which had already been customized and deployed by Kashida. The ALP succeeded in achieving the following:

• During the initial pilot in 2021, we gathered many learnings, and then launched the full program in 2023 with improved functionalities for the ambassador and supervisor dashboards. These allowed the ambassadors to follow up with their students, track their progress, send encouraging messages, and, in turn, be able to rise on the leaderboard.
• We were able to engage and encourage teachers and trainers to sign up as ambassadors on the platform; we had 44,000 ambassadors sign up for the program in the first few months from 47 different cities around the kingdom. And 94 supervisors were assigned in order to manage the different educational districts.
• We had around 1,200,000 students register for three different courses, ranging from children of 8–12 years old to youth in high school.
• The impact was further studied after the completion of the initiative to try to assess the behavioral and effective change the program had on the learners. After completing the program, below are some of the key findings:
  • 86% of learners started saving
  • 68% improved their financial planning
  • 81% gained confidence in their financial planning and dignity
• Finally, 83% of ambassadors responded positively to promoting the ALP amongst their peers.

The above results were achievable only because we were able to work closely with the foundation to identify the needs and gaps, assess the appropriate potential solutions, be able to pilot and test the solutions with beneficiaries through a human-centered design approach, and then launch a program that not only succeeded in achieving the short term objectives, but rather one that still resonates and impacts society on an ongoing basis.

Can you tell us a bit more about Kashida’s proprietary process and how design thinking is involved?

Every project we work on, regardless of how big or small it is, starts with a discovery and analysis phase. This phase includes workshops in which we collaborate with Subject Matter Experts and other key stakeholders to define the learning strategy and approach for the course or program. The stakeholders typically include project owners, Subject Matter Experts, on-the-ground practitioners, and, when possible, a sample of the beneficiaries of the training program.

There are 5 key areas we work to define with them:

1. Key outcomes of the program or course—these could be learning outcomes or other organizational goals.
2. What are the success criteria or evaluation criteria and mechanisms?
3. Who is our target learner? With several parameters to be covered.
4. What are the learning and enabling objectives?
5. Which learning approach should we use? And identify what digital assets need to be produced to create the appropriate learning experience. This could include everything from static documents to highly interactive and video-based learning material.

It is important to note here that this process is important to follow even when the client sometimes seemingly has many of these questions answered, and the reason for this is that we call ourselves investigative journalists in this process. Our LXDs’ role is to trigger and stimulate the stakeholders in order to identify any potential gaps, as well as ensure that the content is delivered with the end learner in mind first and foremost. You can read more about our process here.