Illumin Analytics - LOGO
240-971-8563
Serving Clients World-Wide

Is the tail wagging the dog?

Is the tail wagging your dog? 

There are good reasons why people are skeptical of evaluation. Let's face it; we have all seen evaluation used poorly. Take elementary school evaluation. The purpose of formally testing students was to see how students were doing compared to other students or how students were doing in comparison to achieving a certain level of performance. This information was supposed to provide useful feedback to the school on what could be done better to teach students.


What has been the result of formalized evaluation? Teachers feel pressured into focusing on test preparation instead of teaching their students. They feel the need to maximize test performance, when all they want to be doing is helping students to learn and grow. Thus the tail (test) is wagging the dog (the classroom experience.) 


Some may feel that all evaluation ends up like this, creating more harm than good. This possible negative impact of evaluation has long been known and even has a name: Campbell's Law or Behavioral Drift. High quality evaluators actively make sure that this is NOT the result of evaluation. High quality analysts, like those at Illumin Analytics, know how to select the right measures, design data collection efforts, analyze the data, and how to guide the use of that information in a way that helps organizations understand where they are currently while illuminating the path forward.  This is true regardless if the topic being evaluated is Full Service Community Schools, DEIA efforts, Reducing Recidivism, or the Evidence Act. 


#accountability #aim #goal #evaluation #assessment #impact #indicator #input #objective #outcome #DevelopmentOutcome  #Monitoring #Stakeholder #sustainability #FSCS #FullServiceCommunitySchool #Analysis #Data #DataVisualization #Analyses #Benchmark #BestPractices #Baseline #Construct #Validity #effectiveness #efficiency #evaluability #ExternalEvaluation #DEIA

How can evaluation help your Full Service Community School to thrive?

By 7063179645 • July 3, 2023
The US Department of Education recently announced its request for proposals for the Full Service Community Schools Program Grant. (Full-Service Community Schools Program (FSCS) - Office of Elementary and Secondary Education) As many of you know, a Community School is one that leverages the strengths of the community, families, school, and staff to support and uplift all students, particularly those coming from low resourced environments, to assure academic success in the pre-K through post-secondary space. The Full-Service Community Schools were initially defined in the Elementary and Secondary Education Act of 1965. A program funding FSCS through the US Department of Education has been in existence since 2008. This year there are lots of exciting changes! The sheer amount of funding budgeted for FSCS has increased to the highest levels yet. The US Department of Education anticipates to award 75 new projects across three levels: Capacity Building, Multi-Local Education Agency, & State Scaling. All Full-Service Community Schools are going to need to include a logic model in their proposal. Though not explicit in this call, particularly for the State Scaling grant, it appears that a competitive proposal will need to include a theory of implementation and a theory of change that includes evaluation of such processes and protocols. For “additional competition points” which appear to be critical for a competitive proposal, protocols need to explicitly address: Social, emotional, and academic needs, & Improving coordination across multiple agencies. All submitters need to partner with an external evaluator. There is an invitation priority that focuses on improving transitions between preschool and kindergarten and from kindergarten into first grade. Funded grantees will be required to participate in a national evaluation that will study the implementation of the FY 2023 FSCS proposals that are awarded. Like in prior years, this FSCS project is expected to build upon the four pillars of community schools: integrated student supports, expanded and enriched learning time, active family and community engagement, & collaborative leadership and practices. Over the next two weeks, we’ll be talking further about the ingredients needed to write a competitive FSCS grant. Until then, Illumin is offering two free hours of consulting for organizations writing a FSCS proposal. We can address important issues like the following. How can you leverage your organizations’ strengths to create a high quality, useful Logic Model What characteristics are a part of a high-quality evaluation plan that will meet the requirements of the FSCS program How will your project have to adapt to the 2023 requirement of participating in a national evaluation plan for FSCS grantees? Why is it critical that your external evaluator not only have high skills in complex evaluation, but also be comprised of content matter experts in areas like social and emotional learning, and analysts with lived experiences that align with your service area? Interesting in learning more? Please contact Yonas Ermias ( yermias@assessment-evaluation.com ) to schedule an appointment with our analysts to help guide you through to your best submission! More information on Community Schools can be found here: Community Schools as an Effective School Improvement Strategy: A Review of the Evidence (learningpolicyinstitute.org) The White House tool kit for full service community schools can be found here: 2023-01-13-WHITE-HOUSE-TOOLKIT_Federal-Resources-to-Support-Community-Schools.pdf (whitehouse.gov)

Does diversity matter in your evaluation team?

January 8, 2023
Does diversity really matter in your evaluation team? The answer is, yes. However, what does a diverse evaluation team look like? There are two components of diversity you need to look for in your evaluation team: Skillset and Lived Experiences. A high quality evaluation team will have analysts with the quantitative and qualitative expertise needed to complete the task. There will be at least one measurement expert, called a psychometrician, on the team. Someone else on the team should have expertise in data communication and data visualization. Certainly, it is possible that a single person will be able to fill all of these skills, but depending upon the complexity of your project it is unlikely. Additionally, you may also want someone with content matter expertise. So, if you are looking for an evaluation team to assess your Full Service Community School program, there needs to be people on the team with education expertise. If you have a grant funded program designed to reduce recidivism, there needs to be people on the team with expertise in working with returning citizens. However, merely having the formalized skill set may not get you to the best evaluations. You also need a team that has diversity through lived experience - whether that is racial, ethnic, gender, low income or other forms of lived experiences that can help the team navigate the situation to be evaluated and work with the population under study. Selecting an appropriately diverse evaluation team is a great start in assuring the highest quality evaluation experience. However, there is one critical component that such a team should have - that is humility. A high quality evaluation team knows regardless of their formalized and lived experiences, they do not know it all. So, they ask questions and listen to the people in the know, prior to designing the evaluation plan and throughout the process of selecting measurements, gathering and analyzing the data, interpreting and presenting the findings, and providing accurate and useful data driven recommendations. If you are looking for that kind of evaluation team, remember, Illumin Analytics is a minority-owned business with the right balance of diverse skill sets and lived experiences. Illumin Analytics will create the appropriately diverse evaluation team for your organization's needs, particularly in the area of assessing organizational efforts in DEIA, implementing the Evidence Act in you Federal Agency, evaluation FSCS, and providing direction for your program designed to reduce recidivism among returning citizens.
Share by: