Analyzing the Five Questions Vol 1

Hello again, and welcome back!

Over the past several months, we've spent a great deal of time exploring the implications of the ARC-PA 6th Edition Standards, from syllabus compliance and curriculum mapping to assessment, remediation, and the growing importance of systematic program evaluation. It’s clear that many programs are working hard to understand not only what the new standards require, but also how to implement them effectively.

With that in mind, I'd like to begin a new blog series based on a recent webinar on one of the most important components of the 6th Edition Standards: the five required questions in Standard C1.01.

At first glance, these questions appear straightforward. However, as many program leaders are discovering, answering them requires much more than simply gathering data. Programs must determine which data are most relevant, analyze trends and relationships, draw meaningful conclusions, and decide whether areas needing improvement exist. Perhaps most importantly, they must be prepared to explain and defend those conclusions.

One of the most common questions I've been hearing about the 6th Edition Standards is surprisingly simple: How do you know if you're right?

Not right about a benchmark, a data set, or a template. Right about your conclusion.

As programs begin working through the C1.01 decision framework, many are discovering that the challenge is not simply collecting data. The challenge is determining what that data mean and deciding whether the evidence truly supports a conclusion about program effectiveness.

A Different Kind of Evaluation

Under previous standards, much of the work focused on gathering information and demonstrating compliance. Programs still need to do that, of course, but the 6th Edition asks something more. The standards increasingly require programs to evaluate their own effectiveness and defend the reasoning behind their conclusions. That naturally creates uncertainty.

Ambiguity is something I continue to think about. The 6th Edition Standards are still relatively new, and we are all learning how ARC-PA reviewers will interpret various approaches. Different readers may view the same information through different lenses. Because of that, I believe the most important thing a program can do is establish a systematic evaluation process. A strong process makes your conclusions more defensible, regardless of the specific outcome.

The Five Questions

The C1.01 framework centers on five questions:

1. Are program faculty effective in operating the program outside of teaching?

2. Is the admissions process effective in selecting students who can successfully complete the program?

3. Is the didactic curriculum effective in preparing graduates for clinical practice?

4. Is the clinical curriculum effective in preparing graduates for clinical practice?

5. Overall, does the program successfully prepare graduates for clinical practice?

One of the biggest mistakes programs can make is assuming these questions are interchangeable.

There is certainly overlap in the data. However, each question asks you to evaluate a specific aspect of the program through a different lens. Before selecting data or drawing conclusions, it is important to ask: What is unique about this question?

That simple step helps keep the analysis focused and prevents unrelated variables from driving conclusions.

Critical Analysis Means More Than Looking at Numbers

ARC-PA has emphasized critical analysis for years, but the 6th Edition broadens the scope of that analysis.

The webinar focused on four key concepts that programs should incorporate into their evaluation process:

Comparison

How do data sets relate to one another?

Perhaps course grades align with PACKRAT performance in a specific content area. Perhaps remediation patterns align with lower summative scores. Looking at data side by side often reveals relationships that individual reports cannot show on their own.

Trends

What direction is the data moving over time?

A benchmark may still be met, but what happens if performance has steadily declined over several years? Is that a concern? Potentially, yes. One of the more challenging aspects of trend analysis is determining what to do when performance remains above benchmark but continues to decline over time. As discussed in the webinar, these situations often require careful judgment and contextualization rather than automatic conclusions.

Trend analysis allows programs to look beyond a single point in time and evaluate the broader direction of performance.

Triangulation

One data point rarely tells the whole story.

Triangulation uses multiple data sources to evaluate a single issue, increasing confidence in the conclusions that are drawn. Rather than reacting to one isolated metric, programs can determine whether multiple sources point toward the same finding.

Contextualization

Data do not exist in a vacuum.

If a weakness appears in one area, are there contributing factors elsewhere in the program that help explain it? Context often provides insight into whether an issue is isolated, systemic, or influenced by external variables.

Taken together, these concepts create a more complete picture of program effectiveness.

Compliance and Areas Needing Improvement

One of the most interesting aspects of the 6th Edition Standards is that ARC-PA does not provide an exact definition of what constitutes an Area Needing Improvement (ANI). Programs must establish a rationale for making that determination themselves.

That responsibility can feel intimidating at first.

However, it also creates an opportunity to develop a thoughtful and consistent methodology. Some programs may define an ANI through triangulation, requiring three or more aligned data sets to fall below benchmark. Others may focus on sustained declines over time or significant drops in critical indicators.

The important point is consistency.

Whatever process you choose, you should be able to explain how you arrived at your conclusion and demonstrate that the conclusion was supported by the evidence available.

Looking Ahead

In future posts, we will explore the C1.01 questions in greater detail, beginning with one of the most challenging areas for many programs: evaluating faculty effectiveness outside the classroom.

This is the question that often generates the most discussion because there are relatively few direct data sources available. Programs frequently ask how to gather sufficient, meaningful evidence to support a defensible conclusion.

The good news is that you are not alone. Many programs are working through these same questions right now, and thoughtful, systematic analysis remains the best path forward.

I hope you’ll join me next time as we take a closer look at Question One and the challenge of evaluating faculty effectiveness beyond the classroom.