Analyzing the Five Questions Vol 4

I’m glad you’re joining me again as I continue my series on analyzing the five questions contained within ARC-PA Standard C1.01.

In our previous posts, we've discussed how to build a defensible process for answering the C1.01 questions, how to identify Areas Needing Improvement (ANIs), and how programs can approach the often-challenging task of evaluating faculty effectiveness outside of teaching responsibilities.

In this post, we turn our attention to two additional questions:

1. Is the admissions process effective in selecting students who can successfully complete the program?

2. Is the didactic curriculum effective in preparing graduates for clinical practice?

Unlike the faculty effectiveness question, these areas typically provide programs with substantial data. The challenge is often not finding evidence, but determining what that evidence means and whether it supports the conclusions being drawn.

As with every C1.01 question, the goal is not simply to report data. We want to analyze it thoughtfully and determine whether the process being evaluated is effective.

Evaluating Admissions Effectiveness

At first glance, admissions effectiveness may seem easy to assess. Programs often have access to extensive admissions data, including:

• Number of applicants

• Interview scores

• Acceptance rates

• Academic metrics

• Demographic information

While these data may help describe the admissions process, they do not necessarily indicate whether it is effective.

To answer that question, we must look beyond admissions itself and examine student outcomes. After all, the purpose of admissions is not simply to fill seats. The purpose is to identify students who can successfully complete the curriculum and enter clinical practice.

That means programs should consider data such as:

• Attrition rates

• Academic progression

• Remediation rates

• Graduation rates

• PANCE performance

• Other indicators of student success

These outcomes help determine whether the admissions process is accomplishing its intended purpose.

For example, if students consistently struggle academically after admission, experience high remediation rates, or fail to progress through the curriculum successfully, programs may wish to examine whether admissions criteria are identifying the right indicators of future success.

Conversely, strong student outcomes may support the conclusion that the admissions process is functioning effectively.

As always, no single data point should drive the analysis. Programs should compare multiple sources of evidence and look for patterns that support or challenge their conclusions.

Looking for Trends, Not Just Benchmarks

One of the recurring themes throughout this webinar was the importance of looking beyond individual benchmarks.

Meeting a benchmark is certainly important. However, it does not always tell the entire story.

Consider a situation in which attrition remains below a program's established benchmark but gradually increases over several years. Technically, the benchmark may still be met. Yet the trend itself may warrant further investigation. Likewise, a temporary decline in one area may not indicate a larger problem if surrounding data remain strong.

This is where trend analysis becomes valuable.

Programs should ask:

• Is performance stable?

• Is it improving?

• Is it declining?

• Do multiple data sets tell a similar story?

The answers help create a more complete picture of admissions effectiveness than any single measure can provide.

Evaluating Didactic Curriculum Effectiveness

The third C1.01 question asks whether the didactic curriculum effectively prepares students for clinical practice.

Like admissions effectiveness, this question often provides programs with an abundance of available data.

Potential sources of evidence may include:

• Course performance

• Summative examinations

• PACKRAT scores

• Competency assessments

• Readiness evaluations

• Other measures of student learning

The challenge is determining whether these data collectively support the conclusion that students are adequately prepared to enter the clinical phase of the program.

Again, triangulation becomes important. A strong examination score may be encouraging, but a single assessment rarely tells the entire story. Programs should examine how multiple indicators relate to one another and whether they consistently demonstrate student readiness.

This is where comparison becomes especially useful.

1. Do course outcomes align with PACKRAT performance?

2. Do summative assessments support conclusions drawn from course evaluations?

3. Do students demonstrate readiness consistently across multiple measures?

Looking at data sets side-by-side often reveals insights that individual reports cannot provide on their own.

Critical Analysis and Context Matter

One of the most important concepts in the 6th Edition Standards is that data must be interpreted within context. For example, a decline in one content area may initially appear concerning. However, when viewed alongside other indicators, the issue may prove isolated rather than systemic.

Similarly, strong overall performance may occasionally mask weaknesses that deserve attention.

This is why critical analysis remains so important. Programs should avoid the temptation to focus exclusively on positive outcomes or isolated concerns. Instead, they should evaluate the totality of the evidence and ask whether the curriculum is accomplishing its intended purpose.

Applying a Consistent Process

Whether evaluating admissions effectiveness or didactic curriculum effectiveness, the same principles we've discussed throughout this series continue to apply.

Programs should:

1. Identify relevant data sources.

2. Establish appropriate benchmarks.

3. Review trends over time.

4. Compare related data sets.

5. Use triangulation whenever possible.

6. Consider contextual factors.

7. Determine whether the evidence supports the conclusion being drawn.

When this process is applied consistently, conclusions become more defensible and more meaningful. Most importantly, programs are better positioned to identify genuine opportunities for improvement while avoiding unnecessary overreaction to isolated findings.

Looking Ahead

In our next and final installment, we'll examine the remaining two C1.01 questions:

• Is the clinical curriculum effective in preparing graduates for clinical practice?

• Overall, does the program effectively prepare graduates for clinical practice?

These questions require programs to step back and evaluate the broader outcomes of the educational process. We'll also conclude the series by discussing how all five questions fit together within a comprehensive framework for program assessment and continuous improvement.

I hope you'll join me as we bring this discussion to a close.