Content Area Analysis
Jesse Pittsley, MA
Assistant Professor
Departmetn of Human Performance and Exercise Science
Winston Salem State University
Winston Salem, NC
"A man with a new idea is a Crank, until the idea succeeds." -- M.
Twain
Several weeks ago, in the Department of Human Performance and Exercise
Science at Winston Salem State University, Dr. Cynthia Stanley a professor
of Therapeutic Recreation, presented her thoughts and ideas about the evaluation
and documentation of student competency in specific content areas.
She stated that, despite the noble attempts of many faculty members to
document objectives and test for specific content areas, it is always difficult
to know if students are proficient in the content areas. Just because
students earn a high grade on a final exam, it does not mean that the students
are competent in all areas of the course content. In fact, when applied
on a larger scale, it is possible that an entire class of graduates may
complete a program and lack competency in specific content areas.
The would be unfortunate because university teachers often assume that
students with high grades have mastered the academic program and its specific
content. Dr. Stanley proposed that more objective steps should be
taken to evaluate if students and the courses have met the appropriate
learning outcomes. In this article, I outline how Dr. Stanley
documented competency in a specific content area using an exercise physiology
example. Also, I exam breifly the feasibility of this method and
its role in the design of an exercise physiology program.
Professional Organizations and the Quest for Competency
Before outlining the method proposed by Dr. Stanley, it is useful to
emphasize that professional organizations are responsible for the implementation
of programs that encourage professional competency and ethical thinking.
To standardize the educational programs and progression, professional organizations
establish accreditation guidelines and certification procedures.
These programs are designed to monitor curriculum content and enhance the
students’ critical thinking skill before entering the profession.
As a result, accreditation and certification are important aspects of the
standardization and quality control of the profession.
Accreditation, in particular, is designed to develop consistency in
the primary courses, learning objectives, and the clinical and/or corporate
hands-on experiences offered by institutions that have specific academic
programs. When an academic program applies for accreditation with
a professional organization, representatives from the professional organization
examine the curriculum, faculty, and relevant teaching resources such as
laboratory equipment and space to see if the program complies with national
standards. Following the evaluation, the professional organization or,
more specifically, the accrediting branch of the organization states that
the academic program is accredited or that it fails to meet the accreditation
guidelines.
Theoretically, once students graduate from an accredited institution,
they should then sit for the certification exam that is linked to the accreditation
and the respective profession. The exam is created by the professional
organization to determine if the graduates of accredited institutions meet
the designated learning outcomes to become a member of a profession.
To create the exams, the learning objectives of the academic courses are
listed and test questions or situations (during the “practical” or “hands-on”
portion of the test) are created from the specified content areas.
As a result, the tests are typically comprehensive exams that measure the
students’ academic performance across the spectrum of content areas.
Most professions have designed accreditation and certification procedures
to ensure quality and consistency among their members. Professions
such as nursing, physical therapy, and teaching have long established accreditation
and certification processes. In exercise physiology, there have been
several certification exams offered by various professional organizations
for over 20 years. Only just recently has the American Society
of Exercise Physiologists (ASEP) implemented the first full accreditation
process for exercise physiology. As a result, exercise physiologists
involved in accreditation are quickly becoming no stranger to academic
standardization and quality control.
Academic Department's Role in the Development of Competency
Considering that a major role of an academic department is to develop
a comprehensive program that prepares students for their respective certification
exams and eventual employment, it is important that each department has
an understanding of its success in teaching specific content areas.
For example, certain exercise physiology programs may have very developed
units and courses teaching topics such as resistance training, but may
lack emphasis in cardiovascular testing and ECG analysis. Or, in
a more hidden sense, departments may consistently produce graduates that
are weak in certain areas despite a good curriculum. Either way,
it is important for the departments to know this information.
One possible approach to exam student performance in specific content
areas is test item analysis. Test item analysis allows exam proctors
to evaluate if certain questions are more consistently answered correctly
or incorrectly. Furthermore, after examining a bulk of tests, it
is possible to detect improperly worded test items or content areas where
student performance is weak. For example, the American Society
of Exercise Physiologists (ASEP) accreditation document [1] requires
that exercise physiology programs teach materials from several course areas
including: muscle physiology, cardiovascular physiology, function anatomy,
ECG interpretation, and other relevant areas. The Exercise
Physiologist Certified (EPC) exam offered by ASEP includes items regarding
those topics. If ASEP were to perform a test item analysis of the
completed exams, certain trends may become evident regarding the difficulty
of certain questions and, from a more global perspective, testing performance
in certain areas.
These types of analyses are not uncommon and are consistently performed
on large-scale standardized tests. Secondary and post-secondary
institutions use the analysis of these exams to evaluate their students’
performance and/or evaluate what type of students attend their institutions
(i.e., students at MIT may score high on the math sections of the SAT).
Unfortunately, these analyses are less common with academic departments
and these data are often not available from professional organizations
offering the certification exams. As a result, this burden may fall
upon the shoulders of the specific academic departments to analyze competency
of certain learning outcomes.
The Itemization of Competency
Dr. Stanley set out to examine this issue on a smaller scale.
Instead of diving head first into a department level learning outcome assessment,
she itemized and examined the performance on one final exam in one course.
To do this, the faculty member must perform the following steps: (1) examine
each item of the test and place each specific question in a knowledge category
that was representative of the course and program objectives; (2) determine
which student responses were correct and incorrect; (3) calculate the total
correct responses in each category from a student and class perspective;
(4) evaluate the student performance of the specific content categories.
This process is not either new or unusual. Although this example
was performed by hand, there are several software packages that offer test
item analysis. For example, one of the most common educational tools
used on college campuses is BlackboardTM
[2].
Item Analysis of an Exercise Physiology Exam
The purpose of this section is to provide a quick example of how the
categorization of test items into their respective content areas allows
for an in-depth analysis of student performances. The test items
below were taken from a practice version of ASEP’s EPC exam.
These questions and answers may be found in the EPC manual [3].
Example Questions from the EPC Exercise Physiology Test
1. The respiratory centers are located in the ___.
a. cerebral cortex
b. medulla oblongata
c. thalamus
d. central thorax
2. The cuff pressure at which the last sound can be detected is the
___ pressure.
a. systolic
b. diastolic
c. mean
d. mean arterial
3. The initial increase in heart rate with exercise appears to be
a result of a(an) _____, and later during exercise a(an) _____.
a. decrease in accelerator tone/increase in vagal tone
b. withdrawal of vagal influence/increase in SNS tone
c. increase in PsNS tone/increase in sympathetic tone
d. none of the above
4. When the inspiratory center is triggered into action, the ___
space becomes ___ subatmospheric, thus causing the ___ space to become
subatmospheric and air moves ___ the lungs.
a. intrapleural/less/intrapulmonary/out
b. intrapulmonary/less/intrapleural/in
c. intrapulmonary/more/intrapleural/out
d. intrapleural/more/intrapulmonary/in
5. When doing a squat exercise, which phase is considered eccentric for
the quadriceps muscle group ___.
a. when the hips, knees, and ankles undergo flexion
b. when the hips, knees, and ankles undergo extension
c. both flexion and extension are eccentric
d. neither a nor b is eccentric
6. All of the following muscles flex the shoulder joint except ___.
a. subscapularis
b. biceps brachii
c. coracobrachialis
d. anterior deltoid
7. Of the following types of resistance training protocols, which is static?
a. circuit
b. isotonic
c. isometric
d. pyramid
8. Regarding the following types of research, which one examines cause-and-effect
relationships?
a. qualitative
b. descriptive
c. longitudinal
d. experimental
9. In research, ___ variables are manipulated by the investigator.
a. dependent
b. representative
c. independent
d. control
10. Researchers use correlational research because it allows them to ___.
a. predict relationships
b. establish cause-and-effect between two variables
c. provide proof to support conclusions
d. all of the above
Now that the test items are listed, I’ve created Table 1 which is a
fictional example of an item analysis and content analysis after 10 students
completed the exam. Each test item is listed with the amount of correct
responses reported in its respective content area. For example, eight
of the 10 students answered question one from the respiratory physiology
category correctly.
Table 1. The amount of correct answers per test item and
the average amount of correct responses per content area in a 10-item exercise
physiology exam taken by 10 students.
Content Areas
| Test Item |
Respiratory Physiology |
Cardiovascular Physiology |
Training |
Research Methods |
|
1
|
8
|
|
|
|
|
2
|
|
7
|
|
|
|
3
|
|
9
|
|
|
|
4
|
8
|
|
|
|
|
5
|
|
|
9
|
|
|
6
|
|
|
3
|
|
|
7
|
|
|
7
|
|
|
8
|
|
|
|
2
|
|
9
|
|
|
|
4
|
|
10
|
|
|
|
3
|
| Average |
8
|
8
|
6.3
|
3
|
The data indicate that students performed strongly in the content areas
of cardiovascular and respiratory physiology, but were weak in the content
area of research methods. On closer analysis, one may also find reason
to investigate questions #6 since few students responded correctly despite
the otherwise strong performance in that category.
Final Thoughts
The example was designed to present the general concept that test items
may be divided into specific content areas and used for analysis.
A short 10-item test was divided into four content areas and the performance
of 10 students was documented. The data indicate that the students
were strong in two categories and possibly weak in another. Overall,
this is good information that could be used to modify course content or
a curriculum. Interestingly, despite the quality information, one
must question if this is feasible, considering the possible effort to perform
such analysis.
Obviously, the analysis would be difficult to perform on a substantial
number of tests through the course of an academic year. The amount
of time and effort each test would require would make the task daunting
considering the rewards. In contrast, it may be feasible to investigate
only comprehensive final exams since the quantity of information would
match the quantity of work. Furthermore, there are several test analysis
programs available that would make this procedure more realistic.
Unfortunately, these systems may limit the type of questions offered since
multiple choices tests are usually required for the quick analysis.
An idea that may make content area analysis more realistic is to offer
a comprehensive exit exam that students would take upon completing a program.
The exam could be offered during the final months of a degree and contain
questions from all the major topic areas covered in the curriculum.
It could also be used as a “warm-up” for the final certification exams
offered by professional organizations. The test items could follow
a multiple-choice format that could be directly entered into a computer
or easily graded. Such an exam would allow programs to assess
if the documented learning outcomes had been addressed properly.
Aside from assessing students at an individual level, exercise physiology
programs could investigate if too much of an emphasis was placed in one
topic area versus another.
As an educator, I am curious how my students have retained the information
that has been provided, and I consider content area analysis as a potential
means of finding the answers.
Note: The author of this paper would like to thank Dr. Cynthia
Stanley, a professor of Therapeutic Recreation here at Winston Salem State,
for making an excellent and inspiring presentation on her adventure with
learning outcome analysis.
References
1. American Society of Exercise Physiologists. (2004). The ASEP Accreditation
Document. [Online]. http://www.asep.org/accreditation/
2. The BlackboardTM Homepage. [Online]. http://www.blackboard.com
3. American Society of Exercise Physiologists. (2004). The ASEP- EPC
Manual. [Online]. http://www.asep.org/epcmanual/