DePaul University Center for Teaching & Learning > Assessment > Assessing Learning > Methodology
Assessment is an ongoing process, rather than a collection of “one-off” projects. Results of assessment should be used to inform continuous improvement of student learning. Commonly, this process is referred to as the assessment loop. The Higher Learning Commission (HLC) provides the following graphic to represent this loop:
DePaul University collects evidence of each curricular and co-curricular program’s involvement in this assessment loop through each program’s definition of program learning outcomes and the annual assessment projects and reports. HLC also asks that we “close the loop” by ensuring assessment is an ongoing, iterative process of collection of evidence of student learning and improvement. Collected over time, reports from each program’s assessment projects demonstrate that DePaul closes the assessment loop.
At the program level, assessment is information or a collection of information centered on student learning. Assessment looks at how much knowledge students attain from their coursework over a program. The results of assessment help in making decisions to improve student learning within that program. Two basic forms of assessment are direct assessment and indirect assessment. These two types of assessment are not mutually exclusive and often work hand-in-hand to give program faculty the best picture of student learning within their programs.
Direct assessment of student learning is tangible, visible, and compelling evidence of exactly what students have and have not learned.
Direct assessment comes in a variety of forms:
Direct assessment asks students to show what they know. Students must be able to execute the skills they learn from the classroom and co-curricular activities.
Indirect assessment consists of proxy signs that students are probably learning. Indirect evidence is less clear and convincing and comes in a variety of forms:
(Palomba & Banta, 2013)
Any of the examples mentioned above, either under direct assessment or indirect assessment, could be either type of assessment. To determine whether an assessment methodology is direct or indirect, there are two important considerations:
*How are course grades indirect assessment?Course grades are based on many iterations of direct measurement. However, they are an indirect measurement of any one learning outcome because:
Challenge:Limited or inconsistent exposure to students (especially in co-curricular units)
Challenge: Limited faculty/staff time
Challenge: Student motivation to fully participate and engage in the assessment. How do we know assessment results are valid if students are not putting forth their best effort?
In the Center for Teaching and Learning (CTL), Lewis 1300 Suite, you can find free books and books in the office library that you can check out, with more information on direct and indirect assessment.
Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and General Education, 2nd Ed. by Barbara E. Walvoord.
Assessment in Practice: Putting Principles to Work on College Campuses by Trudy Banta, Jon P. Lund, Karen E. Black, & Frances W. Oblander.
Assessing General Education Programs by Mary J. Allen.
Assessing Academic Programs in Higher Education by Mary J. Allen.
Learner-centered Assessment on College Campuses: Shifting the Focus from Teaching to Learning by Mary E. Huba & Jann E. Freed.
Outcomes Assessment in Higher Education: Views and Perspectives edited by David Allen
Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education by Catherine A. Palomba & Trudy W. Banta.
Actions Matter: The Case For Indirect Measures in Assessing Higher Education’s Progress On The National Education Goals, by Peter T. Ewell & Dennis P. Jones. Available here: http://www.jstor.org/stable/27797182?origin=JSTOR-pdf
Direct and Indirect Writing Assessment: Examining Issues of Equity and Utility by Ronald H. Heck & Marian Crislip. Available here: http://journals.sagepub.com/doi/pdf/10.3102/01623737023003275
The Philosophy and Practice of Assessment and Evaluation in Higher Education by Alexander W. Astin & Anthony Lising Antonio.
Higher Education: Student Learning, Teaching, Programmes and Institutions by John Heywood.
As the name suggests, an assessment prompt is a “prompt” to provide some type of information. A prompt defines a task; it is a statement or question that tells students what they should do (E.g., in a survey item, essay question, or performance). An effective assessment prompt should “prompt” students to demonstrate the learning outcome that is being assessed.
Suskie, L. (2009). Assessing student learning: A common sense guide. Jossey-Bass: San Francisco, CA.
These prompts limit the way in which students may present information. Multiple choice items are an example of a restricted response prompt.
These prompts give students latitude in deciding how to respond or provide information. Students may have flexibility in the format, length, and/or construction of their responses.
“You are there” ScenariosThese prompts ask students to put themselves into a situation and respond. “You are there” prompts are a good way for students to demonstrate their ability to integrate and apply knowledge they gained from a particular program.
Below are a few potential “you are there” prompts, but the possibilities are endless. An advantage of this type of prompt is the ability to customize the prompt to the types of situations students are likely to face in your particular field.
Surveys will generally be used as a secondary assessment methodology since the primary assessment methodology should be a direct measure of student learning. Please note that surveys may be a direct measure of student learning if they ask students to demonstrate what they have learned, rather than asking for students’ opinions about what they have learned, the program, etc.
Triangulation is defined as “multiple lines of evidence that lead to the same conclusion.” Experts recommend that a student’s learning should be measured in several different ways. When we triangulate, or use different types of measures, more accurate conclusions about student learning are produced. (Allen, 2004, p. 172) Triangulation is more than just using different types of assessment measures, but also assessing students in multiple time intervals, such as the end of a chapter or unit, end of the quarter, or end of the semester. Through multiple assessment measures and time intervals, patterns and inconsistencies in student learning can be found and tackled. (Landrigan, C. & Mulligan, T.)
Multiple methods used to triangulate include:
(Banta et al., 1996, 101-104) (Landrigan, C. & Mulligan, T.)
Allen, M.J. (2004). Assessing academic programs in higher education. Bolton, Massachusettes: Anker Publishing Company, Inc.
Banta, T.W., Lund, J.P., Black, K.E., & Oblander, F.W. (1996). Assessment in practice: Putting principles to work on college campuses. San Francisco: Jossey-Bass Publishers.
Landrigan, C. & Mulligan, T. Triangulating: The importance of multiple data points when assessing students. Retrieved from http://www.choiceliteracy.com/articles-detail-view.php?id=525
Reliability and validity are important concepts in any form of inquiry, including assessment. However, it is also important to note that there are usually not the same demands for reliability and validity in assessment as there may be in research. While both are important considerations in the assessment of student learning outcomes, it’s also important to not become paralyzed in your perceived ability to draw conclusions from your assessment projects because you are unsure of the reliability or validity of your measurement. Instead, it may be more useful to allow these considerations to temper your conclusions.
As the name suggests, reliability refers to how consistent (or reliable) the results you achieve by assessing student learning are. How much a student has learned over the course of a program should be relatively stable and not change drastically in a short period of time or depending on who assesses the student.
Commonly, the issue of reliability comes up when multiple people are rating a piece of student work to determine how much a student has learned. Faculty and staff looking at an identical piece of student work should be arriving at identical (or very similar) conclusions. A few strategies to help improve the reliability of your assessment based on this issue include:
Another common issue with reliability arises when using assignments from multiple courses or course sections to assess program learning outcomes. In this situation, reliability issues may occur because assignments from different courses or course sections may not be asking students to demonstrate the same knowledge, skills, or abilities. The best strategy to address this sort of reliability issue is to work on better communication among faculty and staff about a variety of issues, including:
Validity is concerned with the accuracy of the conclusions you draw based on the use of a measurement instrument. Validity is a concept that refers primarily to the conclusions you draw from conducting an assessment project. A common misconception is that validity refers to the instrument being used to measure a learning outcome. However, validity is not inherent to a measurement tool or instrument. In fact, any instrument that has high validity for one purpose will almost surely not be valid for another. For example, a driver’s test may have high validity for making conclusions about a person’s ability to safely drive a car, but would be ridiculous for making determinations about a student’s ability to communicate effectively in writing. Also, validity may be specific to the population that was studied and one should not assume because an instrument was valid for one population it will necessarily be valid for a different population. For example, an instrument that you used to measure undergraduate students’ achievement of a learning outcome may not be valid for measuring a similar learning outcome for graduate students.
There are different aspects of validity and while these were formerly assessed separately, Samuel Messick (1989) suggests that all aspects fall under construct validity in what is commonly referred to as the unified theory of validity. Messick advocates for making an argument for the validity of a measurement tool based on these different aspects of construct validity. For an inference or conclusion to be ‘valid,’ all aspects of validity should be considered. In other words, a single aspect of validity should not be considered sufficient for drawing conclusions about the validity of conclusions or inferences being drawn.
At its simplest level, construct validity simply means that the instrument being used to measure a particular construct is fully measuring that construct and only that construct.