DePaul University Center for Teaching & Learning > Assessment > Resources > Methodology
The Center for Teaching and Learning is frequently asked questions like, “how should we collect assessment data?” or “what methodology would be best for our assessment project?” Despite the huge variety of programs and learning outcomes at DePaul University, we have the same answer for everyone: it depends. While this may not be a particularly satisfying answer to many who hope assessment of student learning is black and white with clear distinctions between what methodologies would be correct and which are wrong, it is the most honest answer we can give.
Much like research, picking a methodology for assessment starts with asking questions. The choice of an assessment methodology depends on a variety of factors:
Answers to these types of questions can help guide decisions about the most appropriate methodology to use to assess students’ achievement of the learning outcome you are assessing.
The Center for Teaching and Learning is happy to facilitate discussions with program faculty and staff regarding best assessment methodology given the nature of the specific assessment project you are pursuing.
Assessment is an ongoing process, rather than a collection of “one-off” projects. Results of assessment should be used to inform continuous improvement of student learning. Commonly, this process is referred to as the assessment loop. The Higher Learning Commission (HLC) provides the following graphic to represent this loop:
DePaul University collects evidence of each curricular and co-curricular program’s involvement in this assessment loop through each program’s definition of program learning outcomes and the annual assessment projects and reports. HLC also asks that we “close the loop” by ensuring assessment is an ongoing, iterative process of collection of evidence of student learning and improvement. Collected over time, reports from each program’s assessment projects demonstrate that DePaul closes the assessment loop.
At the program level, assessment is information or a collection of information centered on student learning. Assessment looks at how much knowledge students attain from their coursework over a program. The results of assessment help in making decisions to improve student learning within that program. Two basic forms of assessment are direct assessment and indirect assessment. These two types of assessment are not mutually exclusive and often work hand-in-hand to give program faculty the best picture of student learning within their programs.
Direct assessment of student learning is tangible, visible, and compelling evidence of exactly what students have and have not learned.
Direct assessment comes in a variety of forms:
Direct assessment asks students to show what they know. Students must be able to execute the skills they learn from the classroom and co-curricular activities.
Indirect assessment consists of proxy signs that students are probably learning. Indirect evidence is less clear and convincing and comes in a variety of forms:
(Palomba & Banta, 2013)
Any of the examples mentioned above, either under direct assessment or indirect assessment, could be either type of assessment. To determine whether an assessment methodology is direct or indirect, there are two important considerations:
*How are course grades indirect assessment?Course grades are based on many iterations of direct measurement. However, they are an indirect measurement of any one learning outcome because:
In the Center for Teaching and Learning (CTL), Lewis 1300 Suite, you can find free books and books in the office library that you can check out, with more information on direct and indirect assessment.
Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and General Education, 2nd Ed. by Barbara E. Walvoord.
Assessment in Practice: Putting Principles to Work on College Campuses by Trudy Banta, Jon P. Lund, Karen E. Black, & Frances W. Oblander.
Assessing General Education Programs by Mary J. Allen.
Assessing Academic Programs in Higher Education by Mary J. Allen.
Learner-centered Assessment on College Campuses: Shifting the Focus from Teaching to Learning
Outcomes Assessment in Higher Education: Views and Perspectives edited by David Allen
Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education by Catherine A. Palomba & Trudy W. Banta.
Actions Matter: The Case For Indirect Measures in Assessing Higher Education’s Progress On The National Education Goals, by Peter T. Ewell & Dennis P. Jones. Available here: http://www.jstor.org/stable/27797182?origin=JSTOR-pdf
Direct and Indirect Writing Assessment: Examining Issues of Equity and Utility by Ronald H. Heck & Marian Crislip. Available here: http://journals.sagepub.com/doi/pdf/10.3102/01623737023003275
The Philosophy and Practice of Assessment and Evaluation in Higher Education by Alexander W. Astin & Anthony Lising Antonio.
Higher Education: Student Learning, Teaching, Programmes and Institutions by John Heywood.
As the name suggests, an assessment prompt is a “prompt” to provide some type of information. A prompt defines a task; it is a statement or question that tells students what they should do (E.g., in a survey item, essay question, or performance). An effective assessment prompt should “prompt” students to demonstrate the learning outcome that is being assessed.
Suskie, L. (2009).
Assessing student learning: A common sense guide. Jossey-Bass: San Francisco, CA.
These prompts limit the way in which students may present information. Multiple choice items are an example of a restricted response prompt.
These prompts give students latitude in deciding how to respond or provide information. Students may have flexibility in the format, length, and/or construction of their responses.
Suskie, L. (2009). Assessing student learning: A common sense guide. Jossey-Bass: San Francisco, CA.
“You are there” ScenariosThese prompts ask students to put themselves into a situation and respond. “You are there” prompts are a good way for students to demonstrate their ability to integrate and apply knowledge they gained from a particular program.
Below are a few potential “you are there” prompts, but the possibilities are endless. An advantage of this type of prompt is the ability to customize the prompt to the types of situations students are likely to face in your particular field.
Surveys will generally be used as a secondary assessment methodology since the primary assessment methodology should be a direct measure of student learning. Please note that surveys may be a direct measure of student learning if they ask students to demonstrate what they have learned, rather than asking for students’ opinions about what they have learned, the program, etc.
Survey Fatigue – You are not the only one asking your students to complete surveys (E.g., institutional surveys, other programs, restaurants, grocery and retail stores, etc.)
Instructions should clarify the purpose of the survey and provide respondents with expected procedures for responding to the survey instrument.
Example: “This survey is designed to determine your feelings about several current human rights issues. Please read each question carefully. For each question, please consider each response option and choose the one option that best matches your feelings about the issue raised in that question.”
The higher the time commitment to complete, the less likely students are to complete it.
Group questions about similar concepts/topics together.
Avoid double-barreled questions.
Example: In the question, “How satisfied are you with the depth and breadth of content covered in this class?” respondents must indicate their satisfaction with two different things ‘depth’ and ‘breadth.’ In this case, they may have different levels of satisfaction with depth than they do with breadth, making it difficult to provide one response to this question.
Avoid questions that request second-hand knowledge.Example: “How happy was your cohort with the experience of taking all core courses as a group?” In this question the respondent only has access to their own happiness with the cohort experience. In responding to this question they may indicate their own happiness, assuming everyone felt the same way, or have to guess how happy the rest of the students seemed with the experience.
Avoid retrospective questionsExample: “How comfortable did you feel with this content before you started this course? How comfortable do you feel with it now?” Students may not reliably have access to their comfort level before they started. A pre- post- design may be better if this is the information you are trying to access.
Avoid leading questions.Example: “Considering the horrible human rights atrocities in countries such as Russia and North Korea, how do you feel about Communism as a form of government?” Clearly, the person who wrote this question is not a proponent of Communism and the question leads respondents to provide a similar viewpoint. People will be sensitive to social desirability cues from the leading nature of this question and will be unlikely to indicate support for Communism.
Avoid questions that make assumptions.Example: “To what extent do you agree that a change in state laws would be the most effective way to support gay rights in the United States?” This question assumes the respondent is a proponent of gay rights. It will not be clear when someone disagrees with this question whether they are disagreeing with the idea that the country should support gay rights or if they are disagreeing with the proposition that changes in state laws are the most effective way to support gay rights in the country.
Example: In the question “How satisfied are you with the variety of electives available to you in this program?” you could reasonably expect that anyone in the program should have an opinion about the variety of electives available – even if it is only slightly positive or slightly negative. In this case, it is better not to include a neutral option since a neutral or middle category may indicate a variety of things other than true neutrality (i.e. someone who is unwilling to respond to the question, someone who is ambiguous, someone who does not understand the question, someone who feels there is no response option that describes their opinions or attitudes, etc.).
For scaled response options, four response options is generally adequate if you are not including a neutral response option and five response options is adequate if you will include a neutral response option.
Triangulation is defined as “multiple lines of evidence that lead to the same conclusion.” Experts recommend that a student’s learning should be measured in several different ways. When we triangulate, or use different types of measures, more accurate conclusions about student learning are produced. (Allen, 2004, p. 172) Triangulation is more than just using different types of assessment measures, but also assessing students in multiple time intervals, such as the end of a chapter or unit, end of the quarter, or end of the semester. Through multiple assessment measures and time intervals, patterns and inconsistencies in student learning can be found and tackled. (Landrigan, C. & Mulligan, T.)
Multiple methods used to triangulate include:
(Banta et al., 1996, 101-104) (Landrigan, C. & Mulligan, T.)
Allen, M.J. (2004).
Assessing academic programs in higher education. Bolton, Massachusettes: Anker Publishing Company, Inc.
Banta, T.W., Lund, J.P., Black, K.E., & Oblander, F.W. (1996).
Assessment in practice: Putting principles to work on college campuses. San Francisco: Jossey-Bass Publishers.
Landrigan, C. & Mulligan, T.
Triangulating: The importance of multiple data points when assessing students. Retrieved from http://www.choiceliteracy.com/articles-detail-view.php?id=525
Reliability and validity are important concepts in any form of inquiry, including assessment. However, it is also important to note that there are usually not the same demands for reliability and validity in assessment as there may be in research. While both are important considerations in the assessment of student learning outcomes, it’s also important to not become paralyzed in your perceived ability to draw conclusions from your assessment projects because you are unsure of the reliability or validity of your measurement. Instead, it may be more useful to allow these considerations to temper your conclusions.
As the name suggests, reliability refers to how consistent (or reliable) the results you achieve by assessing student learning are. How much a student has learned over the course of a program should be relatively stable and not change drastically in a short period of time or depending on who assesses the student.
Commonly, the issue of reliability comes up when multiple people are rating a piece of student work to determine how much a student has learned. Faculty and staff looking at an identical piece of student work should be arriving at identical (or very similar) conclusions. A few strategies to help improve the reliability of your assessment based on this issue include:
Another common issue with reliability arises when using assignments from multiple courses or course sections to assess program learning outcomes. In this situation, reliability issues may occur because assignments from different courses or course sections may not be asking students to demonstrate the same knowledge, skills, or abilities. The best strategy to address this sort of reliability issue is to work on better communication among faculty and staff about a variety of issues, including:
Validity is concerned with the accuracy of the conclusions you draw based on the use of a measurement instrument. Validity is a concept that refers primarily to the conclusions you draw from conducting an assessment project. A common misconception is that validity refers to the instrument being used to measure a learning outcome. However, validity is not inherent to a measurement tool or instrument. In fact, any instrument that has high validity for one purpose will almost surely not be valid for another. For example, a driver’s test may have high validity for making conclusions about a person’s ability to safely drive a car, but would be ridiculous for making determinations about a student’s ability to communicate effectively in writing. Also, validity may be specific to the population that was studied and one should not assume because an instrument was valid for one population it will necessarily be valid for a different population. For example, an instrument that you used to measure undergraduate students’ achievement of a learning outcome may not be valid for measuring a similar learning outcome for graduate students.
There are different aspects of validity and while these were formerly assessed separately, Samuel Messick (1989) suggests that all aspects fall under construct validity in what is commonly referred to as the unified theory of validity. Messick advocates for making an argument for the validity of a measurement tool based on these different aspects of construct validity. For an inference or conclusion to be ‘valid,’ all aspects of validity should be considered. In other words, a single aspect of validity should not be considered sufficient for drawing conclusions about the validity of conclusions or inferences being drawn.
At its simplest level, construct validity simply means that the instrument being used to measure a particular construct is fully measuring that construct and only that construct.