Successful Assessment Grant Proposal: Sociology

Focus: Building exam question banks calibrated for difficulty level and representative of program content and a standardized process for administering, reporting, and analyzing the exam results.

Project information

Project title: Development of Online Jury Assessment Platform for Applied Music Instruction

Contact person: David Merolla
Role/Position: Department Chair
Department/Unit: Sociology
Program name: Bachelor of Arts in Sociology
Participating colleagues/collaborators: 

• R. Khari Brown, Sociology
• Zachary Brewster, Sociology

Reason for proposal:

1. How is program-level assessment currently done in your program?

Starting in AY20-21, the sociology department changed our assessment approach from a qualitative assessment of papers written in the Sociology Capstone (SOC 4996) class to an objective multiple-choice exam which students take in the Sociology Capstone class.  The current assessment instrument is a 60-question multiple choice exam with 3-7 questions for each of our 11 program learning outcomes. We administer the course in SOC 4996 because this is the final course most sociology students take before graduation.

This change was made to better align our assessment method with our program learning outcomes and curriculum map.  For instance, we often noticed that we were assessing writing skills, which are not part of the sociology curriculum map or a program learning outcome.  In addition, it was difficult to assess all program learning outcomes depending on the specific topic each students' paper.  The new method has proven to assess students on a wider breadth of sociological knowledge and provide more insight into what students know when they complete a BA in sociology compared to a review of the capstone papers. This additional insight can help shore up problem areas where our curriculum is falling short of our program learning outcomes.

2. What, specifically, needs development or improvement with respect to those assessment practices, processes, or instruments?

There are three aspects of the current assessment instrument that we would like to improve upon.  First, the assessment instrument needs to be better calibrated to both difficulty level and what instructors teach in courses across the sociology curriculum map. Some questions on the exam are very difficult and somewhat ambiguous.  Second, we would like to develop question banks for each learning outcome so that questions can be alternated, and every student does not get the exact same questions each semester.  Third, we need a standardized system for the instructor of Sociology Capstone to efficiently administer the exam and get the exam results for the assessment process.  The past two years, the instructor has simply given the assessment coordinator the paper copies of the exam that was given in the Sociology Capstone course. The standardized online process for administration will also allow the assessment coordinator to more readily identify problematic questions that can be reviewed and revised or deleted for future exams. Specifically, we will develop automated processes (i.e., macros) in SPSS or excel to identify potentially problematic questions (e.g., those with extremely high correct or incorrect answers).

3. What factors or conditions have contributed to the area(s) needing development or improvement?

Several factors have contributed to the need for improvement for the assessment instrument and process.  First, the department had significant turnover of administrative positions (undergraduate director, graduate director, department chair) during the COVID19 pandemic.  This timing led to less-than-optimal continuity in the assessment process and effectively only three faculty members with any substantive role in our assessment process. Second, the various responsibilities for assessment have been unclear. While the departmental undergraduate director has been the default BA assessment coordinator, the instructor of Sociology Capstone has a key role in integrating the exam into their course.  Third, the department planned a more inclusive approach to developing and refining the exam, yet this effort was mothballed during COVID shutdowns and disruptions. Thus, the current exam was primarily written by the three members of the undergraduate committee with limited input from other faculty who teach important courses in the sociology curriculum map.

Proposed actions and expected impact:

1. What steps will you take to improve your program's learning outcomes assessment practices, processes, or instruments if you receive funding? Be as specific as you can about the link between the needed development or improvement(s) and your proposed actions.

We are proposing to engage in a comprehensive review and update of the current assessment instrument and administration and to engage all faculty more actively in the assessment process.  Specifically, we propose to work with faculty who teach various courses in our curriculum map to solicit feedback to improve existing questions or create new questions for each program learning outcome.  The goal is to have a question bank of at least 10-15 questions supplied and/or approved by faculty teaching relevant courses for each program learning outcome. This process will ensure that the questions on the exam are of appropriate difficulty level and are reflective of content taught in courses across the sociology curriculum. The engagement of all faculty will also lead to more recognition among faculty that everyone has a role in program assessment.  In addition, in winter 2023 we will hold a department assessment review retreat during which we will go over each question as a team and have additional faculty centered discussions on how to operationalize our learning outcomes with effective exam questions. We will also develop and pilot the online exam through Canvas, Qualtrics or other system to make the exam easier for the instructor of the capstone class to administer and get the results to the assessment coordinator, regardless of who is instructing the course.

2. How will your assessment practices, processes, or instruments grow or improve as a result of your project?

The result of this project will make our assessment process more effective and efficient and will engage more faculty in the assessment process. For instance, the assessment coordinator can share information with the faculty regarding specific questions with high or low rates of correct answers, so faculty can see what kinds of material students are retaining from their course.  If faculty have supplied the exam questions or given input on the process, they will have an increased amount of ownership over the results- a process that will lead to more faculty buy in and engagement with program assessment. Further, by streamlining the assessment process, our department can engage faculty in the substantive aspects of evaluation.

3. If this program has received a WSU Program Assessment Grant in the past, how does the current project differ from the previous grant project?

NA

Assessment expertise:

1. What experience in assessment does your team bring to the project? (NB: The goal of this question is to understand the existing assessment expertise among the collaborators. Prior experience in assessment is not a requirement of the grant.)

David Merolla has served as assessment coordinator for both the undergraduate and graduate programs and was undergraduate director at the time of the switch to a multiple-choice assessment format.  As current department chair, he oversees all assessment activities in the sociology department.

Zachary Brewster has served as assessment coordinator for both the undergraduate and graduate programs.  He regularly teaches SOC 4996 Capstone in Sociology.

R. Khari Brown is the current undergraduate coordinator.  He has overseen assessment of the BA program in the past and regularly teaches SOC 4996 Capstone in Sociology.

2. What assistance, if any, do you need from experts in assessment or in other areas to carry out your project? Examples might include survey, test, or activity design support, statistical analysis, etc.

Any guidance on possible systems to use to create a portable exam that can be administered by any instructor would be very much appreciated.

Deliverables, responsible parties, timeline:

Funding request:

Comments, explanations, or clarifications of funding request:  

Our project consists of hiring a student assistant to coordinate the revision and implementation of the new assessment exam.  A process that will take place over the AY22-23.  We will begin to conduct our review of the current exam, how it relates to our curriculum map and where the improvements are most needed in the fall 2022 term.  During this term, we will begin to revise the exam with a higher priority on getting a version of the current exam into an online format for administration and sharing across different faculty members.  Our main goal will be to administer a version of the exam online during fall 2022.  As we move into winter 2023, we will continue to revise our exam question bank with input from all faculty.  Our goal will be to have the revised exam ready to administer in the W23 Capstone course.   The costs for the project are primarily for the student assistant (roughly 5 hours per week, 30 weeks, $15 per hour= 2,250) and to provide food for a department assessment retreat held during winter 2023.

Return to main WSU Program Assessment Grants page