Ten curriculum assessment tools every dean needs. Part 5: Alumni Surveys

Theological school deans are not just theological leaders for their institution, they must be EDUCATIONAL leaders. That is, they must implement sound educational practices related to curriculum, instruction, supervision, assessment, and administration. There is a variety of ways to assess the effectiveness of the curriculum, and there are several levels of assessment (program-level, course-level, student testing, student projects, etc.). While faculty members can focus on course-level and individual student learning assessment, academic deans need to focus on program-level assessment in order to evaluate the effectiveness of the school's curricular course of study. Here are ten basic curriculum assessment tools every academic dean needs:

Outcomes alignment worksheet
Syllabus assessment worksheet
Curriculum maps
Program-level rubrics
Alumni survey
Program retention and completion rate worksheets
Grade Distribution report
Entering student profile
Graduating class profile
Student course evaluations

In this series we review these ten assessment tools every dean needs. This month:

5. Using alumni surveys for program-level assessment.

Assessing your graduates' experiences and perceptions is one way to measure the quality and success of your program. Alumni surveys can be an effective way to find out what graduates think about the effectiveness of your curriculum, that is, how well the program of study prepared them for the practice of ministry or scholarship. The results can give administrators and faculty insight into which areas in the program of study, and of the seminary experience, need improvement. Using this data as part of the school's formative assessment can help ensure current students will be better prepared to face challenges post-graduation.

Equally as important, the survey may highlight the areas where your program excels. Positive data about the effectiveness of your program from alumni is great for promotional material. Indirect assessment and anecdotal information from alumni are also valid sources of information for accreditors.

Here are things to keep in mind when creating an alumni survey:

 

• Create a valid instrument. Be sure your alumni surveys are well-designed and solicit genuinely helpful information. Few things inhibit getting good information from your alumni more than a poorly designed survey.
• Allow respondents to remain anonymous---you'll usually get better data.
• Use online surveys to make taking the survey easier for respondents. Online surveys will also simplify gathering and analyzing the results
• Include in the survey an offer for the graduate to update their alumni information. This will help you, and them, stay connected with your school. Share the information with the alumni and the development offices to help keep the institutional databases up to date.
• Consider including an incentive for completing the survey, like a chance to win a gift card, a school coffee mug, or a copy of a professor's latest published book.
• Set up a periodic schedule for sending alumni surveys. For example, send the survey one, three, and five years after graduation. While comparative data in the first five post-graduation years is helpful, after year five, consider that you may need a different survey to solicit a different data set and a different profile.
• While more challenging, and though it takes more effort, consider administering a survey to your graduates' employers. That feedback will provide additional data about how well equipped graduates were for their current position. If you do so, alert your alumni you will be contacting their employees for data related to alumni. Explain the purpose and describe how the information will be used.

   

  What To Ask on the Alumni Survey
  1. Basic demographic profile data. The demographic profile section of the survey will help you do co-factor analysis of your data. Here are sample items for this section:

  Gender
  Age
  Ethnicity/Race
  Marital status
  Year of graduation
  Degree program
  Geographical location of current position
  Additional degree/schooling post-graduation
  Age when you entered the degree program

  2. Basic ministry demographic profile data. This data will help you do co-factor analysis on the survey responses. Sample questions to ask include:

  Number of churches or organizations served since graduation
  Current place of service and length of service
  Current context (rural, urban, suburban, small town)
  Current size of church or organization
  Longest tenure at ministry location; shortest tenure at ministry location
  Questions about educational debt
  First position post-graduation (first call); current position

  3. Helpful data on curricular programs. Strive to uncover data helpful to evaluate and make decisions about your curricular programs. Ask questions about the perceived effectiveness or value of:

  Supervised ministry and internship experiences
  Coursework
  Degree program for current position
  Would he/she attend your school again?
  Would he/she pursue a different degree program?
  Most helpful components of program of study
  Least helpful components of program of study

  

  4. Helpful data on career and vocation. Gaining data on the vocational and professional experience of graduates can help inform curricular changes to respond to changing ministry contexts. This kind of information is particularly important when faculty engages in curriculum review and revision:

  Denominational affiliation during seminary and at time of survey
  Ordination status during seminary and at time of survey
  Length of time between graduation and first ministry position
  Helpfulness of school in finding first call position
  First ministry position or job after seminary
  Second position or job after seminary
  Current ministry position or job
  Length of tenure in current position
  Number of ministry positions or jobs held after seminary
  Current job profile: full-time, part-time, bi-vocational
  Preferred or anticipated next ministry position or job

  The Association of Theological Schools provides a standard alumni survey for theological schools. The advantage of using it is that it will provide you with comparative data to evaluate your school's alumni responses with those of other theological schools. For example, using data from the ATS Entering Student Questionnaire (ESQ), the Graduating Student Questionnaire (GSQ) with the Alumni Questionnaire (AQ) can provide comparative date helpful for interpreting the scope of the experiences and needs of the students, and, for informing decisions about the curriculum.

  

  Using the data

  It must be stressed that the point of evaluation instruments like the alumni survey is not merely to gather data or information, but to use the information to evaluate and make necessary changes in the programs of study. The design of your alumni survey should allow you to gain insight about what your alumni say through co-factor analysis. A longitudinal comparative study will reveal trends and changing needs of your alumni. You should then use the information, appropriately interpreted, to determine if any changes are necessary to your curriculum, or, to what you offer your alumni by way of continuing education or professional development.

  

  Israel Galindo is Associate Dean, Lifelong Learning at the Columbia Theological Seminary. Galindo writes for the "Along the Journey" blog of the Columbia Theological Seminary, and, the Digital Flipchart blog. His most recent books are Mastering the Art of Instruction: The Nine Essential Instructional Skills Every Teacher Must Master, Stories of the Desert Fathers: Wit And Wisdom for Today's Bewildering Times, Seeking the Holy: An Introduction to the History and Practice of Spiritual Direction for Today's Churches, and Theories of Learning: Approaches to Teaching and Learning for Christian Educators and Theological Faculty