Rubric on How to Evaluate an Institutional Effectiveness Report


© 1999 by Dr. Thomas W. MacFarland -- All Rights Reserved





Academic Center ..... XXXXXXXXXX
Dean ................ XXXXXXXXXX
Contact ............. XXXXXXXXXX 
Reviewer ............ XXXXXXXXXX
Major(s) ............ XXXXXXXXXX
                      XXXXXXXXXX
Draft # _ ........... X of X  
Due Date ............ XXX-XX-XX
Review Date ......... XXX-XX-XX

Greetings:

Place a general greeting and broad comments here.

The completed rubric comes after the comment section.

Best wishes.

Tom MacFarland

Typical Comments
----------------

1.  Avoid acronyms.

2.  Identify individuals by title, not name.

3.  Provide discussion about enrollment trends and do not say "Review Table
    X for information about five-year enrollment trends." 

4.  Be sure to compare your document against the University style sheet.  
    As an example, be sure to review the appropriate use of the word 
    "percent" v.  use of the percent symbol.  Did you use the University
    Style Manual for this presentation?

5.  It might be better to say "part-time faculty" instead of "adjuncts."

6.  Are you sure that you want to go into detailed discussion on breakout
    faculty evaluation by type of terminal degree?

7.  In Section X you offer Mean and SD for student satisfaction with the 
    distance education delivery format.  That is grand.  Do you have a 
    similar set of statistics for face-to-face instruction.  If you do, 
    please add it to offer comparison and context.

8.  Do you need to add any additional information to the introduction?

9.  For Assessment Area 1, please write "Using data provided by the Office
    of [ ]" if another University office provides the data.

10. Throughout this report, you mention the use of inferential statistics,
    such as the use of Analysis of Variance (ANOVA).  I'll ask three
    broad questions:  

    (1) Why do you wish to use ANOVA?

    (2) How will you effect the analysis?

    (3) Have you selected .05 or .01 as the declared significance level?

13. You also mention the use of graduate survey.  Let me ask:

    (1) Can you please edit your discussion on the survey?  You have more
        detail than needed in the Methods of Assessment section.

    (2) Are you OK with time and process on survey distribution and data
        entry/analysis?

    (3) Will you be able to present survey outcomes, specific for your
        program, in table format?


Rubric
------


             Section I    

             Program Goal

01.  Y or N  Does the report identify the primary goal of
             the major?

             Yes or No

02.  Y or N  Was the goal stated in a single sentence?    

             Yes or No

03.  Y or N  Did the single goal statement capture the 
             primary purpose of the program?

             Yes or No


             Program Overview

04.  Y or N  Does Section I include an overview/history of
             the program at the University?

             Yes or No

05.  Y or N  Does Section I provide the reader with a sense 
             of the program, especially for an outside reader 
             who is not familiar with the discipline or the
             University?

             Yes or No

06.  Y or N  Does Section I address questions that are
             frequently asked by outside constituencies:
             general information, enrollment trends, degrees 
             awarded, information on program faculty?

             Yes or No

07.  Y or N  Does Section I include additional information
             and/or subsections?

             Yes or No

08.  Y or N  Does Section I provide any interpretation
             of the data?

             Yes or No

09.  Y or N  Does Section I provide any discussion on the
             significance of observed trends? 
            
             Yes or No


             Program Overview - General Information

10.  Y or N  Does Section I include a listing of professional
             or special accreditations for the program?

             Yes or No

11.  Y or N  Does Section I include a listing of any other
             special recognitions on the quality of the 
             program?

             Yes or No

12.  Y or N  Does Section I include any reference to the
             following:

             -- target populations for the major

             Yes or No

             -- admissions requirements (GPA, SAT, etc.)

             Yes or No

             -- graduation requirements

             Yes or No

13.  Y or N  Does Section I discuss the delivery modes used
             in the major?

             Yes or No

14.  Y or N  Does Section I discuss if the entire degree program
             can be completed online or through any other unique
             delivery format?

             Yes or No


             Program Overview - Enrollment and Degrees Awarded

15.  Y or N  Does Section I include enrollment trends data,
             focusing on the last five years and including
             breakouts by race/ethnic and gender?  Was the
             data resource identified?

             Yes or No

16.  Y or N  Does Section I include an interpretation and
             discussion of enrollment trends data?

             Yes or No

17.  Y or N  Does Section I include data on the number of
             degrees awarded in the major during the 
             previous academic year.  Was the data resource
             identified?                    

             Yes or No


             Program Overview - Program Faculty

18.  Y or N  Does Section I include a brief overview of faculty
             teaching courses in the major:

             -- N full-time faculty teaching courses in the
                major during Fall Term 20XX

             Yes or No

             -- N part-time faculty teaching courses in the
                major during Fall Term 20XX

             Yes or No

             -- student:faculty ratio in the major, using the
                algorithm

                         (N Fall Term 20XX Enrollment)
                -----------------------------------------------
                ((N Full-Time Faculty) + (N Part-Time Faculty))

             Yes or No

             -- information on faculty diversity

             Yes or No

             -- information on faculty quality

             Yes or No


             Section II   

             Assessment of the Expected Results of the 
             Curriculum

01.  Y or N  Does Section II have any discussion on the
             use of alternate delivery methods and subsequent
             comparison of non-traditional format(s) v.
             traditional format(s)? 

             Yes or No

02.  Y or N  Was Section II organized using the format:  
             (1) statement of expect results, (2) methods 
             of assessment, (3) results of assessment?

             Yes or No

03.  Y or N  Were expected results presented in measurable
             terms, written in future tense?

             Yes or No

04.  Y or N  Were expected results focused on expected   
             levels of student achievement?

             Yes or No


             Assessment Area 1 - Graduation Rates

05.  Y or N  Did the report include an expected result   
             for graduation rates over a prescribed period 
             of time?  Was the data resource identified?

             Yes or No


             Assessment Area 2 - Summative Evaluation of 
                                 Student Knowledge

06.  Y or N  Did the report include an expected result   
             specific to levels of student achievement 
             in areas as diverse as mastery of the 
             knowledge base, competencies, and skills 
             associated with the program?  As data apply,
             student achievement data could include:

             -- pass rates in capstone course(s)

             Yes or No

             -- pass rates in standardized exams

             Yes or No

             -- pass rate for internship/practicum

             Yes or No

             -- pass rate for comprehensive examinations

             Yes or No

             -- pass rate for thesis research      

             Yes or No

             -- pass rate for dissertation research 

             Yes or No

             -- pass rate for external examinations
                (i.e., state boards, national boards,
                licensing exams, certification exams)

             Yes or No


             Assessment Area 3 - How Well the Program 
                                 Prepared Graduates

07.  Y or N  Did the report include an expected result    
             specific to student achievement and consequent
             levels of employment, advanced study, or other 
             indicators associated with career development 
             and career/personal advancement?

             Yes or No

08.  Y or N  Did the presentation for Assessment Area 3
             focus on methods and data that allowed some
             level of quantification?  Specifically, did
             the presentation for Assessment Area 3 avoid
             anecdotal comments that may not represent 
             group outcomes? 

             Yes or No

             As data apply, student achievement data could 
             include results from:

             -- exit survey of current graduates

             -- followup survey of prior graduates

             -- employer survey

             Is a copy of any survey instrument(s) on file
             with appropriate University offices? 

             Yes or No

09.  Y or N  If a survey instrument were used for Assessment
             Area 3, was the instrument described in adequate
             detail?  Was survey design explained?  Were the
             response scales identified?  Was a table of 
             results provided and did the table provide
             descriptive information for all statements:
             N, Mode, Median, Mean/SD, Percent Response, etc.

             Yes or No


             Other Issues 

10.  Y or N  Did the report include other statements of
             expected results, beyond the three measures 
             that are required?

             Yes or No

11.  Y or N  Did the assessment method(s) meet standard    
             reporting requirements, such as identification
             of time period, description of participants,
             description of data and methodology?

             Yes or No

12.  Y or N  Do the results summarize the data that were
             actually collected?

             Yes or No

13.  Y or N  Overall, does Section II meet the methodology
             and reporting requirements typically associated 
             with social research?       

             Yes or No


             Section III  

             How Assessment Findings Were Used to Guide 
             Changes for Improvements in the Program/Major

01.  Y or N  Does the report include a discussion on the 
             significance and implications for each of the
             three (or more) expected results?

             Yes or No

02.  Y or N  Does the report include an analysis of how  
             results had an impact on the curriculum?

             Yes or No

03.  Y or N  Does the report include an analysis of how  
             results can be used to effect improvements?

             Yes or No

04.  Y or N  Does the report include a discussion on     
             strengths and weaknesses of the program?

             Yes or No

05.  Y or N  Does the report include discussion that links
             assessment results with changes to the curriculum 
             (or policy, administrative practices, etc.) that 
             were implemented, as a result of this reporting 
             process, either in the short-term or the long-term?

             Yes or No

06.  Y or N  Was any historical information provided that 
             demonstrated improvements over time?

             Yes or No


Editorial Notes and Conventions

Report Template

             Did the writer use the standardized report
             template, prepared by the Office of XXX?      

             Yes or No

             Was the interim report presented in paper-copy
             only?

             Yes or No

             Was the final report presented in both paper-copy 
             and electronic format (using the template prepared
             by the Office of XXX)?

             Yes or No

Report Length

             Was the report restricted to 15 pages in 
             length?

             Yes or No

             Were all data and tables included in the 
             report and NOT presented in attachment/appendix 
             format?

             Yes or No

Stylistic Conventions

             Did the program goal start with "To ..."?

             Yes or No

             Were statements of expected results of
             the curriculum presented as "It is expected
             that ..."?

             Yes or No

             Reports from all units will be published as a 
             compendium. Therefore, it is important that all 
             reports are in the same layout and format, and
             that the writing is of publishable quality. 

             Yes or No

             Were acronyms and abbreviations avoided?

             Yes or No

             Was technical jargon avoided?

             Yes or No

             Were individuals identified by position title, 
             not name?

             Yes or No

                               ##

Let me know how I can assist you in the areas of institutional research, statistics, and computer science education.


Please send comments or suggestions to Dr. Thomas W. MacFarland

There have been visitors to this page since February 1, 1999.