RATING SCALE – FULL REVIEW

The rating scales are based on several existing frameworks including the Better Evaluation Rainbow Framework and the National Health and Medical Research (NHMRC) Body of Evidence matrix. The scales have been designed to be flexible and inclusive of different types of programs and services, and to take into consideration the variance in research and evaluation evidence behind different programs and services.

 

 

Part 1: Research / Evaluation quality

 

 

 

 

Review criteria

 

Rating scale

1

Plan and Scope: Evidence provided that the research/evaluation was well managed (e.g. evidence of stakeholder engagement; evidence that ethical and quality evaluation standards were defined and followed; evidence that planning documents were developed such as an evaluation framework and/or program logic model; appropriateness of staff that undertook the research/evaluation).

0. Not at all

1. Minimal

2. Moderate

3. Strong

2

Frame: Evidence provided that clear parameters were set for the research/evaluation in terms of its purpose and key evaluation questions.

0. Not at all

1. Minimal

2. Moderate

3. Strong

3

Describe: Clear description provided of: the sample/participants; measures; how data were collected; how data were managed; how data were analysed; key findings.

0. Not at all

1. Minimal

2. Moderate

3. Strong

4

Report and support use: Evidence provided that research/evaluation findings have been presented in ways that are useful for the intended users of the evaluation, and/or evidence the research/evaluation findings have been used.

0. Not at all

1. Minimal

2. Moderate

3. Strong

 5a

Score:

= Sum of responses to questions 1 to 4 where:

Not at all = 0; Minimal = 1; Moderate = 2; and Strong = 3

5b

Research/evaluation quality rating:

Very strong = 10 or higher

Strong = 8 or 9

Sufficient = 7

Weak in methodology/conduct = 6 or lower

 

  • Very strong
  • Strong
  • Sufficient
  • Weak in methodology/conduct

 

 

 

 

 

 

Part 2: Overall assessment

 

 

 

 

Review criteria

 

Rating scale

6

Research/evaluation quality score (take result from Part 1 scoring)

A. Very strong

B. Strong

C. Sufficient

D. Weak in methodology/conduct

7

Impact

A. It can be demonstrated that the program or service is causing a change among those who have used the program or service (where the change aligns with the program or service aims) by showing less impact among those who don't receive the program or service.

 

B. Data have been gathered that show some change among those who have used the program or service (where the change aligns with the program or service aims).

 

C. There is a reason (or reasons) to conclude the program or service has the impact(s) it aims to achieve.

 

D. There is no indication that the program or service has the impact(s) it aims to achieve OR There is indication that the program or service has a negative impact(s) and/or adverse outcomes.

8

Generalisability

A. The population(s) studied in the body of evidence are the same as the target group of the program or service.

 

B. The population(s) studied in the body of evidence are similar to the target group of the program or service.

 

C. The population(s) studied in the body of evidence differ to the target group of the program or service but it is reasonable to apply the evidence to the target group of the program or service.

 

D. The population(s) studied in the body of evidence differ to the target group of the program or service and it is hard to judge if it is reasonable to apply this evidence to the target group of the program or service.

9

Applicability

A. Directly applicable to outcomes and impacts identified for The Hub.

 

B. Applicable to outcomes and impacts identified for The Hub with a few caveats.

 

C. Probably applicable to outcomes and impacts identified for The Hub with some caveats.

 

D. Not applicable to outcomes and impacts identified for The Hub.

 

10

Final rating

Rating scale:

 

Excellent = A. for Evaluation quality; A. for Impact; A. or B. for Generalisability; A. for Applicability

 

Good = A. or B. for Evaluation quality; A. or B. for Impact; A. or B. for Generalisability; A. or B. for Applicability

 

Satisfactory = A., B. or C. for Evaluation quality; A., B., or C. for Impact; A., B. or C. for Generalisability; A. or B. for Applicability

 

No rating The Hub = Any other scenario

 

11

Overall assessment rating

(using rating scale above in section 10)

 

 

  • Excellent
  • Good
  • Satisfactory
  • No rating on The Suicide Prevention Hub

 


RATING SCALE – PROMISING REVIEW

 

 

 

 

 

Review Criteria

 

 

Rating Scale

1

To what extent do you agree that the design of the program or service has been adequately informed by an appropriate range and source of evidence?

1. Not at all

2. To some extent

3. Very much so

2

To what extent do you agree that the design of the program or service is innovative?

1. Not at all

2. To some extent

3. Very much so

3

To what extent do you agree that the research/evaluation activity that is underway or planned will be well conducted and provide useful insight into the implementation and/or effectiveness of the program or service?

1. Not at all

2. To some extent

3. Very much so

4a

Score:

 

Sum of responses to questions 1 to 3 where: Not at all = 1; To some extent = 2; and Very much so = 3

Score of 6 or more = Promising listing

Score of less than 6 = No listing on The Hub

4b

Final rating:

  • Promising
  • Not suitable for The Suicide Prevention Hub