I. Policy Evaluation
1. Although evaluation is being done, must be continued
implementation and reinforcement
2. Evaluation involves- examining the gap between
the desired and the actual goal
"Evaluation is the systematic examination of activities to provide
information on the full
range of the policy's short and long term affects on the target audience."
- Comparing over time (are goals met at the specified times). Did the
gap decrease? Or increase?
- Does the policy have the intended impacts?
- Look at the desired process and outcomes
- What caused the outcome?
(This is also referred to as "impact analysis")
- What factors are responsible for the affect
- Evaluation is needed to provide information, analyze strategies- (can
we do this better)
and rewrite goals
3. Evaluation consists of 2 factors (global)
a. Has the policy fulfilled the mission of what it set out to do (does
it represent groups well. This does not mean improvement.)
b. Have improvements occurred due to the policy
4. Evaluation: 3 steps
a. Establish standards or benchmarks i.e. 1000 new patients will register
at the clinic during the 1st year
b. Measure performance against the standards - comparing what is true
(real) to what is our goal (planned)
c. Set steps to correct deviations
i. Creative
ii. May need to restructure the entire policy
5. Evaluation should be ongoing but it is time-consuming - especially on national policies. E.g. demographic variables are always changing
II. Qualitative and Quantitative Evaluation Methodologies
1. Evaluation employs the research process especially
the design phase
- Assess reliability- consistent results upon repeated administration
- Assess validity- are we measuring what we want to ( will the data collected
answer our questions)
2. Qualitative Evaluation Methodologies
a. In analysis, we examined three types of surveys. Surveys are a major
part of qualitative evaluation.
- Can we put results in a database and statistically find correlation's.
b. Records provide data for evaluation
- Records show utilization of covered services
- Did they increase? To the appropriate groups?
| Advantages | Disadvantages |
| Inexpensive |
Difficult to go back and add on new data that you may now need due to time changes or the external situation of people |
| Quick | |
| People familiar with using records |
Qualitative evaluation:
| Advantages | Disadvantages |
| Gets to the "grass roots" people through the survey |
Difficult to do large scale |
| Uses "non-experimental designs" ( do not attempt to control variables, just gather data) |
3. Quantitative Evaluation Methodologies
Quantitative evaluation methodologies measure amounts. E.g. money, time
a. Experimental design
Example: reduce the clinic fee and observe if this increases utilization
(or did the nice weather increase utilization). Here, the treatment group
and control group are in fair weather climate and the treatment group
is given a 50% price reduction- does it make a difference
Control for all variables, then make a judgment if the one changed variable made a difference.
TX Group (Large slope ?)
Control Group (best is to have no slope ? here, then you know all of the
variables were truly controlled for)
C. Time Series Methodology
- See trends over time at multiple points in time
- See what happens at particular points in time when changes are introduced.
E.g. introduce reduced fees, visits increase
- Also can do multiple time series where you reintroduce the variable
to assess if the change increases even more
D. Different Time Series Evaluations
E. Post Treatment Control Evaluation
Compare before and after the change is introduced (pre/post) on the 2
groups (TX- Control).