Topic

Key Points Raised

Feedback on project and proposed tools

· Evaluation questions

· CR: Wonderful materials.  Looking at the matrix of evaluation questions and timelines, it is hard to understand questions 1 and question 4.  Should we ask about whether (1) partnership is operating effectively before we ask (4) if there is value added?

· HD: Agree that we need to re-state those questions.  

· We want to ask q.4, but cannot measure it and it will always be just the perception of the participant.  This will only be someone’s opinion but still valuable to have those opinions.

· GG: How would you characterize member organizations? Are they statewide?

· DL: Membership- that is our denominator.

· EH: Organizational members are people who rotate; based on if they are appointed to the alliance by official means. Those are the members we should ask about partner satisfaction.

· Evaluation Q1. To what extent is the CPM meeting the needs of those affected?  Coalition quality and effectiveness?

· It is hard to measure the effectiveness of coalitions.

· Do we need to measure the hope of the participants that the coalition is effective?  There is not much evidence base for coalition effectiveness.

· HD: L. Green wrote an article/editorial on partnerships.

· Timeline

· HD: On the timeline, drop the first quarter (Q3) 2005 of work.  Then do document review in Q4.

· EH: Or are you assessing impact? Is this a baseline survey in Q3?

· JK: Could be a satisfaction survey in the 1st quarter and again later to assess change.

· HD: Timing of how often we ask questions should be even.  This timeline is not even.

· Satisfaction survey

· CR: Build into title of survey something about satisfaction—mention participation.

· JK: Could be satisfied with agency involvement, but involvement of individual could be small

· PY: Add question about level of involvement vs. committee vs. general

· CR: Who is the denominator pool for this survey?  Who are the targets for this survey?  Organizational members? People who attend meetings?

· JK: Explanation of planned membership structure

· EM: Want to be deliberate in who we ask & how we ask them to respond.  What to target specific group of participants. Challenged by mail survey response rates.  We are not doing research, and do not have to strictly follow scientific methods.

· JK: Build in responsibility to participate in evaluation into member commitment.  Survey a couple of people from each org.

· CR:  It might be possible to use “scantron” response forms for survey, to make data entry easier.

· PY: Online survey or internet survey forms work well

· HD: This survey is a nice subset of other coalition instruments I have seen.

· Is the coalition brining people together to work on common topics?

· This survey is not about effectiveness in changing cancer.  It’s about how well the partnership is operating.  Is the partnership operating to the satisfaction of its members? Answers to that question would help us to guage member frustration. 

· Do we want to worry about people who are not members?

· We should provide more room on the questionnaire for people to write in why they are dissatisfied. 

· Are we using the survey to head off problems before they occur?

· Work groups and committees will fill out this survey. We know where they come from.

· Which organizational members have no people on any work groups or committees? We could do a special survey for them.  Why haven’t you participated? Are you satisfied?  Are you doing anything related to the cancer plan?

· Could we ask what organization they belong to on the membership satisfaction questionnaire?

· No way to say why dissatisfied and what they suggest we change on this survey.

· GG: keep track of non-participants who complete the survey.  Identify based on configuration and demographics

· Portions of this evaluation survey could be pilot tested this fall

· Pilot or ask this survey of outgoing SC in November?

· Stakeholder phone interview

· HD: question about two instruments.  Phone call instrument is measuring impact, paper survey is measuring satisfaction, Liz will make observations based on phone calls

· LS: How have participants instituted parts of CPM within their organizations?

· Phone interview will get at value added.  What is the value added of MCA for their organization.  This is story gathering.

· CR: maybe regularly ask questions about how people institute CPM strategies in their regular work.

· HD: Diabetes steering committee has ‘sharing time’ at each meeting.  They go around the table and each person answers: ‘what is your organization doing with the Cancer Plan?  Then a student follows up later by phone.

· We could do something at the annual meeting to give all participants a chance to tell their stories about how they are using the plan.

· This should be an interview of people who are involved in the process.

· When you go after people who do not show up, they sometimes forget what they had signed up for.

What should we discuss during CDC site visit on 8/24/05?

• Attribution, Sustainability and Participation
• Description of process for development of evaluation plan
• Heather D. & diabetes program has $ match expectation & a way of estimating FTE. Will comp cancer be expected to report on In-kind contributions?

• What to expect we have to report in future?
• What are minimal CDC expectations for evaluation?
• Where is CDC going with evaluation in future?
• Do they see a long-term investment in these kinds of comprehensive chronic disease programs?
• If state isn’t getting evaluation money, do we need to do a partner satisfaction survey?
• If CDC is making case for more money, what is the #1 thing we can provide them? Anecdotes or big numbers?
• DL: Summarize meeting before CDC visit.  Confirm and agree about what we need to do,
• Ask CDC if they prefer in house or professional evaluation?