Feedback on this page
At some point, you will need to know whether you have been successful in changing the situation and keeping it that way. There are two parts to this measurement, Impact Evaluation and Process Evaluation. (PARTNERS)
Click DIRECTORY to see list of products and services relevant to this advice
Impact Evaluation - Did it work?
Whenever people present a problem and all the things they and others have done, there is a killer question that will come up and one that everyone poses which is “Well, did it work?” For example, the strap line on a number of commercially available diet plans state that “the weight goes and stays off”. The emphasis on the “stays off” has come about as a result of comments like “this diet is all very well to lose weight but then it all goes back on again”.
Questions you need to ask yourself
- Was the aim met? If not, why not?
- Did my actions cause the change?
- Did the change make it worse for someone else?
Was the aim met? If not, why not?
Either the aim was or was not met. To help you with this, you need to make use of the baseline. The baseline was the measurement you took when you researched your problem. As a reminder, it was a ‘snapshot’ of where you were at the start so that you had data to measure your success against. It is a description of the situation at the start of a problem solving initiative before any work has been carried out.
For example, you may have wanted to increase sales and had chosen to do it by launching an extensive mail shot, encouraging people to visit your website and in turn place an order for your product or service. You would want to know if there had been an increase in the number of hits on your company website as a result of your promotion.
Therefore, you could use two baselines.
- The first measure would be how many hits you were getting on your website before your mail shot went out.
- The second baseline would be how many of those hits had previously been converted to a sale.
Your evaluation would then be all about measuring whether you had an increase in hits during the campaign and also how many of those hits converted to sales. One method to distinguish the regular hits, from those resulting from the mail shot is to provide the customer with a code. The code would be given to provide a discount. This would then identify them as someone prompted by the mail shot.
How to undertake your Impact Evaluation Who should do the evaluation?
Preferably anyone other than yourself or your team, as people will not believe that you will be impartial, especially if what you did was a great success. People may doubt the way you collected the data.
For example, I was invited to assess the performance of the front counter staff. A survey had been conducted to measure customer satisfaction. However, the survey forms had only been given to those who were happy with the service. Anyone who was not satisfied was never given a form.
When should it be done?
At the end of the initiative. This should have been described in the aim. For example,” by.....date the following will have been achieved”.
What is going to be measured?
We have to make sure that we have not “hit the target but missed the point” This focus on specifics has become the bane of Problem Solving, where efforts made to measure something result in the wrong thing being measured, which then creates activity to hit the measure as opposed to actions that are in place to meet the long term aim.
Evaluations can be made using Quantitative and Qualitative measures.
Quantitative data involves numbers
You can count such measures before and after your actions, and note the difference. Quantitative measures allow you to use statistics to estimate the impact of your action. An example could be the decrease in graffiti tags in a certain area.
Qualitative data relates to opinions and feelings
Qualitative data is generally harder to gather than quantitative data but can often be more revealing. For example, how did you feel about the way a cashier handled your purchase. The answer, however, is normally converted to a quantitative measure to enable the results to be compared over time. For example, “From 1 to 5, 1 being the lowest, how well did the sales assistant appear to understand your needs?”
Contact the people who made the demand
When we undertook our Research we needed to find the people, department or organisation who were making the demand for something to be done about a problem. What they wanted and why it was important to them was covered in your Research of the Problem.
When undertaking the Qualitative Evaluation, I strongly recommend that you make sure that the people who made the demand are interviewed.
Make sure they are happy about the new situation. There is a chance that they are not approached at the end of your initiative. Yet the issue was raised by them in the first place and it would be best to verify with them that the problem has been resolved. The questions could be based on comparisons, for example “Compared to the situation last year, are things better, the same or worse?”
If these were customers, what a smart move for a company to get them to confirm all is well. Otherwise, they may have moved on to new issues and, assuming you had done nothing, may tell other people how poorly you had performed.
Give these same people, who had made the demand, details of people or departments to contact if the problem emerges again. If you are able to give them a reference that would be ideal, as those employed to work on the problem may have changed.
Did my actions cause the change?
There may have been other influences that occurred which had an impact on your problem.
For example, a Police Force in the UK had a dramatic reduction in cars being broken into over the year. An evaluation took place to see what had been done. The reason for the reduction was that it was the year Foot and Mouth disease had broken out and a National Park in their area had been closed. The car parks within this National Park had always accounted for a large number of the yearly car crimes, but as there were no tourists, there were no cars to steal from. The crime figures went back up after a while, when the car park was reopened.
Was the problem solved or displaced?
The question of whether a problem is actually solved or just displaced is a valid question and one you need to consider and measure. For example, it could be that a problem of pollution was resolved because higher chimneys were built to disperse the soot.
Let us look at Crime Prevention for information to help us consider how problems can be displaced. Displacement Theory (Felson and Clarke, 1998) argues that removing opportunity for crime or seeking to prevent a crime by changing the situation in which it occurs, does not actually prevent crime but merely moves it around. There are five main ways in which this theory suggests crime is moved around:
- crime can be moved from one location to another
- crime can be moved from one time to another
- crime can be directed away from one target to another
- one method of committing crime can be substituted for another
- one kind of crime can be substituted for another
This is a review of what else went on. It means looking back over the work done and, regardless of results, asking some questions:
- What went well?
- Why did it go well?
- What didn’t go well?
- Why not?
- What could be done better next time?
- What would I never do again?
Feedback is a gift...though not always welcomed
You need to remember that not everyone will want to receive your feedback, however tempting it is to share it. As an advisor I always have to tread carefully when offering feedback. If you are able to give your feedback, make sure it is something they can actually use, and always balance negative with positive comments. Also seek feedback from them about something you have done as well, including how you gave your feedback.
Everyone’s a winner
Unfortunately, most projects always appear to be a success. That’s the way of the world. But important learning is not always captured and shared. Or if it is, then it’s done privately, so you will need to get the trust of those involved in sharing with you what did not go well.
We have looked at the crucial role Evaluation plays in Problem Solving to confirm what works; what does not work so well and what does not work at all.
In effect, this is the end of a Problem Solving Process, but there is also Good Practice and the next two articles look at the need to reward people and how you can share what you have learnt.
If you want to know more about this topic then please contact Neil via his DIRECTORY entry at this link
All materials Copyright © 2011 - 2015 Sixth Sense Training Limited. You may use this material for non profit educational purposes, but please reference its source to Sixth Sense Training Limited. You may not use this material for any other purpose unless you have the written permission of the author.
Quick Links to each Article:
- Introduction to Problem Solving on the Crime Prevention Website
- Article 1 How to correctly define a problem
- Article 2 Setting your aim
- Article 3 Undertaking your Research and Analysis
- Article 4 Thinking Creatively
- Article 5 Negotiating the Changes
- Article 6 Evaluation
- Article 7 Recognition and Reward
- Article 8 Sharing Good Practice
Updated February 2015