In my last post, I talked about my two main goals when designing a website.  Now I want to talk about why I have a lot of trouble telling if I actually accomplished those goals after I come up with a new design.  I'll brainstorm some ways I can improve my testing methods, but I'd be really interested to hear if you can think of anything I could do differently.

So to recap, the two things that I'm shooting for are to control the initial impression and the next step that the user takes (or action item).  Here's the problem: by definition, if you ask someone to look at a website and give feedback, these two items won't be the same for them as they would be for a random visitor that isn't trying to judge the design.

For example, I recently decided to play around with new logo ideas at work.  The old one was fine, but it was very conservative and our company is trying to become more of an innovative tech company and less of a traditional health benefits company.  So I started playing around with new ideas and eventually I came up with eight designs.  I added them into a mockup I made of a new website (so there were eight different websites that were identical except for the logo).

I took these eight mockups and showed them to everyone at work and all my friends that were on instant messenger at the time.  The problem is, when I asked people for feedback, they have no choice but to look critically at the website.  They suggest things like changing the colors, the font, the size, etc.  I obviously appreciate all the feedback that I get, but that's not how a normal visitor will see things.

A new visitor doesn't consider the fact that the colors could be different.  They would look at my design and just accept that our company colors are blue and green.  They don't consider the font and they don't try to figure out what the logo means.  They take one look at the site and form an opinion based on instinct.  That's what I want to measure.

This reminds me of all that crazy stuff in physics where observing a system fundamentally changes that system's behavior so you can't really know what's going on when you're not looking.  There are things like A/B testing which allow you measure which of two options performs the best, but we don't have the resources to hire testing groups and with something as important as our logo, we can't experiment with our real customers.  A/B testing works better for little things like deciding where to put a graphic on your site.

One of my friends took a pretty smart approach.  Before trying to give me any suggestions, he went through all eight designs and told me what the logo reminded him of.  He basically said, "1 - the company from resident evil, 2 - looks like you're copying facebook, 3 - some trendy web 2.0 site that doesn't actually have a business model..."

I think this is about as good as I can hope for given the resources and time I can devote to testing designs.  In the future, I'll ask all my friends to just glance at a design and give me feedback on that initial impression before thinking about it in any detail.  I still won't be measuring what their reaction would have been if I hadn't asked them, but it's pretty close.

Sorry to drag on, but here's a slightly different example.  On the new corporate page I'm making for my company, there are three images that link to our newsletter, blog and customer testimonials.  Several co-workers are now complaining about the way they look, but I'm not sure how much value to put in their feedback.  I watched them the first time they saw the new design and they all looked at the elements on the page in the correct order (my action items seem to have worked).

It wasn't until after they spent a while staring at the page that they objected to my image links.  No users will ever do this.  They won't spend 5 minutes thinking about what could be better.  Maybe it's a problem that people think the images look bad, but maybe it's not since no one seemed to mind right away and the images perfectly managed to be noticed without distracting from the more important elements on the page.

So what do you think?  How can I use my friends and co-workers to better test designs?  Should I put much stock in people disliking a design decision that seems to work very well for people that aren't specifically trying to critique the site?

Finally, thanks to everyone that offered their opinion.  I'm not trying to suggest that you guys aren't a huge help (you definitely are).  I'm just trying to figure out how to make the system even better.