experimentation

There are 2 posts tagged experimentation (this is page 1 of 1).

Reading Kaushik (Part 5): Qualitative Analysis

Got Surveys? Recommendations from the Trenches

  • Work with an expert
  • Benchmark against industry: http://www.foreseeresults.com/, http://www.theacsi.org/, http://www.iperceptions.com/
  • Great insights are in the raw answers to your survey
  • Don’t show the survey too early
  • Think about your target segment that you want to survey
  • Treat them as a ongoing measurement system

Build A Great Web Experimentation & Testing Program

  1. Get over your own opinions
  2. State hypotheses
  3. Create goal and success metrics beforehand
  4. Don’t neglect side-effects of testing
  5. You can start small but get bigger
  6. Get people involved in testing: e.g. let them bid on outcomes
  7. Know the techniques and theories
  8. Evangelize people about testing

The Three Greatest Survey Questions Ever

  • What is the purpose of your visit to our website today?
  • Were you able to complete your task today?
  • If you were not able to complete your task today, why not?

Experiment or Die. Five Reasons And Awesome Testing Ideas

  • Reasons:
    1. It’s no expensive
    2. It’s quite fast
    3. It’s allows you to measure change
    4. You have the ability to take controlled risks
    5. It’s quite easy
  • Ideas:
    1. Fix the worst landing pages and be bold
    2. Test single page vs. multi page checkout
    3. Test number and placement of ads
    4. Test different pricing / selling tactics
    5. Test box layouts and other online stuff

Qualitative Web Analytics: Heuristic Evaluations Rock!

  • Trying to get tasks complete: e.g. place an order, look for support, etc.
  • You can try this in a group
  • Process:
    1. Write down task that you want to see completed
    2. Try to establish success benchmarks
    3. Walk through each task and note important findings
    4. Check with a best practices checklist
    5. Create a report: screenshots / screen recording
    6. Create a prioritized list with all the problems

Video: Successful Web Analytics Approaches by Avinash Kaushik

Again a great video, this time about Web Analytics by Avinash Kaushik. I just love his no-BS style.

  • Ask the metric: So what? Three times, if it don’t give an action it’s useless
  • Data should drive action
  • Give people the information they need – don’t send them everything => no death by data
  • Home pages of websites, are no longer the home page you want
    • Where do people come from?
    • What are they looking for?
  • Context matters: previous months, years, etc.
  • Relative numbers more important than absolute numbers
  • Compare different metrics, e.g. conversion rate and page views
  • non e-commerce sites:
    • averages hide truth effectively
    • How often do they visit?
    • How recent did they visit?
    • Depths of visit
    • => Understand the value: Loyalty
  • Segment people
  • Survey people: What do they think about the content?
  • Bounce rate: Came and left
    • Segment by source, entry-page, landing pages, etc.

Rules for Revolutionaries

  • 10/90 Rule: $10 Tools, $90 People: Understand Data & Business, Able to analyze => to extract value
  • Reporting is not analysis: Reporting -> provide data; Analysis -> prove insights
  • Data Quality can be low, but is still better than other data
  • Faith-based initiative: e.g. magazine ad without tracking
  • Make decisions, don’t argue about the quality of the data
  • Over time understand why quality is different -> confidence will get better

    Conclusion

  • Decision making is a journey, not a destination
  • => Put some level of process in place, mostly for tasks, e.g. what happens to implement a test, etc.
  • if HiPPo (highest paid person’s opinion) makes the most decisions
  • => make experiments
  • Learn to be wrong, quickly
  • => You probably don’t know what your customers want
  • => Experimentation