Reading Kaushik (Part 5): Qualitative Analysis

Got Surveys? Recommendations from the Trenches

  • Work with an expert
  • Benchmark against industry:,,
  • Great insights are in the raw answers to your survey
  • Don’t show the survey too early
  • Think about your target segment that you want to survey
  • Treat them as a ongoing measurement system

Build A Great Web Experimentation & Testing Program

  1. Get over your own opinions
  2. State hypotheses
  3. Create goal and success metrics beforehand
  4. Don’t neglect side-effects of testing
  5. You can start small but get bigger
  6. Get people involved in testing: e.g. let them bid on outcomes
  7. Know the techniques and theories
  8. Evangelize people about testing

The Three Greatest Survey Questions Ever

  • What is the purpose of your visit to our website today?
  • Were you able to complete your task today?
  • If you were not able to complete your task today, why not?

Experiment or Die. Five Reasons And Awesome Testing Ideas

  • Reasons:
    1. It’s no expensive
    2. It’s quite fast
    3. It’s allows you to measure change
    4. You have the ability to take controlled risks
    5. It’s quite easy
  • Ideas:
    1. Fix the worst landing pages and be bold
    2. Test single page vs. multi page checkout
    3. Test number and placement of ads
    4. Test different pricing / selling tactics
    5. Test box layouts and other online stuff

Qualitative Web Analytics: Heuristic Evaluations Rock!

  • Trying to get tasks complete: e.g. place an order, look for support, etc.
  • You can try this in a group
  • Process:
    1. Write down task that you want to see completed
    2. Try to establish success benchmarks
    3. Walk through each task and note important findings
    4. Check with a best practices checklist
    5. Create a report: screenshots / screen recording
    6. Create a prioritized list with all the problems

#4/25: Experimental Auctions

I decided to write in bullet-point type of stile because these summaries/reviews slowly become wall of texts which for one take long to write and make it, in my opinion, harder to read.

  • Values are revealed by actions — if you pay $2 for some chocolate than you value the chocolate more than your $2
  • How can we measure the value of non-market goods?
  • => Revealed preference methods, i.e. (particular) auctions.
  • Erases the problem of hedonistic pricing which requires market prices a priori.
  • Also better than stated preference methods, i.e. surveys and trails — the big problem is that these methods are hypothetical and allow strategic behavior (manipulation)
  • Experimental auctions are a better choice because of:
  • a) They use real money and real goods.
  • b) They can be carried out in real environments, e.g. supermarkets

Value theory

  • Willingness to Pay (WTP): if person do not own the good, how much is the person willing to pay to get the extra good?
  • Willingness to Accept (WTA): if person owns the good, how much does person need to accept the loss of the good?
  • However difference between WTP and WTA depends on the risk behavior. Problematic: We have to estimate Arrow-Pratt but measuring risk-aversion is hard and destroys the sense of experimental auctions.
  • Furthermore, we have to include time. Proposed formula: WTP today = Expected Value – commitment costs, which includes uncertainty, lack of information, patience, reverse transaction, freedom, etc.


  • What’s the objective? Do we want to test, explore or generalize?
  • Simple experiment design: Attribute vs. Attribute, however we need 2^\text{\#Attributes} different test cases.
  • Better: Fractional factorial design. Assumes there are no interaction effects which makes the main effects separately identifiable.
  • e.g. three attributes (H vs. L, S vs. L and B vs. R). Normal design:

    Factorial design, just needs:
  • Randomize treatment but control for time effects (time of the day, weekday, etc.)
  • Furthermore, control within-subject change. E.g. with ABA designs, i.e. first treatment A, then B, then A.
  • For experimental auctions however a problem, i.e. demand reduction effect. Solution: Only one treatment will be binding and will be picked randomly.
  • Furthermore, external effects, e.g. superior information about market prices. Solution: Inform people as good as possible; you can also provide substitute products for sale at a fixed price.

Field vs. laboratory

  • pro lab: better control, less confounding variables
  • pro field: more realistic, self-selecting population -> less sample-selection bias, more knowledge in established environment & experience, reduced costs
  • Problem: Hawthorne effect, i.e. if people think that they are being watched they change their behavior

Conducting experimental auctions

  • Qualitative study prior designing to learn about decision-making processes and general info
  • Focus groups and pretests to met objective
  • Teach them about your mechanism and optimal behavior
  • Real-money practice rounds with another good helps to learn how auctions work
  • You can do the auction anonymously — but that can be relaxed

Endowment vs. full bidding approach

  • Endowment bidding: Give participants the inferior good or they have it beforehand. Let them bid on how much they want to pay or receive to give up endowed good. This measures the real value between the endowed and auctioned good.
  • Full bidding: Here you don’t get the exact value because of transaction costs, however you have an estimate for the absolute value.
  • Generally things to consider: Is consumption important? Is there reference dependence/loss aversion (esp. endowment bidding)? Does the endowment signal something about prices?


  • English auction: works very well in practice and is highly accurate; furthermore people mostly understand how it works
  • Second price auction: everybody gives one bid, the highest wins and pays second highest price — Works quite well, especially for high value bidders
  • Random nth price: like 2nd price but n is drawn randomly, i.e. the people with the n highest bids get the good for the nth highest price — Similar to 2nd price but works better for low value bidders.
  • BDM mechanism: people bid and random number is drawn, if number smaller than bid, then person gets good for drawn number — good for field studies but creates no real market environment

Some of the case studies
The books presents a lot of different case studies which are quite detailed. I won’t go into them because this would be beyond the scope of this short summary.

Auction design problems

  • novelty of the experiment experience: learning effects; price increases because people learn about the optimal strategy
  • preference learning: high bids to learn about unfamiliar products
  • off-the-margin bidders are problematic => solution: random nth price auction


  • preference reversal: arbitrage causes value adjustments not preference adjustments
  • hypothetical auctions still not revealing => idea: calibration function.
  • => can work in some environments, e.g. trading cards
  • => doesn’t work so well in others, esp. new products

Influences on choice and valuation

  • context matters
  • goods matters
  • information matters
  • exchange institutions matter
  • market experience matters
  • price information sometimes matters
  • substitutes and complements matter

idea: consequential vs. inconsequential mechanisms

  • consequential means that the probability of binding lies between 0 and 1
  • sadly not revealing, real auction still better but not as worse as hypothetical auction

Empirical results

  • internal validity of experimental auctions is high
  • reliability is high, but influences on choice and valuation still matter
  • some anomalies don’t exist in the long run – however, there’s also things like prospect theory and loss-aversion.
  • preference reversal: market creates rationality through natural selection or budget constrains
  • even with anchoring the valuation is still valid in relative terms


  • great tool for the future
  • lots of opportunities for research

I love this book. It has enough theory and a ton of case studies — the authors don’t shy away from real problems and resent possible solutions. I seems like a rather new area but I can see a lot of potential in using experimental auctions. Especially e-commerce sites like amazon could profit highly by providing a service for producers to get WTP estimates. All in all, a superb book, more of such books, please!