In this paper we present an empirical study comparing user studies and expert evaluations based on a specific set of heuristics for evaluating information appliances with a heuristic walkthrough. The study looks at an e-book reader as well as a digital music player. In the user study, question-answer protocols are used as means of intervention during the experiments. To gain insight into performance of the evaluation methods the identified problem sets were analyzed. Results for the thoroughness, validity and effectiveness are presented and compared with prior studies.