Short-term memory for visually presented verbal items is impaired by task-irrelevant background speech that the participants are instructed to ignore. This so-called “irrelevant speech effect” (ISE) has been attributed to interference with phonological representations, interference with representations of serial order, or attentional capture. In order to learn more about the task characteristics necessary for ISE evocation, we designed 3 verbal working memory tasks that were comparable with respect to task structure, but differed in the cognitive processes involved. In each task, sequences of 7 German nouns were presented pictorially. In Exp. 1, participants had to reconstruct the serial order of the items. In Exp. 2, participants had to decide which of the 7 words represented by the pictures had no “partner” with the same initial phoneme. In Exp. 3, participants had to decide which of the 7 items had no “partner” with respect to semantic category. Serial order reconstruction and phonological categorization were significantly impaired by irrelevant speech, whereas semantic categorization was unaffected. The results indicate that serial order retention is not a necessary precondition for ISE evocation. Implications for theoretical accounts of the ISE and practical implications are discussed.