This book constitutes the refereed proceedings of the Second International Workshop on Human Behavior Understanding, HBU 2011, held in Amsterdam, The Netherlands, in November 2011, in conjunction with Am I-11, the International Joint Conference on Ambient Intelligence. The 13 revised full papers presented together with 2 keynote talks and one summarizing paper were carefully reviewed and selected from 32 submissions. The papers are organized in topical sections on analysis of human actions and activities, face and gesture analysis, persuasive technologies, and social interactions.