AI must accommodate diverse human preferences to be genuinely helpful, but navigating trade-offs requires input from social sciences and careful control mechanisms. Utilitarian AI, in particular, faces challenges in respecting both individual and collective preferences.