They get wedged in there pretty well, so it takes a little bit more force than you would expect sometimes. If you are standing with your knees at the headlights, facing the back of the car, you should twist in a clockwise direction to get the bulb loose from the housing.
Look at the bulbs you are going to replace the old ones with, that should give you a clue as to how those things fit in there.