The press is one of the most popular exercises, however it is frequently done poorly.
1) Most people stick their elbows out to the side at around shoulder height to form a T shape. This isn't a very efficient position and can lead to injury. Try bringing your elbows down closer to your torso.
2) It's common for the shoulders to lift up. Actively push them down away from your ears. Forget about the exercise for a moment and think of creating a strong posture: shoulders back and down, chest high and core and glutes tight.
3) Think about using your back muscles, especially your lats.