Horror in roleplaying games

Scaring your players when they are sitting around your table with some snacks in front of them will obviously be very hard.

You can get them nervous though, giving them the chills and having them doubt their actions.

Fortunately the player's imagination will do most of the work for you. All you have to do is getting their imagination going. 

What is it we are the most scared of? I would say the unknown, thinking there is something without knowing what it is, where it is and what it wants. 

This is mainly how you will get your players to experience horror in a roleplaying game.

Don't just throw your critters at your players. Instead build up the tension. Start by giving them hints, noises, irrefutable proof that there is something nasty stalking them, or hiding in a dark cave or basement they need to explore. Keep everything mysterious.

Let them react and bait them to find out more. In the mean time, their imagination is running wild, trying to make sense of what is happening and filling the missing links with horrible stuff
.

If you go with "While investigating the corpse washed up on the shore, a deep one jumps out of the river and attacks you", you killed the tension. The players know it is a deep one. No matter how awesome your descriptions will be, the mystery is gone. They know it's a deep one.

Instead you should take your time. Talk about the water being stirred up and about air bubbles rising to the surface. Let them react and ask you questions. Don't give straight answers, only give obscured half answers that will rise more questions.
The go ahead and maybe describe how that weird smell that has been in the air all the time is turning into an unbearable stench and how that thing underwater is now moving towards them at increasing speed.
Never tell them exactly what they are facing, or what is happening.

Give the players again the chance to react. Some of them might already have fled the scene.

This works of course better in games like Call of Cthulhu where characters have the habit of not surviving most of the critters out there, but that doesn't mean it doesn't work in other settings.

So take your time to build up tension! Keep the players guessing and give them plenty of time to react. Never fully describe what they see, keep it obscured by fog, rain, darkness, shadow. Don't let them know exactly what it is. The mystery will be more horrific and interesting than anything you will tell them.