Generally I used to recommend Omnibus 3, since its got a fairly good supply of stories in it (then omnibus 1 then 2). However I hear the Penguin editions (one, two, three, which sadly I don’t own) are better bound and more accurate versions of the stories (you also get good commentary on them).
However I don’t know the Penguins so well and having had a quick glance at their indexes I’m hard pushed to recommend one over the others, they all seem to have a good supply of really good stories in.
If your feeling cheap and don’t mind reading on screens (or can print) then wikisource has a good selection of stories from Lovecraft. If I was to suggest a handful in a good order it’d be something like this (should give a good mix of styles and lengths, and introduce some topics before others to make things flow well):
Dreams in the Witch House
The Rats in the Walls
The Call of Cthulhu
The Shadow Over Innsmouth
The Strange High House in the Mist
The Dunwich Horror
The Music of Erich Zann
The Whisperer in Darkness
The Case of Charles Dexter Ward
The Colour Out of Space
The Cats of Ulthar
Dream Quest of Unknown Kadath
The Shadow Out of Time
At the Mountains of Madness
Edit: I thought I had a copy of “The King in Yellow” somewhere but I can’t find it, if I’ve lent it to anyone can they let me know.