Calling north, south, east and west is part of calling a circle / erecting a temple. Fire, water, air and earth are associated to the compass directions. It's never been a challenge and it doesn't take much time.
For a lot of folks it's not so much an issue of it being a challenge or time consuming, it's just plan old not part of their practice, so why do it? Not saying no one should do it, but it really comes down to what works for the individual, what makes sense, and what tradition one follows. There's very little in the way of ceremony and ritual structure that is common to all who practice magic.
A lot of places go as far as to suggest spells done without the circle are going to simply end up ineffective, Not getting results.
And a boat load of traditional witches would disagree, myself included.

Really though, the origins of the circle as found in Wicca are with Gardner and, somewhat, in the ceremonial magic which inspired him. Casting a circle isn't universal, and it's just not found in a lot of non-Wiccan witchcraft traditions, in a lot of forms of folk magic, etc - but if these folks weren't seeing results, these other traditions probably wouldn't continue to exist, so we must be doing something right.