Wine of the Dreamers by John D. MacDonald. Magazine version in Startling Stories May 1950, the first hardcover was 1951, and first paperback edition was 1953. One of 4 Science Fiction books by this famous author of mysteries and thrillers, best known for his Travis McGee series.

The Watchers were pale creature living in a remote, sealed world. Their game, their religion, their release - is to dream, and those dreams carry across a galaxy to lodge in the minds of men, on Earth. When Earthmen laid plans to travel beyond their planet, the Watchers stepped in - for escape from Earth was the dream that must be destroyed…