I don't mean to pick exclusively on Google here, but they're synonymous with the kind of shotgun approach to searching that we poor consumers have come to know and love (or hate as my case may be).
The usefulness of top-level search engines has been rapidly deteriorating for me over the past 6-12 months or so, not because they're getting worse, but because the shine is rubbing off. I find search results are usually non-authoratative, which is generally the kind of info I'm after (I can only tolerate random musings from random people for so long). I like to have some context for content I consume. If I search for "nice Paris hotels" in Google, I get a blast of random results; some of them useful, most of them not. If, however, I go to an authoritative (by my definition) travel site, and do the same search, I get meaningful results that allow me to form a more useful opinion (perhaps the results are standardized/familiar, or I understand and have come to rely on a known rating system, etc.).
Indexing the entire web is actually not very useful unless you channel the top-level query somehow. One way to accomplish this is to constrain the context of all searches coming into the search field of some site. For example, MapQuest searches are constrained to location specific outcome. Searching consumerreports.org yields product specific results. I'm finding the data I share with others, and consider of personal high value and relevancy, is coming much less frequently from Google, and rather through vertical searches I do at tailored, context specific sites.
If I want to get reputable news, I go to reputable sites (NY Times, Wall Street Journal, etc), and happily pay subscription fees for said content. If I want medical research, I go to discipline specific sites and ante up the money they ask for in return for a report/document. The adage "you get what you pay for" comes to mind.
A major problem with Google-type crawling and indexing is that it will never reach the dark web (the content out there that is only accessible via usernames and logins). I suspect that this content dwarfs the amount of content openly available on the web. Some search products are on board with the value of vertical search. a9.com has hundreds of vertical search plugins that you can checkbox on/off to constrain your searches (and yes, you can search through the list of vertical search plugins :-) ). I recently found an obscure medical study that provided evidence of a hunch I've had all my life for a specific medical condition surrounding headaches; I wouldn't have found that report in a million years using Google.
Consumer oriented products are too general to be very useful. If these products added just one level of indirection, our experiences would be significantly enhanced. Disambiguation at search.aol.com (left hand column) is a great way to start to narrow things. As is, vertical search plugin selection at a9.com.
Imagine how much more useful Tivo recommendations would be if, after you thumbs up'ed something, you were asked to answer a multiple choice question: "why did you thumbs up this program: you liked the actors, you liked the subject matter, you liked the length, etc?"
Imagine how much more useful the web would be if you could further constrain your thinking when you set off on your quest for information.
Business Week gets it: "A Search Engine for Every Subject"
Wall Street Journal gets it: "Beyond Google"