W Power 2024

Not Your Grandma's Search Engine

Welcome to the Deep Web and beyond

Published: Jun 20, 2009 03:15:00 PM IST
Updated: Jun 20, 2009 04:06:02 PM IST

Imagine a trawler in the ocean. Its net captures what swims close to the surface. That’s what search engines do.
But the Internet, like the ocean, is deep — widely estimated to be 500 times larger than the space most search engines have managed to index — and to truly explore it, you need specialist tools.
Quick example.

Let’s take India’s claim that it exports software worth $30 million to the US.

Image: lii.org
To authenticate this, you’d need to cross-check American imports of Indian software. Answers such as these are generated only when you ask for them, from databases; they’re “dynamic” content. Most search engines deal with static content.

What you need therefore, is an engine that queries a specific database, and throws up the answer you’re looking for.

Take my hand. Trust me.

Welcome to the Deep Web. Let’s start with Librarians Internet Index . Careful navigation will eventually lead you to the Interactive Tariff and Data Web. A few minutes of figuring-the-system-out later, the answer will come up on your screen — downloadable as a spreadsheet too — completely out of whack with India’s claims.
Get the drift?

If these sites feel intimidating, try Noodle Tools . And if Noodle intimidates you, perhaps you’d better go play in the shallows.

Still with me? There’s more: the Dark Net, a.k.a. the Invisible Net, a.k.a. the Private Net, territory that cannot be browsed even by Deep Web search engines. Where you can ferret out rare books, music and films from fortress-like machines.
Entry is by invitation only, encryption is heavy, and it takes patience to break into
its labyrinths. But once in, no sysadmin, no business entity, no government can stop you from looking for what you want to see.

The gates open at Free Net . Follow instructions, install the software and start looking around.
For the more enterprising and experienced, there’s WASTE . Yanked off the Internet after AOL bought its developer (AOL went on to call it illegal), it was adopted by a coalition of programmers who believe in file sharing and free software. It can now be downloaded from wasteagain.sourceforge.net.

Big tip: Visit Fravia — the father of search on the Web. Now terminally ill, he has stopped updating his site; but his archives reside on searchlores.org , loaded with wit, charm and subversiveness.
Go visit. It’s where you start learning to search


Click here to read on the alternate search engines available to mine the internet.

(This story appears in the 03 July, 2009 issue of Forbes India. To visit our Archives, click here.)

Post Your Comment
Required, will not be published
All comments are moderated
  • Vivek Kumar

    This is an eye opener of an article. Great that you are looking at Tech topics again. I look forward to exploring the resources mentioned. BTW what was the result of the query "American imports of Indian software"?

    on Jun 22, 2009