Long ago and far away, I was starting a job as a help desk minion at a then-small ISP, and since the job would include creating helpful web pages (to provide support to the users who could actually get online and access the web, yet still had questions) my employers-to-be suggested that I create a web page about something, anything, as practice.
I picked chocolate. Why? Well, I wasn't a huge fan of chocolate - and despite what's transpired since, I'm still not one. Or at least, I'm rather picky about which chocolate. My then-girlfriend at the time, on the other hand, was such a fan. And I'd had no luck getting her to use a text-based web browser. So I figured this would help persuade her.
I scoured the net and came up with seven sites. Over time, the list grew, and the site changed and moved and changed some more and eventually wound up being what it is today:
A searchable, human-reviewed index of over 1,300 sites in a specific topical area.
Along the way it also attracted a bit of media attention, a lot of users, and a lot of links (which are not bad things to attract, overall), required the development of several generations of code to keep up with the growth, and taught me two very interesting lessons about its place in the universe.
Lesson #1 was that human-reviewed indices couldn't be truly encyclopedic. The original incarnation of Yahoo! tried to do that, with human reviewers looking over every site that was submitted, on every topic - and it just didn't scale. The Open Directory Project tried to do the same thing with a vast number of volunteers... and it still didn't scale, for some reason. However, a suitably narrow topic could be human-reviewed by a very small workforce... for example, one person. And so, my site wound up listing vastly more chocolate resources than Yahoo, and quite possibly in a more useful manner, though I can't say that for sure.
Lesson #2, a few years later, was that fully automated indices couldn't be sufficiently discriminating. Google claims to have somewhere over 19 million results for "chocolate." (Yes, I was quite flattered that mine was in the top 10.) Yet rather quickly, one runs into things that aren't about chocolate in the simplest sense - movie trailers, companies with "chocolate" in their names that actually sell things other than chocolate, and so on. And so, my site manages to attract the users who're looking for something a little more... focused.
I have, over time, been a little astounded by this. With giant companies like Yahoo! and Google, and large-scale implementations of both human-reviewed and automated indexing, it would seem that "niche" sites like mine would be obsolete... but perhaps not.
