The internet does forget (if you want it to)

by Stefanos Kofopoulos • On 16-08-2006 • AT 2:45 am • 6 ΣΧΟΛΙΑ


I’ve been a loyal reader of Paul for quite sometime now and i’m really happy he made it to Yahoo! as an intern. I was digging my RSS feeds today when i spotted Paul’s post on the new Yahoo! Social Media Podcast on Yodel Anecdotal. Among the obvious talking on Social Media, Richard Crowley suddenly said: “The Internet doesn’t forget does it?” and Paul agreed.

The short dialog went like this:

RICHARD CROWLEY: The Internet doesn’t forget does it?

PAUL STAMATIOU: That’s right.

DOREEN BLOCH: No, it doesn’t. Go ahead.

I was shocked by the statement and I’m afraid i have to disagree with all three of them. The Internet does forget if we want it to.

Remove sites or individual URL from search index

It’s quite easy to remove a web site or individual pages from a search engine’s index. Google offers a service for removing a URL from its index. There’s also the option to remove Usenet posts from Google Groups. Google states that “URL removal system will be removed from the Google index temporarily for six months” which gives plenty of time to vanish anything such as outdated URLs or embarrassing pictures.

On the Yahoo! campus, there’re easy to follow suggestions on how to remove pages from the index.

Use robots.txt

There’s always the option to use a robots.txt file to prevent web spiders and bots from accessing all or part of a website. Again Google offers help and tools on robots.txt. So does Yahoo! with info on robots.txt

There’s also an free online robots.txt generator here.

Use the noindex meta tag

A way to remove specific pages from getting indexed is by using the noindex meta tag. Place the meta tag into the section of the web page you want to get out of the index.

Here’s an example that prevents all robots from indexing a page and following outgoing links on your site.

Google shares a bit more of tips than Yahoo! on how to remove content from the their index.

The internet does forget (if you want to)

The Internet is exploding with thousands if not millions of pages created every day. User generated content is a must and will continue to grow now that broadband and online tools for file sharing and socializing are easier than ever to use.

Some want to keep their pages secret, away from the public eye and search index on Google or Yahoo!. Thankfully, there’re tools and techniques to easily remove individual URL or pages and at the same time preventing bots from crawling specific parts of the web. Βρείτε τώρα τη δουλειά που σας ταιριάζει!


  1. All very valid points, most of which I’ve forgotten about or never known. But with the exception of the first bit about removing your pages from the search index, this post focuses more on keeping the Internet from doing the initial learning, rather than making it forget after the fact. A high profile site is likely quoted and cited by many third parties (especially in embarassing cases), so it proves to be very difficult to be certain you got rid of a piece of content. Add to that sites like and you start to see the kind of memory the Internet has.

  2. Richard, my thought process is more or less this: If you’re not indexed you don’t exist.

    If information is not available on a search index or 404s occur is likely that no one will bother to dig further. Even if third parties quote you and the pages are removed there’s no reason to worry about.

    Back in the “old” USENET days i can recall postmasters removing posts on per person basis when requested.

    I wasn’t aware of – yeah you can make fun of me! :-)

  3. Two points:

    1. “Even if third parties quote you and the pages are removed there’s no reason to worry about.” Uhm, why exactly is that? If the search results bring up people who say you did something, I’m most likely to think their story has grounds — even if the link to your site gives me a 404 (that’ll increase my suspicions, actually).

    2. RE: — you can have pages removed from, per your request.

  4. If you want to have your pages removed there’s a good reason for that. If they’re removed you’re done. Quotes are quotes not the whole story. Big picture is gone and you can explain why you wanted to have paged removed from the search index. In a nutshell it’s a right, an option we have or maybe given. The internet does forget.

  5. You’re rephrasing the same arguments again.

    – “Quotes are quotes not the whole story.” Exactly, they’re not the whole story, but they’re enough to reproduce a big part of it.
    – “Big picture is gone…” Again: with the quotes the big picture is there — it’s the whole picture that isn’t.

    Do you really think people will take the trouble to check both sides of the story? Actually, scratch that — this isn’t about checking both sides of the story in the traditional sense — i.e. check to see what the other party’s saying because what the one party’s saying is libel, or something.

    People here maybe want to check both sides of the story, but when you’re taking down the page, you’re removing your side of the story. Which argument is used for that move by the way? That the people who quoted you and pointed to that page are all liars?

    As Richard said, there are cases where the Internet does not forget.

  6. OK, i see where you’re coming from. In that sense, yes both of you’re right. The Internet doesn’t forget.


Show Buttons
Hide Buttons