beatnic - just wondering

Reading the e-Consultancy interview with Lou Rosenfeld on the importance of site search analytics, I was reminded of when I was product manager of intranet (or enterprise) search at the BBC.

It was back in 2002. People complained that search was broken, but we had neither quantitative nor qualitative data to analyse.

After consulting with my colleague Martin Belam, who at the time was looking after search on bbc.co.uk, I put in place a system for capturing search engine usage data.

In a nutshell, this included where people came from, what they searched for, and where they went.

We also conducted user research to gather qualitative data about the experience of using the intranet and what it was like to find things on it.

After only a few weeks, the usage data started to settle and patterns began to emerge.

We noticed that search term frequency had a long tail (though we didn’t know that’s what it was called at the time!).

The top 25 search terms accounted for half of all searches. The top 50 terms accounted for 75% of all searches made.

We tried searching for the top 10 terms, including “training”, “expenses”, “ariel” (the BBC in-house weekly newspaper) and “jobs”.

The results were worrying to say the least. Few of them returned the result expected on the first page of search results.

If we could find the most likely link (or links) for each of our top search terms, and return that as the top result, we should be able to guarantee providing most people with what they’re looking for.

So we set about building what became the Best Bets system. (How we did this I will write about another time, but needless to say it involved an SQL database, some asp code and some hacking in to the Microsoft Site Server system.)

We then set about working with the numerous intranet site managers to determine what the best link should be for each of the top 50 search terms. Further usage analysis, research and testing showed that this working. Success! People were clicking on the Best Bets result almost 95% of the time we had a search term match.

Over time we extended to cover the top 100 search terms, then 200.

Regular usage analysis showed us when new terms appeared and we took editorial decisions on whether they should have a Best Bet.

In spite of the success of the Best Bets system, we certainly couldn’t rest on our laurels. This was an improvement for sure, but as we well knew, it was only a part of the overall search and navigation user experience. There was plenty more work to be done!

Any search engine worth its salt these days comes with best bets functionality as standard. My advice is to take the time to study and understand how people are using your search engine. Test the top search terms yourself to see what the results look like.

Maintaining a best bets system is a relatively low cost exercise, and encourages regular monitoring of search usage and behaviour, which in turn will provide excellent information about your users’ expectations and your intranet’s findability performance.

A word of caution from experience: bear in mind that the formatting of the best bet in the search results page should be minimally different from the natural results, otherwise it risks ending up in the user’s blind spot and be regarded as a “Sponsored Link” rather than something genuinely useful.

As mentioned in a previous post, you can ask twitter to text message or instant message you whenever any term you’re interested in is mentioned in a tweet.

As well as tracking East Dulwich, I’ve been tracking a few others including “intranet”

It makes for some pretty interesting reading but was hard to share online until I came across Tweet Scan courtesy of David Sterry the other day.

It’s a twitter search tool with an RSS feed of your search results…

You can filter your search to individual people on Twitter or have it search the entire public timeline.

Also you can add the search to your browser’s dropdown list of search engines.

That’s mighty handy.

And through it I’ve found blogs by Anu Gupta and Jeremiah Owyang, which I’ve added to my intranet reading list.

This series of short articles explains some simple and free ways to use the internet to keep tabs on the subjects you’re interested in. I’m using East Dulwich as the example subject.

Previously: Part 1 - Google Alerts, Part 2 - Technorati Watchlists, Part 3 - Twitter tracking.

This article is about following Delicious bookmark tags using RSS.

In a nutshell: use your RSS reader to notify you whenever someone adds a bookmark to Delicious and tags it “eastdulwich“

Delicious

The website del.icio.us (pronounced as “delicious”) is a social bookmarking web service for storing, sharing, and discovering web bookmarks. The site was founded by Joshua Schachter in late 2003, and was acquired by Yahoo! in 2005.
[Source: wikipedia]

Instead of using Firefox “Bookmarks” or “Favorites” (sic) in Internet Explorer, you can save URLs (web addresses) of the pages and sites you want to remember to your account on Delicious.

This means you can access your bookmarks wherever you go, rather than being tied to using the same computer.

You can also share your bookmarks, and see other people’s bookmarks. There is an optional setting to make any bookmark private.

When you save a bookmark you can add tags - or labels - to describe it, to make it easy to find and to group it with other similar bookmarks.

This also means that you can use delicious to track the tags you’re interested in.

There is an RSS feed available for all tags in delicious. Adding a tag’s feed to your RSS reader means you’ll be notified whenever your reader picks up a new item.

The page for all latest public bookmarks for East Dulwich is at http://del.icio.us/tag/eastdulwich, and the RSS feed at http://del.icio.us/rss/tag/eastdulwich.

As well as subscribing to feeds for tags you can also subscribe to feeds from people with accounts on delicious. My latest public bookmarks are at http://del.icio.us/beatnic and the RSS feed at http://del.icio.us/rss/beatnic

I have my own public delicious bookmarks automatically published to this website, resulting in posts with a title beginning “links for yyyy-mm-dd” (where yyyy-mm-dd is the date I saved the bookmarks).

Other social bookmarking websites are gaining in popularity - see this list on wikipedia - delicious is easy to use and one of the most popular, so should give a reasonable representation of what’s getting noticed on the web.

In this series of short articles, I’m looking at different ways of subject-tracking on the internet.

Previous articles covered Google Alerts and Technorati Watchlists.

This article is about Twitter and its tracking feature, using East Dulwich as the example subject.

Twitter

Twitter lets you share your thoughts with the world.

You can do this by text message (SMS), IM (instant messaging), via the Twitter website, or using a downloadable desktop application like Twitterific (Mac only).

Once you’ve set up your Twitter account you can start “tweeting” your updates. Sometimes called micro-blogging, it’s a bit like writing status updates in Facebook.

Your tweets will appear on your page on the Twitter website. My page is at http://www.twitter.com/beatnic and is public.

People who want to subscribe to your updates can become your “followers” - don’t worry it’s not as cultish as it sounds.

Your updates will appear on the Twitter public timeline. If you’d prefer not to be so public you can “protect” your updates - in this case people will need to request your permission to follow you.

If you’ve set yourself up with an RSS reader, you can add Twitter update feeds of the people you’re following.

People are using Twitter in all sorts of ways, including:

• The BBC tweets news headlines and links to the full articles on its website
• US presidential candidate Barrack Obama tweets about his campaign
• Transport for London tweets traffic updates (using a service called twitterfeed to convert RSS items to tweets.

Twittervision is a mesmerising website showing what people are tweeting right now, and where they are in the world. It’s a mash-up of Twitter - using a feed from the public timeline - and Google maps.

Tracking subjects on Twitter

One of the features of Twitter is the ability to “track” subjects.

You can do this using by text message (SMS) and IM. At the moment, this is limited to text message (SMS) only. [thanks to Andrew M for the correction - see comments]

Text or instant message Track East Dulwich to Twitter. You will immediately recieve a confirmation message.

That’s it. You’ll now receive updates for any public Twitter update mentioning the subject/s your tracking.

If you’re logged in to IM your updates will be by instant message only. Your text message updates from Twitter will be switched off until you log out of IM.

To stop tracking a subject, send a text or instant message with the words Untrack East Dulwich. Again, Twitter will send you an immediate confirmation message.

The rate of updates will very much depend on the subject you choose and the timing.

I’ve received one update for East Dulwich in the last three days.

Last night I tracked England and Croatia and received about 300 texts!

Cost of using Twitter with SMS in the UK

Sending a text to Twitter costs your mobile phone company’s standard text-message rate - watch out if you’re with 3 or T-mobile, according to this article on TechCrunch uk:

Note also that the 07624 in Twitter’s number (+44 762 4801423) means it is actually billed as “international” by 3 and T-Mobile, making it a pricey service for those who like to tweet via SMS.

Receiving text updates from Twitter is free in the UK. In the United States you pay.

The same Techcrunch article suggests a newly added a UK limit of 250 incoming texts per week.

How do you keep on top of everything everyone’s saying about East Dulwich, or any other subject, online?

In this series of short articles I’m going to run through a few things you can set up quickly and for free to follow what people are saying about the things you’re interested in. [Also in this series: Part 1 - Google Alerts, Part 3 - Twitter tracking, Part 4 - Delicious bookmark tags]

This article looks at Technorati Watchlists.

I’m using East Dulwich as an example, but you can do this for any number of subjects.

To get the most out of these tools and techniques, you’re best bet is to set yourself up with an RSS reader or aggregator. Don’t be put off if this sounds a bit geeky. The popular RSS readers are fairly intuitive to set up and free to use. For further information about RSS and how to set up a reader, see this page on the BBC website.

Technorati Watchlists

Set up a Watchlist on Technorati, and find out when somebody writes a blog post mentioning East Dulwich (or whichever subjects you choose). Currently Technorati tracks 112.8 million blogs on our behalf.

Technorati is a search engine which covers the “World Live Web” - a subset of the World Wide Web - and claims to be no more than 10 minutes out of date. Read more about how Technorati works here.

Setting up an account on Technorati is quick and free - look for the link titled “Join”. Once you have your account set up, here’s what you do:

1. Visit the Technorati Watchlist page
2. Enter your subject in the Add to your Watchlist box
3. Hit the Add button.

That’s it. You’re now watching the “World Live Web” for the subject you entered.

To view an example of what a Watchlist looks like for East Dulwich, click on the thumbail image below.

You now have 3 choices for keeping up to date with your Watchlist:

• Bookmark the web address for your Technorati Watchlist
  This is not the most efficient method, as you’ll need to remember to visit fairly regularly to avoid missing anything.
• Subscribe to the RSS feed for your Watchlist
  Using this method, your RSS reader does the work by regularly visiting your watchlist and looking for updates. Any new content will be listed in your RSS reader, a bit like new email in your inbox. Then you can scan this list in your reader whenever most convenient.
• Subscribe to an RSS to email service
  If you’d rather not user an RSS reader, you can always have updates emailed to you. Technorati doesn’t offer this service, so you’ll need to use another website to do this for you. I’ve set up an account on FeedBlitz - I will review this separately, but so far it has not scored high on usability!

Is anyone blogging about you? Why not set up a Watchlist for your name.

How do you keep on top of everything everyone’s saying about East Dulwich, or any other subject, online?

In this series of short articles I’m going to run through a few things you can set up quickly and for free to follow what people are saying about the things you’re interested in.

This article looks at Google Alerts.

[Also in this series: Part 2 - Technorati Watchlists, Part 3 - Twitter tracking]

I’m using East Dulwich as an example, but you can do this for any number of subjects.

As well as for personal interest, these techniques are useful if you want to keep an eye on what people are writing about your industry, company, product, service, or your competition, not to mention your favourite sports team, tv programme, pop singer… So if you work in marketing, product development, corporate communication, the press office, public relations - to name but a few - take note.

Google alerts

Google Alerts is a service which emails you when it finds a new mention of your chosen subject.

So rather than you having to search for East Dulwich every now and then, you can get Google to do the searching for you.

You can have alerts sent to your preferred email address, but you’ll need a google account (free and quick to set up) if you want to edit and manage your alerts.

Here’s what you have to do:

1. Visit google alerts
2. Type in the subject you want to track, e.g. East Dulwich or SE22
3. Choose the type of search you’d like. The choices are news, blogs, video, web, groups or comprehensive (an aggregate of recent results)
4. Select how often you’d like google to email you. The options are as it happens, once a day and once a week.

That’s it. Now you’ll never miss another mention of your chosen subject - as long as google picks it up of course.

You can set up as many alerts as you need.

The alerts are in the same format as the search results. They highlight where your subject is mentioned and link to the original source.

Yesterday I was due to make a Skype call with a friend in Denmark at 1530 CET. I needed to double-check I’d got the right time scheduled so I did a little research.

It turns out that if you want to know the time in, say, Copenhagen you can simply type “time copenhagen” in to your search engine of choice and hey presto!

• Check the time now in Copenhagen using Google
• London time using the UK version of Ask (only Google appears to offer disambiguation between London UK, London Ontario and London Arkansas)
• Current time in New York using Yahoo!

On further investigation, all three search engines also show you current weather conditions (e.g. “weather Paris“) and links to forecasts:

• Check the current weather in Paris using Ask - this will also show the current time and news headlines for Paris in a side-dish
• Current weather in Rome using Yahoo!
• Current weather in Sydney using Google

Handy.

Sounds fun, I haven’t tried it yet. My friend Paul R says it’s only got 1 out of 3 of his so far!

songtapper.com

Song Search and Retrieval by Tapping

This site lets you search for a song, by tapping the rhythm of its words (lyrics).

Let’s say you have a song stuck in your head, and you don’t know the name of it. Never fear! Load up our search screen, and try tapping the rhythm of the song on your space bar while humming the tune. Tap the space bar for each syllable that you sing. It’s that simple!

Every so often an idea comes along that has the potential to change the game. When it does, you find yourself saying - “Sheesh, of course that was going to happen. Why didn’t I predict it?” Well, I didn’t predict this happening, but here it is, happening anyway.

In short, Alexa, an Amazon-owned search company started by Bruce Gilliat and Brewster Kahle (and the spider that fuels the Internet Archive), is going to offer its index up to anyone who wants it. Alexa has about 5 billion documents in its index - about 100 terabytes of data. It’s best known for its toolbar-based traffic and site stats, which are much debated and, regardless, much used across the web.

[...]

I am quite sure this means that Yahoo and Google will have to stare hard at their own (somewhat limited) search services and APIs, and think what they might do to compete, that much is certain. And if this starts to gain traction, all of a sudden, Amazon is a major search player, right next to Yahoo, Google, MSN, and IAC. A9+Alexa+web services= hmmmm….

Via John Batelle’s Searchblog

Occasionally I search for my name on the web in the vain hope that one day I will no longer be buried beneath hundreds of thousands of prices for network interface cards (NICs).

And to my great astonishment that day has arrived (at least on Google anyway)!

I had to capture this as no doubt tomorrow the search algorithms will be adjusted to counter vanity attacks like this (ok and the search spammers as well).

A piece in today’s Guardian Online lists some of the web’s top search engines as a “new search war breaks out”

I’ve set them up below for easy access.

The groupings are taken from the paper version of the Guardian. (One or two missing from the original list until I have time to get them to work)

When I search Google for tat gallery…

…and it says “Did you mean tate gallery?”

I’d like to tell it “No”

Top search result for Bill Gates this morning on the new MSN Search takes you to… Bill Gates as Mabus

Today Microsoft launches MSN Search which it hopes will give Larry and Sergei a run for their money.

Looks like the folks behind it have taken a leaf out of the minimalist design book, and the results are delivered in a very familiar style to Google’s.

However good it is, personally it would take an awful lot to make me switch.

Even though I don’t always get what I think I want from Google, it’s built up one hell of a lot of loyalty points with me over the last few years (I think that alone will help keep them ahead of the rest for quite a while).