Sherman Meter Blends New Hotel-Review Cocktail

The travel industry is grappling with a fix for sometimes-suspect hotel reviews, and the latest entity to offer a solution is Shermans Travel, a deal-publisher and media company.

Shermans Travel believes it has found a credible formula in its new Sherman Meter Rating.

This new beta test of the Sherman Meter Rating closely follows the move by professional-review website Oyster Hotel Reviews to require consumers to use Facebook Connect if they want to pen hotel reviews. Oyster feels having consumers shed their anonymity will lead to more reliable reviews.

And, the context for the Oyster.com and Shermans Travel review experiments is months of controversy about the usefulness of user-generated content in general, and hotel reviews on TripAdvisor.com, in particular.

In its announcement, Shermans Travel gets a little carried away in its hype for the Sherman Meter, billing it as a "proprietary meta-search tool."

It may be proprietary, but it is not a metasearch tool in the sense that you cannot compare reviews from various sources in grid-like fashion.

But, still the Sherman Meter indeed offers a hedge against review manipulation by hotels, and it is a useful contribution to the debate.

The Sherman Meter rates hotels by aggregating consumers' reviews from sites like TripAdvisor, Yelp and Yahoo Travel; tosses in "professional" critiques from Concierge, Frommer's and Fodor's; sprinkles it all with its own expert assessments; and comes up with a percentage of positive reviews with a corresponding color code.

If the Sherman Meter is green, then the reviews generally are positive. Yellow means the reviews are mixed, and red points to negative reviews.

Darren Frei, editorial director for ShermansTravel.com, tells me that the aggregated user reviews, the professional reviews and Shermans' own expert reviews get "roughly equal weight in our proprietary algorithm."

Hence Sherman has figured out a way -- however imperfect -- to hedge against hotel industry/pr/marketing gamesmanship.

That's because Shermans utilizes the "wisdom of the masses" in sometimes hundreds or even more than 1,000 consumer reviews from a variety of websites for a particular property; the opinions of the professional or guidebook crowd; and also Shermans' own staff-formulated reviews, with the latter being "a qualitative evaluation independent" of the Sherman Expert Review displayed on the website, Frei says.

Given that Shermans Travel -- which publishes deals, provides a travel search engine and has a print magazine -- likely makes most of its money in the media/advertising business, is there an advertising bias in the Sherman Expert reviews?

Frei says: "All of Sherman's Expert Reviews are selected and managed by our in-house editorial staff, which evaluates and edits hotel reviews without input from the advertising department."

So, what are the results?

I just spent the last eight hours or so checking out the Shermans Meter for a variety of properties and found similar patterns.

Let's consider, for example, the Waldorf-Astoria in New York City.

The Sherman Meter assesses 1,310 user reviews from a variety of websites, "professional reviews" [more on this later] from Fodor's and Frommer's [Concierge doesn't rate the Waldorf], and the opinions of Shermans' own experts, and gives the Waldorf a 76 percent, or green, rating.

Incidentally, the Sherman Expert Review for the Waldorf is descriptive, but neutral and doesn't really tell me whether the wallpaper is pealing or whether I am guaranteed to get a Brangelina sighting.

Independent of Shermans Travel, TripAdvisor, using 1,051 traveler reviews of the Waldorf, gives the property four stars and a 72 percent TripAdvisor Traveler Rating, in the same ballpark as the Shermans Meter.

Then we arrive at the so-called professional reviews.

It turns out that Concierge.com doesn't rate the Waldorf.

Fodor's rating actually isn't a professional rating. Instead Fodor's provides a 3.4 "member rating," meaning the Fodor's rating turns out to be another consumer-review compilation.

However, I found the Fodor's Review of the Waldorf valuable because, unlike Sherman's Expert Review, at least the Fodor's Review offered pros and cons about the property, including Fodor's opinion that the "rooms [are] not contemporary; very public lobby includes loads of tourists hoping for Brangelina sighting."

And, I have problems with the Frommer's review methodology and the fact that the Shermans Meter gives it substantial weight.

Someone correct me if I'm wrong, but it appears that Frommers.com, which features Arthur Frommer periodically railing against user-generated content, doesn't post negative reviews. Its hotel ratings are: 0 stars (recommended), 1 star (highly recommended), 2 stars (very highly recommended) and 3 stars (exceptional).

Thus, there are no reviews of hotels that are not recommended. Where is the credibility in that?

Frommer's rates the Waldorf three stars [its highest rating], or exceptional, and is lone among the above review sites to do so.

And, the actual Frommer's Review of the property, like Sherman's Expert Review, is descriptive, but neutral and not very insightful.

Meanwhile, Oyster.com, which relies on journalist reviews and just added consumer reviews using Facebook Connect, gives the Waldorf four stars out of five, and doesn't yet offer a consumer comment about the hotel.

Like Fodor's -- and unlike Shermans and Frommer's -- Oyster.com at least serves up some pros and cons about the Waldorf, including its opinion that the property has a "top-notch gym and spa (among New York's best)."

So, where does this leave us?

Shermans, with its mix of consumer, professional and its own expert reviews on the Waldorf, has the same overall view of the Waldorf as TripAdvisor does, despite all the allegations about TripAdvisor's suspect consumer reviews.

TripAdvisor doesn't come out looking so bad, although I know I am discussing merely one, large, well-known property. (And, it is easier to manipulate rankings for smaller hotels.)

The expert reviews offered by Fodor's and Oyster are cool because at a minimum they are unafraid to ruffle some feathers.

And, Frommer's review stands out as the questionable one in the crowd because Frommer's stands alone in its assessment that the Waldorf is exceptional.

Meanwhile, there's raveable, which slices and dices 1,469 user reviews of the Waldorf and analyzes what might seem like a morass of complexity and transforms it into some clear and effective metrics.

raveable outlines its methodology here.

Instead of having to wade through 1,469 reviews and murky star ratings, raveable gives the Waldorf an overall rating, plus ranks the property on its service, the rooms and value. It also outlines where the Waldorf stands not only when measured against all other New York City hotels, but also evaluates its stature amongst the other four-star hotels (49th of 62).

The question then becomes: Would travelers rather rely on the opinions of admittedly seasoned experts on Frommer's, unvetted consumer reviews displayed on TripAdvisor, concise analysis of consumer reviews on raveable, journalists' critiques on Oyster.com, or Shermans sometimes-flawed mix of user reviews, professional reviews and its own experts?

The jury is out.

And, consumers likely will vote with their feet -- or their browsers.

RELATED POSTS

TripAdvisor Launches Family Vacation Critic in its Own Image

TripAdvisor: The Beat (of Hawaii) Goes On

Hotel-Review Dilemma: Expedia's TripAdvisor Gives Notice, Expedia.com Mum, But...

Update and Flashback: The Truth About TripAdvisor and Professional Reviews

On TripAdvisor, Oyster and Professional Hotel Reviews: My Totally Reasonable Solution

TripAdvisor vs. Kayak: Paid Reviews Don't Float My Boat

Why Facebook Connect Fits Oyster.com Strategy

I have some additional thoughts about what I see as the positive step that Oyster.com took when it began requiring that its consumer reviews be penned through Facebook Connect.

It's great that Oyster.com's user-generated content won't be created anonymously and this adds to review credibility.

However, using Facebook Connect is a limited solution because so many potential review writers don't use Facebook or even other social-media websites.

From a business perspective, that's OK for Oyster.com while it wouldn't work necessarily for businesses like TripAdvisor.

That's because TripAdvisor lives and dies by its traveler-written hotel reviews and the advertising money it takes in from the ads positioned "around the rails" of those reviews, while Oyster.com's business is based on journalist-written hotel reviews.

Unlike websites like Professional Travel Guide, which sees professional reviews as the end-all and be-all, Oyster.com is smart enough to know that it has to dabble in consumer-written hotel reviews and social media.

But, if Oyster.com doesn't attract a fraction of the user-generated content that TripAdvisor does when Oyster.com mandates vetting through Facebook Connect, then Oyster.com will have to live with that.

Oyster.com's differentiation in hotel reviews is through its journalist-written reviews.

And, that differentiation has a lot to do with the, well, differences between Oyster.com's and TripAdvisor's hotel review policies.

That is not to excuse TripAdvisor from its obligation to do more to ensure the credibility of its own reviews.

But, from a business standpoint, this may explain some of the differences between the two approaches.

On TripAdvisor, Oyster and Professional Hotel Reviews: My Totally Reasonable Solution

I've been giving the following idea a lot of thought because of all the noise in the marketplace about the integrity of TripAdvisor's hotel reviews, its efforts to police them and the entry of new players, like Oyster.com, with its reviews by journalists sneaking anonymously into hotels.

Adding further documentation to the debate, the Chris Elliott blog even found a hotelier and a former restaurateur who bear witness to the alleged manipulation of their TripAdvisor rankings.

So I have revisited the question and have totally abandoned my former advice that consumers would be prudent to consult hotel reviews from an array of sources, including fellow consumers, journalists, guide books and professional-ratings services.

My new mantra? If you want to figure out if a hotel is worth staying at, then trust me only!

I look at it this way: I have impeccable taste (according to me) and my situation is consistent with the demographics of virtually every traveler.

After all, I am a male; single; divorced (therefore formerly married); I have kids; at one juncture I didn't have kids; I once was in my 20s; my parents are grandparents (so I know their travel issues); I know women who have executed fantastic girls' getaways; I have heterosexual and gay friends; and I have lived in Connecticut, New York, New Jersey, Massachusetts, Israel and Germany.

I drove a yellow cab in New York City so I know the hotel space from that angle, as well.

In the 1980s, I worked as a nachtportier at the Palace Hotel in Berlin and therefore, coupled with my journalism experience, I am an expert reviewer on all matters regarding lodging.

So, obviously, TripAdvisor's user-generated content has absolutely no worth even though there may be dozens or even hundreds of reviews about a property to balance questionable reviews, and despite the fact that even some of TripAdvisor's critics concede they often consult its reviews before stepping out.

Guidebooks? Forget about them. They add nothing.

Ratings services? They all have axes to grind and little information.

Oyster.com and its review by journalists? Not even worth a look because the scribes are paid experts and probably have crazy deadlines.

So, before you take a trip, if you want to find out if the hotel you plan on booking is a flea trap or a five-star wonder, my advice is to look at things in a totally one-sided manner.

Read my hotel reviews only.

Hotel-Review Dilemma: Expedia's TripAdvisor Gives Notice, Expedia.com Mum, But...

The dilemma about the user-generated hotel-review model hits home at Hotel Renew, a 3.5- to 4-star property (depending on who's rating you use) in Honolulu.

Expedia Inc.'s TripAdvisor flags Hotel Renew for alleged manipulation of travelers' reviews, as I noted in previous posts.

TripAdvisor, which doesn't require proof that the reviewer actually stayed at the hotel, displays the hotel despite questioning the integrity of a portion of the 544 reviews, gives it a rating of 4.5 out of 5, and ranks the property #2 of 102 Honolulu hotels.

Meanwhile, Expedia.com, which requires, an Expedia-booked and completed stay at the property before authorizing a guest review, offers just 35 reviews of Hotel Renew, or just 6.4 percent of the 544 reviews that TripAdvisor offers.

Expedia.com displays no information about whether Hotel Renew allegedly played fast and loose with Expedia.com's review system and rates the property a 4.7 out of 5. Expedia.com has a contractual relationship with Hotel Renew. As an Expedia Special Rate property, Hotel Renew offers Expedia.com a net rate for Expedia.com's merchant-model business.

There could be many totally justifiable reasons that Expedia.com provides no notice about Hotel Renew allegedly gaming the review system, including the possibility that Expedia.com's tighter review requirements headed off any such hanky-panky.

And Expedia Inc.'s Hotels.com, which requires a completed stay prior to review-writing, rates Hotel Renew 4.0 out of 5 and offers just 38 reviews.

And, here's an industry-oriented review of Hotel Renew from Professional Travel Guide, a sister company of Travel Weekly, which I write for.

So, TripAdvisor, with its more loosey-goosey review requirements, gets a depth of content that Expedia.com and Hotels.com would die for.

And, that richer content translates into more advertising revenue for TripAdvisor, which is riding this model for global expansion, because 23 million consumers navigate to TripAdvisor for trip planning -- and often trip-booking through TripAdvisor's advertisers.

The dilemma for TripAdvisor is how to maintain the stickiness of its websites and all of that great content without undermining its integrity to consumers and worth to advertisers. TripAdvisor needs to avoid another controversy like the one that engulfed its subsidiary, Cruise Critic, regarding Royal Caribbean and the Royal Champions.

One option would be to ban the display of hotels that are caught incenting guests or enticing employees to write fraudulent reviews.

And, for consumers, here's a real issue: Would you put more stock in 544 reviews of a property, knowing that some of them are cooked, or would you give more weight to a couple of dozen reviews, knowing that at least the reviewers stayed at the hotel?

I don't think the answer is necessarily a slam dunk for the Expedia.com and Hotels.com model because hoteliers can manipulate this system, too, although not as readily.

It all points again to how complex travel planning and booking are these days: Does an online consumer need to spend all day getting professional advice or reading user-generated hotel reviews of various origins to figure out where to stay?

Are travel agents an answer?

Update and Flashback: The Truth About TripAdvisor and Professional Reviews

After writing my previous post about the issue of hotel-review manipulation on TripAdvisor and the merits or demerits of professional reviews, someone at TripAdvisor reminded me of a fascinating Travel Weekly roundtable discussion that focused on some of the same themes.

Check out this exchange between Arnie Weissmann, Travel Weekly's vice president and editor in chief, who moderated the discussion; Tim Zagat, co-founder of Zagat Surveys; and Arthur Frommer, founder of Frommer's Travel Guides:

"Weissmann: If a consumer sent something to the Zagat Survey that struck you as insightful and authoritative, but was the only person to voice that opinion, would you print it?

Zagat: If the opinion says what other people are saying, or more or less says what people are saying, we will print it. And we like to find opinions that are especially well said.

Weissmann: So if somebody had an observation that struck you as insightful, but they were a minority of one ...

Zagat: We would not print them. We never print a minority of one. But you don't find that. What you find is that there are themes. I mean, these reviews practically write themselves.

The average restaurant in New York has got 1,000 comments on the survey, but the first 100 people say it all, and the others say the same things again and again and again. Pick a restaurant. How many ways can you describe it? Or a hotel?

Frommer: Wasn't it Ibsen who said that the majority is always wrong?

Would you not rather rely on the prognostications or the appraisals by Frank Bruni, the ...

Zagat: No.

Frommer: ... restaurant critic ...

Zagat: No.

Frommer: ... of the New York Times ...

Zagat: No.

Frommer: ... than on 100 New Yorkers ...

Zagat: No.

Frommer: ...who have no ...

Zagat: No, no, I would absolutely not, and I will tell you why. First of all, 100 people who filled out our surveys on a specific restaurant will have gone there an average of 11 times each. That means the restaurant has been visited 1,100 times. Frank Bruni has to write another review next week, so he goes once, twice, three times, maybe four.

Frommer: But Frank Bruni has a basis for comparison.

Zagat: What the hell does he have? A Ph.D in taste buds?"

I love that last quote: "What the hell does he have? A Ph.D in taste buds?"

The answer for consumers -- and to travel businesses -- is that you should probably blend restaurant and hotel reviews from the Frank Brunis of this world with hotel reviews written by guests and restaurant patrons, too.

But, as you can see from the above exchange, people like Frommer, who as a guide-book publisher has a vested interest in promoting professional reviews, have a deep distrust for the opinions of average travelers because Joe and Betty Vacation haven't earned their Ph.Ds in room service or hotel-check-out.

Yes, Mr. Frommer, the majority can be wrong, and solitary and minority opinion can inform.

But, in hotel or restaurant reviews, don't dismiss the majority opinion, either.

After all, Joe and Betty Vacation may have an insight that Bruni wouldn't have considered in 5 million years.

@BeatofHawaii kicked off the latest -- what @UpTake characterizes as -- "@TripAdvisor review manipulation kerfuffle," by pointing to TripAdvisor's practice of posting warning notices alongside hotels that allegedly have gamed the hotel-review system.

And BeatofHawaii (Jeff) subsequently clarified his position, tweeting that he values traveler reviews, but "92 fraud badges don't scrape surface of problem. Feels like TripAdvisor wants traffic and reviews at virtually any cost."

We agree that TripAdvisor's reputation and business is at stake when hoteliers muck up the works and encourage fraudulent reviews so I'm sure TripAdvisor will do everything it can to police its hotel reviews.

I like Priceline's approach, which it recently modified.

Priceline stated: "Most hotels now feature a detailed guest scorecard. The scorecard ranks the hotel’s popularity among Priceline hotel guests and shows when a hotel is ranked among the top-rated hotels for a specific city. Also, each hotel receives a guest satisfaction score in 5 different satisfaction categories -- overall quality, cleanliness, staff, location and dining. Unlike other hotel review services that allow anyone to post a review, the priceline.com scorecards are based on surveys that priceline.com hotel guests receive after they’ve completed their stay."

In addition, Priceline took steps to make it easier to enable travelers to search reviews based on their own demographic profile or interest.

Priceline states: "Now, guest hotel reviews can be sorted and grouped according to 7 different traveler types -- solo travelers, groups, seniors, couples, families with teens, families with young children and business travelers. This gives potential hotel guests the ability to search their hotels through the eyes and experiences of like-minded travelers. Priceline.com also offers exclusive hotel and restaurant ratings and reviews from Zagat Survey, LLC."

Any hotel-review system can be manipulated to some extent, and that's especially true, as @RobertKCole pointed out, when it comes to smaller, non-chain properties.

Priceline's policy of limiting hotel reviews to authors who booked their stay on Priceline diminishes the number of potential reviews, but greatly enhances their credibility.

I don't know if TripAdvisor would be able to institute such a policy since it is a research and planning site and doesn't handle bookings.

But, one thing TripAdvisor could and should do immediately is to take the additional step of banning, for a set time frame, the display of hotels that haven't played by the rules and have earned review-manipulation notices.

Then, TripAdvisor, we'd know that you really are serious about the problem.

The Truth About TripAdvisor and Professional Hotel Reviews

The blogosphere is abuzz with "news" that TripAdvisor has conceded that some of its user-generated content is cooked.

Arthur Frommer, a longtime critic of guest-written hotel reviews, is at it again on his blog, blasting the concept of user-generated hotel reviews under the headline, "TripAdvisor is now warning you that numerous hotels are submitting phony recommendations for themselves."

This latest round of TripAdvisor-stomping apparently originated with the Beat of Hawaii blog, which found more than 90 hotels with TripAdvisor review-manipulation warning notices and then trumpeted the news of "TripAdvisor's new stand on fraud."

When TripAdvisor catches a property that allegedly games the system, it prominently displays the following notice: "TripAdvisor has reasonable cause to believe that either this property or individuals associated with the property may have attempted to manipulate our popularity index by interfering with the unbiased nature of our reviews. Please take this into consideration when researching your travel plans."

For example, Hotel Renew in Honolulu, rated #2 out of 102 Honolulu properties on TripAdvisor, earned a notice.

And, while Frommer and the Beat of Hawaii dismiss the worth of guest-written reviews, Chris Elliott takes a more reasoned stance, noting that he reads TripAdvisor reviews on the road, "but I do so with the knowledge that the travel industry is successfully manipulating the site. I ignore the best and worst reviews (those are typically the fake ones) and whenever I read phrases like 'best hotel ever' or 'incomparable service' I roll my eyes and wonder about that fabled algorithm that’s supposed to catch counterfeit reviews."

I would disagree with Elliott on the matter of the travel industry "successfully manipulating the site."

With its more than 23 million hotel reviews, I don't think the crush of bogus reviews rises to the level of hotels "successfully manipulating the site," although certainly some of them would want to.

I find tremendous value in guest-written hotel reviews, but read them skeptically.

It's the same approach I take when I read newspaper websites, book reviews, advertisements and, oh yes, professionally written travel reviews.

There is nothing holy about the professionals. And, there often is wisdom in the masses.

There are great hotel-rating services out there, and you should use them in conjunction with guest-written reviews to get a well-rounded picture.

But, "professionals" have their own prejudices because, after all, they are human.

The dirty little secret about getting advice from a travel agent is that they have business relationships with "preferred" suppliers who compensate them especially well because of all the guests that the agents send to the hotels. So some of these agents might steer their clients to a Marriott property instead of a Starwood hotel because Marriott might give the agent a better deal.

Of course, online-travel sites have preferred relationships, too, and many bias their displays.

And, travel agents, like trade journalists, often get discounted rates or free rooms to entice them to visit and say nice things about particular properties.

This is not to say that the professionals are bought off. Hopefully, the vast majority still have integrity.

But, when scoping out a hotel, use your brain, talk to people, and read both professionally written and guest-written reviews. Don't take any of them in isolation.

One of the shortcomings of the TripAdvisor reviews is that TripAdvisor, unlike some other review sites, demands no proof that the reviewer actually stayed at the property.

That could be a tough requirement for TripAdvisor to impose, even if it wanted to, because TripAdvisor doesn't actually book any travel, but funnels consumers off to advertisers for bookings.

TripAdvisor takes fake reviews seriously because, when you take its business model into account, guest-written hotel reviews are its bread-and-butter.

TripAdvisor's advertising/media revenue is contingent on consumers visiting TripAdvisor over and over, and spending some time there, going through all of that content.

When I'm traveling, I consult TripAdvisor's hotel reviews and find them to be a valuable tool, warts and all.

I think the hidden agenda of some of TripAdvisor's critics, like Frommer, is that they consider the democratic empowerment of the consumer and the Web in general to be one big blemish.

Royal Champions, Royal Rooters ... Who Knew

I like the Red Sox, but didn't know a thing about the team's historic Royal Rooters, a scattershot collection of brewski and baseball buffs that traced their origins to 1897, until Ted Kennedy threw out the first pitch on opening day at Fenway Park the other day.

It turns out that Kennedy's grandfather, John "Honey Fitz" Fitzgerald, the mayor of Boston, threw out the very first pitch at Fenway Park in 1912. The Kennedy grandad was a member of the Royal Rooters.

Today, the Royal Rooters of Red Sox Nation even have a blog.

So, it seems that Royal Caribbean's Royal Champions, which made for unflattering headlines about user-generated content a few weeks ago, weren't the first "Royal" rooting section.

No word, though, on whether the Boston Red Sox in the early 1900s encouraged the Royal Rooters to post puffy reviews on CruiseCritic.com:)

Reviewing Review Policies on TravelPost and TripAdvisor

Lost in the Royal Champions' controversy, in which Royal Caribbean organized and wooed this group of frequent cruise-reviewers, is the fact that a lot of review websites these days incentivize reviewers to pen their opinions.
As I wrote here a few days ago, I oppose the practice of review compensation.
Now, as it turns out, Kayak-owned TravelPost, in its quest to give TripAdvisor a run for its media dollars, is aggregating and posting user reviews from some sites that compensate their reviewers.
To its credit, Travelpost itself doesn't compensate people who write reviews for TravelPost. And, neither does TripAdvisor pay reviewers.
But, TravelPost is partnering with sites, including IgoUgo, Epinions.com, and BedandBreakfast.com , that compensate reviewers to varying degrees.
IgoUgo offers reviewers Go Points from American Airlines and Amazon; Epinions provides a revenue share; and BedandBreakfast.com makes reviewers eligibile for a $1,000 gift card.
TravelPost provides a way for consumers to filter in or out the source of TravelPost reviews.
But TravelPost, which says it values transparency, should provide explicit information for consumers about the review policies of its partners.
In that way, consumers would be able to make up their own minds about whether they should read or disregard reviews from incentivized critics.
It would only enhance TravelPost's reputation if it does so.