How Twitter Slayed the Fail Whale

  • Share
  • Read Later
yiyinglu.com

The original Fail Whale, originally titled "Lifting a Dreamer"

On Friday, August 2, two very surprising things happened on Twitter. At 7:21 PM Pacific Time, users sent 143,199 tweets in a single second, a new record for Twitter activity. Most of the posts came from Japan, where TV viewers tweeted in unison as the classic anime Castle in the Sky reached its climax. The deluge of messages, about 25 times more than the average of 5,700 tweets per second, was an unprecedented strain on the social network’s infrastructure.

Even more shocking: the Fail Whale, the image Twitter serves up to users when it’s over capacity, was nowhere to be seen. In fact, sightings of the social media users’ most loved (and most loathed) marine mammal have been few and far between recently. As Twitter prepares for its IPO Thursday, the company seems to have finally nixed its most glaring technical issue. “Twitter’s done a really great job of getting a handle on their service reliability,” says James Caverlee, a professor in the department of computer science and engineering at Texas A&M University. “If you go back a few years, you’d see the fail whale all the time.”

A Whale of a Problem

In 2007, when Twitter first came to prominence at the South by Southwest interactive festival, the social network became notorious for its downtime. Between server overload and scheduled maintenance, the site was offline for almost six days total that year, according to Pingdom, a service that tracks website performance. Things were not much better in 2008, when the site crashed during Steve Jobs’ keynote at MacWorld in January. Co-founder Biz Stone described the site as being in constant “emergency maintenance mode” at the time. By May the quickly growing site had created a standalone blog just to tell users when Twitter was down.

None of this mattered to Yiying Lu, an artist and designer based in Australia who did not have a Twitter account. She enjoyed drawing animals and had created an unusual image of a serene whale being lifted out of the ocean by eight orange birds. It was a virtual birthday card she sent to friends in other countries. The whale signified the size of her wish to be with her friends, and the birds were working against all odds to make the impossible happen. “The original image was a very positive image,” she says. “The whale was designed as a blessing to the people that I miss a lot of overseas.”

She eventually put the image on iStockPhoto.com, where Stone spotted it and decided it could serve as an error message that was both cute and symbolic. Better than the LOL-cats they were using at the time, anyway (remember, this was 2008). “I wanted to find something that was still fun but that wasn’t so jokey,” he told NPR. “This idea [is] that it’s a big job but we’re all working together to do it.”

pale-whale-640

TBS

Yiying Lu designed the “Pale Whale” specifically for the promotion of Conan O’Brien’s TBS late night show.

Users first encountered the whale around May 2008. Early Twitter user Nick Quaranto seems to have been the one to coin the actual term “Fail Whale.” It was an instant hit. There were Fail Whale fan clubs, T-shirts and coffee mugs. There were fail whale martinis and FailParties. It even became a fake brand of craft-brewed beer, called Fail Whale Pale Ale, for which you can buy pint glasses. Conan O’Brien commissioned a version of the image, dubbed the Pale Whale, to help him promote his new TBS talk show in 2010.

By accident, Twitter’s technical problems had attracted a cult following.  “You could potentially view it in the early days as sort of a cultural marker,” Caverlee says. “If you knew what the Fail Whale was, you were part of the inside group.”

Twitter was ambivalent about the whale’s popularity. When co-founder Evan Williams received a box of Fail Whale T-shirts from fans of the creature, he tweeted that he had “mixed feelings.” Stone said at the time that some people inside Twitter weren’t happy that the public had taken to calling it the Fail Whale. “For some reason…people love that whale,” he said. “Or they hate it. They love and hate it.”

Lu, meanwhile, became a minor celebrity among Twitter geeks. She was showered with praise for her creation during her first trip to the United States in 2009.  “It’s so easy to make friends when you tell people you created the whale,” she says. “Birds in real life are never going to lift a whale, but in this picture it’s presented and it’s very affirming. It’s kind of like an impossible dream come true. People like something that makes them happy, makes them smile.”

Harpooning the Whale

Even if the Fail Whale was a branding coup, the root cause behind it—Twitter downtime—was hurting business. Whale appearances, which typically occur for a subset of users when the site is operating too slowly, became an increasing concern as the site expanded to a broader audience. The social network buckled (along with most of the Internet) when Michael Jackson died in June 2009, and again during the World Cup in June 2010. That was a particularly poor month for Twitter performance. ‘The influx of Tweets –– from every shot on goal, penalty kick and yellow or red card –– repeatedly took its toll and made Twitter unavailable for short periods of time,” Rafi Krikorian, Twitter’s vice president of platform engineering, wrote in a recent blog post about Twitter’s downtime issues.

It was after the World Cup debacle that Twitter finally began a concerted effort to tame the Fail Whale. The company began a large-scale endeavor to shift many of its back-end processes from Ruby-on-Rails, an open source web application framework, to a series of programming languages that were compatible with the Java Virtual Machine framework. The company also revamped its engineering workflow to allow different teams to work on different parts of the Twitter backend without affecting the whole system. The site’s code base was separated into different silos for functions like storing tweets and sending messages. In his blog post, Krikorian explained that the changes have allowed the company to spend less time troubleshooting outages—he calls them “whale hunting expeditions”—and more time developing new features.

Since summer 2010, Twitter’s uptime has increased steadily, only dipping below 99% in October 2011, according to Pingdom. The site has experienced zero downtime in seven out of eleven months in 2013. The company is also continuing to beef up its engineering muscle. The social network launched a new initiative called Twitter University in August to provide engineers more on-the-job training and acquired a start-up called Ubalo in May to help improve the site’s back-end infrastructure.

 “They’ve been able to handle huge, huge events like Election Day 2012 without a problem,” Caverlee says. “You don’t really see [the Fail Whale] very much, and that’s a credit to their engineering team.”

Other Whales Persist

Though the Fail Whale has been subdued, Twitter has hardly shed all the technical baggage of its start-up days. Spam has been an ongoing issue for the social network, peaking at around 11 percent of all tweets in August 2009, according to a company blog post. Though spam has been drastically reduced since then, phishing scams sent through private direct messages continue to be a problem.

Fake accounts, which Twitter estimates comprise less than 5% of monthly active users, are also an ongoing concern. And the hacking of high-profile accounts, like Burger King and Jeep, are becoming more common. Such incidents can have influence outside the twitterspehre—when a hacked Associated Press account tweeted that the White House had been attacked in April, the Dow Jones Industrial average plummeted more than 150 points. If not curbed, account hacking could make companies wary of investing time and advertising dollars into the social network.

burgerking-twitter-hack-1_620x350

Twitter

Burger King was one of several big-name brands whose Twitter account was hacked early in 2013.

Twitter is working to combat these issues. In May the company finally introduced two-step authentication, a more secure way for users to log into their accounts. The company is also planning to encrypt private direct messages, according to The New York Times. But it’s unlikely that all spam will be eradicated or that account hacking will be eliminated. “It’s a tough balance for a company like Twitter that wants growth,” Caverlee says. “They could add in some really stringent authentication mechanisms. But the risk there is that you may make the user experience much less appealing for regular users.”

The social network’s technical glitches have not phased Wall Street, though. The company upped its IPO stock price to $23 to $25 in anticipation of strong investor demand for shares. “When they’re public they’re going to have to pay even more attention [to spam],” says Michael Pachter, an analyst at Wedbush Securities. “But my bias is that they do a really good job and they’ll just do a better job.”

As for the Fail Whale, Twitter Creative Director Doug Bowman declared it dead after the social network survived 2012’s election night unscathed. Lu designed a new animal for Twitter execs, the Success Loch Ness, to more accurately reflect the state of Twitter’s servers in 2013. Though the whale hasn’t been formally retired, millions of the site’s users don’t even know it exists. If Twitter succeeds in becoming one of the big tech companies of this generation, the fail whale may just be a hazy memory from its time as a scrappy, error-prone start-up.

Lu says she’s not disappointed that fewer people are seeing her whale these days because the creature has already accomplished a lot. “The whale was actually kind of the pioneer of error pages having something visual,” she says. “I was very grateful that this art—because of the love of the community—could become an iconic image in the social media era.”