Why government search engines can’t handle misspellings

  • Share
  • Read Later

In my post Thursday on the failure to connect the dots pointing to the underpants bomber (can dots point?), I declared without offering any evidence that:

Any halfway affluent individual can assemble a better set of communication devices and networks on her own than she’d ever get from the IT department of a large corporation (or large government agency).

Well here’s Noah Schachtmann at Wired’s Danger Room riffing (via Boing Boing) on the Obama administration’s report on what went wrong:

Government search tools weren’t even flexible enough to handle simple misspellings. As the White House review notes:

A misspelling of Mr. Abdulmutallab’s name initially resulted in the State Department believing he did not have a valid U.S. visa. A determination to revoke his visa however would have only occurred if there had been a successful integration of intelligence by the CT [counterterrorism] community, resulting in his being watchlisted.

This is a problem that commercial software firms largely solved years ago. (Try typing “Noa Schactmann” into Google, and see what comes up.) How it could persist in the CT community, I just don’t understand.

I think CT stands for “counterterrorism,” although I’m not ruling out “chicken tenders.” In any case, I don’t understand why Schachtmann (or Schactmann) doesn’t understand why the CT community uses a crappy search engine. One of the key technological realities of our age is that free software available on the Web is generally much better than the expensive stuff purchased by corporations and government agencies. I’m not entirely sure why this is so—although I know the cloud must be involved and maybe the singularity is too. Perhaps it’s just a passing phenomenon. But it really shouldn’t be surprising that in 2009 some proprietary search software purchased by the State Department didn’t work nearly as well as Google or Bing.