Why government search engines can’t handle misspellings

In my post Thursday on the failure to connect the dots pointing to the underpants bomber (can dots point?), I declared without offering any evidence that:

Any halfway affluent individual can assemble a better set of communication devices and networks on her own than she’d ever get from the IT department of a large corporation (or large government agency).

Well here’s Noah Schachtmann at Wired’s Danger Room riffing (via Boing Boing) on the Obama administration’s report on what went wrong:

Government search tools weren’t even flexible enough to handle simple misspellings. As the White House review notes:

A misspelling of Mr. Abdulmutallab’s name initially resulted in the State Department believing he did not have a valid U.S. visa. A determination to revoke his visa however would have only occurred if there had been a successful integration of intelligence by the CT [counterterrorism] community, resulting in his being watchlisted.

This is a problem that commercial software firms largely solved years ago. (Try typing “Noa Schactmann” into Google, and see what comes up.) How it could persist in the CT community, I just don’t understand.

I think CT stands for “counterterrorism,” although I’m not ruling out “chicken tenders.” In any case, I don’t understand why Schachtmann (or Schactmann) doesn’t understand why the CT community uses a crappy search engine. One of the key technological realities of our age is that free software available on the Web is generally much better than the expensive stuff purchased by corporations and government agencies. I’m not entirely sure why this is so—although I know the cloud must be involved and maybe the singularity is too. Perhaps it’s just a passing phenomenon. But it really shouldn’t be surprising that in 2009 some proprietary search software purchased by the State Department didn’t work nearly as well as Google or Bing.

Related Topics: terrorism, Technology & Media
  • Latest on Business

    LM Otero / AP

    Senate Approves Hike in Airline Security Fees

    (WASHINGTON) — A Democratic-controlled Senate panel Tuesday approved a $2.50 increase in airline security fees that would double the per-passenger fee for those taking nonstop flights.

    Why Greece Isn't Leaving the Eurozone YetSlate

    Associated Press

    Stocks Rally Further in Run-up to EU Summit

    MOSCOW — Global stocks enjoyed one of their best days in weeks on Tuesday ahead of a summit of European leaders that’s expected to be dominated by calls to boost economic growth.

    Europe remains the focus of attention across all financial markets in the run-up to the June 17 Greek election that could go a long way to determining the country’s membership of the euro as well as the future of the single currency zone.

  • dochosvet

    Heck my Mac can’t spell Obama correctly so why should the government get it right. I suppose I could correct it so it quit underlining it every time but why. And now my power just went out. Hope the worlds OK.

  • curmudgeon57
  • jomiku

    Think about this for a moment. Let’s say the search tools found all sorts of possible misspellings. How many names are close to one another and thus how many people are going to be “false positives”? What if the name was misspelled in 3 places or 1 or 5 or maybe the misspelling was in a double letter or in the last position? We can look at this one long name and think it would have been easy to pick this one guy out, but is that true? Aren’t a number of possible suspects likely to have Arabic sounding names? Aren’t a lot likely to have “abdul” in them? (And “muttalab” is a version of another common usage.) How do we assume that the system would then have clearly picked this one guy out? Is it that we have names like Fox or Smith?

    Think again. Why would a person, a human being, a trained human being using a search system for intelligence purpose, rely only on the spelling of a long name? That isn’t intelligent on a human level. The question of the nature of the search system is subordinate to the searching done by a trained intelligence operative who must have had ways to examine the data other than by typing in a specific spelling of a specific name.

  • parakori

    Readers are plentiful; thinkers are rare….

    http://japan-russia.jimdo.com/world-press/

    Beware when the great God lets loose a thinker on this planet.

blog comments powered by Disqus