Smaller startups with new ideas take on big data

By Marisa Kendall 

SAN JOSE, California — Delivering an eBay order in under 30 minutes, mining 11 million financial documents for evidence of illegal activity, helping humans reach Mars more quickly — three seemingly unrelated feats made possible using a new data analysis tool that is sweeping Silicon Valley.

Forget spreadsheets. Its all about graph databases, which map information using an intricate web of connections between data points. Looking at data that way, an idea popularised by industry leaders like Facebook and Google, lets users spot relationships that otherwise might be missed. Experts say it is helping revolutionize the field of data as the world is flooded with more information than ever before.

“Data does feel like the new oil. It’s kind of the commodity that makes everything go,” said Zavain Dar, a principal at Menlo Park-based venture capital firm Lux Capital. “And it’s really on the enterprise now to have as fine-tuned of an engine as possible.”

Collecting the data is no longer the hard part — it’s sucked up when you search and post online, and gathered by everything from autonomous cars, to satellites, to smartphones and funnelled to companies or government organizations. Tech companies’ challenge now is figuring out how best to analyse that data.

Silicon Valley startups are attacking that problem, a trend that experts say could ultimately be problematic for incumbents of the big-data industry — such as Palantir — which risk losing some of their market share to the newcomers.

Neo Technology, a startup named after the main character in The Matrix movie trilogy, is one of the early pioneers of graph database technology. Previously, that type of data analysis was reserved for companies with big wallets and deep talent pools, like Facebook and Google. Neo Founder and CEO Emil Eifrem says his team offers its own version of that “little piece of Silicon Valley magic” to the masses.

For example, last year the International Consortium of Investigative Journalists used Neo’s technology to dig through the more than 11 million records revealed in the Panama Papers leak, searching for evidence of corrupt offshore tax activity.

NASA uses Neo technology to manage the space agency’s database of more than 10 million documents detailing lessons learned from past mission failures and successes. Using Neo, searches in the database take less time — days instead of weeks or months — and return more relevant results, said David Meza, chief knowledge architect at NASA. He said Neo helped one NASA engineer save millions of dollars and up to two years by locating existing research he could use in his work on the Orion, the spacecraft NASA hopes eventually will take humans to Mars.

Closer to home, eBay engineer Volker Pacher demonstrated another use for Neo’s technology. Standing onstage at Neo’s 2014 graph database conference, Pacher ordered a bottle of bourbon on eBay. As he finished the talk half-an-hour later, a courier approached the stage with his delivery, eliciting cheers from the crowd. Neo had facilitated the transaction, working behind the scenes to find the best delivery option.

EBay has since pulled the plug on same-day delivery in the US, but continues testing the service, using Neo, in the UK.

Eifrem said he has seen interest in his industry explode since Neo launched its first graph database product in 2011, holding a meet-up for graph database fans that drew four or five people. Now the events bring in crowds — more than 1,000 attended the company’s recent conference in San Francisco.

Eifrem called that buzz “amazing,” but admitted it has brought more competition.

“The flip side is that in the past 12 to 18 months, IBM has announced a graph database,” he said. “Microsoft has announced that they’re working on several internally; Oracle has launched a graph database.”

There also are smaller companies to contend with. Objectivity, which has been in the data analysis business for two decades, started developing its own graph database two years ago to address the growing demand for the technology. Ayasdi sells data analysis software that uses graph databases and other tools to help hospitals pinpoint the best patient care, or banks uncover money laundering.

Many data companies are eyeing the giant in the room — Palantir. The secretive, $20 billion company has long been known for helping government and corporate clients solve tough data problems — its technology is rumoured to have helped US forces track down Osama Bin Laden.

San Francisco-based data analysis company Gemini advertises itself as “the poor man’s Palantir” because it offers similar services that it says are cheaper and simpler to deploy. Unlike Palantir, which custom-builds platforms for each client and sends its own engineers to help clients get set up, Gemini builds a service that any company can put on top of its existing platform.

That type of competition puts pressure on companies like Palantir, Dar said.

“Once you’re so big it’s harder to pivot and incorporate emerging technology paradigms as they mature,” he said. “Can Palantir successfully incorporate these novel toolkits in a way that meets their customer expectations before getting displaced by faster, more agile upstarts?”