In this second installment from our discussion with John Markus Lervik we discuss the challenges of enterprise search, search as an integration layer, and where it’s all headed (click here to read the first installment). Also, stay tuned for other interviews we’ll be posting and conducting with speakers from the upcoming FASTforward conference.

On the consumer Web side of things you’ve got a near-infinite amount of data scattered haphazardly around the internet. But within the enterprise you’ve got a more controlled environment – more structured data, richer data sets, etc. – which in theory could make for a better search environment. But what are the challenges of analyzing and managing that data?

JML: In some ways it looks harder on the Internet because it’s more unstructured. But in other ways it’s easier in that there are mechanisms such as links that can be leveraged. And so one of the problems within companies is that though there are fewer amount of documents, i.e. hundreds or thousands rather than millions or billions, they often don’t have links.

In addition you have so many different types of data, e.g. different applications with different security entitlements, that make it difficult. The challenge too is that on one’s intranet or closed system one’s not looking for just any related document. Often what you’re looking for within a company is *the* document. For instance, on the Internet when one’s looking for a document, when one’s using Google or MSN for a query to find that one single document, it almost never works. So you have much higher requirements when one’s searching for a document within a company. And that makes finding something much harder when one’s contending with very different types of data and very different types of content structures. And of course that’s also a reflection of the different types of software being employed within companies – from ERP systems to large information integration applications. It’s not as easy as just putting text into a searchable index.

Andrew McAfee said in a recent article in the MIT Sloan Management Review that “Enterprise 2.0″ is an appealing reality for companies now. And a recent McKinsey/Sand Hill Group report claimed: “The software industry is in the midst of a quiet but dramatic revolution. The implications of this revolution – increased innovation, new business models, technology discontinuities, and global capacity shifts – could be even more profound than previous industry transitions.” Are we at a turning or tipping point in the way enterprises develop, implement and integrate enterprise software. Is this true in the search market?

JML: It is most definitely a turning point. I’m travelling all the time – 200 days this year – and I’m sensing in recent months a real change in the ways companies think about search. We started seeing and talking about search as a platform, search not just as about text but also about structured data, about five years ago. But it’s really only in the last 3-6 months that we’re really beginning to see people turned on and getting it. It’s really accelerating now.

Can you talk about how companies are integrating search into their companies at an operational level?

JML: We’re seeing companies now hiring individuals to lead their search initiatives – search experts are being hired at the executive and management level which indicates that search is now a key part of their businesses, a key part of their internal architectures. For example, Reuters or Reed Elsiver – these huge information businesses now make a point of hiring high-level people to head up their search functions.

Companies are also realizing how these capabilities and people can help them solve issues in more efficient ways. They’re working on large integration projects, projects which in the past cost tens of millions of dollars – they can now create an informational access layer which costs a fraction of that and which helps them move much faster to market and more quickly develop applications.

Essentially, you have two key functions: you have the transactional model where you want to store and organize data and then you have the retrieval model where you want to be able to retrieve business intelligence. All of the latter, all of it, is going to be powered by search. So Business Intelligence is going to be entirely driven by search, allowing you to pull analytics or pull data out of internal systems – it’s also going to be a subset of what search is in five years.

Can you point to particularly interesting apps being developed by your partners or customers?

JML: There are a lot of interesting external apps being developed – from searching yellow page-like sites, or for local search, or vertical search, as well, of course, as for B-to-C and B-to-B sites as well as mobile devices.

As for internal applications, some of the banks are doing some interesting things. Merrill Lynch is really getting it, using search as an information access platform in building business applications on top of our platform. They’re using FAST to integrate their data and then turning that around into informational portals – they are, after all, an information business and they’re using all that data and content and analytics from different parts of the company to provide contextual data to whomever needs it, be they brokers, analysts, support teams or customers. One of bank’s main purposes, of course, is to help their customers, help them get the right information, whether it’s a consumer trying to get information about their account, or a professional fund manager – it’s about providing the right information at the right time. And that’s all about search.

And on the mobile front, one of the largest Internet companies in Japan – Rakuten – has recently launched a mobile service in concert with DoCoMo that employs a FAST search solution. It is seeing its traffic rocket – they’re already at 200-300 queries per second – that’s millions of searches a day. That’s amazing.

You talk about how search must go beyond the search box. What does that mean?

JML: It’s about connecting people to content. But not just to information but also to other people, to other services, to other applications.

So how far do you go with that?

JML: For an example, we see more and more that people within companies are searching within their own organizations for experts on certain topics. But you can always further improve, further develop the results and relevance of those searches. The biggest challenge right now is to open the eyes of people to what search can be – it’s so much more than just a search box. We’re just at the very beginning of this.

A general and broad question: what’s your grand ambition for search?

I’ve been in this space for ten years and it’s incredible how much i learn every day about the potential for search, about the new opportunities in search. It’s mind-boggling and much bigger than people think. Even for me for whom it’s a passion, it’s much bigger than I thought a year ago and a year from now I’m sure I’ll feel the same way.

I think everything we do online will be driven by search. We may not think of it as search as we won’t necessarily see a query box. But it’s about matching, about connecting the dots: matching me as a person to another person and matching me to some interest I have. All those things are going to be queries. All those things will trigger a contextual query that will tap into a search engine that tries to match you with the data you’re after, be it a link to another person, to content, to other services. It’s going to be the fundamental foundation of communities and communication. To provide just one, simple example: when you call me that’s going to trigger a message from my cell phone that will call up emails you’ve sent me or any other data that’s relevant to you calling me, i.e. all these things will be triggering queries.

And going forward, internet portals, web portals, mobile portals – all will basically be search engines with graphic interfaces laid over them. The portals aren’t going to change the way they look. But they are going to change the way they’re generated – they’ll be generated through a search layer rather than a static database. Search will increasingly become an intelligent layer, an access layer that virtually sits there as an integration layer and one that can present things appropriately and contextually for each user – there will be no need to have all the hardwiring. And in this case we will use a search engine to understand what a user wants and present it and that may not be in a search result list. It may present information as graphs, as tags, as a summary of a news article, as other types of content – however you want to present it.

And I think the main thing for us, the good thing for FAST is that we’ve seen this coming for six years. Our technology is built not just around text but also around structured data. We’ve been prepraring for this ability to scale, for analyzing streaming data, not just static data, and we’ve thought about search as a horizontal application. That’s why we are very confident that as we continue our innovation, together with our biggest customers around the world, that we’re going to be in the lead. We believe search is going to be as big as databases in a couple years – the capabilities and the market. And if we play our cards right we’re going to have a fair share of a very significant market opportunity. Of course our customers will be B-to-C companies, consumer companies, ecommerce companies, and online media companies, but they’ll also be B-to-B, or banks or pharmas or government so we’re focusing on the largest opportunities.