A customer writes:

Dear Dr. Search,

We've been enjoying our new search engine, FAST ESP 5.3, and it's certainly a very powerful platform.

But today on our development server we can't seem to spider any more pages, and I'm seeing a weird error:

session_servant (27): DocAPI manually suspended. Operation denied.

Can you help?

Dr. Search replies:

This is an odd one, and I confess even I had to consult with our guru, Deep Code.

It turns out this isn't technically an "error", it's just a particular state the FAST system is in, and it's being kind enough to proactively tell you. As you know, most larger software applications are composed of multiple subsystems and ESP is no exception.  The DocAPI is one of those subsystems, and it is involved in processing newly spidered content that needs to then be added to the index.  For whatever reason, that system was told to shut down at some point, and dutifully obeyed that command.

But enough theory, First the Fix:

I'll circle back as to the "why" later.

This can be easily fixed from the command line.  As a reminder, to run FAST commands, you need to first CD to the $FAST/bin directory and run the setupenv.cmd file (or on UNIX variants you need to source the setupenv.sh)

Then enter the command:

indexeradmin resumedocapi

Our lab rats verified that this does work.

And as to "Why" this Happened:

We're guessing you never intentionally shut that process down.  Our clue is that you mentioned this was a development box.  I'm curious, have you been making many changes to your index profile?  That's the global schema file that ESP uses.  When developing applications you'll frequently decide you want to define additional search fields or adjust other settings.  We suspect that repeatedly uploading new index profiles would cycle various subsystems over and over, and subsequent reindexing can temporarily strain resources.  We suspect that as part of these normal operations the DocAPI subsystem was restarted in order to be reinitialized with the new schema, but that the second half of this process failed, it didn't start back up for some reason.

The good news is that if this is the cause, then it's unlikely to happen in production, since index profiles are rarely updated in production.  We'd also suggest that you take a look at your dev box's hardware.  Does it have enough CPU and memory to comfortably hold all of the FAST components?  And do your developers have lots of other software running on it?  Again, in a production system, FAST subsystems are deployed onto multiple machines that have been specifically configured to run it.  Whereas on development systems there's a tendency to stuff the entire FAST system onto one box, along with 2 or 3 databases, some source code control systems, a GUI environment for the developer running Eclipse or Visual Studio, maybe some mp3's playing the background, etc.  If this happens again you might want to upgrade that developer's workstation, with at least a bit more memory, or maybe a more modern system – we doubt they'll object.

In Closing

Like all cutting edge technology products, ESP has many subsystems, and they can get out of sync once in a while. We have not seen this on a production system, and if our theory is correct, you shouldn't either. And the good news is the system was smart enough to explain what was happening, even though it didn't technically considere this to be an error. FAST has a pretty extensive logging facility, if you haven't had the chance to check out it. FAST ESP remains one of the two most scalable systems on the market today, although many upstarts will attempt to challenge that in the years to come.

We're also seeing a trend to "starve" development and testing servers; we've seen this at various client sites with multiple search vendors. We advise companies to not skimp too much on the non-production hardware; if saving $100 on RAM costs days of a developers' time, this is no bargain!

Got a question, or maybe some weird symptoms? Drop ol'doc a line at drsearch@ideaeng.com