get all docs in a particular novus collection from wln?

I have a document collection that currently has only about 50 docs. For the beta release of our wln feature project it'll have 600 docs. In 2014, it should have 6k docs. At app init time, the search module needs to load data from these docs. My understanding that the most kosher way to get this data is to have the content search team create a berkeley file that contains this data. An alternative (which I'm currently exploring) would be to make a call to Novus to get all docs in this collection at app init time. My questions are:

1. Does Novus have a method for getting all docs in a particular collection? Can someone point me to this documentation?
2. Assuming the answer to #1 is yes, at what point would I need to worry about performance implications? I assume asking Novus for 50 docs is no big deal. Is asking for 600 docs a big deal? 6k?

Thanks,
Ben

Find more posts tagged with

refinitiv-internal

Accepted answers

Erik Hyrkas

If the collection set is small, you can do a search for =n-document, but this will return a maximum of 10,000 documents. The implications are the same as doing a search for any 10,000 results. There are real-life examples where this is done, but again, they are rather small collection sets. To my knowledge, there is no way to get more than the 10,000 documents without doing multiple queries and having an additional constraint (where each query finds documents that the others don't.)

Kevin McKeown (who already answered) is going to know far more than I do and give you more detailed options. I'm just giving you the quick and dirty answer.

All comments

Kevin McKeown

Ben,
I am the Novus Consultant for WestlawNext, and there are implications to the question you are asking, but it requires further review of your use case and alternatives. Please contact me directly and we can discuss this. Thanks!

Erik Hyrkas

mtaylor

Is this "configuration" data that you are going after at "app initialization" time? If so, you might consider putting it into PMD. If you are just trying to get the data and cache it, then you might consider utilizing a coherence cache. Not knowing your use case, it does sound like Berkley is likely the "proper" method for doing what you are attempting to do (any reason you don't want to utilize that mechanism)?

As Kevin indicated, it would really depend on your use case. Remember that dpeneding on how many instances of the app are being deployed, if you are calling Novus to get the initialization data, it could create a large, unexpected burst of usage against Novus that increases as the # of application instance increases.

Generally speaking Novus would not be the preferred solution for what you are trying to do (app initialization) but that doesn't mean that it doesn't make sense in your specific instance (there are instances where "caches" are populated from Novus document collections.

Erik Hyrkas

Mark is correct in that querying Novus at start up isn't ideal and that PMD would be the best spot for configuration and Berkeley databases would be the best place for frequently used data (presuming you don't need it updated more than once per day.)

For the record, I was strictly answering your question, not saying that you should do it.

Benjamin Anderson

"dpeneding on how many instances of the app are being deployed"
good point - something I hadn't considered. I was only thinking about the need for one instance in one environment. It starts to add up if consider every instance in every environment.

EXPLORE OUR SITES