A brief introduction to indexing in Jama

By Shabnam posted 02-21-2015 00:23

  
This article is intended to explain indexing with the previous Jama model present in 2015.5 and below.

Currently Jama uses Lucene to search for objects, whether the object is an item, a node in the explorer tree, a revision, a comment, etc. Lucene uses the indexes to quickly find the objects that comply with a search criteria. Jama application creates and writes the indexes in search folder in contour_home. If you have On-premises version of Jama you have access to this folder. If you are hosted, this folder will be on the server and you do not have access to it.
Below is the snapshot of the search folder in the contour home:



Each folder contains the indexes for that object. These are not tables in the database; these are Jama objects. This means the values for the indexes in each object may be fetched from several tables, e.g. some of the fields that are indexed in contouritem are name, documentType, testRunSet (in case the contouritem is a test run). The value for these fields are in tables document, documenttype and testset respectively.

Full re-index:


During a full re-index all the indexes will be rewritten from the current values in the database. If there is any missing indexes, it will rewrite all the indexes for all the objects. 

Project level re-index:

During a project-level re-index only the items for that project will get re-indexed.  This means that the objects that are not project-specific like user and role will not get re-indexed.


As for the other objects—comment, evententry, contouritem, revision_item and revision_user— Jama will look into the project id associated with each entry and only those with the project id equal to the project that is being re-indexed will get updated. 

#installation #administration
24 comments
191 views

Comments

12-03-2015 22:20

Sudha, this is a very good question. We also deal with this question pretty frequently when troubleshooting tickets; determining whether a project level re-index can fix the problem or do we want to ask customers to do a full a re-index which requires a maintenance window and a root access. 
It is not always easy to determine that, however, we can provide some hints that can help:

If you have issues with user list, then it is definitely time for a full re-index. The project level re-index will not update indexes for users.  

A project level re-index updates indexes for the items that belong to a specific project. It also only "updates" indexes vs. a full re-index clears all the search folder and re-builds all the indexes again (more effective). The type of issues that a project level can solve is inconsistencies. For example, you search for an item and it shows up but you cannot find it in the explorer tree. This is a type of issue that a project level re-index may help. 

If a filter does not show its results at all, it is most likely because of a corrupted index and you may require a full re-index. When there is a corrupted index file that needs to be removed, you usually do not see any results at all or you see an error message in logs or in a pop up window. This is the time to do a full re-index. However, as Swoo mentioned, it is always better to do a project level re-index first to check whether that can solve the problem because it has less impact on users of Jama. 

We also encourage you to open a support ticket if you cannot determine whether it is time to do a full re-index or not. We can check the logs and run scripts to verify your search folder that can better help us come up with the right decision. 

12-03-2015 21:56

Hi Sudha,

When I notice something that could be wrong (any kind of inconsistencies and strangeness--but I don't have a list of known symptoms and problems) with the Jama Web UI, the first thing I'll do is to use Chrome and clear the browser histories. This takes seconds.

If that does not fix it,  then I'll do a project re-index. This takes minutes (or hours depending on number of items in that project)

If that does not fix it, then I'll ask the database admin to do a full server re-index, but that could be very long (maybe days) and tedious based on number of projects and database items.

Then I'll file a bug report if strangeness still exist. So the fix is based on how much time the re-index will take.

Good luck.

swoo

12-03-2015 21:48

Is there a list of conditions/symptoms that determine whether a project re-index or full re-index is needed? For example, a filter doesn't  display its results properly, will project re-index fix it? How is the determining factor to decide which type of re-index needs to be performed?

Thanks,
Sudha

12-02-2015 18:05

Thank you Swoo. You are always ahead of us. That is correct, a user with Manage Project permissions can run a project level re-index. 

12-02-2015 18:03

Thanks Swoo!

12-02-2015 17:59

Sudha, please look for project/configure project/index. Toward the upper left hand corner. A project admin can do this.

12-02-2015 17:56

How do we perform project level re-index? Can users with manage project access perform it or does it have to be done by Org admin?

Thanks,
Sudha

08-21-2015 19:31

Thank you swee for your reply! That is correct. Jama re-index will not change any value for any fields. 

08-21-2015 15:48

Chris, all global and item ID's are retained with either a project or full database re index swoo.

08-21-2015 15:18

Does re-indexing change the Global ID's?

05-07-2015 15:53

Update: Please note that with 2015.1 release and back ground indexing tomcat can not be restarted as the background indexing will be interrupted. Therefore from 2015.1 release, a proxy should be used to block and further allow users' access to Jama.

05-04-2015 18:50

Hi Steve, for hosted Jama instances full re-index can only be done by Jama Support. We usually do that if our hosted users are facing an issue that is related to a bad index. For example a filter is not showing its results properly or items are not listed in the list view as they should. 

05-04-2015 18:02

Hi Shabnam,  for hosted users, is there a way to do a full index?  

03-03-2015 18:20

This Article has an example of re-directing Jama to a maintenance page.

03-03-2015 18:01

Hi George,
Thank you for your question. The easiest way to do that is to change the tomcat port in the server.xml file. This requires a server restart. Once you change the port, users can not access tomcat with the same URL. Once the re-indexing is done, you need to change the port back to what it was and restart the tomcat server again.
If Jama is behind a proxy, you can have requests to Jama re-directed to a maintenance page when you are doing a re-index.

03-03-2015 17:15

What method does everyone use to exclude users from Jama before indexing?
I have teams in a lot of countries so there's a risk that someone remote could login.

02-27-2015 22:18

Hi Bob, sorry for getting back to you with delay. I wanted to check with our Engineerings and Consultants to check whether there is a preference in using one method (adding Test Cases across boundaries) vs. another one (re-using Test Cases and keep everything within a Project). From Engineering perspective we don't see a performance impact on either of the methods. Re-use in general is an expensive process in terms of transactions with database. Our only recommendation is to that in small batches and in less busy hours. Keeping Test Cases within a project can prevent confusions with filtering. For example, searching for a Test Runs with the scope of project returns the Test Runs that are from the Test Cases that belong to that project only and you need to change the scope to All Projects. 
From the Consultant's perspective, we recommend either way based on how your Test Cases are designed. If there is a library of Test Cases that need to be used in all projects then a re-use is the way to go. If there are sets of Test Cases distributed among different projects and each project need to use them based on their needs then what you are doing right now makes more sense. 
I hope my answer makes sense.  

02-24-2015 19:51

Hey Shabnam-

I'm always crossing project boundaries and I've had a sense this might not be optimal for Jama.  This might be getting a little off-topic but do some forms of re-use lessen this impact?

Thanks,
Bob

02-24-2015 19:08

Hi Jason, thank you for your question.
The Test Plans can include Test Cases that are from other projects. Therefore, if there is an indexing problem in a Test Plan, a project level re-index may not resolve it. This is the scenario that makes sense in your instance.

02-24-2015 18:57

Thank you for a very valid question: Do users need to be excluded during a project re-index?
The re-index process is a read operation and in theory it should be ok to run the re-indexing in the back ground while users are working with Jama. However there is a possibility that a user updates/creates an item on the exact same time that the item is being re-indexed. In this rare situation the indexed item may not reflect the most updated information or become corrupted. 
Therefore as a rule of thumb we highly recommend users to be logged out of Jama especially during a full re-index that all items are affected and is a more lengthy process.
In the case of project re-index, if you have so many items and users work with project heavily then it is recommended to exclude users. I hope this answers your question. Let us know if you need more clarification.

02-24-2015 14:03

Hi Shabnam - Thanks for the post.  It is helpful to see some insight into how jama uses indexes.  Are there other items that a full index does when compared to the project index?  as you know we have seen project level issues resolved with a full index when a project level one did not resolve them.

02-24-2015 14:00

If something is being actively edited during the index (full or partial) there is potential for data issues.  Based on my conversations with Jama we should ensure that users are not actively modifying any items until the index is complete

02-24-2015 13:16

I was wondering about that as well.  I try to do it "off shift", but last night I realized I hadn't warned my team in advance, let alone asked them to log out...  (No complaints this morning though).

Cheers,
Bob

02-24-2015 07:29

Shabnam,

when doing a full re-index Jamas strong recommendation is to exclude users from Jama. Does that also apply to the project re-index?

Regards
Anna