Gathering Empirical Data for Troubleshooting

By Godiva posted 11 days ago


Gathering Empirical Data for Troubleshooting

This document is primarily intended for self-hosted customers; cloud/SaaS customers will not have access to the servers on which Jama is run, so only the Browser Developer Tools section applies for them. You can then attach the aforementioned file to a support ticket on Jama's support portal. Note that Jama Software does not provide support on the installation, administration, or use of the tools listed in this document, although we encourage their use to assist in troubleshooting technical issues.

Client Side Tools:

How To Collect HAR files for Support Troubleshooting
When performance degradation caused by external issues interfere with Jama Connect's functionality, a HAR file can be generated to gather specific data about most issues. When recording a HAR file, use features in Jama Connect that may be generating the type of activity in question. For example, if you have an error popping up in the review center, exercise the functions used to create the error pop up. A HAR file allows anyone with the HAR file to impersonate your account and all the information that you submitted while recording. The data within a HAR file can include private or personally identifiable information (cookies, passwords, credit card numbers, etc.) All network traffic during the time a HAR file is being recorded is also included. If we ask for a HAR file, it's recommended to open the HAR file using your text editor of choice and delete or obfuscate any sensitive information that you submitted through a form. Customers are responsible for any data they provide to Jama (personal details, passwords, credit card numbers, etc.) You can navigate to where it says "cookies" in the file and delete any lines regarding sensitive information, such as session IDs.

Google Chrome:
Depending on your browser version, the path names may vary to open the console:
   1. Navigate to the area in Jama Connect where you will perform the slow or faulty action.
   2. From the Chrome menu (top right) , select Tools > Developer tools or More tools > Developer tools as applicable. Alternatively, you can right-click anywhere and select Inspect Element
   3. Select the Network tab.
   4. Select the red record icon.
   5. If applicable, select the Preserve log check box to save network transaction data.
   6. Before trying to reproduce the issue, select the Clear icon to remove unnecessary header information.
   7. Perform the action which is causing or suffering from the issue (select the button, run through the reuse process, open a review, etc)
   8. Once the issue is experienced, right-click on any of the items within the Network tab and select Save all as HAR with Content (preferred) or take a screenshot of all the requests.
   9. Upload your HAR file to your ticket or attach it to your email so that our Support team can analyze it.

Firefox version 41 and higher comes with the ability to copy the transactions to a HAR file.
To capture data with the network tab in Firefox:
   1. From the application menu, select Tools > Web Developer > Network or press Ctrl+Shift+I (Windows/Linux) or Cmd+Option+I (OS X).
   2. Select the Clear option (lower right corner of the Developer window) to remove unnecessary header information.
   3. The recording autostarts when you start performing actions in the browser.
   4. Navigate to the place in Jama Connect where you will perform the slow or faulty action.
   5. Perform the action which is causing or suffering from the issue (click on the button, run through the reuse process, open a review, etc.)
   6. Capture the information by right-clicking the transaction list in the console and selecting either Copy All As HAR to place the data in your buffer, or Save All As HAR to copy it to a file
   7. Upload your HAR file to your ticket or attach it to your email so that Support can analyze it.

Internet Explorer:
   1. Open Internet Explorer and go to the Jama Connect page where the issue is occurring.
   2. Press F12 on your keyboard (or click the gear icon > F12 Developer Tools)
   3. Click the Network tab.
   4. Reproduce the issue that you were experiencing before, while the network requests are being recorded.
   5. Once done click the Save button.
   6. Give the trace a filename and click the Save button which will save it as a .har file or .xml file.
   7. Upload your HAR file to your ticket or attach it to your email so that we may analyze it.

Microsoft Edge:
   1. Open Edge and go to the Jama Connect page where the issue is occurring.
   2. Open the menu in the top right > More tools > Developer Tools > Network.
   3. Navigate to the place where you will perform the slow or faulty action.
   4. Refresh the page to start capturing the traffic between the browser to the server, or click on a link for the affected page.
   5. Perform the action which is causing or suffering from the issue (click on the button, run through the reuse process, open a review, etc.)
   6. Press Ctrl + S or select the Export as HAR option in the top right corner, designed to look like a floppy disk, and Save As HAR file.
   7. Upload your HAR file to your ticket or attach it to your email so that we may analyze it.

Apple Safari:
   1. Enable Develop menu: Safari > Preferences > Advanced > Select Show Develop menu in menu bar.
   2. Select Develop > Show Web Inspector > Select Network tab.
   3. Within the Network tab, check the Preserve log option.
   4. Navigate to the place where you will perform the slow or faulty action.
   5. Refresh the page to start capturing the traffic between the browser to the server, or click on a link for the affected page.
   6. Perform the action which is causing or suffering from the issue (click on the button, run through the reuse process, open a review, etc.).
   7. Once the page is loaded, select Export on the top right in the window of the Network tab.
   8. Upload your HAR file to your ticket or attach it to your email so that we may analyze it.

When is this useful?
  • The network output is useful any time a page or its elements are slow to load. It may also be helpful in certain situations to help identify what code in the back-end is being called during a particular action in the front-end.  This will allow our support engineers to dig deeper into the issue. Generally speaking, you should always include this output when filing a ticket.
  • The javascript console is useful anytime data or the UI is displaying abnormally, items or data appear to be missing, or clicking on buttons does not appear to do anything. Generally speaking, you should always include this output in any ticket as well.

Server-Side Tools:

Support Bundles
A support bundle includes all of the logs we'd traditionally ask for in support. It also includes logs for Docker and Replicated, detailed system information and all of the configuration and environment variables in use in a customer's instance.

A support bundle can be generated in one of two ways: using the Replicated web portal user interface (UI) and the command line interface (CLI).

Using the UI:
This is the easiest way to generate a support bundle and is the method you will use most often.
   1. Log into the Replicated portal for your Jama instance. This will be located at https://<IP or hostname>:8800
   2. Select the Support tab.
   3. Select the Download Support Bundle button to download the bundle to your local machine.
   4. You can find information on how to generate the Support bundle from the UI here.

Using the CLI:
If you cannot access the Replicated UI, you can still download the support bundle from the Replicated CLI by running the following commands on the server:

replicated apps
replicated support-bundle <app_id>

Where <app_id> is the id of your application taken from the output of the first command. This is available in versions of Replicated 1.2.73 and higher.

Note: If your self-hosted console has a password you will need to authenticate on the CLI.

Thread Dumps
For details on how to take thread dumps, see our community article on taking thread dumps.

In the case of application-wide performance degradation affecting all users in an on-premises installation, thread dumps may be critical to understanding what is occurring within the application at the time of the issue. A thread (short for "thread of execution") is generally a subset of instructions (code) that is designed to run concurrently (i.e. in parallel) to other code.  Threads are what allow the computer to multi-task, or simulate parallel operation, such as when you stream music and type up a document at the same time. By breaking sequences of operations (a.k.a. code) into separate threads of execution, the computer can switch between them rapidly to run them in a manner that appears "simultaneous" to the end user. A thread dump is a snapshot of the state of the application at the moment in which the dump is started. Because it's a snapshot of a given point in time, it's important that thread dumps are taken as the issue is occurring. By running multiple dumps over a short interval, we can determine what the application was attempting to do over the course of that interval and identify particular threads (i.e. sequences of operation) which may be taking a long time or are stuck. Long-running threads may be expected.

Thread dumps can only be captured by someone with direct access to the server on which Jama is hosted, and the method in which they will be collected will be dependent upon which OS the server uses. Typically, a minimum of 3-5 thread dumps taken 3-5 seconds apart (thereby spanning 9-15 total seconds) is necessary for the data to be useful. If you wish to gather samples from a longer period of time, it's better to take more dumps than to increase the time interval.

When is this useful?
This information can be illuminating in a lot of cases, but it is not relevant to every situation. Thread dumps are most useful when investigating performance issues, some memory usage issues, and some issues that do not generate any exceptions or error messages in the logs. Generally speaking, Jama Support will direct you to gather thread dumps if they're necessary to troubleshoot your issue. However, there are a couple of instances where gathering them before filing a ticket can be helpful:

  • If the application has stopped responding completely, and the Tomcat process is still alive, it is critical that you gather thread dumps before restarting the machine. Our ability to troubleshoot or RCA the application "crash" will be severely hampered if thread dumps are not gathered prior to restart.
  • For issues in which a particular functionality is slow for all users regardless of other factors, thread dumps captured while a user is carrying out that action can help determine what may be causing that action to be slow.
  • For site-wide performance issues, thread dumps captured at any time can be helpful to get a picture of what is going on within the application. More targeted thread dumps may be required at the discretion of support.

Heap Dumps
A heap dump is a snapshot of all objects currently stored in memory. Generating a heap dump requires more effort than gathering a log file or gathering thread dumps and can be somewhat cumbersome to deal with, but may be helpful to troubleshoot memory-related issues. Generating a heap dump does stop the Java Virtual Machine for the duration of the time it takes to generate the file, which effectively pauses the Jama application for that duration. Because of this, Jama Support will generally only request Heap Dumps when absolutely necessary.

To generate a heap dump, we recommend using the jmap command built into the Oracle JDK. To run this, you would execute the following in a command prompt (windows) or shell (linux):

  • Linux: jmap -dump:format=b,file=heap.bin <pid>

  • Windows:  %JAVA_HOME%\bin\jmap -dump:format=b,file=heap.bin <pid>

In both cases, <pid> would be the process ID for the Java process that is running Tomcat. This will generate a file in the directory in which it is run that is at least as large in size as the amount of memory you have allocated to your heap. Therefore, it's important that you have enough disk space within the directory to hold the file. Jama Support will provide you with a link to save the file to our secure cloud storage. 

When is this useful?
Heap dumps are generally useful only when memory issues have been identified, either through analysis of the Garbage Collection logs or through OutOfMemory errors in the contour.log. Generally, due to the limited scope of issues where heap dumps can be useful and the overhead and impact of gathering heap dumps, we recommend only gathering a heap dump when advised by Jama Support via a ticket, rather than preemptively before filing a ticket.

Wireshark (cross-platform) or TCPDump (Linux):

Both tools are a form of a "packet sniffer", i.e. a tool that intercepts and reads network packets as they are being transmitted. While Jama Software doesn't support third-party network infrastructure, nor do we commit to troubleshooting it, these tools can help us to gather additional data about the state of a network connection in between the client and the server to help better narrow down whether the issue is a Jama Connect issue or a network issue.

When is this useful?
In certain situations where Jama is attempting to communicate with outside services such as an email server or an integration provider (Tasktop, Jira, Rally, TFS, etc.) and the issue is determined to be a network issue, packet information can help determine where in the network the issue lies. Generally speaking, preemptively gathering packet dumps isn't necessary for most issues; Jama Support will request them as needed. 

Monitoring Tools:

There are a wide range of application monitoring tools available, both free and as paid services.  Jama Software uses multiple monitoring solutions to ensure the stability and uptime of its cloud solution.  While Jama Software does not provide support for third-party tools, including monitoring solutions, self-hosted customers are encouraged to investigate deploying real-time monitoring solutions for their Jama application.  For customers who choose to implement a monitoring solution, providing data to Jama Support from your monitoring tools when reporting intermittent or performance-related issues can be an invaluable auxiliary data source.

Tomcat Manager (prior to 8.x only)
Of particular note is the Tomcat Manager application, which is a built-in administration tool that ships with the Tomcat servlet container (a required system component).  While not a robust real-time monitoring solution, the Tomcat Manager app provides some basic statistics and useful functionality, particularly around user sessions.  Jama Software strongly recommends enabling (or being prepared to enable upon request) the Tomcat Manager app to assist in administering and monitoring your Tomcat installation. 

Database Tools:

SQL Server
The Microsoft Technet and Microsoft Developer Network websites have a collection of articles detailing tools and methods to assist in troubleshooting database issues.  Jama Support may recommend configurations or changes from this list, including (but not limited to) enabling or disabling trace flags or additional logging.  You can view the full lists at:

The MySQL performance schema and the slow query log can be used to help track down issues with database performance. Jama Support may request information from these sources if we determine that the issue may lie on the database server.  Additional logs or information may also be requested depending on the situation.