Troubleshooting an outage on Jama 8.0 and up

By Jason posted 02-15-2019 17:28

  

Let me start by saying that if your Jama production instance goes down, you should submit a ticket to Jama support here right away. But if a sandbox Jama instance goes down and you are feeling a bit adventurous, here are some steps you can try to bring Jama back up. Note: all commands should be made from the Jama server command line and assume root access. 

Before getting started it is a good idea to backup your /data/contour/ directory which can be done with the command cp -R /data/contour/ /tmp. This will create a copy of your data/contour directory in your root /tmp directory. The /data/contour/ directory is where Jama stores things like attachments, avatars and reports. Jama itself is stateless, so as long as you have this directory and your database you can kill and restart Jama as much as you want without causing issues. You can always do a fresh full install of Jama and copy these files back to your data/contour directory and be fine. 

 
Check the Jama server resources:

 Running low on memory or disk space can cause issues with Jama, so check that you have enough of both.

 The command top -o %MEM will list your available memory on the system. If memory is running low on the server, adding more memory or stopping other processes on the server could help the issue. Similarly, the command top -o %CPU will let you see if you are running low on processing power. Lastly, check that you are not running out of storage. df -h will show your current capacity and usage. If you are reaching near 100% disk usage, consider clearing out some space on the drive, old log files are a good place to start. 
 

Check for conflicts:

There are a few other programs that Jama doesn’t play well with. The first is firewalld. If you have recently rebooted the server it is possible that firewalld has restarted too. To check if it is running you can use the systemctl status firewalld command. If firewalld is running, you can stop it with the command systemctl stop firewalld.

Next check if apache is running. Apache causes port conflicts with Jama so they both can’t be running on the same server. Use the command systemctl status apache to see if it is running. If apache is running on the server, you can stop it with systemctl stop apache.
 

Check the Docker containers:

Jama uses 5 docker containers to run. These are Jamanginx, Jamacore, search, elasticsearch, and diff. To see if all of the containers are running, use the command docker ps. This shows a list of all running docker containers. If one of the containers is down, you can check the docker logs for that container by first running docker ps -a to get the ID of the container that is down and then running docker logs <container ID>. The last entries in this log can often point you to what caused the container to go down.

If one or more docker containers are down, you can restart all of the docker containers using the command systemctl restart docker.

  

Check replicated: 

Replicated handles the Jama admin console, the docker containers, and the Jama license. It’s what starts Jama up and keeps it running, so it is very important that replicated is up. To check the status of replicated use replicatedctl app status inspect. Here you can see the status of replicated and if it is reporting any errors. If replicated is down, or if it is running but Jama is down, try restarting replicated. It is a good idea to make sure you have a back up the data/contour/ directory before restarting replicated. There are also a few directories on the Jama server that need to be rebuilt when replicated restarts, so it’s a good idea to remove them before starting replicated back up, but these don’t need to be backed up first. Here are the steps to restart replicated:

 

1. Stop replicated:

 replicatedctl app stop

 
2. backup the data/contour/ directory:

  cp -R /data/contour/ /tmp

 

3. Remove the activeMQData, elasticsearch, and tenant directories from the /data folder on the Jama server:

  cd /data

  rm -rf activeMQData, elasticsearch, tenant

 

4. Start replicated:
  replicatedctl app start


If you have restarted replicated and Jama is still having issues, create a support-bundle from the admin console or from the command line of the Jama server by running replicated apps copy the ID of the Jama app then run replicated support-bundle <ID> and submit a ticket to Jama support and upload the support bundle here.

0 comments
10 views