Troubleshooting an outage on Jama 8.0 and up
Let me start by saying that if your Jama production instance goes down, you should submit a ticket to Jama support here right away. But if a sandbox Jama instance goes down and you are feeling a bit adventurous, here are some steps you can try to bring Jama back up. Note: all commands should be made from the Jama server command line and assume root access.
Before getting started it is a good idea to backup your /data/contour/ directory which can be done with the command cp -R /data/contour/ /tmp. This will create a copy of your data/contour directory in your root /tmp directory. The /data/contour/ directory is where Jama stores things like attachments, avatars and reports. Jama itself is stateless, so as long as you have this directory and your database you can kill and restart Jama as much as you want without causing issues. You can always do a fresh full install of Jama and copy these files back to your data/contour directory and be fine.
Check the Jama server resources:
Running low on memory or disk space can cause issues with Jama, so check that you have enough of both.
The command top -o %MEM will list your available memory on the system. If memory is running low on the server, adding more memory or stopping other processes on the server could help the issue. Similarly, the command top -o %CPU will let you see if you are running low on processing power. Lastly, check that you are not running out of storage. df -h will show your current capacity and usage. If you are reaching near 100% disk usage, consider clearing out some space on the drive, old log files are a good place to start.
Check for conflicts:
There are a few other programs that Jama doesn’t play well with. The first is firewalld. If you have recently rebooted the server it is possible that firewalld has restarted too. To check if it is running you can use the systemctl status firewalld command. If firewalld is running, you can stop it with the command systemctl stop firewalld.
Next identify if any services are running that may have listening port conflicts, or that can consume system resources causing negative affects to the performance of Jama Connect. One example is Apache, if not properly configured Apache causes port conflicts with Jama so they both can’t be running on the same server. Use the command systemctl status apache to see if it is running. If apache is running on the server, you can stop it with systemctl stop apache. You can substitute other services names in the command if you find any other port conflicts.
Check the Docker containers:
Jama uses 5 docker containers to run. These are Jamanginx, Jamacore, search, elasticsearch, and diff. To see if all of the containers are running, use the command docker ps. This shows a list of all running docker containers. If one of the containers is down, you can check the docker logs for that container by first running docker ps -a to get the ID of the container that is down and then running docker logs <container ID>. The last entries in this log can often point you to what caused the container to go down.
If one or more docker containers are down, you can restart all of the docker containers using the command systemctl restart docker. Before doing this you should stop the replicated service first. You can do using the replicatedctl app stop command.
Check replicated:
Replicated handles the Jama admin console, the docker containers, and the Jama license. It’s what starts Jama up. To check the status of Jama Connect use replicatedctl app status inspect. Here you can see the status of Jama Connect and if it is reporting any errors. you can also check the docker replicated logs to see if they point to any issues by using docker logs replicated. If replicated is down, or if it is running but Jama is down, try restarting replicated.
1. Stop replicated if it is up:
replicatedctl app stop
2. backup the data/contour/ directory:
cp -R /data/contour/ /tmp
3. Start replicated:
replicatedctl app start
If you have restarted replicated and Jama is still having issues, create a support-bundle from the admin console or from the command line of the Jama server by running replicated apps copy the ID of the Jama app then run replicated support-bundle <ID> and submit a ticket to Jama support and upload the support bundle here.