Jama 8 Installation and Upgrade Troubleshooting Tips

By Jamal posted 06-08-2017 14:50

  

If you encounter issues installing or upgrading Jama 8, walk through the checklist below before submitting a support ticket. Many of the issues that our customers have reported while installing Jama 8 have been due to one (or more) of the following.

 

Does the Jama application server meet the specifications for Jama 8?

Is Jama configured behind a proxy?

Is a firewall blocking the communication between Jama’s containers and the server?

Is the Jama server connecting with the database?

Have you changed the Jama server IP address?

Are you pointing an existing Jama 8 environment to a new database?

Are all Replicated and Jama containers running?

Have you restarted Docker and Replicated services properly after a configuration change?

Is the application no longer starting after syncing a new license?

Useful commands


Does the Jama application server meet the specifications for Jama 8?

  • Check the supported software and system requirements to make sure your Jama application server is compliant. The installation can fail if one (or more) of the system requirements is not met.
  • Pay special attention to the operating system, the total memory of the Jama server, the Docker version that is installed and whether it is supported by Docker. The errors due to an unsupported operating system or missing dependencies for installation of Docker usually show up while running the install script.
  • Verify if your Jama server has enough memory. Even if you are running Jama on a test server with potentially few users, you still need to allocate the memory required for the SMALL category.
  • Make sure Linux is installed and configured properly.
  • If you are using RHEL make sure you have an active subscription.
  • Docker repository on GitHub is a good place to consider if you face issues during the installation of Docker.
  •  Running devicemapper in loopback mode is discouraged for production. It has known performance problems and a different storage driver should be used. Please visit the article Devicemapper Warning for recommended storage drivers for Docker.

 

Known symptoms:

  • Dashboard gets stuck at “Waiting for Jama Core.”
  • Docker installation fails due to a missing dependency.
  • Install script halts with the error:

Either your platform is not easily detectable, is not supported by this installer script, or does not yet have a package for Replicated.

- If you face the above error, you are using a Linux distribution that is not yet supported by the install script. Please follow the manual instructions instead. Open a support ticket and include your Linux distribution and we will file a request with our Product team to add support for it.

  • Install script shows a warning on Docker devicemapper.

Running devicemapper in loopback mode is discouraged for production. It has known performance problems and a different storage driver should be used. Please visit the article Devicemapper Warning for recommended storage drivers for Docker.

Note: Troubleshooting Docker installation issues is out of Support's scope.

Is Jama configured behind a proxy?

Proxy misconfiguration has been the root cause of most of the issues that our customers have faced during installation of Jama 8.

  • Make sure that you follow the instructions to Install the Admin Console and configure the NO_PROXY environment variable.
  • Verify that both HTTP_PROXY and NO_PROXY environment variables are set properly and are picked up by Docker:

- Download a Support Bundle and go to scheduler > node > {node_id} > docker > docker_info.json.

- Look for "HTTPPROXY" and "NOPROXY" attributes and make sure the right values are set for both.

  • Verify that replicated file has HTTP_PROXY variable set in REPLICATED_OPTS and it has the correct value:

- Find the replicated file and open it.

Debian and Ubuntu: /etc/default/replicated
CentOS and RHEL: /etc/sysconfig/replicated

- Locate the line that starts with REPLICATED_OPTS

- Make sure "-e HTTP_PROXY={proxy_url}" is set properly

  • If your proxy requires authentication, the URL should include the user: password e.g. http://user:password@proxy_url. Make sure you are using the right credentials.
  • Check to see if the proxy variables are indeed working to verify communication outside of application server

curl -s http://help.jamasoftware.com | grep title

curl -s http://[some_intranet_url]

 

Known symptoms:

  • Admin Console throws an error while pulling images. e.g.:

API error (500): Get http://{ip_address} EOF

  • Snapshots are failing.
  • Jama and the Admin Console are running but the Admin Console is stuck at "Starting" on the Dashboard.
  • Jama license does not get updated after a license sync even after restarting Jama.

 

Is a firewall blocking the communication between Jama’s containers and the server?

Firewalls can block communication among containers or block communications from outside of the Jama server if the required ports are not open. We recommend disabling firewalls on the Jama server if possible. If you need to have firewalls, make sure that routing rules are not interfering with the rules that Docker adds.

 

  • Make sure all the required ports are open on the Jama server.
  • If you are running CentOS or RHEL make sure that firewalld is not running. There is a conflict between firewalld and Docker that may cause communication issue among running containers. 
  • Disable the firewall on your server and restart the setup to verify that the problem is, in fact, the firewall.
  • If you need to have firewall enabled on your Jama app server, Jama requires all the ports to be open for internal IP addresses used by Docker. You can use the following iptables rule to allow traffic on all ports for 172.17.0.0/24 subnet which is used by Docker. 

-A INPUT -p tcp -s 172.17.0.0/24 -j ACCEPT

 

Known symptoms:

  • Pre-flight checks show an error

Can connect to address - ERROR: dial tcp {ip_address}:9874: getsockopt: no route to host

  • Admin Console throws an error while deploying search or elastic search containers.

Is the Jama server connecting successfully?

Dashboard shows the error Can't populate database.

It is very likely that your Jama server cannot communicate properly with the database. There are several common reasons:

 

  • Your Jama server cannot communicate with the database server. You can use the telnet command to check the connection

telnet {database_url or ip_address} {database_port}

e.g.

telnet mysql.jama.com 3306

  • If you are using MySQL, verify if Jama database user has permission to connect from application server host.
  • You have entered the wrong credentials for the database user. Try logging into your database directly using the same credentials to make sure it works.
  • Your database user does not have enough permission to create and modify the database. See the help guide.

You can look into the following log files on your Jama server to get more details on what can be wrong:

/logs/tomcat/contour/contour-c3p0.log
/logs/tomcat/contour/contour.log

Have you changed the Jama server IP address?

If you have to change the IP address of your Jama 's Application Server, please follow the instructions to Update Jama's IP Address.
You may get an error in Pre-Flight checks (Can't connect to address - ERROR: dial tcp {ip_address}:9874: getsockopt: no route to host) showing the old IP Address of the server if you miss a step.

Are you pointing an existing Jama 8 environment to a new database?

If you want to point an existing Jama 8 environment to a new database, or you are dropping an existing database and creating a new one, make sure that you are following the instructions:

1. Stop Jama from Admin Console.

2. Drop the existing database and create a new one with the same name.

3. Go to the /data/ folder and remove the following folders:

ActiveMQData , elasticsearch , tenant

4. If loading from an xml backup, put the xml file in restore and make sure it has all the permissions as stated on the settings page.

5. Go to the Settings page and make sure all the database fields are filled properly, then select Save.

6. Start Jama.

 

Are all Replicated and Jama containers running?

If none of the above scenarios apply to you, use the following command to check if any of the main Replicated or Jama containers have exited.

sudo docker ps -a

The main Replicated containers are:

  • replicated
  • replicated-ui
  • replicated-operator

The main Jama containers are:

  • jamanginx
  • jamacore
  • search
  • elastic-search

If any one of the above containers are not running, you can take a next step and check the logs for the specific containers using:

sudo docker logs {container_id}

 

Have you restarted Docker and Replicated services properly after a configuration change?

If you have changed a configuration file in Docker or Replicated you need to restart those services properly to make sure they take effect.
There are times when restarting Docker and Replicated services can fix an installation issue.

Note: Restarting Docker and Replicated services means stopping Jama.

1. Stop Jama from the Admin Console Dashboard. 

2. Go to the Jama application server and stop all running container using the below command.

sudo docker stop $(sudo docker ps –a –q)

Skip the steps above if the Admin Console is not accessible. 

3. Restart Docker.

sudo systemctl restart docker

4. Restart all replicated services.

sudo systemctl restart replicated replicated-ui replicated-operator

5. Go to the Admin Console and start Jama.

 

Is the application no longer starting after syncing a new license?

Check to ensure that your current user account settings are compliant with the new license (i.e. you have the same number of users configured as your license entitles you to have).

 

Known Symptoms:

Review the stderr.log log inside the Support Bundle > scheduler > container > tenantmanager to confirm the following entry:

[timestamp information] INFO __main__: sending status to Replicated: problem: problem set application license: Error license update for 'contour'; Cannot load this license. It would put the organization out of compliance.

Next Steps:

  • If you believe you received an errant license, contact your Account Manager to remedy the issue.
  • If you received the correct license, but forgot to update your Jama user accounts, contact our Support team and they will provide you with steps to manually edit the existing user accounts.

 

Useful commands


You need root access to run docker commands.

docker info
A useful command to verify the Docker version, the total number of Docker containers and their status, kernel version and OS.

docker ps -a
Lists all the docker containers and their status (whether they are currently running or not).

docker logs {container_id}
Shows the logs for the container specified by container_id. Very useful for initial troubleshooting if a container is not running.

telnet {database_host_server} {database_port}
Useful command to verify if Jama server can communicate with database server.


You can always open a Support ticket and attach a Support Bundle if you still cannot determine the problem.

 

 



#Jama8 #installation #bestpractices
0 comments
1093 views