Database reset broke jama

Mark Moorcroft
Mark Moorcroft Member Posts: 2
edited March 13 in Peer-to-Peer Support Forum

Support had us recreate the SQL database to remove all the trash sample frameworks, before they would schedule the onboarding meeting. Now I have connect-drainer-xxxx pod in error condition and jama doesn't start up. The database appears to have reprovisioned as you might expect. I have deleted the pod and rebooted several times. I tried a redeploy and that fails with backoff limit exceeded.

{"log":"  File "/usr/bin/entry", line 269, in u003cmoduleu003e
","stream":"stderr","time":"2023-12-07T00:12:39.855531968Z"}
{"log":"    main()
","stream":"stderr","time":"2023-12-07T00:12:39.855538468Z"}
{"log":"  File "/usr/bin/entry", line{"log":"2023-12-07 00:12:39,166 INFO __main__: Starting Connect Drainer
","stream":"stderr","time":"2023-12-07T00:12:39.258826881Z"}
{"log":"2023-12-07 00:12:39,258 WARNING shared_lib.manifest: manifest being read in non-strict mode, 
please refactor this image
","stream":"stderr","time":"2023-12-07T00:12:39.258889981Z"}
{"log":"2023-12-07 00:12:39,258 INFO shared_lib.manifest: loading manifest file from /entry/manifest.yaml
","stream":"stderr","time":"2023-12-07T00:12:39.258974482Z"}
{"log":"/usr/local/lib/python3.6/site-packages/shared_lib/manifest.py:13: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
","stream":"stderr","time":"2023-12-07T00:12:39.268807672Z"}      
{"log":"  return yaml.load(open(self.file_path).read())
","stream":"stderr","time":"2023-12-07T00:12:39.268843473Z"}
{"log":"2023-12-07 00:12:39,268 INFO __main__: Checking if Jama Core is available at http://core:80/rest/tenant
","stream":"stderr","time":"2023-12-07T00:12:39.268850073Z"}
{"log":"2023-12-07 00:12:39,699 INFO __main__: Jama Core is available
","stream":"stderr","time":"2023-12-07T00:12:39.700048938Z"}
{"log":"2023-12-07 00:12:39,699 INFO __main__: request base url will be: "http://core:80"
","stream":"stderr","time":"2023-12-07T00:12:39.700214639Z"}
{"log":"2023-12-07 00:12:39,700 INFO __main__: host url: http://core:80
","stream":"stderr","time":"2023-12-07T00:12:39.70032114Z"}
{"log":"2023-12-07 00:12:39,846 ERROR __main__: problem: provisioning is in an unexpected state: error
","stream":"stderr","time":"2023-12-07T00:12:39.855487067Z"}
{"log":"Traceback (most recent call last):
","stream":"stderr","time":"2023-12-07T00:12:39.855524868Z"}
{"log":"  File "/usr/bin/entry", line 269, in u003cmoduleu003e
","stream":"stderr","time":"2023-12-07T00:12:39.855531968Z"}
{"log":"    main()
","stream":"stderr","time":"2023-12-07T00:12:39.855538468Z"}
{"log":"  File "/usr/bin/entry", line 264, in main
","stream":"stderr","time":"2023-12-07T00:12:39.855543868Z"}
{"log":"    raise exception
","stream":"stderr","time":"2023-12-07T00:12:39.855549668Z"}
{"log":"  File "/usr/bin/entry", line 245, in main
","stream":"stderr","time":"2023-12-07T00:12:39.855554968Z"}
{"log":"    if tenant.is_provisioned():
","stream":"stderr","time":"2023-12-07T00:12:39.855560368Z"}{"log":"  File "/usr/bin/entry", line 82, in is_provisioned
","stream":"stderr","time":"2023-12-07T00:12:39.855565568Z"}
{"log":"    raise RuntimeError("provisioning is in an unexpected state: error")
","stream":"stderr","time":"2023-12-07T00:12:39.855570768Z"}
{"log":"RuntimeError: provisioning is in an unexpected state: error
","stream":"stderr","time":"2023-12-07T00:12:39.855576568Z"}
------------------------------
Mark Moorcroft
AMA Inc.
Santa Clara CA
------------------------------

Comments

  • Mark Moorcroft
    Mark Moorcroft Member Posts: 2
    edited December 2023

    I dropped the db a couple more times, then eventually a redeploy finally worked. 

    Now the error state is:

    NAME                                  READY   STATUS      RESTARTS   AGE
    activemq-0                            1/1     Running     0          35m
    connect-drainer-pcppm                 0/1     Completed   0          10m
    core-0                                1/1     Running     0          35m
    diff-0                                1/1     Running     0          34m
    elasticsearch-0                       1/1     Running     0          34m
    hazelcast-0                           1/1     Running     0          35m
    kotsadm-85c97cb449-qpwn7              1/1     Running     22         11d
    kotsadm-postgres-0                    1/1     Running     0          34m
    kurl-proxy-kotsadm-5c6bb84446-rgsf6   1/1     Running     8          11d
    nginx-0                               1/1     Running     0          35m
    oauth-0                               1/1     Running     0          35m
    saml-0                                1/1     Running     1          35m
    search-0                              1/1     Running     0          34m
    tenant-manager-5nz5w                  0/1     Error       0          9m53s
    tenant-manager-74csx                  0/1     Error       0          10m
    tenant-manager-d68dt                  0/1     Error       0          10m
    tenant-manager-ggt2m                  0/1     Error       0          9m33s
    tenant-manager-hgn7p                  0/1     Error       0          8m53s

    ------------------------------
    Mark Moorcroft
    AMA Inc.
    Santa Clara CA
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 12-07-2023 11:17
    From: Mark Moorcroft
    Subject: Database reset broke jama

    Support had us recreate the SQL database to remove all the trash sample frameworks, before they would schedule the onboarding meeting. Now I have connect-drainer-xxxx pod in error condition and jama doesn't start up. The database appears to have reprovisioned as you might expect. I have deleted the pod and rebooted several times. I tried a redeploy and that fails with backoff limit exceeded.

    {"log":"  File "/usr/bin/entry", line 269, in u003cmoduleu003e
","stream":"stderr","time":"2023-12-07T00:12:39.855531968Z"}{"log":"    main()
","stream":"stderr","time":"2023-12-07T00:12:39.855538468Z"}{"log":"  File "/usr/bin/entry", line{"log":"2023-12-07 00:12:39,166 INFO __main__: Starting Connect Drainer
","stream":"stderr","time":"2023-12-07T00:12:39.258826881Z"}{"log":"2023-12-07 00:12:39,258 WARNING shared_lib.manifest: manifest being read in non-strict mode, please refactor this image
","stream":"stderr","time":"2023-12-07T00:12:39.258889981Z"}{"log":"2023-12-07 00:12:39,258 INFO shared_lib.manifest: loading manifest file from /entry/manifest.yaml
","stream":"stderr","time":"2023-12-07T00:12:39.258974482Z"}{"log":"/usr/local/lib/python3.6/site-packages/shared_lib/manifest.py:13: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read <a href="https://msg.pyyaml.org/load">https://msg.pyyaml.org/load</a> for full details.
","stream":"stderr","time":"2023-12-07T00:12:39.268807672Z"}      {"log":"  return yaml.load(open(self.file_path).read())
","stream":"stderr","time":"2023-12-07T00:12:39.268843473Z"}{"log":"2023-12-07 00:12:39,268 INFO __main__: Checking if Jama Core is available at <a href="http://core:80/rest/tenant
" ,"stream":"stderr","time":"2023-12-07t00:12:39.268850073z"}"="">http://core:80/rest/tenant
","stream":"stderr","time":"2023-12-07T00:12:39.268850073Z"}</a>{"log":"2023-12-07 00:12:39,699 INFO __main__: Jama Core is available
","stream":"stderr","time":"2023-12-07T00:12:39.700048938Z"}{"log":"2023-12-07 00:12:39,699 INFO __main__: request base url will be: "http://core:80"
","stream":"stderr","time":"2023-12-07T00:12:39.700214639Z"}{"log":"2023-12-07 00:12:39,700 INFO __main__: host url: <a href="http://core:80
" ,"stream":"stderr","time":"2023-12-07t00:12:39.70032114z"}"="">http://core:80
","stream":"stderr","time":"2023-12-07T00:12:39.70032114Z"}</a>{"log":"2023-12-07 00:12:39,846 ERROR __main__: problem: provisioning is in an unexpected state: error
","stream":"stderr","time":"2023-12-07T00:12:39.855487067Z"}{"log":"Traceback (most recent call last):
","stream":"stderr","time":"2023-12-07T00:12:39.855524868Z"}{"log":"  File "/usr/bin/entry", line 269, in u003cmoduleu003e
","stream":"stderr","time":"2023-12-07T00:12:39.855531968Z"}{"log":"    main()
","stream":"stderr","time":"2023-12-07T00:12:39.855538468Z"}{"log":"  File "/usr/bin/entry", line 264, in main
","stream":"stderr","time":"2023-12-07T00:12:39.855543868Z"}{"log":"    raise exception
","stream":"stderr","time":"2023-12-07T00:12:39.855549668Z"}{"log":"  File "/usr/bin/entry", line 245, in main
","stream":"stderr","time":"2023-12-07T00:12:39.855554968Z"}{"log":"    if tenant.is_provisioned():
","stream":"stderr","time":"2023-12-07T00:12:39.855560368Z"}{"log":"  File "/usr/bin/entry", line 82, in is_provisioned
","stream":"stderr","time":"2023-12-07T00:12:39.855565568Z"}{"log":"    raise RuntimeError("provisioning is in an unexpected state: error")
","stream":"stderr","time":"2023-12-07T00:12:39.855570768Z"}{"log":"RuntimeError: provisioning is in an unexpected state: error
","stream":"stderr","time":"2023-12-07T00:12:39.855576568Z"}
    ------------------------------
    Mark Moorcroft
    AMA Inc.
    Santa Clara CA
    ------------------------------

Categories