Support

Expand all | Collapse all

REST Authentication

  • 1.  REST Authentication

    Posted 03-29-2017 12:59
    I've been trying to stress test the REST API to provoke 429 errors as I want to test some integrations in adverse server load to make sure they manage the queue and throttling nicely with Jama.  To do this, I have the giant hammer of a script below that makes thousands of concurrent REST requests.  Just replace MYSERVERHERE with your Jama server to run the Python (and use a linux system)

    I've discovered some strange responses.  I'm yet to get a 429 error though but I do get lots of 401s and a few 599s.

    We run on-premises with LDAP as authentication.  I suspect that as every request has a basic authentication, that each request is being authenticated against our corporate LDAP server.  This server then really cant keep up with the load

    Could someone confirm that Jama does not cache LDAP and authenticates every API request please?  

    Looking at the logs, I see we are getting LDAP errors so this looks likely

    Thanks

    Tim


    2017-03-29 19:46:04,132 ERROR http-bio-8080-exec-1144 jama [d04dd9] [com.jamasoftware.contour.dao.LdapUserDao] - org.springframework.ldap.CommunicationException: LDAPSERVER:389; nested exception is javax.naming.CommunicationException: LDAPSERVER:389 [Root exception is java.net.ConnectException: Connection refused (Connection refused)]
    2017-03-29 19:46:04,134 INFO http-bio-8080-exec-1444 jama [0f4f13] [com.jamasoftware.contour.security.UserAuthenticator] - [LDAP] - The LDAP user credentials you entered were not valid, please try again.
    2017-03-29 19:46:04,133 DEBUG http-bio-8080-exec-1220 jama [3a48d9] [com.jamasoftware.contour.rest.filters.JamaThrottleInterceptor] - Returning from throttle at concurrency count 173
    2017-03-29 19:46:04,112 WARN http-bio-8080-exec-1427 jama [e04b65] [com.jamasoftware.contour.util.auth.LdapProviderManagerImpl] - Ldap Provider Error.
    org.springframework.ldap.CommunicationException: LDAPSERVER:389; nested exception is javax.naming.CommunicationException: dcnwd.web.analog.com:389 [Root exception is java.net.ConnectException: Connection refused (Connection refused)]



    import collections
    import tornado.httpclient

    class BacklogClient(object):
    MAX_CONCURRENT_REQUESTS = 500

    def __init__(self, ioloop):
    self.ioloop = ioloop
    self.client = tornado.httpclient.AsyncHTTPClient(max_clients=self.MAX_CONCURRENT_REQUESTS)
    self.client.configure(None, defaults=dict(connect_timeout=150, request_timeout=300, validate_cert=False))
    self.backlog = collections.deque()
    self.concurrent_requests = 0
    self.callback = {}

    def __get_callback(self, function):
    def wrapped(*args, **kwargs):
    self.concurrent_requests -= 1
    return function(*args, **kwargs)

    return wrapped

    def flush(self):
    while self.backlog and self.concurrent_requests < self.MAX_CONCURRENT_REQUESTS:
    request, callback = self.backlog.popleft()
    self.client.fetch(request, callback=callback)
    self.concurrent_requests += 1

    def fetch(self, request, callback=None):
    wrapped = self.__get_callback(callback)
    self.backlog.append((request, wrapped))
    self.flush()



    import time
    import getpass
    from tornado import ioloop, httpclient


    class TornadoBacklog:

    def __init__(self):

    self.queue = 0
    self.debug = 1


    def handle_request(self, response):

    print(str(response.code) + ", " + response.reason)

    if self.backlog.backlog and self.backlog.concurrent_requests > 0:
    self.backlog.flush()
    else:
    ioloop.IOLoop.instance().stop()


    def launch(self, user, password):

    self.ioloop = ioloop.IOLoop.current()
    self.backlog = BacklogClient(self.ioloop)

    for y in range(0,20000):
    self.backlog.fetch(
    httpclient.HTTPRequest(
    "https://MYSERVERHERE/rest/latest/items?project=230",
    auth_username=user, auth_password=password, auth_mode="basic",
    ),
    self.handle_request
    )

    self.ioloop.start()


    def main():
    user = getpass.getuser()
    password=getpass.getpass("Please enter password for user " + user + ": ")
    start_time = time.time()

    scraper = TornadoBacklog()
    scraper.launch(user, password)

    elapsed_time = time.time() - start_time
    print('Process took %f seconds processed %d items.' % (elapsed_time, len(scraper.toProcess)))


    if __name__ == "__main__":
    main()


    ------------------------------
    Tim Kerby
    Principal Systems Engineer
    Analog Devices
    Edinburgh
    ------------------------------


  • 2.  RE: REST Authentication

    Posted 03-29-2017 13:12

    Based on the experiences I reported, I think the authenticate every time is  the behavior.  The LDAP server  problem was fixed and the 401 errors went away

     

    Sam

     






  • 3.  RE: REST Authentication

    Posted 03-29-2017 13:47
    Thanks Sam!

    It's looking that way. I doubt we will be able to do much on the LDAP side as it's managed by a different team elsewhere in the network and supports about 20k accounts.

    Might need some architectural improvements from Jama as at the very least this will significantly slow down API operations.


    ------------------------------
    Tim Kerby
    Principal Systems Engineer
    Analog Devices
    Edinburgh
    ------------------------------



  • 4.  RE: REST Authentication

    Posted 03-30-2017 11:16
    Hi Tim,

    You're correct that we authenticate each request individually for our REST API. For a variety of reasons, we don't want to create a persistent session for a single call, which unfortunately does mean some extra overhead for users with federated authentication. One way we've addressed this concern in our hosted environment is by providing an OAuth token service, which allows users to request a token tied to their user account and then use the token to authenticate locally until it's expired. I'm not sure where there are any plans to bring that functionality to on-premises users, but it might be something to look into if you are able to jump over to our hosted environment.

    Cheers,
    Alexander

    ------------------------------
    Alexander Kahn
    Software Engineer
    Jama Software
    Portland OR
    ------------------------------



  • 5.  RE: REST Authentication

    Posted 03-31-2017 08:54
    Hi Alexander

    Thanks for the confirmation. That's a big issue for us as our LDAP infrastructure cant handle that load and we have no option to move to hosted due to both security and functional safety requirements.

    Incidentally, it appears you do create a session for every single call.  You can see from the (concatenated) access log below that thousands of sessions are created when I run my REST tests and in fact, I do get back these session keys as cookies with the returned value.

    I'm yet to test if these sessions persist but that may be a way round the issues - to use basic authentication on the first call then authenticate with the cookie for following calls. I know that's undocumented but is there a reason why we shouldn't do that?

    2017-03-29 20:03:26,231 http-bio-8080-exec-1283 jama [434ef4] [INFO ] - [510 ms] PRBDIJN9 - - 208C240F785ACE19B3BDBFB4133FB9AA 286178 https://MYJAMASERVER/rest/latest/items
    2017-03-29 20:03:26,231 http-bio-8080-exec-1507 jama [9244ce] [INFO ] - [499 ms] PRBDIJN9 - - 05B4EC98ABE9D16666348F4A1F256469 286612 https://MYJAMASERVER/rest/latest/items
    2017-03-29 20:03:26,232 http-bio-8080-exec-1579 jama [404278] [INFO ] - [449 ms] PRBDIJN9 - - 349C217251DD5DEF41D7ACE41ECE3847 286790 https://MYJAMASERVER/rest/latest/items
    2017-03-29 20:03:26,232 http-bio-8080-exec-1455 jama [114b3e] [INFO ] - [470 ms] PRBDIJN9 - - B814D9CCFE6AF59484728E7F5E1D6FAE 286530 https://MYJAMASERVER/rest/latest/items
    2017-03-29 20:03:26,233 http-bio-8080-exec-1647 jama [344e73] [INFO ] - [356 ms] PRBDIJN9 - - 8C271D9127F6F646FA6631C05567D995 326078 https://MYJAMASERVER/rest/latest/items
    2017-03-29 20:03:26,234 http-bio-8080-exec-1493 jama [5c420b] [INFO ] - [477 ms] PRBDIJN9 - - 86452631401E027D05E885583C546BCA 286596 https://MYJAMASERVER/rest/latest/items


    ------------------------------
    Tim Kerby
    Principal Systems Engineer
    Analog Devices
    Edinburgh
    ------------------------------



  • 6.  RE: REST Authentication

    Posted 04-02-2017 21:03
    Edited by Dana Medhaug 06-30-2017 13:53
    Tim,

    The 429 response will only be returned when your team has opted to enable the in app throttling option. This is not something we have enabled by default as our on-premises customers have many different needs, hardware and may opt to throttle their calls with options outside of Jama Software.

    Steve

    Throttle Jama REST/SOAP API Requests - 8.0 and above - Jama Software Community
    Jamasoftware remove preview
    Throttle Jama REST/SOAP API Requests - 8.0 and above - Jama Software Community
    Overview Jama provides an option to help system admins protect their Jama servers. The system can stop aggressive integrations from overwhelming the
    View this on Jamasoftware >


    ------------------------------
    Steve Gotsch
    Product Manager
    Jama Software
    ------------------------------



  • 7.  RE: REST Authentication

    Posted 04-03-2017 02:39
    Thanks Steve- that's good to know.

    I think our problem at the moment lies mainly with the LDAP authentication though.  I'm wondering if throttling on the Jama server would help slow the number of requests to the LDAP server?  Is the request authenticated at the start or the end of the queue? 

    Thanks

    Tim

    ------------------------------
    Tim Kerby
    Principal Systems Engineer
    Analog Devices
    Edinburgh
    ------------------------------