How to avoid and work around illegal control characters in Jama

By Carmen posted 08-02-2017 04:41


Due to a bug affecting Jama versions before 8.0—SOS-DEF-773—it was possible to add unicode characters through imports or copying-and-pasting content from other systems/tools.

These characters are now automatically stripped during Jama imports and saving items, though previously added characters may still be present in any instance. They won’t be causing any errors since Jama is able to handle them. However, issues may arise when placing data into XML, since XML parsing libraries can't handle these characters. Exports or integrations are examples of features that might be affected by this.

All the control characters can be searched using an ASCII table such as this one. Note that not all control characters may cause problems: some ASCII codes such as Horizontal Tab, Line Feed, and Carriage Return characters are normal and valid XML. Here's a list of what characters are valid or not.

Customers that haven't upgraded to Jama 8.x yet, should ensure that any data planned to be imported, or copied-and-pasted into Jama is free of these characters. 

Adding content as plain text will avoid bringing these characters into Jama. These unicode control characters are designed to be non-visible, so they won't render in most programs. Sublime Text (and possibly other text editors) will typically show these characters and will allow for users to search for them using a regex pattern, which may be helpful for finding and removing these characters before attempting to paste text into Jama.

Customers that have upgraded to Jama 8.x may have added illegal control characters into the system before this bug was fixed and may still come across these issues. If that happens, here are the options to remove the characters:

  • Manually delete the characters from the UI. You will need an SQL query to look for the illegal characters. The affected items can be identified by documentKey and Id. 

  • Strip these characters directly in the database using a query to programmatically cleanse the data. If you choose this option, you should test the query in a test instance and perform a full backup of your data prior to running it in Production.

Here is an example of a JIH sync error where illegal characters play a role.