Reveal AI Integration API
(NIA for short)
September 9, 2021
1 New MSA Instructions
Begin by editing the NIA.config file. It will be on the DB server at a location like D:/Reveal/NIA/NIA.config. You should see several sections like this:
[MSA - 000000]
User: <some username>
Password: <some password>
URL: https://trunk.revealdata.com/rest/api/
NIA needs a user and password configured to be able to log into Reveal, so copy/paste that config section and edit the MSA number, username, password, and Reveal API URL appropriately.
Generally the easiest thing to do here is to re-use the processing-user login we create for processing.
The MSA number should be the same number as set in the instance’s System Settings under the setting MSANumber.
You should not need to restart any of the NIA services for the updates to take effect. It re-reads the config before it uses the values.
On the Reveal side, the following system settings need to be edited to link into Reveal correctly:
NexLP API TenantName NexLP Tenant name for creating databases and interacting with Reveal AI.
e.g. Default
NexLP API Database Suffix Suffix to place after new database names created within Reveal AI. Mainly important for all-in-one database server setups where we do not want review’s database name to conflict with Reveal AI’s.
e.g. RevealAI
NexLP API Database Server Hostname for Reveal AI’s database. This must also match the name of a Reveal AI datasource in the Tenant you have set below.
e.g. REVEALAI-SANDBOX-1
NexLP API Login User name with which to log into Reveal AI’s API.
e.g. nexlp-admin@revealdata.com
NexLP API Password Password for the NexLP API user.
e.g. fooBarBangBiz
NexLP API URL URL to Reveal AI’s API.
e.g. https://nexlp-sandbox.revealdata.com/StoryEngineWebApi/api
NexLP Integration API URL URL to NIA’s API.
e.g. http://10.0.130.56:5566/nia
MSANumber MSA number for the instance as previously mentioned.
e.g. 000000
Reveal AI Version Reveal AI version number that is currently running.
e.g. 2.80.
2 Network Requirements
Ports with a non-standard designation (5566) can be altered in code as necessary.
Source | Dest | Port | Required |
|---|---|---|---|
NexLP Integration Worker | Reveal API NexLP API | 443 443 | X X |
Reveal Index Batch | NexLP Integration API NexLP API | 5566 443 | X X |
Reveal Ops Center | NexLP API | 443 | X |
Reveal Web | NexLP API | 443 | X |
NexLP Integration Worker | NexLP Database | 1433 | X |
NexLP Integration API | NexLP Database | 1433 | X |
NexLP Integration Worker | Email Server | 465 |
Reveal AI integration will send out email alerts for high-level unhandled errors with syncing. This can be disabled if desired.
3 Troubleshooting Help
The following is a breakdown of the statuses of a NIA job:
processing_job_status_to_desc = {
-1: ’Unknown’, 0: ’Created’,
’Submitted’,
’InProcess’,
’Complete’,
’Error’,
’Cancelled’,
’CancelPending’,
’Deleted’,
’Modified’,
’Processing Job Submitted’,
’Deletion Job Submitted’,
’Deletion Identify Job Submitted’,
’Deletion Job Started’ }
You can check the current status of a particular job by running the following on the nia-database database:
select *
from nia_job_queue
select *
from nia_job
order by 1 desc
The first query gets you anything sitting waiting in the queue. Realistically, this should almost always be empty. The second query gets you a historic record of jobs which have status in the column nia job status.
Generally, the NIA services should be on the Reveal AI Database server located under:
D:\Reveal\NIA
You should also find a config file in that folder that you can double-check settings on. In that config file, passwords must have the dollar sign and percentage sign escaped by putting the offending character in twice.
To double check that the NIA API is running, you can navigate to the following:
http://10.0.73.128:5566/nia/version
Replace the IP address with the IP of the database server, or just localhost if you’re on the server itself.
4 Network Communication Flow
Definitions:
NIA API Runs on the Reveal AI DB Server under the name NexLP Integration API service. Our homegrown API.
NIA API Worker Runs on the Reveal AI DB Server under the name NexLP Integration API Worker service. A worker separate from our homegrown API but developed by us.
NexLP API Runs on the Reveal AI Web Server. Written by Reveal AI.
Reveal API Runs on the Reveal Web Server. Written by Reveal.
Index Batch Service Runs on the Reveal Lightweight Services Server.
4.1 Text Syncing
Initiated by a user indexing documents or deleting documents from Reveal.
Index Batch Service sends a request to the NIA API via port 5566 to create a job. From this point forward, the Index Batch Service will continue to periodically poll the NIA API via port 5566 to get status on the job.
NIA API Worker watches the nia-database for new work. When it sees the new job, the NIA API Worker queries the Reveal API via port 443 to get text metadata for the job.
The NIA API Worker writes out the text it receives to a configured text storage location. At the same time it builds a load table on the Reveal AI Database Server inside the Reveal AI Source Database associated with the job.
After the NIA API Worker finishes loading to the load table, the NIA API Worker queries the NexLP API via port 443 to create a processing job for the database.
The NIA API Worker will continue to poll the NexLP API via port 443 until it sees the processing job is done.
When the job is done, the NIA API Worker updates the status in the nia-database. This is eventually also returned to the Index Batch Service which finishes out text syncing workflow.
4.2 Score Syncing
Initiated periodically by the Index Batch Service.
The Index Batch Service queries the NIA API via port 5566 to get a list of the newest model update jobs from a given starting time.
The Index Batch Service checks these values against the relevant Reveal database to determine if it needs to get new scores from Reveal AI.
For every case for which it needs to get scores, the Index Batch Service queries the NIA API via port 5566 to get the newest scores for a model.
The NIA API queries the NexLP API via port 443 to get the model identifiers and the SQL database name for the Reveal AI Storybook.
For each model, the NIA API queries the Reveal AI Storybook Database directly via port 1433 to get current scores for the model.
The NIA API takes these scores and compares them to the cached scores in a configured location.
The NIA API returns the new scores to the Index Batch Service, and also caches the scores it just read to a configured location.
The Index Batch Service saves these scores into the Reveal database via port 1433.
The Index Batch Service commits the changes to the NIA API by running a query on port 5566.
5 New Instance Instructions/On-Prem Installation
5.1 Storage Requirements
Reveal AI integration needs some sort of ephemeral network storage. When a new document sync job is submitted, the Reveal AI integration worker will start pulling text for a case locally to be loaded to Reveal AI. This means that the Reveal AI integration worker will need storage with at least as much space as the text size of your largest case.
The Reveal AI processing servers will ingest this text during processing. This means that the datasource must be visible by both NIA and Reveal AI’s processing services.
5.2 Setup Instructions
Download the packaged services from here:
s3://revealdata-software-private-build-repository/RevealIntegrationAPI/ NexLPIntegrationAPI.zip
Unzip that and it should look like the following:
Installation Logs
NIA.config
NiaAPIService
NiaWorkerService
Score Syncing
Text Syncing
Place the unzipped contents wherever you want the install to be. In this document we’re going to assume everything is located here:
D:\NexLP\Reveal\NexLP Integration API\
This root directory is important to note because it is used in further installation commands. We will call out when this is important.
The Installation directory has two commands to use for installation and a C++ install redistributable:
Install Commands.txt
VC_redist.x64.exe
This redistributable installer may not be necessary to install depending on what is already installed on the server. Safer to install though as it won’t hurt anything to have it.
The installation instructions should look something like this:
cat Installation/Install\ Commands.txt
New-Service -Name "Reveal-NexLP Integration" -DisplayName "Reveal-NexLP Integration" -Description "Reveal-NexLP Integration" - BinaryPathName ’D:\NexLP\Reveal\NexLP Integration API\ NiaWorkerService\NiaWorkerService.exe -c "D:\NexLP\Reveal\NexLP Integration API\NIA.config"’
New-Service -Name "Reveal-NexLP Integration API" -DisplayName " Reveal-NexLP Integration API" -Description "Reveal-NexLP Integration API" -BinaryPathName ’D:\NexLP\Reveal\NexLP Integration API\NiaAPIService\NiaAPIService.exe -c "D:\NexLP\ Reveal\NexLP Integration API\NIA.config"’
Note
Note the double-quotes around the configuration file path. This is necessary so be careful if you’re dealing with filepaths that have spaces in them.
Note
Note the four locations in the above where the root path appears.
D:\NexLP\Reveal\NexLP Integration API\
Make sure to change this root path if you have the files downloaded somewhere else.
Run the above install commands from an admin powershell to install the services. If you find you didn’t set the paths correctly you can run the following to remove the services for re-installing:
Remove-Service -Name "Reveal-NexLP Integration"
Remove-Service -Name "Reveal-NexLP Integration API"
Edit the config file. In our example this would be at:
D:\NexLP\Reveal\NexLP Integration API\NIA.config
It should look like this:
# Dollars signs need to be doubled to escape.
[MSA - 999999]
User: username
Password: password
URL: https://foo.hostname.com/rest/api/
[NIA]
Temporary Text Output: D:\NexLP\Reveal\NexLP Integration API\Text
Syncing
Worker Load Threads: 2
Worker Batch Size: 2000
API Log Location: D:\NexLP\Reveal\NexLP Integration API\Logs\nia_api.log
Worker Log: D:\NexLP\Reveal\NexLP Integration API\Logs\nia_worker.log
Verify SSL: False
mass edit batch size: 500
[NIA Score]
Score Storage Path: D:\NexLP\Reveal\NexLP Integration API\Score Syncing
[Database]
Type: MSSQL
[Review SSO]
User: foo
Pass: bar
[MSSQL]
NIA Database Name: nia-database
Host: nexlp2
Driver: SQL Server Native Client 11.0
[NexLP]
User: nexlp_username
Pass: nexlp_password
URL: https://nexlp.revealdata.com/StoryEngineWebApi/api/
[email]
enabled: False
emailFrom:
no-reply@revealdata.comemailTo: to
emailHost: host
emailPort: 465
emailPassword: Password
emailUser: User
From here the following sections need to be filled with the correct linking information:
MSA - 999999 The 999999 should be replaced with whatever the MSANumber system setting is set to in review. Make sure this setting is set in Review. The User, Password, and URL settings should be altered to point to whatever site you’re linking into.
NIA Generally these can be left as is as long as you edit the root paths. Again in this example the root path is D:\NexLP\Reveal\NexLP Integration API\. One thing to note is that wherever you point the Temporary Text Output setting needs to have enough disk space to sync down text for at least your largest dataset to sync.
NIA Score Edit the score syncing root path.
Database Nothing to edit here.
MSSQL The Host value needs to match whatever the Database server is named as within Reveal AI’s datasources. If you need to double-check, Reveal AI stores this information in the [nexlp].[DatabaseServer] table in the Name column.
For the Driver value, as long as you’re installing this on the Reveal AI database server you can likely leave this as it. However, to double-check you can confirm the configured value is in the output of the following powershell command:
Get-ODBCDriver
That will print out the ODBC drivers installed on the server for use. Windows should, however, come pre-packaged with the configured value.
NexLP As with the MSA section, this just needs to be re-configured with the appropriate information required to connect to Reveal AI’s API.
email Edit settings here if desired. They are all self-evident. Email can be turned off by leaving the enabled setting at False.
On the Reveal AI database server you’re running this on, create an empty database named nia-database. The API will automatically create the necessary tables in this database on startup.
Start the API service followed by the Integration service. Do not run the Integration service if the API fails to start.
6 Build Instructions
If we need to re-build, put code on disk on a Windows machine somewhere. (e.g. D:/NIA/). Open up a powershell window. Run:
$env:PYTHONPATH = ’D:\NIA’
That will set your pythonpath. Then do:
cd D:\NIA\NIA\Service
pyinstaller.exe .\NiaWorkerService.py --clean --noconfirm
pyinstaller.exe .\NiaAPIService.py --clean –noconfirm
That should build the services into the dist directory. If it’s not already there, put a NIA.config file at the top of the dist directory.
7 Notes
If you get errors when trying to run the source because of some error like:
ImportError: cannot import name 'cached_property' from 'werkzeug' (C:\Program Files\Python37\lib\site-packages\werkzeug\__init__.py)
Run this:
pip install werkzeug==0.16.1