Skip to main content

Reveal Review Publication

5. Custom Entity Types

Story Engine allows you to create custom entity types keying off your data and adapted to your workflow. You build these entity types through AI-driven entity models informed by your data and your annotations. To feed the model-building you can annotate the data in several ways. You might identify exemplary text components through pattern searches (by way of regular expressions). Or you might use lists of keywords or wildcards, or engage in user annotation of the document text.

Once this is done, your admin can let the model building go to work using your examples as a guide. You can refine the resulting entity model by further annotation, identifying additional relevant content or, alternatively, by identifying false positives.

Your System Administrator can store the models used to generate these new entity types in a library for use in future projects. Through a succession of projects in which new data and new work product are applied, your bespoke entity models will embody accumulated expertise. With these tools you can make quick work of new case assessment.

A. Entity Search and Extract

Choose the Exploring Tab, then the Term reports Icon and then Create Report...

603e732590151.png

and then choose Entity search and extract.

603e73278f97c.png

You will see the first of two pages for your Entity search and extract report:

603e73293a1df.png

Enter the following:

  • Name: Choose a name for the report.

  • Entity Type: Choose an existing entity type or choose the plus sign to create a new one.

  • Keywords and patterns (Type or paste in keywords, wildcards, or regular expressions. One term per line.)

    • A keyword is purely alphanumeric such as abc456”.

    • A single wildcard word is alphanumeric but also includes one or more asterisks such as “def*123*”.

    • A regular expression is anything else “(ab%d*f). (See Appendix F: Examples of Useful Regular Expressions.)

  • Notes: Add any notes you want.

Choose Continue.

The Term report continues as shown below. Here you can review each keyword, wildcard or pattern and their term type and entity type and edit or delete before running. You then choose Create or Create and Run Report.

Note

Once you have run the report you can no longer delete an individual term or pattern of the entity type.

603e732ad94da.png

Once you have created the entity search and extraction you click on its name in the Term Reports list...

603e732c8a9ad.png

And then review the status of each keyword, wildcard or pattern:

603e732e2d847.png

The following options and information are provided:

  • + Add Term: Choose this to add additional lines to your search

  • Run Full Report: This runs a report on all search terms.

The report columns:

  • Keyword or pattern: The separate keywords, wildcards and patterns that comprise the search.

  • Term type: Regex, wildcard or keyword.

  • Entity type: The chosen entity type.

  • Status: Successful or Never Run.

  • Last run time: The last time this term or pattern was run.

  • Actions: You may delete an individual term or pattern, but only if it has never been run.

To review results,

  • under the Exploring Tab choose Entities and then

  • choose the relevant custom entity type.

  • Open the Detection list and

  • choose Found by Term Report,

  • hit Apply.

  • Note the number and composition of hits.

    603e732f7fc9c.png
B. User Annotation

In addition to entity search and extraction, you may engage in user annotation by highlighting relevant words and phrases in document text prior to building a model. Also, once an entity model is built and run you can improve a model by engaging in further user annotation. (See below Section C: Building, Running and Exploring a Custom Entity Model.) By highlighting additional words or phrases you can broaden the model. By coding already highlighted text where appropriate as “False Positives” you can focus the model and enhance accuracy.

To manage the entity highlighting:

  • First enter the TRAINING page by choosing an item from the Snippet view on the right.

  • Make sure to expose the Thread Information Panel in the middle of the Training page by clicking the left pointing arrow as shown:

    60424d57db3cf.png

Note that the entity-based highlighting can result from any of the following:

  • Pattern searches by RegEx.

  • Term Searches.

  • User Annotation (executed in document review).

  • Entity Model extrapolation from any of the above.

In the Thread Information Panel, select + Entities to see detailed entity highlighting information organized by entity type.

Note

The highlighting is color-coded by entity type.

60424d59e32ea.png

By choosing Legend...

603e73375b9da.png

You see a list of icons that appear next to highlighted terms.

603e73390be15.png

These icons direct you as to the origin of the highlighting that appears in the text. For instance, all the words here highlighted and indicated by the star have been chosen by the entity models:

603e733aaab9c.png

Multiple colors on a single word or phrase identify it is a member of more than one entity type:

603e733c7830a.png

By clicking the gear icon, you see the following items:

603e733e26466.png

Configure entities - you can view a list of all the entity types to reorder or to hide.

603e7340026c9.png

You may choose either or both Auto-save and Auto-resolve:

603e7341c16e6.png

Both Auto-save and Auto-resolve are toggle switches – you enable by selecting, which causes a checkmark to appear.

Auto-save: When enabled, annotations will be automatically saved when navigating away from the document. See Section 5.B.i. Adding User Annotations below for use in workflow.

Auto-resolve: When enabled, annotations will combine with an existing entity if an exact match is found, If no exact match is found, a new entity will be created. See Section 5.B.i. Adding User Annotations below for use in workflow.

You may use the colored icon next to each entity type name to reveal or hide the associated entity highlighting in the text.

603e73432cbca.png

NOTE that a colored line appears underneath the entity type which gives an idea of the number of mentions. The total number of entities appears in parentheses. By choosing the plus sign...

603e73444f228.png

...you can open the list of entities to view the number of mentions for each entity.

603e734588c3a.png
603e7346e4767.png

You may click an entity to navigate to it in the document.

603e73489a695.png

Note

If the entity was discovered prior to Version 2.21, the click navigation functionality is not available.

603e734a2f688.png

If you choose the ellipsis to the right of an entity type, you see the following options:

603e734bd3571.png
  • Show only [entity type]: This displays only the annotations of this entity type.

  • Rename [entity type]: This allows the renaming of this entity type.

  • Customize color: This allows you to change the color of the highlighting.

Highlighting settings persist from session to session per storybook per user.

  1. Adding User Annotations

As mentioned above, you may engage in user annotation by highlighting relevant words and phrases in document text prior to building a model. Also, once an entity model is built and run (see below Section C: Building, Running and Exploring a Custom Entity Model) you can improve a model by engaging in further user annotation.

For example, as part of building a model for the entity type “Crimes” you might highlight “battery”:

603e734d783bf.png

By double-clicking the word, if you have not enabled Auto-resolve a window appears that allows you to create a new entity or choose an existing one. (If you have enabled Auto-resolve then this window does not appear; instead, annotations will combine with an existing entity if an exact match is found. If no exact match is found, a new entity will be created.)

603e734f400ab.png

You choose “Crimes” and the word “battery” is highlighted with the color reserved for “Crimes”, marked on the lower right corner with an icon meaning “user annotation”. This begins a list of annotations in the Entities window to be Saved before moving to the next thread.

603e7350ef995.png

If you have not enabled Auto-save then you should choose Save before you navigate away from the document.

603e73529f7d9.png

If you attempt to navigate away from the document without choosing Save under your list of changes then this message will appear:

603e735446dc6.png

If you have enabled Auto-save, then your annotations will be automatically saved when you navigate away from the document.

When performing a series of annotations for one particular entity type, you can facilitate the workflow by activating the marker icon so that you will automatically choose this particular entity type when annotating.

603e7355d7bdb.png

Now when you choose a word or phrase it will highlight for the designated entity type.

603e7357924c7.png
  1. Candidate Entities & Validating/Invalidating Examples

For custom entities only (see Building Custom Entity Model below), you may tell the system if an entity is valid or invalid. By default, they are not assigned as “valid (+)” or “invalid (-)”. These entities are in a “candidate” state.

When an entity has been annotated incorrectly by a custom entity model, or picked up by an entity search and extract query and is NOT a specific entity, you may annotate to that effect. By correctly annotating such false positives you may rapidly improve model accuracy.

For example, in this instance shown below “arm” is misidentified by an entity search & extract report as a “Body Part”:

603e735948adf.png

By default, since it was detected by an entity search & extract report, it is not assigned valid (+) or invalid (-).

You could simply remove the annotation by choosing the trash can icon on the right to delete. However, to help refine the model you may instead click the negative sign:

603e735addf96.png

This tells the system that this particular use of the term is not relevant. The highlighted word now appears with a strikethrough.

Alternatively, you may decide that “arm” is a VALID example, and you want the system to learn that it is a valid example. In this case, you would press the “+”:

603e735c7924a.png

Note

The options to delete or change the sign of an annotation are unavailable for entities that were found prior to Version 2.21.

C. Building, Running and Exploring a Custom Entity Model

Once a sufficient number of annotations have been accomplished (this can often include several hundred user annotations), you may proceed to build a Custom Entity Model. The Entity Model is created in System Administration. See Admin Guide Section 1 > E. Entity Models.

After the admin has built and run the entity model in System Administration, you may go to the Exploring tab and review results for the associated custom entity type by the following:

  • Under the EXPLORING Tab choose Entities.

  • Choose the relevant custom entity type.

  • Open the Detection list.

  • Choose Found by Entity Model.

  • Hit Apply and note the number of hits.

These are the hits found by the AI Model built on entity examples.

603e735ee41df.png

You can review the highlighted entities in the documents found by the entity model. The model can be repeatedly refined through further user annotation and expanded by further entity search and extraction as described above in this section. The resulting improvements in the model information retrieval (Recall, Precision and F1) are available for review by the Admin. (See Admin Guide Section 1 > E. Entity Models.)