Skip to main content

Reveal Review Publication

User Annotation

In addition to entity search and extraction, you may engage in user annotation by highlighting relevant words and phrases in document text prior to building a model. Also, once an entity model is built and run you can improve a model by engaging in further user annotation. (See below Section C: Building, Running and Exploring a Custom Entity Model.) By highlighting additional words or phrases you can broaden the model. By coding already highlighted text where appropriate as “False Positives” you can focus the model and enhance accuracy. 

To manage the entity highlighting:

  • First enter the TRAINING page by choosing an item from the Snippet view on the right.

  • You may also begin to code a COSMIC model by selecting TRAINING from the Flyout Menu. That will open the COSMIC Queue Selector:

    603e73318a848.png
    • Under Focus Model, select the COSMIC Group created in COSMIC Mission Control via the Applied AI entry in the Flyout Menu.

    • Under Document List, select one from among Current Results, several Model-specific queues, or Saved searches.

    • Click Apply to begin training using the selected COSMIC Group - in this case, Relevant.

  • Make sure to expose the Thread Information Panel in the middle of the Training page by clicking the left pointing arrow as shown:

    603e7333846fe.png

Note that the entity-based highlighting can result from any of the following:

  • Pattern searches by RegEx.

  • Term Searches.

  • User Annotation (executed in document review).

  • Entity Model extrapolation from any of the above.

In the Thread Information Panel, select + Entities to see detailed entity highlighting information organized by entity type.

Note

The highlighting is color-coded by entity type.

603e7335c82a1.png

By choosing Legend...

603e73375b9da.png

...you see a list of icons that appear next to highlighted terms.

603e73390be15.png

These icons direct you as to the origin of the highlighting that appears in the text. For instance, all the words here highlighted and indicated by the star have been chosen by the entity models:

603e733aaab9c.png

Multiple colors on a single word or phrase identify it is a member of more than one entity type:

603e733c7830a.png

By clicking the gear icon, you see the following items:

603e733e26466.png

Configure entities - you can view a list of all the entity types to reorder or to hide.

603e7340026c9.png

You may choose either or both Auto-save and Auto-resolve:

603e7341c16e6.png

Both Auto-save and Auto-resolve are toggle switches – you enable by selecting, which causes a checkmark to appear. 

Auto-save: When enabled, annotations will be automatically saved when navigating away from the document. See Section 5.B.1. Adding User Annotations below for use in workflow. 

Auto-resolve: When enabled, annotations will combine with an existing entity if an exact match is found, If no exact match is found, a new entity will be created. See Section 5.B.1. Adding User Annotations below for use in workflow. 

You may use the colored icon next to each entity type name to reveal or hide the associated entity highlighting in the text.

603e73432cbca.png

Note

A colored line appears underneath the entity type which gives an idea of the number of mentions.  The total number of entities appears in parentheses.  By choosing the plus sign...

603e73444f228.png

...you can open the list of entities to view the number of mentions for each entity.

603e734588c3a.png
603e7346e4767.png

You may click an entity to navigate to it in the document.

603e73489a695.png

Note

If the entity was discovered prior to Version 2.21, the click navigation functionality is not available.

603e734a2f688.png

If you choose the ellipsis to the right of an entity type, you see the following options:

603e734bd3571.png
  • Show only [entity type]: This displays only the annotations of this entity type.

  • Rename [entity type]: This allows the renaming of this entity type.

  • Customize color: This allows you to change the color of the highlighting.

Highlighting settings persist from session to session per storybook per user.

1. Adding User Annotations

As mentioned above, you may engage in user annotation by highlighting relevant words and phrases in document text prior to building a model. Also, once an entity model is built and run (see below Section C: Building, Running and Exploring a Custom Entity Model) you can improve a model by engaging in further user annotation.

For example, as part of building a model for the entity type “Crimes” you might highlight “battery”:

603e734d783bf.png

By double-clicking the word, if you have not enabled Auto-resolve a window appears that allows you to create a new entity or choose an existing one. (If you have enabled Auto-resolve then this window does not appear; instead, annotations will combine with an existing entity if an exact match is found. If no exact match is found, a new entity will be created.)

603e734f400ab.png

You choose “Crimes” and the word “battery” is highlighted with the color reserved for “Crimes”, marked on the lower right corner with an icon meaning “user annotation”. This begins a list of annotations in the Entities window to be Saved before moving to the next thread.

603e7350ef995.png

If you have not enabled Auto-save then you should choose Save before you navigate away from the document.

603e73529f7d9.png

If you attempt to navigate away from the document without choosing Save under your list of changes then this message will appear:

603e735446dc6.png

If you have enabled Auto-save, then your annotations will be automatically saved when you navigate away from the document.

When performing a series of annotations for one particular entity type, you can facilitate the workflow by activating the marker icon so that you will automatically choose this particular entity type when annotating.

603e7355d7bdb.png

Now when you choose a word or phrase it will highlight for the designated entity type.

603e7357924c7.png
2. Candidate Entities & Validating/Invalidating Examples

For custom entities only (see C. Building, Running and Exploring a Custom Entity Model below), you may tell the system if an entity is valid or invalid. By default, they are not assigned as “valid (+)” or “invalid (-)”. These entities are in a “candidate” state.

When an entity has been annotated incorrectly by a custom entity model, or picked up by an entity search and extract query and is NOT a specific entity, you may annotate to that effect. By correctly annotating such false positives you may rapidly improve model accuracy.

For example, in this instance shown below “arm” is misidentified by an entity search & extract report as a “Body Part”:

603e735948adf.png

By default, since it was detected by an entity search & extract report, it is not assigned valid (+) or invalid (-).

You could simply remove the annotation by choosing the trash can icon on the right to delete. However, to help refine the model you may instead click the negative sign:

603e735addf96.png

This tells the system that this particular use of the term is not relevant. The highlighted word now appears with a strikethrough.

Alternatively, you may decide that “arm” is a VALID example, and you want the system to learn that it is a valid example. In this case, you would press the “+”:

603e735c7924a.png

Note

The options to delete or change the sign of an annotation are unavailable for entities that were found prior to Version 2.21.