Skip to content

Deep linguistic analysis overview

Deep linguistic analysis is a type of document analysis that combines the following interdependent processes:

The analysis is "deep" because:

  • It performs the common linguistic analysis.
  • It disambiguates the terms of the text, i.e. it determines the exact meaning of the text after considering all the other possibilities.

Deep linguistic analysis also performs knowledge linking: Knowledge Graph information and open data—Wikidata, DBpedia and GeoNames references—are returned for text tokens corresponding to syncons of the expert.ai Knowledge Graph. In the case of actual places, geographic coordinates are also provided.

The API resource carrying out deep linguistic analysis has the following endpoint:

/api/analyze

In the manual's reference section you will find all the information required to perform deep linguistic analysis, specifically:

Here is an example of performing deep linguistic analysis on a short English text:

This example is based on the Python client you can find on GitHub.

The client gets user credentials from two environment variables:

EAI_USERNAME
EAI_PASSWORD

Set those variables with your account credentials before running the sample program below.

The program prints a JSON representation of the results and the list of tokens' lemmas with their part-of-speech.

from expertai.nlapi.edge.client import ExpertAiClient
client = ExpertAiClient()

text = "Michael Jordan was one of the best basketball players of all time. Scoring was Jordan's stand-out skill, but he still holds a defensive NBA record, with eight steals in a half." 

output = client.deep_linguistic_analysis(text)

# Output tokens' data

print("Output tokens' data:");

print (f'{"TEXT":{20}} {"LEMMA":{40}} {"POS":{6}}')
print (f'{"----":{20}} {"-----":{40}} {"---":{6}}')

for token in output.tokens:
    print (f'{text[token.start:token.end]:{20}} {token.lemma:{40}} {token.pos:{6}}')

This example is based on the Java client you can find on GitHub.

The client gets user credentials from two environment variables:

EAI_USERNAME
EAI_PASSWORD

Set those variables with you account credentials before running the sample program below.

The program prints a JSON representation of the results and the list of tokens' lemmas with their part-of-speech.

import ai.expert.nlapi.security.Authentication;
import ai.expert.nlapi.security.Authenticator;
import ai.expert.nlapi.security.BasicAuthenticator;
import ai.expert.nlapi.security.DefaultCredentialsProvider;
import ai.expert.nlapi.v2.API;
import ai.expert.nlapi.v2.edge.Analyzer;
import ai.expert.nlapi.v2.edge.AnalyzerConfig;
import ai.expert.nlapi.v2.message.AnalyzeResponse;
import ai.expert.nlapi.v2.model.AnalyzeDocument;

public class Main {

    public static Authentication createAuthentication() throws Exception {
        DefaultCredentialsProvider credentialsProvider = new DefaultCredentialsProvider();
        Authenticator authenticator = new BasicAuthenticator(credentialsProvider);
        return new Authentication(authenticator);
    }

    public static Analyzer createAnalyzer() throws Exception {
        return new Analyzer(AnalyzerConfig.builder()
                                          .withVersion(API.Versions.V2)
                                          .withHost(API.DEFAULT_EDGE_HOST)
                                          .withAuthentication(createAuthentication())
                                          .build());
    }

    public static void main(String[] args) {
        try {
            String text = "Michael Jordan was one of the best basketball players of all time. Scoring was Jordan's stand-out skill, but he still holds a defensive NBA record, with eight steals in a half.";

            Analyzer analyzer = createAnalyzer();

            AnalyzeResponse disambiguation = analyzer.disambiguation(text);


            // Output JSON representation

            System.out.println("JSON representation:");
            disambiguation.prettyPrint();


            // Tokens' lemma and part-of-speech

            System.out.println("Tab separated list of tokens' lemma and part-of-speech:");
            AnalyzeDocument data = disambiguation.getData();
            data.getTokens().stream().forEach(c -> System.out.println(c.getLemma() + "\t" + c.getPos()));
        }
        catch(Exception ex) {
            ex.printStackTrace();
        }
    }
}