NeoVoc

From eneoli wikibase

ENEOLI WG1 Task 1.2

The workflow described below relates to ENEOLI WG1 Task 1.2, NeoVoc multilingual equivalents and NeoCorpus Zotero.

Tasks to be carried out in Zotero "NeoCorpus" collection and on this Wikibase
Who What Description
Collaborative task Add Neology articles to Zotero "NeoCorpus" collection Research and handbook articles in multiple languages are added to Zotero, see “Working with Zotero” document
Task lead Send validated article metadata to this Wikibase and process full text (convert to TEI-XML or TXT and lemmatize), and enrich article metadata with NeoVoc terms occurrence information (example) Employed tools: Zotwb, GROBID, SpaCy
Task lead Include new validated French (or English) terms coming from task 1.1 into NeoVoc concept entries The first set of Neology terms was defined in Task 1.1. based on the Lambert-Lucas corpus, manual validation and provision of short glosses in French by the ENEOLI panel of French experts. More terms will be added.
Collaborative task Provide or validate drafted multilingual equivalents (and short glosses) on this Wikibase See task description at NeoVoc page.
Task lead Create lexical entries (example) Create lexical entries for validated equivalents (terms in our working languages)
Task lead Find validated terms in lemmatized articles, and add the information about term occurrences to article metadata Done for a first set of articles, example. Employed tool: Flashtext

Other ENEOLI tasks will reuse and/or enrich the dataset put together on Zotero and on this Wikibase:

  • All other tasks may reuse the NeoCorpus article TXT (lemmatized or not), e.g. for feeding them to the Sketch Engine and extract more terms as candidates to be included in the NeoVoc glossary.
  • NeoVoc concept entries and lexical entries (we decided to have both, linked together on the sense level) may be enriched with more descriptions, e.g. domain labels, lexical and terminological relations, etymology, etc. This will be the data source for a lexical-conceptual resource targeting end users.

NeoVoc

SPARQL queries

All NeoVoc concept entries

#title: All NeoVoc concepts with labels in your browser's language (with English or French as fallback), and English and French descriptions.

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>

select distinct ?concept ?conceptLabel (iri(concat(str(wd:),?wd)) as ?wikidata) ?descript_en ?descript_fr
       
where { 
  ?concept endp:P82 enwb:Q12. # part of NeoVoc collection
  ?concept rdfs:label [].
  optional {?concept endp:P1 ?wd.}
  optional {?concept schema:description ?descript_en. filter(lang(?descript_en)="en")}
  optional {?concept schema:description ?descript_fr. filter(lang(?descript_fr)="fr")}
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en,fr". }
}

Try it!


All NeoVoc concept CANDIDATE entries

This query lists concepts that have come in from Task 1.1 and currently marked as “instance of NeoVoc candidate concept”, that is, concepts that have not yet been validated and declared “instance of NeoVoc Concept”. 

#title: All NeoVoc candidate concepts with labels in your browser's language (with English or French as fallback), and English and French descriptions.

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>

select ?concept ?conceptLabel (iri(concat(str(wd:),?wd)) as ?wikidata) ?descript_en ?descript_fr
       
where { 
  ?concept endp:P5 enwb:Q14. # instances of "NeoVoc candidate Concept"
  optional {?concept endp:P1 ?wd.}
  optional {?concept schema:description ?descript_en. filter(lang(?descript_en)="en")}
  optional {?concept schema:description ?descript_fr. filter(lang(?descript_fr)="fr")}
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en,fr". }
}

Try it!


All NeoVoc lexical entries

#title: All NeoVoc lexical entries with linked concept nodes

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>

select ?lexical_entry (lang(?lemma) as ?lang) ?lemma ?sense ?concept ?conceptLabel (iri(concat(str(wd:),?wd)) as ?wikidata) ?descript_en ?descript_fr
       
where { 
  ?lexical_entry a ontolex:LexicalEntry; endp:P82 enwb:Q12; wikibase:lemma ?lemma; ontolex:sense ?sense.
  optional {?sense endp:P12 ?concept.
  optional {?concept endp:P1 ?wd.}
  optional {?concept schema:description ?descript_en. filter(lang(?descript_en)="en")}
  optional {?concept schema:description ?descript_fr. filter(lang(?descript_fr)="fr")}
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en,fr". }}
}

Try it!

NeoVoc lexical entries: occurrences in NeoCorpus articles

#title: NeoVoc lexical entry lemmata, and in how many articles they occur

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>

select ?lexical_entry (lang(?lemma) as ?lang) ?lemma (count(?bib_item) as ?in_how_many_articles)
       
where { 
  ?lexical_entry a ontolex:LexicalEntry; endp:P82 enwb:Q12; wikibase:lemma ?lemma.
  optional {?bib_item endp:P65 ?lexical_entry.}
} group by ?lexical_entry ?lang ?lemma order by desc(?in_how_many_articles)

Try it!


Authors and the NeoVoc terms they use

#title: Authors, and the terms that occur in their texts

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>

select ?author ?authorname (group_concat(distinct str(?lemma);SEPARATOR=", ") as ?terms) # (count(?bib_item) as ?in_how_many_articles) (lang(?lemma) as ?lang)
       
where { 
  ?lexical_entry wikibase:lemma ?lemma.
  ?bib_item endp:P40 ?author; 
            endp:P65 ?lexical_entry.
  ?author rdfs:label ?authorname. filter(lang(?authorname)="en")
} group by ?author ?authorname ?terms  order by ?authorname

Try it!

Authors, and in how many of their articles NeoVoc terms appear

#title: Authors, and in how many of their articles NeoVoc terms appear

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>
PREFIX enp: <https://eneoli.wikibase.cloud/prop/>
PREFIX enps: <https://eneoli.wikibase.cloud/prop/statement/>
PREFIX enpq: <https://eneoli.wikibase.cloud/prop/qualifier/>
PREFIX enpr: <https://eneoli.wikibase.cloud/prop/reference/>
PREFIX enno: <https://eneoli.wikibase.cloud/prop/novalue/>


select ?author ?lastname ?authorLabel (group_concat(distinct ?lang;SEPARATOR=", ") as ?language) (group_concat(distinct concat(str(?lemma)," (",str(?count),")");SEPARATOR=", ") as ?terms) # (count(?bib_item) as ?in_how_many_articles) (lang(?lemma) as ?lang)
       
where { 
  ?author endp:P52 ?lastname.
  
 { SELECT ?author ?lang ?lemma (count(?bib_item) as ?count)
                    where {
    ?bib_item endp:P40 ?author; endp:P7 [rdfs:label ?lang]. filter (lang(?lang)="en")
    ?bib_item endp:P65 [wikibase:lemma ?lemma].
                          } group by ?author ?lang ?lemma ?count order by ?count desc(lcase(str(?lemma)))
         
 } SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en,de,fr,pt". }
} group by ?author ?lastname ?authorLabel ?langauge ?terms  order by ?lastname

Try it!


All articles that contain terms denoting one concept

#title: All articles that contain terms denoting one concept, here: Q1112, blending'

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>

select ?concept ?conceptLabel ?article ?articleLabel ?article_languageLabel
       
where { bind(enwb:Q1112 as ?concept) # change "Q1112" for another concept
  ?article endp:P5 enwb:Q2; endp:P7 ?article_language; endp:P65 [ontolex:sense [endp:P12 ?concept]]. 
  
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}

Try it!

All concepts that appear in articles together with one certain concept

#title: All concepts that appear in articles together with one certain concept, here: Q1112, blending'

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>

select ?conceptLabel ?other_concept ?other_conceptLabel (count(?article) as ?how_many_articles)
       
where { bind(enwb:Q1112 as ?concept) # change "Q1112" for another concept
  ?article endp:P5 enwb:Q2; endp:P7 ?article_language; endp:P65 [ontolex:sense [endp:P12 ?concept]]. 
  ?article endp:P65 [ontolex:sense [endp:P12 ?other_concept]] filter(?concept != ?other_concept)
  
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en,fr". }
} group by ?conceptLabel ?other_concept ?other_conceptLabel ?how_many_articles order by desc(?how_many_articles)

Try it!

Timeline of articles where a concept appears

#title: All articles that contain terms denoting one concept, here: Q1120, 'derivation'
#defaultView:Timeline
PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>

select ?article ?articleLabel (group_concat(str(?author);SEPARATOR=" | ") as ?authors) ?date ?article_languageLabel
       
where { bind(enwb:Q1120 as ?concept) # change "Q1120" for another concept
  ?article endp:P5 enwb:Q2; endp:P7 ?article_language; endp:P30 ?date; endp:P65 [ontolex:sense [endp:P12 ?concept]]. 
  ?article endp:P40 [rdfs:label ?author]. filter(lang(?author)="en")
  
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
} group by ?article ?articleLabel ?authors ?date ?article_languageLabel

Try it!

Coverage of NeoVoc concepts with multilingual equivalents

All existing equivalents

First, we define multilingual equivalents for the concepts; we do this in the concept entry. Lexical entries will be created after that. This query shows the progress in covering NeoVoc with multilingual equivalents. 

#title:Number of equivalents present in NeoVoc concept entries (including unvalidated equivalents)
#defaultView:BarChart

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>
PREFIX enp: <https://eneoli.wikibase.cloud/prop/>
PREFIX enps: <https://eneoli.wikibase.cloud/prop/statement/>
PREFIX enpq: <https://eneoli.wikibase.cloud/prop/qualifier/>

select ?language (count(?concept) as ?equivalents) 

where {
  ?concept endp:P82 enwb:Q12. # part of NeoVoc collection
  ?concept endp:P57 ?equiv.
  bind (lang(?equiv) as ?language)
} group by ?language ?equivalents order by desc(?equivalents)

Try it!

Validated equivalents

The following query shows the coverage of concept entries with equivalents in different languages, but counting only those that have been validated (i.e., where a "validated by" qualifier has been attached to the equivalent). 

#title:Number of equivalents present in NeoVoc concept entries (validated equivalents only)
#defaultView:BarChart

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>
PREFIX enp: <https://eneoli.wikibase.cloud/prop/>
PREFIX enps: <https://eneoli.wikibase.cloud/prop/statement/>
PREFIX enpq: <https://eneoli.wikibase.cloud/prop/qualifier/>

select ?language (count(?equiv_st) as ?equivalents) 

where {
  ?concept endp:P82 enwb:Q12. # part of NeoVoc collection
  ?concept enp:P57 ?equiv_st. ?equiv_st enps:P57 ?equiv. ?equiv_st enpq:P64 [rdfs:label ?validator]. filter(lang(?validator)="en")
  bind (lang(?equiv) as ?language)
} group by ?language ?equivalents order by desc(?equivalents)

Try it!

For each language, number of concepts with at least one validated equivalent

For each language, this query shows how many NeoVoc concept entries have at least one validated equivalent. 

#title:For each language, number of concept entries with at least one validated equivalent
#defaultView:BarChart
PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>
PREFIX enp: <https://eneoli.wikibase.cloud/prop/>
PREFIX enps: <https://eneoli.wikibase.cloud/prop/statement/>
PREFIX enpq: <https://eneoli.wikibase.cloud/prop/qualifier/>

select ?lang_code (count(distinct ?concept) as ?concepts_with_validated_equiv)

where {
  ?concept endp:P82 enwb:Q12. # part of NeoVoc collection
  ?concept enp:P57 [enps:P57 ?equiv; enpq:P64 []].
  bind(lang(?equiv) as ?lang_code)
  
} group by ?lang_code ?concepts_with_validated_equiv
order by desc(?concepts_with_validated_equiv)

Try it!


Validation statistics

This query lists how many multilingual equivalents have been validated by each validator. 

#title:Number of equivalents validated (equivalents with a "validated by" P64 qualifier pointing to an entity describing a person)

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>
PREFIX enp: <https://eneoli.wikibase.cloud/prop/>
PREFIX enps: <https://eneoli.wikibase.cloud/prop/statement/>
PREFIX enpq: <https://eneoli.wikibase.cloud/prop/qualifier/>

select ?validator_entity ?validator (count(?equiv_st) as ?validated_equivalents) 

where {
  ?concept endp:P82 enwb:Q12. # part of NeoVoc collection
  ?concept enp:P57 ?equiv_st. ?equiv_st enps:P57 ?equiv. 
  ?equiv_st enpq:P64 ?validator_entity.
  ?validator_entity rdfs:label ?validator. filter(lang(?validator)="en")
} group by ?validator_entity ?validator order by desc(?validated_equivalents)

Try it!

This query lists which equivalents have been validated by each validator 

#title:Lists of equivalents with a "validated by" P64 qualifier pointing to an entity describing a person

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>
PREFIX enp: <https://eneoli.wikibase.cloud/prop/>
PREFIX enps: <https://eneoli.wikibase.cloud/prop/statement/>
PREFIX enpq: <https://eneoli.wikibase.cloud/prop/qualifier/>

select ?validator_entity (sample(?validator) as ?validator_name) (group_concat(distinct concat('"',str(?equiv),'"@',lang(?equiv))) as ?validated_equivalents) 

where {
  ?concept endp:P82 enwb:Q12. # part of NeoVoc collection
  ?concept enp:P57 ?equiv_st. ?equiv_st enps:P57 ?equiv. 
  ?equiv_st enpq:P64 ?validator_entity.
  ?validator_entity rdfs:label ?validator. 
  
} group by ?validator_entity ?validator_name ?validated_equivalents

Try it!


Concept candidates

Candidates still not validated as "NeoVoc Concept"

Concept candidates on this Wikibase (stemming from Task 1.1) are listed using the following query. They are still not part of NeoVoc (i.e. not instance of NeoVoc concept. Instead they are instance of NeoVoc candidate concept. 

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>

select *

where {
  ?concept endp:P5 enwb:Q14; rdfs:label ?fr_label.
  filter(lang(?fr_label)="fr")
  ?concept endp:P71 ?termhood. 
  optional {?concept endp:P1 ?wd.}}
order by desc(xsd:integer(?termhood))

Try it!

Merging candidates

#title: NeoVoc merging candidates: concepts that have the same label (equivalent) in some language

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>

select distinct ?concept1 ?concept1label (lang(?concept1label) as ?lang) ?concept1class ?concept1gloss ?concept2 ?concept2label ?concept2class ?concept2gloss
       
where { 
 {?concept1 endp:P82 enwb:Q12.} union {?concept1 endp:P5 enwb:Q14.} ?concept1 rdfs:label ?concept1label. filter(lang(?concept1label)="en") # change "en" to another language here for looking at matches in another language
 {?concept2 endp:P82 enwb:Q12.} union {?concept2 endp:P5 enwb:Q14.} ?concept2 rdfs:label ?concept2label. filter(lang(?concept2label)="en") # change "en" to another language here for looking at matches in another language
  filter(?concept1 != ?concept2)
  filter(?concept1label = ?concept2label)
  optional {?concept1 schema:description ?concept1gloss. filter(lang(?concept1gloss)="en")} # do not change
  optional {?concept2 schema:description ?concept2gloss. filter(lang(?concept2gloss)="en")} # do not change
  optional {?concept1 endp:P4 [rdfs:label ?concept1class]. filter(lang(?concept1class)="en")} # do not change
  optional {?concept2 endp:P4 [rdfs:label ?concept2class]. filter(lang(?concept2class)="en")} # do not change
} order by ?concept1

Try it!


Checks for inconsistencies

Is there any mismatching equivalent-to-lexemesense link?

This takes time to execute, but it works. 

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>
PREFIX enp: <https://eneoli.wikibase.cloud/prop/>
PREFIX enps: <https://eneoli.wikibase.cloud/prop/statement/>
PREFIX enpq: <https://eneoli.wikibase.cloud/prop/qualifier/>

select ?concept ?equiv_st ?equiv_mylang ?descript_mylang (iri(concat(str(wd:),?wd)) as ?wikidata) ?from_sense ?from_lemma ?to_sense

where {
  ?concept endp:P82 enwb:Q12. # part of NeoVoc collection
  optional {?concept endp:P1 ?wd.}

  ?concept enp:P57 ?equiv_st. ?equiv_st enps:P57 ?equiv_mylang.
  filter not exists {?equiv_st enpq:P58 ?warning.} # no warning
  ?equiv_st enpq:P64 ?validator. # has been validated.
  ?equiv_st enpq:P63 ?to_sense. # equivalent is linked to some sense.
  bind(iri(concat(str(enwb:),?to_sense)) as ?from_sense)
  ?lexeme ontolex:sense ?from_sense; wikibase:lemma ?from_lemma.
  filter (?from_lemma != ?equiv_mylang)
  ?from_sense endp:P12 ?concept. # from no lexeme sense of that language there is a link to this equivalent
  optional {} # is there a lexeme sense already linked from this concept (for the mylang language)?
  optional {?concept schema:description ?descript_mylang. filter(lang(?descript_mylang)=lang(?equiv_mylang))}	
} order by lcase(?equiv_mylang)

Try it!


Is there any linked lexeme sense not linked back from the equivalent?

PREFIX enwb: <https://eneoli.wikibase.cloud/entity/>
PREFIX endp: <https://eneoli.wikibase.cloud/prop/direct/>
PREFIX enp: <https://eneoli.wikibase.cloud/prop/>
PREFIX enps: <https://eneoli.wikibase.cloud/prop/statement/>
PREFIX enpq: <https://eneoli.wikibase.cloud/prop/qualifier/>

select ?lexeme ?lemma ?sense_id ?equiv

where {
  ?lexeme ontolex:sense ?sense; wikibase:lemma ?lemma.
  ?sense endp:P12 ?concept.
  ?concept enp:P57 ?equiv_st. ?equiv_st enps:P57 ?equiv.
  filter(?lemma = ?equiv)
  bind(strafter(str(?sense),str(enwb:)) as ?sense_id)
   filter not exists {
     ?equiv_st enpq:P63 ?sense_id.
     }
}

Try it!

Retrieved from "https://eneoli.wikibase.cloud/w/index.php?title=NeoVoc&oldid=59969"