RC-A1

From UNL Wiki
(Difference between revisions)
Jump to: navigation, search
(→UNLization)
(→The corpus500)
Line 2: Line 2:
  
 
== The corpus<sup>500</sup> ==
 
== The corpus<sup>500</sup> ==
−
*[http://www.unlweb.net/resources/corpus500/corpus500_eng.txt Corpus<sup>500</sup> in English], to be (manually) translated to your target language in order to be used as the input for the UNLization process (with [[IAN]])
+
*The Corpus<sup>500</sup> in one single file
−
*[http://www.unlweb.net/resources/corpus500/corpus500_unl.txt Corpus<sup>500</sup> in UNL], to be used, "as is" (i.e., without any change), as the input for the NLization process (with [[EUGENE]])
+
**[http://www.unlweb.net/resources/corpus500/corpus500_eng.txt Corpus<sup>500</sup> in English], to be (manually) translated to your target language in order to be used as the input for the UNLization process (with [[IAN]])
 +
**[http://www.unlweb.net/resources/corpus500/corpus500_unl.txt Corpus<sup>500</sup> in UNL], to be used, "as is" (i.e., without any change), as the input for the NLization process (with [[EUGENE]])
 +
*The Corpus<sup>500</sup> in different files (the same as above, but divided according to the different phenomena)
 +
{|+Corpus<sup>500</sup>
 +
!Type
 +
!English
 +
!UNL
 +
!Description
 +
!Number of sentences
 +
|-
 +
|Temporary entries
 +
|[http://www.unlweb.net/resources/corpus500/temporary_eng.txt temporary_eng.txt]
 +
|[http://www.unlweb.net/resources/corpus500/temporary_unl.txt temporary_unl.txt]
 +
|[http://www.unlweb.net/resources/corpus500/temporary.txt temporary.txt]
 +
|15
 +
|-
 +
|Numbers
 +
|[http://www.unlweb.net/resources/corpus500/numbers_eng.txt numbers_eng.txt]
 +
|[http://www.unlweb.net/resources/corpus500/numbers_unl.txt numbers_unl.txt]
 +
|[http://www.unlweb.net/resources/corpus500/numbers.txt numbers.txt]
 +
|150
 +
|-
 +
|Determiners
 +
|[http://www.unlweb.net/resources/corpus500/determiners_eng.txt determiners_eng.txt]
 +
|[http://www.unlweb.net/resources/corpus500/determiners_unl.txt determiners_unl.txt]
 +
|[http://www.unlweb.net/resources/corpus500/determiners.txt determiners.txt]
 +
|60
 +
|-
 +
|Prepositions
 +
|[http://www.unlweb.net/resources/corpus500/prepositions_eng.txt prepositions_eng.txt]
 +
|[http://www.unlweb.net/resources/corpus500/prepositions_unl.txt prepositions_unl.txt]
 +
|[http://www.unlweb.net/resources/corpus500/prepositions.txt prepositions.txt]
 +
|40
 +
|-
 +
|Conjunctions
 +
|[http://www.unlweb.net/resources/corpus500/conjunctions_eng.txt conjunctions_eng.txt]
 +
|[http://www.unlweb.net/resources/corpus500/conjunctions_unl.txt conjunctions_unl.txt]
 +
|[http://www.unlweb.net/resources/corpus500/conjunctions.txt conjunctions.txt]
 +
|40
 +
|-
 +
|Noun phrase structure
 +
|[http://www.unlweb.net/resources/corpus500/np_eng.txt np_eng.txt]
 +
|[http://www.unlweb.net/resources/corpus500/np_unl.txt np_unl.txt]
 +
|[http://www.unlweb.net/resources/corpus500/np.txt np.txt]
 +
|40
 +
|-
 +
|Expressions of time
 +
|[http://www.unlweb.net/resources/corpus500/time_eng.txt time_eng.txt]
 +
|[http://www.unlweb.net/resources/corpus500/time_unl.txt time_unl.txt]
 +
|[http://www.unlweb.net/resources/corpus500/time.txt time.txt]
 +
|20
 +
|-
 +
|Verb forms
 +
|[http://www.unlweb.net/resources/corpus500/verbs_eng.txt verbs_eng.txt]
 +
|[http://www.unlweb.net/resources/corpus500/verbs_unl.txt verbs_unl.txt]
 +
|[http://www.unlweb.net/resources/corpus500/verbs.txt verbs.txt]
 +
|50
 +
|-
 +
|Pronouns
 +
|[http://www.unlweb.net/resources/corpus500/pronouns_eng.txt pronouns_eng.txt]
 +
|[http://www.unlweb.net/resources/corpus500/pronouns_unl.txt pronouns_unl.txt]
 +
|[http://www.unlweb.net/resources/corpus500/pronouns.txt pronouns.txt]
 +
|50
 +
|-
 +
|Sentence structures
 +
|[http://www.unlweb.net/resources/corpus500/sentences_eng.txt sentences_eng.txt]
 +
|[http://www.unlweb.net/resources/corpus500/sentences_unl.txt sentences_unl.txt]
 +
|[http://www.unlweb.net/resources/corpus500/sentences.txt sentences.txt]
 +
|65
 +
|}
  
 
== Goals ==
 
== Goals ==

Revision as of 17:30, 30 July 2012

The Corpus500 is an experimental corpus used to prepare the initial versions of the grammar for sentence-based UNLization and NLization, using IAN and EUGENE, respectively. It comprises a list of 500 sentences in English and their corresponding graphs in UNL, and is supposed to cover very basic linguistic phenomena.

Contents

The corpus500

  • The Corpus500 in one single file
    • Corpus500 in English, to be (manually) translated to your target language in order to be used as the input for the UNLization process (with IAN)
    • Corpus500 in UNL, to be used, "as is" (i.e., without any change), as the input for the NLization process (with EUGENE)
  • The Corpus500 in different files (the same as above, but divided according to the different phenomena)
Type English UNL Description Number of sentences
Temporary entries temporary_eng.txt temporary_unl.txt temporary.txt 15
Numbers numbers_eng.txt numbers_unl.txt numbers.txt 150
Determiners determiners_eng.txt determiners_unl.txt determiners.txt 60
Prepositions prepositions_eng.txt prepositions_unl.txt prepositions.txt 40
Conjunctions conjunctions_eng.txt conjunctions_unl.txt conjunctions.txt 40
Noun phrase structure np_eng.txt np_unl.txt np.txt 40
Expressions of time time_eng.txt time_unl.txt time.txt 20
Verb forms verbs_eng.txt verbs_unl.txt verbs.txt 50
Pronouns pronouns_eng.txt pronouns_unl.txt pronouns.txt 50
Sentence structures sentences_eng.txt sentences_unl.txt sentences.txt 65

Goals

  1. To provide the dictionary and grammars necessary to UNLize your translated version of the Corpus500 (with IAN)
  2. To provide the dictionary and grammars necessary to NLize, to your target language, the UNL version of the Corpus500 (with EUGENE)

Methodology

In order to prepare the dictionaries and grammars to deal with the Corpus500, follow the steps below:

UNLization

  1. Corpus
    1. Translate (manually) the sentences of the Corpus500 from English into your native language. Be as close as possible to the original, and provide one single translation for each sentence. This will be your input document file, and your goal will be to provide (automatically, through IAN) the UNL graphs for each sentence
    2. Save the translated text (without the English original) in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>NL FILES.
  2. Dictionary
    1. Localize the dictionary available at eng_ana_dic.txt. Note that "localization" is not the same as "translation". You may need other features (in English, for instance, nouns do not have gender or case) or other entries. In any case, the resulting dictionary should fit your translated version of the corpus (i.e., all the entries appearing in your translated version of the corpus should appear in the dictionary). For further information on localization, see Localization. For information on the dictionary structure, see Dictionary Specs. For an explanation of the structure of the English dictionary, see English Dictionary. In case you need additional features, use only the tags available at the tagset.
    2. Save the NL-UNL dictionary in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>DICTIONARIES.
  3. Grammar
    1. Create the NL-UNL (analysis) transformation grammar necessary to analyze, in UNL, the natural language sentences of the translated corpus. This grammar is the most difficult (and the actual goal) of the whole analysis task. In order to prepare the grammar, study the Grammar Specs. Next, take a look at the structure of the English Grammar for a detailed example. In many cases, it would be simpler just to localize the English grammar to your own locale rather than creating a whole grammar from the scratch. See the instructions at Localization. The grammar is normally developed inside IAN, because we have to test it every time. But you may also prepare a draft grammar, save it in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>T-RULES.
    2. Create the NL-UNL (analysis) disambiguation grammar in order to improve the results of the transformation and to control the process of tokenization. Again, this grammar is normally developed inside IAN (at the D-RULES tab), because we have to test it every time. Consult the English Grammar for a detailed example of a disambiguation grammar.
    3. Test the grammar against the corpus and provide the necessary changes.

NLization

  1. Corpus
    1. Upload the file corpus500_unl.txt to UNLWEB>UNLDEV>EUGENE>UNL DOCUMENT. Do not do any change to this file.
  2. Dictionary
    1. Localize the dictionary available at eng_gen_dic.txt to your locale. Note, once again, that localization is not the same as translation. The localized version must reflect the word list of your translated corpus. In any case, use only the tags available at the tagset. For further information on the dictionary structure, see Dictionary Specs.
    2. Save the UNL-NL dictionary in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>PROJECTS>EUGENE>DICTIONARIES.
  3. Grammar
    1. Export the inflectional grammar of your language from UNLARIUM>GRAMMAR>[YOUR LOCALE]>EXPORT. If the grammar of your language is not available yet, you may:
      1. Provide it through the UNLarium (only for users approved in CLEA700); or
      2. Create the inflectional paradigms only for the inflected forms appearing in the UNL-NL dictionary. In that case, follow the model available at English Inflectional Grammar. The documentation of the English grammar is available at English Inflectional Grammar (only for reference). For further information, see Inflectional paradigms.
    2. Save the inflectional grammar in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>PROJECTS>EUGENE>RULES.
    3. Provide the UNL-NL (generation) T-grammar and D-grammar necessary to generate natural language sentences from the UNL corpus. Follow the sample instructions available for the analysis grammar.

Samples and Examples

The following resources have been used to deal with Corpus 500 in English and may be used as a sample of what is expected to be provided

Recommended Readings

Before starting the activity, and in order to fully understand what is expected to be done, it is important for you to be acquainted with the following documentation:

  • Tagset, because you are expected to use only the tags included in the tagset
  • UNL Dictionary Specs, which is essential to understand the dictionary structure
  • UNL Grammar Specs, which is essential to understand the grammar structure
Retrieved from "http://unlweb.net/wiki/index.php?title=RC-A1&oldid=4042"
Software