RC-A1
From UNL Wiki
(Difference between revisions)
(→The corpus500) |
|||
| Line 104: | Line 104: | ||
**[http://www.unlweb.net/resources/corpus500/eng_gen_tgrammar.txt UNL-EN T-Grammar] (Transformation grammar used for the NLization of the Corpus500) | **[http://www.unlweb.net/resources/corpus500/eng_gen_tgrammar.txt UNL-EN T-Grammar] (Transformation grammar used for the NLization of the Corpus500) | ||
**[http://www.unlweb.net/resources/corpus500/eng_gen_dgrammar.txt UNL-EN D-Grammar] (Disambiguation grammar used for the NLization of the Corpus500) | **[http://www.unlweb.net/resources/corpus500/eng_gen_dgrammar.txt UNL-EN D-Grammar] (Disambiguation grammar used for the NLization of the Corpus500) | ||
| + | |||
| + | == Methodology == | ||
| + | In order to prepare the dictionaries and grammars to deal with Corpus500, follow the steps below: | ||
| + | === UNLization === | ||
| + | #Corpus | ||
| + | ##Translate (manually) the sentences of the [http://www.unlweb.net/resources/corpus500/corpus500_eng.txt Corpus500] from English into your native language. Be as close as possible to the original, and provide one single translation for each sentence. This will be your input document file, and your goal will be to provide (automatically, through [[IAN]]) the UNL graphs for each sentence | ||
| + | ##Save the translated text (without the English original) in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>NL FILES. | ||
| + | #Dictionary | ||
| + | ##Extract the word list (i.e., the set of all distinct word forms) appearing in your translation of the Corpus500. | ||
| + | ##Create the NL-UNL dictionary for all the word forms following the English model of the [http://www.unlweb.net/resources/corpus500/dic_ana_eng.txt English Analysis Dictionary]. Note that you may need other features (in English, for instance, nous do not have gender or case). If so, use only the tags available at the [[tagset]]. For further information on the dictionary structure, see [[Dictionary Specs]]. For an explanation of the structure of the English dictionary, see [[English Dictionary]]. | ||
| + | ##Save the NL-UNL dictionary in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>DICTIONARIES. | ||
| + | #Grammar | ||
| + | ##Create the NL-UNL (analysis) transformation grammar necessary to analyze, in UNL, the natural language sentences of the translated corpus. This grammar is the most difficult (and the actual goal) of the whole analysis task. In order to prepare the grammar, study the [[Grammar Specs]]. Next, take a look at the structure of the [[English Grammar]] for a detailed example. In many cases, it would be simpler just to localize the English grammar to your own language rather than creating a whole grammar from the scratch. The grammar is normally developed inside IAN, because we have to test it every time. But you may also prepare a draft grammar, save it in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>T-RULES. | ||
| + | ##Create the NL-UNL (analysis) disambiguation grammar in order to improve the results of the transformation and to control the process of tokenization. Again, this grammar is normally developed inside IAN (at the D-RULES tab), because we have to test it every time. Consult the [[English Grammar]] for a detailed example of a disambiguation grammar. | ||
| + | ##Test the grammar against the corpus and provide the necessary changes. | ||
| + | === NLization === | ||
| + | #Corpus | ||
| + | ##Upload the file [http://www.unlweb.net/resources/corpus500/corpus500_unl.txt] to UNLWEB>UNLDEV>EUGENE>UNL DOCUMENT. Do not do any change to this file. | ||
| + | #Dictionary | ||
| + | ##Localize the [http://www.unlweb.net/resources/corpus500/dic_gen_eng.txt English Generation Dictionary] to your language. The localized version must reflect the word list of your translated corpus. Use only the tags available at the [[tagset]]. For further information on the dictionary structure, see [[Dictionary Specs]]. | ||
| + | ##Save the UNL-NL dictionary in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>PROJECTS>EUGENE>DICTIONARIES. | ||
| + | #Grammar | ||
| + | ##Export the inflectional grammar of your language from UNLARIUM>GRAMMAR>[YOUR LOCALE]>EXPORT. If the grammar of your language is not available yet, you may: | ||
| + | ***Provide it through the UNLarium (only for users approved in CLEA700); or | ||
| + | ***Create the inflectional paradigms only for the inflected forms appearing in the UNL-NL dictionary. In that case, follow the model available at [http://www.unlweb.net/unlarium/grammar/export_grammar.php?type=M&direction=G&lang=en English Inflectional Grammar]. The documentation of the English grammar is available at [http://www.unlweb.net/unlarium/grammar/export_grammar.php?type=M&lang=en English Inflectional Grammar (only for reference)]. For further information, see [[Inflectional paradigms]]. | ||
| + | ##Save the inflectional grammar in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>PROJECTS>EUGENE>RULES. | ||
| + | ;NL-UNL (Analysis) Grammar | ||
| + | ##Provide the UNL-NL (generation) T-grammar and D-grammar necessary to generate natural language sentences from the UNL corpus. Follow the sample instructions available for the analysis grammar. | ||
Revision as of 21:13, 23 July 2012
The Corpus500 is an experimental corpus used to prepare the initial versions of the grammar for sentence-based UNLization and NLization, using IAN and EUGENE, respectively. It comprises a list of 500 sentences in English and their corresponding graphs in UNL, and is supposed to cover very basic linguistic phenomena.
Contents |
The corpus500
- Corpus 500 according to the complexity of the graphs
| Order | Description | Analysis (English original) | Generation (UNL) |
|---|---|---|---|
| 1 | Temporary entries | temp_org.txt | temp_unl.txt |
| 2 | Entries with no attribute or relation | attribute0_org.txt | attribute0_unl.txt |
| 3 | one-attribute entries | attribute1_org.txt | attribute1_unl.txt |
| 4 | two-attribute entries | attribute2_org.txt | attribute2_unl.txt |
| 5 | three-attribute entries | attribute3_org.txt | attribute3_unl.txt |
| 6 | one-relation entries | relation1_org.txt | relation1_unl.txt |
| 7 | two-relation entries | relation2_org.txt | relation2_unl.txt |
| 8 | three-relation entries | relation3_org.txt | relation3_unl.txt |
| 9 | four-relation entries | relation4_org.txt | relation4_unl.txt |
| 10 | five-relation entries | relation5_org.txt | relation5_unl.txt |
| 11 | six-relation entries | relation6_org.txt | relation6_unl.txt |
| 12 | numbers and numerals | numbers_org.txt | numbers_unl.txt |
| 13 | pronouns | numbers_org.txt | numbers_unl.txt |
| 14 | expressions of time | time_org.txt | time_unl.txt |
| 15 | relative clauses | relatives_org.txt | relatives_unl.txt |
| 16 | special issues | problems_org.txt | problems_unl.txt |
- The whole corpus in one single file
- Corpus500 in English, experimental corpus in English (500 sentences), to be manually translated to the target languages, in order to be used as the input for IAN
- Corpus500 in UNL, experimental corpus in UNL (500 graphs), to be used as the input for EUGENE
Resources
The following resources have been used to deal with Corpus 500 in English and may be used as a sample of what is expected to be provided
- Analysis
- EN-UNL Dictionary (English dictionary used for the UNLization of the Corpus500)
- EN-UNL T-Grammar (Transformation grammar used for the UNLization of the Corpus500)
- EN-UNL D-Grammar (Disambiguation grammar used for the UNLization of the Corpus500)
- Generation
- UNL-EN Dictionary (English dictionary used for the NLization of the Corpus500)
- UNL-EN T-Grammar (Transformation grammar used for the NLization of the Corpus500)
- UNL-EN D-Grammar (Disambiguation grammar used for the NLization of the Corpus500)
Methodology
In order to prepare the dictionaries and grammars to deal with Corpus500, follow the steps below:
UNLization
- Corpus
- Translate (manually) the sentences of the Corpus500 from English into your native language. Be as close as possible to the original, and provide one single translation for each sentence. This will be your input document file, and your goal will be to provide (automatically, through IAN) the UNL graphs for each sentence
- Save the translated text (without the English original) in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>NL FILES.
- Dictionary
- Extract the word list (i.e., the set of all distinct word forms) appearing in your translation of the Corpus500.
- Create the NL-UNL dictionary for all the word forms following the English model of the English Analysis Dictionary. Note that you may need other features (in English, for instance, nous do not have gender or case). If so, use only the tags available at the tagset. For further information on the dictionary structure, see Dictionary Specs. For an explanation of the structure of the English dictionary, see English Dictionary.
- Save the NL-UNL dictionary in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>DICTIONARIES.
- Grammar
- Create the NL-UNL (analysis) transformation grammar necessary to analyze, in UNL, the natural language sentences of the translated corpus. This grammar is the most difficult (and the actual goal) of the whole analysis task. In order to prepare the grammar, study the Grammar Specs. Next, take a look at the structure of the English Grammar for a detailed example. In many cases, it would be simpler just to localize the English grammar to your own language rather than creating a whole grammar from the scratch. The grammar is normally developed inside IAN, because we have to test it every time. But you may also prepare a draft grammar, save it in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>T-RULES.
- Create the NL-UNL (analysis) disambiguation grammar in order to improve the results of the transformation and to control the process of tokenization. Again, this grammar is normally developed inside IAN (at the D-RULES tab), because we have to test it every time. Consult the English Grammar for a detailed example of a disambiguation grammar.
- Test the grammar against the corpus and provide the necessary changes.
NLization
- Corpus
- Upload the file [1] to UNLWEB>UNLDEV>EUGENE>UNL DOCUMENT. Do not do any change to this file.
- Dictionary
- Localize the English Generation Dictionary to your language. The localized version must reflect the word list of your translated corpus. Use only the tags available at the tagset. For further information on the dictionary structure, see Dictionary Specs.
- Save the UNL-NL dictionary in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>PROJECTS>EUGENE>DICTIONARIES.
- Grammar
- Export the inflectional grammar of your language from UNLARIUM>GRAMMAR>[YOUR LOCALE]>EXPORT. If the grammar of your language is not available yet, you may:
- Provide it through the UNLarium (only for users approved in CLEA700); or
- Create the inflectional paradigms only for the inflected forms appearing in the UNL-NL dictionary. In that case, follow the model available at English Inflectional Grammar. The documentation of the English grammar is available at English Inflectional Grammar (only for reference). For further information, see Inflectional paradigms.
- Save the inflectional grammar in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>PROJECTS>EUGENE>RULES.
- NL-UNL (Analysis) Grammar
- Provide the UNL-NL (generation) T-grammar and D-grammar necessary to generate natural language sentences from the UNL corpus. Follow the sample instructions available for the analysis grammar.
