AUTOMATIC SYNTHESIS OF RUSSIAN SPEECH

Olga Krivnova
Philological Faculty, Lomonosov Moscow State University, Russia

ABSTRACT

The paper describes the main principles of Russian text-to-speech synthesis developed by Speech group of the Philological Faculty, Lomonosov Moscow University, Russia. The system is organized as a mixture of two methods: concatenation - on the segment level (using the linguistically motivated units - allophones' waveforms spliced together to form the initial speech wave) and the rule-based method on the prosodic level to modify the initial speech wave according to the prosodic characteristics of a phrase being synthesized. The allophonic database is a set of allophone wave files, each file being named accounting the allophone itself and its phonetic context. Signal generation is implemented according to the phrase control file, which describes the phrase as a sequence of allophones' code names with assigned duration, energy and fundamental frequency values. To transform the base allophones to required prosodic values we use procedures that are close to TD PSOLA technology.

1. OVERALL ARCHITECTURE OF THE SYSTEM
The overall structure of our system is in line with the functional organization of a general TTS synthesizer [1]. It consists of several blocks or modules, each of which has its own tasks and functions.
The synthesis of an arbitrary sentence may be split into two larger stages. On the first stage the task is to produce a phonetic (symbolic) transcription of the text to be pronounced including its intonational, accentuation and rhythmical characteristics. On the second stage the necessary acoustic signal (in digital form) should be created on the base of this phonetic representation. These tasks are implemented in our system by a text preprocessing module and digital signal processing module accordingly. These blocks are connected through a module termed as speech control generation which function is to generate a list of allophone codes (names) augmented with necessary prosodic information (frequency, duration, amplitude).
The above mentioned modules in turn include several submodules, which output can be supervised independently. 
The text preprocessing module includes a text normalization submodule and two automatic transcriptors: accent-intonation and phonemic.
The speech control generation module consists of a prosodic parameterization block, an allophonic coding submodule and a block of control file generation.
The digital signal processing module includes two allophonic databases (acoustic inventories) and a block of speech signal generation.
The described structure is shown on Fig.1.
Below the more detailed description of each of the above specified modules will be given (see also [2], [3]).

2. TEXT PREPROCESSING
2.1. Input text normalization
The main task of text preprocessing stage is the construction of phonetic representation of an input text or its fragments, which should contain segmental and suprasegmental information necessary for the subsequent generation of an acoustic signal. 

Figure 1. Overall structure of TTS system for Russian.

For creation of such a representation the input text should be at first converted to the sequence of the orthographical (spelling) words with stress marks, white spaces and admissible punctuation marks must divide the words. Such text form can be conventionally named "normalized".
The operations, which transform the input text into the normalized form, concern to a stage of text normalization, which we understand wider than it is accepted in many TTS systems. We differentiate the following tasks here:
- Processing textual objects, which are not ordinary words (numerals, abbreviations of a type " Mr., etc., i.e. " etc.) and require verbal expansion with a choice of the necessary grammatical forms.
- Processing textual objects, which are not ordinary words, but at the same time do not require verbalization (acronyms of a type "MSU", proper names). As a rule such words are absent in the usual dictionaries.
- Processing ordinary words (stress marker placement, replacement the letter "е" on the letter "ë" in those cases where it is needed for correct word pronunciation, as in Russian texts the letter "ё" practically is never used. 
	Regretfully, we did not elaborate systematically the first two tasks. Nevertheless we have collected a number of preparatory materials for their development: on a material of the newspaper texts a list of acronyms with their frequency of occurrence, phonetic transcriptions and word collocations - sources is made (electronic version 200 entries) and also a list of abbreviations (with period in spelling) used in scientific texts with their verbal expansion in dictionary canonical form (electronic version 1100 entries). Besides we can process numbers in expressions of dates and time of the day. But as a whole we think that many problems arising while implementing the tasks 1 and 2, demand the contextual analysis with the use of at least grammatical information. To receive this information it is necessary to perform morphological analysis of words processed and the same is true for the stress assignment to Russian words. 
	As it is well known, Russian has a so-called free stress with strongly reduced unstressed vowels. Pronunciation errors connected with stress misplacement are very appreciable on hearing and frequently make a word practically unrecognizable. In a number of cases the stress placement has distinctive function (as in /mu'ka/ "torment" vs /muka'/ "flour" or /ru'ki/ vs /ruki'/ as different grammatical forms of a word "hand"). So in Russian the place of a stress is lexically and grammatically conditioned.
Full morphological analysis of ordinary Russian words is not an easy problem. Fortunately, there exists (electronic version also available) the grammatical dictionary created by A.A.Zaliznjak [4], which contains about 100000 words in their initial forms and all information necessary for the analysis / synthesis of their grammatical forms. The most of morphological processors developed for Russian uses this dictionary by and large and we including. 
As against the majority of existing morphological processors for Russian in our analyzer the theoretical model of word inflection described in [4] is used practically without changes. It was achieved by using the well-known principles: calculations of stems, analysis by synthesis, statement and verification of hypotheses by various modules (principle of "a class board ") [5].
In [4] all possible accentuation word types are accounted for and all initial word forms have markers for stress position and position of the letter "e" to be replaced by the letter "ë". Also for each word entry [4] specifies part-of-speech, gender and animacy (for nouns), aspect and transitivity (for verbs), type of inflection paradigm. Due to the principle "analysis-by-synthesis" this information allows not only to assign stresses, replace "e" by "ë" (what is performed in our system), but also to receive all grammatical features possible in case of an isolated word processing. We are going to use these features for contextual analysis, which is necessary for different tasks: unusual symbols to word expansion, homographs and other types of ambiguities, prosodic phrasing etc. Now it is only in a development stage.

2.2. 	Automatic accent-intonation transcription (AITR)
2.2.1. Intonational phrasing and prosodic words grouping. At synthesis of emotionally neutral speech the AITR should locate the intonational phrases boundaries, determine a degree of intonational breaks and set the intonational model parameters, which are making out a phrase.
Major cues to locate the intonational boundaries are punctuation marks. However it is well known that the correspondence between intonation and punctuation is not direct one. Besides, the information about major syntactic breaks and the length of the main constituents are at least needed. Now we conduct work in this direction, however while the intonational phrasing in our system is based exclusively on punctuation. It is possible also to indicate phrase boundaries in the input text manually.
Intonation phrasing is supposed to have some pause after each phrase. We distinguish three degrees of pauses: short (about 250ms), moderate (about 400 ms) and long (about 800 ms). The choice of a pause duration type is also based on punctuation marks. It is the main, though not the only one, prosodic means to create different degrees of intonational breaks.
Prosodic words grouping within an intonational phrase is closely connected with rhythmization and is realized in our system by special feature "degree of prosodic break", assigned to the blanks between words. Here we have three breaks' levels: 0 - after or before full clitics (e.g. preposition and some particles); 1 - after or before functional words, which are not full clitics (e.g. pronouns); 2 - between two content words. This information is used in rhythmization rules and phonemic transcriptor when processing external phoneme sandhi and vowel reduction.

2.2.2. Intonation. To each intonational phrase AITR should assign a number of characteristics determining an intonational model, which is chosen for its pronunciation. The major characteristics are: a type of intonational model and global prosodic parameters such as a voice range area used and its width, degree of pitch salience of lexical stresses for content words, speech rate and loudness.
In our system we use the following intonation models: 1 model of finality, 1 model of non-finality for the affirmative sentences, 3 interrogative models (general question, special question, comparative question), 1 model for exclamation sentences. For all models the possibility of different positions of an intonation center in the phrase is available (in some cases its position is determined by the nuclear phrase stress, while in others by focus accent mark). The choice of the appropriate model is based on punctuation mark and some lexical information. It is obvious that these cues are not enough and more over the relation between punctuation and intonation models is rather difficult one, especially in Russian. This problem is also under investigation. At a present stage of development any phrase can be pronounced with any of accessible intonational models in a hand-operated mode.
As to the global intonational parameters, the possibility of their manual adjustment in any combinations is realized, however automatically the most neutral variant is temporarily used (by default).

2.2.3. Rhythm and accentuation. We assume that word prominence in an intonational phrase is adjusted by two functionally different mechanisms: focus accentuation and rhythmization.
The focus accents (to contrast or emphasize some words) are substantially defined by a speaker intention or by the whole information structure of a text. The last one frequently has no evident cues to determine an accent place and type. Therefore the formalization of focus accentuation represents the most difficult problem for TTS-systems. Our synthesizer is able to synthesize phrases with different focus accents but we have no rules to determine their localization automatically: it should be done manually. If a phrase has words with accent markers, the last of them is considered as the intonational center (nuclear) of a phrase. Otherwise the last content word of a phrase is admitted as its intonational nuclear (or its automatic phrase stress) by default. It is the most typical situation for the narrative Russian texts, which construction is based on the use of neutral linear - intonational structures with a final position of the intonational center.
As far as rhythmization is concerned, AITR assigns some degree of prominence to each vowel in an intonational phrase as its rhythmical feature. We distinguish three degrees of vowel prominence within a word (stressed, strong unstressed, weak unstressed) and four degrees for lexically stressed vowels (1 for full clitics, 2 for functional words, 3 for nonnuclear content words, 4 for nuclear content word). In Russian the prominence markers are very important to determine correctly the duration of sounds.

2.3. Automatic phonemic transcription (PHTR)
The segment inventory used in our system includes 56 units. It slightly differs from that prevalent in the phonological descriptions of Russian. But for the purpose of synthesis it is convenient in some cases to have different symbols in the transcription even for those sounds that are in no phonological contrast. The list of sounds we differentiate is given below, further we name them phonemes (our transcription is based on Russian alphabet, so for convenience the Russian symbols are replaced here with their IPA conformity). 
Stressed vowels: [A], [], [], [], [], []Strong unstressed vowels: [A], [], [], [], [], []Weak unstressed vowels: [], [], []Non-palatalized consonants: [],  [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []Palatalized consonants:[], [], [], [], [], [], [], [],[], [], [], [], [], [],  [],  [], [], [], [], [], [  ].
A set of regular phonological rules implements the mapping "letter - phoneme" and "phoneme - phoneme" which include such operations as elimination of spelling fictions, removing of hard and soft symbols in spelling with simultaneous interpretation of palatalization of near by consonants, assimilation processes in consonant clusters, vowel reduction and so on. Irregular pronunciation of some word classes (e.g. loan words) and even individual words is accounted for by using special word lists. Currently there are 54 such exception lists in PHTR [6].
All phonetization procedures are based on expert rewrite rules, which take into account the standard Russian pronunciation described in [7].

3. SPEECH CONTROL GENERATION
3.1. The block of prosodic parameterization
Rules of this block define the amplitude, duration and fundamental frequency values, which should be attributed to phonemes forming the output transcription of an intonational phrase. According to these calculated physical values the modification of the base acoustic allophones chosen for concatenation is carried out at the stage of acoustic signal generation. 
Consider briefly as it is carried out in our system.

3.1.1. Amplitude. Basically those characteristics are kept which are inherent to allophones as the concatenation units, i.e. in most cases no updating of the initial data occurs. We use some rules to control the changes of amplitude on the final syllables in a phrase and some rules of adjusting type. Intensity is usually thought as the least relevant for realization of suprasegmental phonetic distinctions in Russian. Sharing in whole this point of view, we consider at the same time that further research on the contribution of energy parameter in the quality of synthesized speech is necessary.
3.1.2. Duration. The duration of vowels and consonants are formed by separate blocks of rules. In both cases the hierarchical system of the factors determining an allophone's duration is used. 
For Russian vowels primary factors (in decreasing order of importance) are their phonetic identity (especially vowel height feature), prominence level (rhythmical feature), the distance from beginning of a phrase and of a phonological word. These factors are captured by the quantitative model, which has been described in [9] and is close to the known Klatt’s model [10]. In our model the effect of all above mentioned suprasegmental factors is described as shortening of the intrinsic duration of a vowel in question. Duration values received as a result of model application are consired as base ones. These base durations can be modified further by some additional rules which take into account such lengthening factors as prepausal position, the class of near by consonants (approximant, trill, nasal, voiced nonsonorant) and vowel hiatus. 
The consonant duration rules are organized similarly. Here the primary factors are phonetic identity, prominence level of a syllable, to which a consonant belongs to, phrase position (initial, medial, final), cluster entering and position in a cluster (final / non final). Except for base rules, which take into account the specified factors, the additional rules are used which consider phonetic identity of near by consonants in clusters and presence of a length feature.
All numerical estimates needed are obtained empirically and based on various phonetic experiments, which include reading of large enough texts.

3.1.3. Pitch and fundamental frequency. The pitch characteristics of sounds are defined in accordance with intonational, accent and rhythmical markers assigned to an intonational phrase as the output of accent-intonation transcription.
The generation of F0 curve is carried out in two steps [11]. At first pitch values are set in semi-tone scale with respect to the average pitch (or base tone) of a speaker. Then these semi-tone values (or pitch contour) are transformed into F0 curves. It allows to use the same tone rules for synthesis of voices with different pitches. Besides there is an opportunity to control a base tone level, if the phrase should be sounded in the marked voice register - lowered or raised rather neutral. 
	The strategy of pitch contour generation is as follows: the contour is formed as a result of concatenation of tonal objects of two types - tonal figures and tonal plateaus. 
The tonal figures are aligned with lexically stressed syllables and according to tradition can be named as pitch accents. They are ordered in accordance with their functional and phonetic importance, what defines a degree of reflection of phonetic details in their tone rules. The most important is the figure that anchored to the nuclear syllable (intonation center of the model), then follows the figures anchored to the first and the last stressed syllables of a phrase (if they аre not nuclears), and then figures anchored to stressed syllables of the rest content words ( if there is no indication that they should be made atonal in the given intonational model). The main parameters of pitch accents are a type of pitch movement, the realization domain (part of a phrase on which the figure is realized, stressed syllable including), the localization of pitch targets points of a figure in a speaker pitch range and realization domain.
The tonal plateaus are aligned with unstressed and atonal stressed syllables in the beginning and end of a phrase and also in the intervals between pitch accent realization domains. The controllable parameters in this case are pitch values at the margins of intonational phrases and an interval of pitch change.
The number of tone rules, which form the described tone objects, depends on their phonetic complexity and functional - phonetic importance. For example, the detailed phonetic elaboration of the nuclear pitch accent of non-finality intonational model is carried out by a set of 27 rules, which take into account a position of the intonational center in a phrase (initial, medial, final), position of a nuclear syllable in a word (initial uninitial), sound structure of the nuclear syllable, variations in the realization domain. 
As a result of application of these rules each phoneme in a phrase is attributed by at least two pitch values as its starting and end points. If it is necessary three and even more values can be assigned to any point inside the phoneme. So, on the whole any pitch contour is approximated by linear tonal movements. 
	As for duration all tone rules are hand-written and based on phonetic and acoustic analysis of read-aloud texts.

3.2. Allophonic coding module
This module converts the phoneme symbols used in phonemic transcription into the codes (names) of appropriate concatenation units. As a rule these units are phoneme-size acoustic segments (i.e. allophones) and their codes contain just the information on their phonemic identity and on the classes of their right and left phoneme contexts. There are two exceptions from this trend, which require some additional processing. So, stops, affricates and trills can be spliced on several acoustic segments and on the other hand some two-phoneme sequences (i.e. vowel hiatus) can be merged into a single acoustic segment.

3.3. Generation of speech control file
The outputs of prosodic parameterization block and allophonic coding module are united in a general control file where a synthesized phrase is represented as a sequence of concatenation unit codes with assigned duration, F0 and amplitude values. At this stage some readjustment prosodic rules are applied which take into account small divergences between the structure of phonemic transcription and the sequence of concatenation units necessary to sound it.

4. DIGITAL SIGNAL PROCESSING
4.1. Synthesis Allophonic Databases
As it was already mentioned, we use for concatenation such acoustic segments, which usually have phoneme size. Each unit is stored as a single coded file. There are two acoustic inventories in our synthesizer: one for male voice (SR 11025Hz; SS 8-bit; TV-announcer) with 158 consonant allophones and 530 vowels; the other for female voice (SR 22050Hz; SS 16-bit; RD-announcer) with 200 consonant allophones and around 1000 vowels. All vocalic wave files in the databases are marked semi-automatically according to their pitch periods to generate the output speech signal. 
	The main principles of the database design and its preparing was described in [12]. The basic requirement was to cover possibly all acoustically and perceptually relevant effects of coarticulation, not exceeding at the same time the reasonable bounds of database size.
4.2. Signal Generation
 Signal generation is implemented according to the speech control file the structure of which was described above. To transform the base allophones to physical  values given by the phrase control file, we use algorithms and procedures that are close to PSOLA technique in the time domain [13]. 
To increase the naturalness of synthesized speech at last stage of a signal generation the final sequence of the modified pitch periods is transformed from linear approximated curve into cosine form and some casual fluctuation of period duration's is implemented.

5. CONCLUDING REMARKS
Our TTS system for Russian is designed as a research prototype. At the same time we wanted to build such a device that could be easily edited, tested and changed. Therefore special attention in our work is given to development of special toolkit, which can be used for operative testing of results of both: all system and its separate modules [8]. Besides the system is used in the educational purposes: in the courses on Russian phonetics and speech technologies. It gives us an opportunity to make natural testing and evaluation of the synthesizer at different stages of its development. Also we carried out special tests to evaluate the intelligibility and naturalness of the synthesized speech, which have given good results. At the same time we realize, that a lot of work is to be done, especially at the NLP processing stage. The main problems for us here are the lack of necessary speech and text corpora and special tools for their automatic analysis and linguistic and acoustic modeling. 

NOTES
Synthesizer is realized in library DLL for platform Win32 (Windows 95 or NT); the place occupied on a disk about 10Mb (without wave compression). The demo-examples of Russian speech synthesized by our system (in wave format) can be found in INTERNET to the address http://isabase.philol.msu.ru/SpeechGroup.

REFERENCES
[1] Dutoit, T. 1997. Аn Introduction to Text-to-Speech Synthesis. Dordrecht-Boston-London, Kluwer Ac. Publishers.
[2] Zinovieva, N.V., Krivnova, O.F., Zaharov, L.M. 1995. Automatic synthesis of Russian speech (synthezator "Agafon"). In Narin'yani A.S. (ed.), Proceedings of the International Workshop on Computational Linguistics and its Applications "Dialogue95". Kazan. (In Russian).
[3] Krivnova, O.F. 1998. TTS synthesis for Russian language (second version with female voice). In Narin'yani A.S. (ed.), Proceedings of the International Workshop on Computational Linguistics and its Applications "Dialogue98". Kazan: Hater. (In Russian).
[4] Zaliznjak, A.A. 1977. Grammatical Dictionary of the Russian Language. Moscow, Russkij Yazyk.
 [5] Sidorov, G.O. 1996. System for automatic stressing of Russian words. II International Simposium IARLLT (International Association of Russian Language and Literature Teachers) "Phonetics in the Language System". Teachers). Moscow, Unikum-Centre. (In Russian).
[6] Zaharov, L.M. 1997. Text transcription at synthesis and analysis of Russian speech: untrivial cases. In Narin'yani A.S. (ed.), Proceedings of the International Workshop on Computational Linguistics and its Applications "Dialogue97". Moscow. (In Russian).
[7] Orthoepic Dictionary of the Russian Language. Avanesov, R. I. (ed.) Moscow, Russkij Yazyk, 1997.
[8] Strokin, G.S. Tools for speech synthesis system development. In Narin'yani A.S. (ed.), Proceedings of the International Workshop on Computational Linguistics and its Applications "Dialogue98". Kazan: Hater. (In Russian).
[9 ] Krivnova, O.F. 1987. Durational Pattern of Rissian Syntagma: The Standard Scheme and its Modification. Proceedings of the XIth ICPhS, 5, 122-125. Tallinn.
[10] Klatt, D.H. 1973. Interaction between two factors that influence vowel duration. JASA, 54, 1102-1104.
[11] Krivnova, O.F. 1997. Modeling and synthesis of phrase intonation on the basis of target points of pitch contour. In Narin'yani A.S. (ed.), Proceedings of the International Workshop on Computational Linguistics and its Applications "Dialogue97". Moscow. (In Russian).
[12] Zinovieva, N.V. 1995. Phonetically sufficient allophonic database for concatenation synthesis of Russian speech. Proceedings of the XIIIth ICPhS, 2, 358-361. Stockholm, Sweden.
[13] Babkin, A.V. 1998. Automatic synthesis of speech - problems and methods of speech signal generation. In Narin'yani A.S. (ed.), Proceedings of the International Workshop on Computational Linguistics and its Applications "Dialogue98". Kazan: Hater. (In Russian).