Research on business English grammar detection system based on LSTM model

2.1 Word segmentation technology

After dividing a paragraph into single sentences, we also need to extract the words from a sentence, which is the smallest unit of language study. Therefore, word segmentation is a very critical step, which is related to the problem analysis of the whole sentence, and is an important step in text processing.

English word segmentation is relatively simple compared with Chinese, because English itself has more obvious characteristics, there are spaces or other symbols between the words. However, English word segmentation is also difficult, and there are many ambiguous scenes. For example, abbreviations of words: it’s and couldn’t, it’s can be used as a word or split into it is, couldn’t can be split into could not or as a separate word. Similarly, the representation of the number can also affect the segmentation, for example, 222333 can be written as 222,333 or 222.333, and the use of different punctuation marks to represent the number causes some ambiguity. There are also certain words, such as New York and Los Angeles, which should be treated as a complete word in word segmentation, and should not be separated to facilitate part of speech tagging [10,11].

At present, the commonly used word segmentation methods mainly include the following types: rule model, which defines a series of word segmentation rules combined with regular expressions to divide words in sentences. The other is the word segmentation technology based on statistical probability model. Based on the corpus and some commonly used language models such as N-gram and Hidden Markov model, the segmentation results are compared and analyzed to select the most reasonable way of word segmentation [12].

2.2 Part-of-speech tagging

Part-of-speech tagging is a common technique in natural language processing. Part-of-speech tagging is to label parts of speech for each word unit on the basis of word segmentation, such as verbs, nouns, adjectives, adverbs, and so on. The grammatical correctness of a sentence is closely related to the part-of-speech of a word. Some grammatical errors are caused by the wrong tense of a word. The biggest difficulty in part-of-speech tagging is to solve various ambiguity problems. Words and parts of speech are not all one-to-one correspondences, and some words have different parts of speech in different scenes. For example, here are two examples:

“I bought a book yesterday.”

“I have booked a seat at the restaurant.”

The book in the first sentence is the noun “book” and the verb “book” in the second sentence. It is not reasonable to distinguish the parts of speech only according to morphology, and the general processing method will label the parts of speech according to the context. The above example is only a typical example of ambiguity in part-of-speech tagging. Brown corpus is one of the most famous corpora in natural language processing. According to the statistics of part-of-speech tagging results, there are 4,100 words with multiple tagging and 35,340 words with only one tagging, which shows that ambiguity in English part-of-speech tagging is very common. The important task of part-of-speech tagging is to deal with these ambiguous scenarios.

There are two types of part-of-speech tagging: regular model and statistical model. Most of the early adopters were rule models, in which linguists defined the part-of-speech candidate set and formulated a series of part-of-speech tagging rules according to their understanding and familiarity with the language [13]. This method relies heavily on the linguist’s knowledge of the language, and English is a very complex language, and thousands of rules may be formulated for some large corpora. The work is cumbersome, consumes a lot of manpower and material resources, and after too many rules, it is difficult to ensure correctness and consistency, and there may be consistency and inconsistency. The rule model has to do special processing for each language, and the work is extremely tedious. Therefore, the annotation method of the rule model is rarely used at present, at most as an auxiliary method.

Compared with the rule model, statistical method is much simpler. The development of statistical model is closely related to the construction and improvement of corpus [14]. According to the corpus and related algorithms, a tagger is first trained and learned, and then annotated. Although there is also the problem of manual initial labeling of data, the labeling here is much easier than making rules, and inconsistency before and after labeling is allowed because probability is used. Commonly used methods include Viterbi algorithm, CLAWS algorithm, and improved VOLSUNGA parts-of-speech tagging algorithm [15]. They all use Markov hypothesis principle, and apply dynamic programming technology to reduce the complexity of the algorithm and improve the speed of annotation. On the whole, the statistical model has lower requirements on the original labeled data, and the labeling process is simple, which has greater advantages than the regular model.

2.3 Syntax error correction algorithm

Syntax analysis is a common technique in natural language processing, and open source tools can perform syntax analysis and build corresponding syntax trees. Generally speaking, if the construction of a sentence’s grammar tree fails, then the sentence should exist. Finally, according to the voting importance results, the grammar corrected results were obtained. Of course, syntactic analysis also has limitations and shortcomings [16]. The current syntactic analysis ability is limited, and the analysis ability of complex clauses is limited. For some special sentences, it is necessary to add parsing rules. One of the most important problems with syntax-based language is to establish a complete grammatical system for the language, which requires a lot of work by language experts. At present, the theories of grammar system construction by language experts are not consistent, and there are great controversies on some difficult points. This also leads to different results in the syntactic analysis of some sentences, and there is ambiguity.

The rule-based approach is to build the rule base first, and then match the input word sequence according to the rules. If the sentence has a syntax error, it will not match the rules in the rule base. Some rule bases are also built based on the parts of speech sequence of words, but the basic principle is no different from the word sequence principle. The word check feature of Microsoft Word uses the rule base method and can check multiple languages. And its error detection function is more powerful, for common word spelling errors, phrase errors and conjunction errors can be recognized, and can give suggestions for modification. Rule based syntax error correction is very convenient to implement, search and match according to the rules on the line, from the perspective of system implementation is simple and fast. In CoNLL2014, many papers mentioned the use of rules for grammar error correction, which is more used as an auxiliary function [17]. The biggest limitation of rule grammar correction is that rules are limited, grammar errors are complicated, English grammar errors are difficult to count, and with the expansion of the rule base, the cost of adding rules is getting higher and higher. It becomes more difficult to conclude new rules, there may be conflicts between rules, and the proliferation of rules cannot be stopped.

2.4 LSTM model

When tagging sequence data, the tagging information of the current word generally depends on the context information. Traditional sequence labeling methods rely on statistical or fusion methods, while recurrent neural networks can extract sequence information well by establishing sequence relationships of hidden layers. By setting threshold units and cells, LSTM can effectively avoid the problems such as gradient disappearance and gradient explosion that may occur in the training of traditional recurrent neural networks [18].

Compared with one-way LSTM model, which only accumulates the information before the current moment, two-way LSTM can accumulate the context information of the current moment, so that the model can synthesize the context information for sequence annotation. Bidirectional LSTM processes data from both directions by using two separate hidden layers, both of which are then fed forward to the same output layer. The whole network structure is shown in Figure 1.

Figure 1

Schematic diagram of bidirectional LSTM network.

Through time T to 1 and time 1 to T, the bidirectional LSTM network uses the formulas (1)–(3) and the input vector sequence x = ( x 1 , ⋯ , x N ) to calculate the feedforward hidden layer sequence h → = ( h → 1 , ⋯ , h → N ) , feedback hidden layer sequence h ← = ( h ← 1 , ⋯ h ← N ) , output sequence y = ( y 1 , ⋯ , y N ) , and network output layer, respectively.

where W represents the weight matrix; b is the bias vector; and f is the hidden layer function, specifically f ( u ) = 1 1 + e − u .

3.1 N-gram search service and knowledge base

The syntax error correction strategy is based on N-gram frequency statistics, so it is necessary to establish N-gram search service first. The N-gram sources used were all news for 2020 from about 12,000 news sites, with statistics shown in Table 1.

In order to improve its query efficiency, the open source search engine is used to establish inverted index and provide search services.

Both articles and prepositions have limited variations and are in closed sets. Build a finite confusion set for articles and prepositions. The article confusion set contains three cases: the, a/an, and null. Null indicates that no article is used. The preposition confusion set contains the common 17 prepositions: on, about, into, with, as, at, by, or, from, in, of, over, to, among, between, under, and within. Unlike articles and prepositions, nouns and verbs are open sets. Therefore, the variation tables are established for noun errors, verb forms, and subject-verb inconsistency errors, respectively.

3.2 Moving windows and N-gram voting strategies

The GEC method for confounding set as open set is based on moving window and N-gram voting strategy.

The moving window is defined as follows:

where ω i is the ith word in the sentence, k represents the window size, and j is the distance between the first word in the window and ω i . The selection of the window size k and the value range of j directly affect the effect of GEC. For different error types, different k and j values are selected.

The N-gram voting strategy simulates a real-life voting mechanism, with an N-gram fragment containing a syntactically incorrect candidate representing a candidate who may have the right to vote. Due to the limited corpus, the frequency of N-gram fragments may be very sparse. This policy sets a minimum effective frequency. Only when the queried frequency is higher than the minimum effective frequency, the N-gram fragment has the voting right.

In real life, the importance of votes cast by different people in different fields is different, and the importance of votes cast by experts in fields is higher than that of ordinary people. This strategy uses N-gram fragment length to simulate the degree of expertise of the domain expert, and the longer the N-gram, the higher the importance of the vote. Finally, according to the voting importance results, the grammar corrected results were obtained. The specific algorithm is as follows:

Input: N-gram fragment set Nset[window][candidates], least significant frequency set Fset[window], weight set Wset[window], N-grams Search service N-gram.

Output: Vote result set Rset[candidates].

from window = 3 to 5:

maxFrep = 0, vote = null

for candidate in Nset[window]:

   freq = N-gram.getFreq(candidate)

   If(freq > = Fset[window]andfreq > maxFreq):

      maxFreq = freq

      vote = candidate

Rset[vote] + = Wset[window]

return Rset.Max

Fset and Wset are parameters. Fset is the minimum effective frequency. Voting can be performed only when the queried frequency is greater than Fset. Wset is used to adjust the voting weights of N-gram fragments of different lengths.

This algorithm is based on the corpus, because on the one hand, the corpus size is limited and it is impossible to contain all fragments, and on the other hand, there are noise data in the corpus. Therefore, the minimum effective frequency Fset is set. Only when the frequency of the queried N-gram fragment is greater than this frequency, can it be shown that the corpus contains relevant corpus and this N-gram fragment has voting rights. According to the experimental comparison, Fset is set to 100.

The frequency of the N-gram segment with voting rights represents the probability of changing to the corresponding result. Suppose that the frequency of “have an apple” is 2, the frequency of “have the apple” is 1, and the frequency of “have apple” is 1. Then, according to the corpus, the probability of changing to “have an apple” is greater than “have the apple” and “have apple.” The voting strategy is to select one as the voting object from “have an apple,” “have the apple,” and “have apple.” This strategy selects the one with high probability as the voting object.

3.3 Annotation correction strategy based on bidirectional LSTM

Considering article and preposition errors as a special sequence labeling task, this study proposes a GEC method for sequence labeling based on LSTM. First, the training corpus with part-of-speech tagging is preprocessed, and the article part of speech is replaced by a special mark “ART” and the prepositional part of speech is replaced by a special mark “TO,” and the above marks are exchanged with the positions of the articles or prepositions. Then, the model is trained using LSTM. Finally, the trained model is used to test the data. The labeling model based on LSTM is shown in Figure 2.

Figure 2

Labeling model based on LSTM.

The model first converts the word into a word vector as input to the model, entering the word vector at the corresponding position in the sequence at each moment. During the training process, word vectors are updated as parameters. The model uses the word vector as input to the LSTM unit and, at each moment, outputs the corresponding annotation vector. The tagging set is the union of all parts of speech sets and all confounding sets of prepositions or articles. The dimension of the annotation vector is consistent with the total number of annotation sets, and the annotation with the highest selection probability is marked by SoftMax.

The labeling model relies on back propagation through time algorithm and uses stochastic gradient descent for supervised training.

In this study, the grammatical errors with fixed confusion sets, namely, articles and prepositions, are marked and corrected. Before labeling, the article or preposition in the sequence is represented by a uniform identifier. When marking, the uniform identifier is marked with specific prepositions or articles to realize the correction of article or preposition errors in grammar.

3.4 Article error identification and correction

Article errors mainly include: misuse of articles, redundancy of articles, and absence of articles. In this study, article error correction is regarded as a special sequence labeling task, which involves three sub-modules: article error preprocessing module, article error recognition and correction module, and article error post-processing module.

1. Article error preprocessing module.

   The part of speech is replaced by a special mark “ART” to swap the part of speech with the position of the article. In this way, all places in the sentence where the article appears are replaced with “ART,” and the corresponding part of speech is modified to indicate the article that should appear there.

2. Article error recognition and correction module.

   According to the given sentence, it determines the position where there may be incorrect use of articles in the sentence, makes part-of-speech tagging of the sentence, and then identifies all parts of speech labeled as articles (a, an, the). Article errors are identified and corrected using LSTM-based sequence annotation method. First, each word in the input sentence is converted into a word vector representation. Then, through the two-layer LSTM model, the annotation results are obtained.

3. rticle error post-processing module

   Swap the word with the special mark “ART” in the result of the previous step with the part-of-speech mark, and then delete the part-of-speech mark to get the final output result.

3.5 Identification and correction of preposition errors

Preposition errors mainly include: misuse of prepositions, redundancy of prepositions, and absence of prepositions. Preposition error correction is regarded as a special sequence labeling task, which involves three sub-modules: preposition error preprocessing module, preposition error recognition and correction module, and preposition error post-processing module.

1. Preposition error preprocessing module

   Replace the prepositional part of speech with a special mark “TO,” switching the part of speech with the position of the preposition. In this way, all places in the sentence where the preposition occurs are replaced with “TO,” and the corresponding part of speech is modified to indicate the preposition that should occur there.

2. Preposition error recognition and correction module

   According to the given sentence, it determines the possible positions in the sentence where the preposition is used incorrectly, tags the parts of speech in the sentence, and then identifies all the parts of speech labeled as prepositions. The sequence annotation method based on LSTM was used to correct preposition errors.

3. Preposition error post-processing module

   Swap the word with the special mark “TO” in the result of the previous step with the part-of-speech mark, and then delete the part-of-speech mark to get the final output result.

3.6 Noun single and plural error correction

Based on the noun-single and plural change table and N-gram voting strategy, the module mainly corrects the misuse of noun-single and plural. The specific correction process of this module is described as follows:

1. The part-of-speech sequence of the example sentence is obtained by part-of-speech tagging, and the error candidate set is obtained by extracting the words labeled NN and NNS.

2. The corresponding set of correction candidates is obtained by using the table of single and plural noun changes.

3. Based on the correction candidate set, the N-grams fragment set is obtained using a moving window with a size of 3–5. Use the N-gram voting strategy to get the correct candidate with the highest number of votes, and replace the original sentence.

3.7 Verb and subject-verb inconsistency error correction

The verb error correction module mainly corrects the verb form misuse and subject-verb inconsistency. This module relies on verb form conjugation table, verb single and plural conjugation table and N-gram voting strategy. The specific correction process is as follows:

1. The parts of speech sequence is obtained by marking the parts of speech of the words in the sentence. For verb form errors, the words labeled as VB, VBD, VBG, and VBN are extracted as error candidates. For subject-verb inconsistency errors, the words labeled VBP and VBZ are extracted as candidates for errors.

2. The correction candidate set of the error candidate is obtained according to the error candidate and verb form change table/verb single and plural change table.

3. For the correction candidate, the collection of N-grams fragments is obtained using a moving window with a size of 3–5. Use the N-gram voting strategy to get the correct candidate with the highest number of votes and replace it in the original sentence.

The GEC method system architecture based on LSTM and N-gram is shown in Figure 3.

Figure 3

GEC architecture diagram.

4.1 Experimental data

The experimental data came from the GEC evaluation task of CoNLL2013, and the statistical results are shown in Table 2. The CoNLL2013 corpus does not have correct parts-of-speech tagging, and the CoNLL2013 training corpus is smaller than PIGAI parts-of-speech tagging corpus and Brown corpus. Therefore, when the PIGAI parts-of-speech tagging corpus, Brown corpus, and labeled CoNLL corpus were used to expand the LSTM training corpus tagging, the Stanford tagging tool was used for parts-of-speech tagging. Among them, CoNLL2013 and PIGAI corpus are used as ESL corpus, and Brown corpus is used as supplementary news corpus to participate in the training of the model.

The GEC evaluation task data of CoNLL2013 marked a variety of error types, but the evaluation task mainly targeted at the five error types with a relatively high proportion: article error, preposition error, noun error, subject-verb agreement, and verb form error.

4.2 Evaluation method

The evaluation standard of CoNLL2013 is F ₁, which is defined as follows:

where P and R represent the accuracy rate and recall rate, respectively, defined as follows:

where N correct refers to the correct number of syntax errors that the system modifies, N predicted refers to the number of syntax errors that the system modifies, and N target refers to the number of errors in the corpus itself.

4.3 Experimental results and analysis

The experimental results of the GEC method based on LSTM and N-gram on the GEC evaluation data of CoNLL2013 are shown in Tables 3–5, and are compared with the corp-based methods of GEC and correction of English articles and UIUC methods.

As shown in Table 3, for the correction of article errors, the F ₁ value of the proposed method is 5% higher than that of the UIUC method and 5% higher than that of the Corpus GEC method. For the correction of preposition errors, F ₁ value of the proposed method is 21% higher than that of UIUC method and 13% higher than Corpus GEC method, indicating that the LSTM-based GEC method is effective for article and preposition syntax error correction tasks. This is because the word vector contains the main rich context information, and using LSTM better learns the long-term dependency information that determines the use of articles or prepositions, so the results are better.

As shown in Table 4, when only N-gram + vote voting strategy is used to correct noun and verb errors, there is still a certain gap between F ₁ value and UIUC method. This is because the news corpus on which the N-gram + vote strategy is based is different from the article that needs to be corrected, and a large number of correct sentences are changed into incorrect ones. The table of noun and verb conjugations cannot cover all the conjugations of nouns and verbs, which leads to certain limitations in the correction of nouns and verbs.

As shown in Table 5, for the correction of all five types of errors, the proposed method is superior to the UIUC method. The total F ₁ value in the GEC data of CoNLL in 2013 is 33.87%, exceeding the total F ₁ value of UIUC by 31.20%. The experimental results show that the automatic correction method based on LSTM and N-gram is effective.

In order to verify the accuracy of the automatic grammar correction system based on bidirectional LSTM and N-gram, the RNN based Sep2Sep model, LSTM-based Sep2Sep model, CNN-based Sep2Sep model, nested attention neural model, and deep context model are further introduced for comparative analysis. The syntax error correction effects of the six error correction models are shown in Table 6.

As can be seen from Table 6, the automatic syntax correction system based on bidirectional LSTM and N-gram in this study has the highest detection accuracy, with F ₁ value of 46.00, and P value and R value of 66.78 and 35.09, respectively, significantly superior to other error correction models.