Testing Grammarly's Grammaticality Judgements

I was looking back fondly over my past rejected job applications, when I came across a cover letter that I wrote to Grammarly in 2017, a grammar-checking app that came to prominence due to their aggressive YouTube advertising campaigns.   They were advertising for a job for linguists, aimed in particular at people working on natural language processing.  I used Grammarly for a few days, and had some fun devising tortuous variants of ungrammatical sentences to test what it could do.  It does many things well, but it also makes bizarre mistakes that reveal that the app is not attempting to parse the sentences, or was doing so very superficially, and on which it is out-performed by existing parsers such as the Google Natural Language API or the Stanford Parser.  I wrote these points up in a cover letter and sent it to them, which got me invited to an uninformative Skype interview and a polite rejection some time later.  What follows is a brief summary of these points.

Grammarly corrects ‘I am disappointed of course’ to ‘I am disappointed in course’ (treating ‘disappointed of’ as a collocation error).  At the time it also corrected ‘Last year we have been to Thailand’ to ‘Last year we were to Thailand’ (this issue has now gone).  By themselves these problems are minor, but may be part of a large class of ambiguities that could be tackled by having a probabilistic parse of the sentence, such as recognising that ‘of course' normally occurs as an adverbial phrase, or ‘been’ as the past participle of ‘go’ if its object is a prepositional phrase.

I made four sentences below, all of which are ungrammatical in that they have a plural verb (‘are’) where the subject (‘impact’) is singular.  Grammarly only detects the agreement error in sentences 2 and 3, marked with an asterisk.  Its parsing is affected by irrelevant features such as the choice of verb in the subordinate clause, ‘predict’ in (1) compared with ‘expect’ in (3).    It is also thrown by the irrelevant subject of the verb in the subordinate clause, such as ‘theories of economics’ in (4). 

1. The impact, which people predict are bound to be felt sooner or later, could be enormous.
2. *The impact, which they predict are bound to be felt sooner or later, could be enormous.
3. *The impact, which people expect are bound to be felt sooner or later, could be enormous.
4. The impact, which theories of economics expect are bound to be felt sooner or later, could be enormous.

  I used another four sentences to show that it gets confused by different adjuncts, such as ‘maneuvering in markets for oil’ in (1) and ‘maneuvering in markets for oil and coffee’ in (2)).  Again, all of these sentences are ungrammatical ('have' should be 'has', to agree with 'manuevering'), but Grammarly only recognises (1) and (4) as ungrammatical.

1. *The maneuvering in markets for oil have brought billions in profits to banks. 
2. The maneuvering in markets for oil and coffee have brought billions in profits to banks.
3. The maneuvering in markets for oils have brought billions in profits to banks.
4. *The maneuvering for oils have brought billions in profits to banks.

The problem seems to be that Grammarly is not attempting to parse the sentences.  Parsers such as the Google Natural Language API and Stanford Parser are both able to do correctly detect the dependency between 'impact' and 'could', as in their respective trees below.

Parsing each sentence of a text before suggesting corrections may be computationally intensive, but is desirable for detecting errors of agreement in long-distance dependencies, precisely the type of error that writers are most likely to make: the above sentences are variants of two errors that have appeared in print, from Steven Pinker's The Sense of Style (91-92).  Grammarly in fact happens to be slower anyway, taking five seconds to produce an incorrect answer compared with the Stanford Parser and the Google Natural Language API, which each gets the correct answer in under two seconds.

Other mistakes that Grammarly does not pick up on include incorrect uses of definite articles for generic nouns (‘Some of us consider the money as the force which controls our lives’), for indefinite nouns (‘I live in the small apartment in the suburbs’), or in particular idioms (‘to put it in the nutshell’).  Mistakes involving tense are also not picked up on (‘It is raining for two days already’, ‘I go to France tomorrow’).  Problems such as these may have to be tackled by having not only a probabilistic parse of the sentence, but some attempt at probabilistic semantics, such as the knowledge that ‘money’ is often a generic noun; that ‘in a nutshell' is an idiom; and estimating the intended tense and aspect of a sentence given the adverbs used (‘tomorrow’, ‘already’) and semantic properties of the verb (‘raining' cannot be scheduled for a duration, unlike ‘going’ in ‘He is going for two days’).  

Grammarly seems to be a useful tool in some cases, but lacking dependency parsing, knowledge of collocations, or rudimentary semantics that would be needed to detect mistakes that people are most likely to make.  I was reminded of these issues by Douglas Hofstadter's recent article here in which he plays around with Google Translate and makes similar points about its relative shallowness, despite its impressive advances (e.g. see this essay).  

I deregistered from Grammarly soon afterwards, partly because of the weekly spam emails which border on desperate if you do not use it ( "Anybody home?"  "This is awkward.  You seem to have stopped using Grammarly on your browser", "We miss you already"), and which give you patronising compliments if you do, one of them given below to finish on just because it suggests a comical lack of semantics in their language software more generally.