Dependency Parsing vs NGrams

I did a project this time for my Natural Language Processing class, on using both dependency parsing and ngrams to build a language models.

The key idea that motivated me and my friend Ikhyun was that NGram model was having trouble (as we thought) with long dependencies and with sparse data. So we wanted to see a hybrid model, consisting of both dependency model and ngram model, would be any better than ngram model alone.

We build our two language models and tested with various kinds of data. Initially we were not impressed by the results as it was no improvements over ngram model.

Then we tested with bad sets of sentences. These sentences were created shuffling original sentences. But our hybrid model didn't perform very well compared to ngram model alone, as we expected. We tested our results with a special perplexity measure. Since it failed, we drew graphs to see how models performance with ngram model. Both approaches showed no or little improvement.

During the discussion with Professor Gasser, we came up with some interesting ideas for the failure.
1. We used English sentences for this experiment. Perhaps English might not be too rigid about word order and this might be one factor. So carrying out the same experiment with a language like Spanish might give us some better results. But the problem was neither me or Ikhyun speaks or understands Spanish :(

2. The sentences we used to train and test the system didn't have long dependencies within them. But we wanted to have a model which would perform well, irrespective of the nature of the sentences. Actually we identified that dependency model was capturing the same information that ngram model was capturing. When we graph our probability results from the experiment, this was so obvious to us. Even though these two models rely on two different concepts, these observations were really interesting.

3. The worst one. We used Stanford dependency parser for our experiment to get dependency outputs. Being a statistical parser, this was always trying to give the best possible output, irrespective of the nature of the input sentence. For example, this was treating both "I go to school" vs "I to go School" the same way. We were expecting this parser to output something wrong, but since it was outputting correct even for the wrong ones, we were having trouble with our model.

I think we learnt a lot from these experiences. Perhaps we should try to test our model with a different dependency parser and a different language.