Just skipping historical information fields.

Bug: 11281877
Change-Id: I43d2adaa576b7da11ed3ca54990265dbb6f53b08

Change-Id: I468994c98d091be621b9fb3fbe6405c67fc6a465

Change-Id: I1b8eb6ab2ea797d2590495b1f57f9ec9560ea817

Bug: 11245133
Change-Id: I0907a091ac3ae960eaf3b27da78dbb48a24b2ea1

- Version 3 is not supported
- Now passing the right string to open v4 dicts. Fix the tests for this.

Change-Id: I7829330c3568a715b96396ba4e4e69c6e17775ab

This simplifies the code quite a bit.
- GERMAN_UMLAUTS are now handled through a key-value attribute.
  The dictionary generator does not need to know about it any more.
- FRENCH_LIGATURES are deprecated as we handle them with shortcuts now.
- CONTAINS_BIGRAMS is deprecated. Bigram processing is always applied
  regardless of this flag.

Bug: 11281748
Change-Id: If567e52e245a9342adc7f3104a0f7d8d782df8c1

The bulk merge from -bayo to klp-dev should not have been merged to master.

Change-Id: I527a03a76f5247e4939a672f27c314dc11cbb854

This simplifies the code quite a bit.
- GERMAN_UMLAUTS are now handled through a key-value attribute.
  The dictionary generator does not need to know about it any more.
- FRENCH_LIGATURES are deprecated as we handle them with shortcuts now.
- CONTAINS_BIGRAMS is deprecated. Bigram processing is always applied
  regardless of this flag.

Bug: 11281748
Change-Id: I55a11ba61d3589c1584a3fa6c941374b349b7b5c

Bug: 11281748
Change-Id: Ia2490d18bf159990310e10ed45fb6b3a0dee1c5f

This fixes a few bugs that became apparent with the
reinstating of a test.

Bug: 11954718
Change-Id: I1127bc3180f856566429f40d14c944e6f0007e09

Bug: 11073222
Change-Id: I76e47d0399cf43fc3cc18cb1252f166be86b9a69

Bug: 11772864
Change-Id: I3ba59fbb3fed226a1f4b80f33c70e168b3b2b7c5

Change-Id: I95c5fb58e2cab3a2c523d2bc3c6fbad06692ac59

In new format, each bigram entry has flags (1 byte), a terminal id (3 byte),
a time-stamp (4 byte), a counter (1 byte) and a level (1 byte).

Bug: 10920255
Bug: 10920165
Change-Id: I0f7fc125a6178e6d25a07e8462afc41a7f57e3e1

Change-Id: I986ab26faf535fc4bc98443053f534eced9d048f

Change-Id: I2208378b33038771b460abb33f9a690872e998e2

Change-Id: Ic586e46e5a9f59de53d53e59886d635345940974

Change-Id: Iaf1556fec194d17cb4318f2bdcc837f8d79449ef

When there are too many bigrams, we stop reading the file,
so the file pointer is in an inconsistent place. This means we
have no idea what's going to happen next. It's better to crash
right away.

Change-Id: Id3b7b78cbe4fda3493b3c9c46758763e1ab5f6a3

Change-Id: Idd9176c9943dfacac5a06957f1a07187b642b207

Bug: 9618601
Change-Id: I43c5840505c6a847aaf4893a400392ccd45903c0

Bug: 6669677

Change-Id: Id2154db47adea2929559a4187a726f9dfa83363e

Bug: 9618601
Change-Id: I161d2845906f07c1251deb8005fdffe49c5b7940

Bug: 9618601
Change-Id: I173100ac704c03f7d5d0d53477e83cab5d1110d4

Change-Id: I356adb72047ebc43c924fbff1ff45e7460508a31

Change-Id: I259db91d837c67cbcb3b6dc504b21dca23a6a5be

Change-Id: I9d04f64a58f5481cbb64cf1c09b5c485dd4176b4

Bug: 10233675
Change-Id: I7b0eb07d195cd386cd0d9e97cd59bf48fcf24107

Bug: 10434720
Change-Id: I14690a6e0f922ed1bab3a4b6c9a457ae84d4c1a4

Change-Id: Ifb1cc01fa5bc2d6d69671f1acb9b9675a4081d32

Change-Id: I1d6b086f1d9f0dbd8d74f964e29ae62c533af978

Change-Id: I41049b9118b58838e5dedf8e5618d939ca70c5ef

Change-Id: I8939fdfb4f79e55bcd7393633784effb30df3f8f

Change-Id: I4dabf17da7003b1d8204a83dbd10e5be6e8fd805

Change-Id: Ic918822fc1b3a8a7c39ffbcf7defde2c5bf888db

Change-Id: Ibf9b95b658df6e2c2218bdb62e2380f326a03832

Change-Id: I1a1830aaa8ea586b68fc34ff3a27ae52b810e8af

BinaryDictReader -> BinaryDictDecoder.
BinaryDictDecoder -> BianryDictDecoderUtils.

Change-Id: Iadf2153b379b760538ecda488dda4f17225e5f37

Change-Id: I298f86b70d18cd08b240509b6f757c72e1a59ffe

Change-Id: Ib104d5de71c2ab1a07921b407c74c21b0409d9af