bug: 6669677

Change-Id: I4627093811a19c46ce13fe351d1db63cbd78cf4a

Signed addresses are used only in version 3 with dynamic update.

bug: 6669677

Change-Id: Iadaeab199b5019d2330b4573c24da74d64f0945e

bug: 6669677

Change-Id: I82799af199358420f09ac34fc005091e202c5d3b

bug: 6669677

Change-Id: I1a5b90ee05e5cffd74a5c140384a3e37c79e7e70

Change-Id: If04d779db23b1aea2cc12e5e9b8cecfcb35a5737

Change-Id: I45830235ee738233e8eb2bd91d659705b698f58c

Change-Id: Ia5eaae0653179b2eb74c53b0823beaf80377a389

Change-Id: Ic9ff717a9751023d47b02ff3b9d1fbf3115c2501

Change-Id: Ib2f194775cfe5ab05481ac95cd709d6e8e8dd3c6

Change-Id: I60feda815ea08cf73300fccca1ae12b97550f116

Change-Id: Ifc4c64c9cffe4f343c5a604c192db010a1792acc

Change-Id: Idb4b635fcac70cc988e0dd3ce3bf121fba12099c

Change-Id: I232e35598635113bf2c81825669c744aadc79efe

Change-Id: Ifd66bda7d528827ba61c60531121ea206a2325be

Change-Id: Iac0b35d2da05e81237d105e8fe13c56d16038de1

Change-Id: Ia99411d4009857d5e420ca87ef8acf1f1826d3ed

Change-Id: Id98d18e90a8b83b597507728b467f56888c8fd12

Change-Id: Ibad05a5f9143de1156b2c897593ec89b0a0b07e7

Also add the blacklist attribute

Bug: 7005742
Bug: 2704000
Change-Id: Icbe60bdf25bfb098d9e3f20870be30d6aef07c9d

Change-Id: I8a70e43b23f65b5fd5f0ee0b30a94ad8f5ef8a8a

Change-Id: I7967f11211221d4877bf0a0c30183af885f45390

Change-Id: I5be5d62a63ca897e36fe93200ffdca6befb363aa

Change-Id: I8640c994231d5f46bc6e074ce8a5bf5344fed0aa

Change-Id: Ia71561648e17f846d277c22309ac37c21c67a537

Bug: 7005813
Change-Id: Ie0d8d4b2d5eb147838ca23bdd5ec1cecd4f01151

Change-Id: Iff20dcb9ca0d6064bb118247887fe24b812c0c61

Bug: 7005645
Change-Id: Ifd942b3ce242aeeec512e132e1cee31329e994b1

Change-Id: Icd540742073d49d12e70b2d8bd99aaf7ccb5802d

Change-Id: I03236386aea13fbd4fb8eaeee18e0008aa136502

Change-Id: I604da15e65fc3cf807ec4033df4e4cd5ef0196fc

Bug: 6313806
Change-Id: Id0779bd69afae0bb4a4a285340c1eb306544663a

Performance gain is < 2%

Bug: 6394357
Change-Id: I2b7da946788cf11d1a491efd20fb2bd2333c23d1

Bug: 6394357
Change-Id: I00ba1b5ab3d527b3768e28090c758ddd1629f281

We don't merge tails anyway, and we can't do it any more
because that would break the bigram lookup algorithm.
The speedup is about 20%, and possibly double this if
there are no bigrams.

Bug: 6394357

Change-Id: I9eec11dda9000451706d280f120404a2acbea304

This also tests for bigram frequency against unigram frequency

Bug: 6313806
Bug: 6028348
Change-Id: If7faa3559fee9f2496890f0bc0e081279e100854

Rename it, rename parameters, and add a parameter that will
be necessary soon.
Also, rescale the bigram frequency as necessary.

Bug: 6313806
Change-Id: I192543cfb6ab6bccda4a1a53c8e67fbf50a257b0

This is a cherry-pick of Icb602762 onto jb-dev.

Bug: 6355745
Change-Id: Icb602762bb0d81472f024fa491571062ec1fc4e9

This is a cherry pick of I14b67e51 on jb-dev

Bug: 6340915
Change-Id: Iaa512abe1b19ca640ea201f9761fd7f1416270ed

This is not the Right fix ; the Right fix would be to read
the file in a buffered way. However this delivers tolerable
performance for a minimal amount of code changes.
We may want to skip submitting this patch, but keep it around
in case we need to use the functionality until we have a good
patch.

Change-Id: I1ba938f82acfd9436c3701d1078ff981afdbea60

The core reason for this is quite shrewd. When a word is a bigram
of itself, the corresponding chargroup will have a bigram referring
to itself. When computing bigram offsets, we use cached addresses of
chargroups, but we compute the size of the node as we go. Hence, a
discrepancy may happen between the base offset as seen by the bigram
(which uses the recomputed value) and the target offset (which uses
the cached value).
When this happens, the cached node address is too large. The relative
offset is negative, which is expected, since it points to this very
charnode whose start is a few bytes earlier. But since the cached
address is too large, the offset is computed as smaller than it should
be.
On the next pass, the cache has been refreshed with the newly computed
size and the seen offset is now correct (or at least, much closer to
correct). The correct value is larger than the previously computed
offset, which was too small. If it happens that it crosses the -255 or
-65335 boundary, the address will be seen as needing 1 more byte than
previously computed. If this is the only change in size of this node,
the node will be seen as having a larger size than previously, which
is unexpected. Debug code was catching this and crashing the program.

So this case is very rare, but in an even rarer occurence, it may
happen that in the same node, another chargroup happens to decrease
it size by the same amount. In this case, the node may be seen as
having not been modified. This is probably extremely rare. If on
top of this, it happens that no other node has been modified, then
the file may be seen as complete, and the discrepancy left as is
in the file, leading to a broken file. The probability that this
happens is abyssally low, but the bug exists, and the current debug
code would not have caught this.
To further catch similar bugs, this change also modifies the test
that  decides if the node has changed. On grounds that all components
of a node may only decrease in size with each successive pass, it's
theoritically safe to assume that the same size means the node
contents have not changed, but in case of a bug like the bug above
where a component wrongly grows while another shrinks and both cancel
each other out, the new code will catch this. Also, this change adds
a check against the number of passses, to avoid infinite loops in
case of a bug in the computation code.

This change fixes this bug by updating the cached address of each
chargroup as we go. This eliminates the discrepancy and fixes the
bug.

Bug: 6383103
Change-Id: Ia3f450e22c87c4c193cea8ddb157aebd5f224f01