Fix the rate estimate in the deblocking offset search

[yeqing-wu]

search_filter_offsets() had two problems with the rate:

• The rate was passed to RDCOST_DBL_WITH_NATIVE_BD_DIST as plain bits, which expects (bits << AV2_PROB_COST_SHIFT) with distortion as SSE << 4. The distortion here is the plain SSE, so the rate is now (bits << AV2_PROB_COST_SHIFT) >> 4. Before, the rate was almost free.
• The bit count did not follow encode_loopfilter(). Each coded offset has a 1-bit flag, plus df_par_bits when the offset differs from its reference: 0 for the vertical luma offset and for chroma, and the vertical offset for the horizontal luma offset. The old count left out the flags, and for the vertical luma search it coded the horizontal offset against 0 and the vertical one against the horizontal one, which is the reverse of the writer.

The results below are with both this change and "Fix inter tx type cost table fill for small tx sets" (commit: 5315fe2).

Anchor: commit 6169621
Speed 0 (cpu-used=0): VCWG CTC (130 frames, RA and LD)

1) RA
  +------------+-------+-------+-------+-------+------+------+
  | Class      |     Y |    Cb |    Cr |  wAvg | Enc% | Dec% |
  +------------+-------+-------+-------+-------+------+------+
  | A1         | -0.09 |  0.01 | -0.12 | -0.09 |  100 |   99 |
  | A2         | -0.15 |  0.32 | -0.02 | -0.12 |  100 |   98 |
  | A3         | -0.09 |  0.87 |  0.25 | -0.04 |  100 |   99 |
  | A4         | -0.21 |  0.29 |  1.55 | -0.12 |  100 |   97 |
  | A5         | -0.07 |  0.26 |  0.33 | -0.05 |  100 |   99 |
  | B1         | -0.07 |  0.03 |  0.39 | -0.05 |  100 |   99 |
  | B2         |  0.07 | -0.06 |  0.05 |  0.07 |  100 |  100 |
  | Avg w/o B2 | -0.12 |  0.29 |  0.28 | -0.09 |  100 |   98 |
  +------------+-------+-------+-------+-------+------+------+

2) LD
  +------------+-------+-------+-------+-------+------+------+
  | Class      |     Y |    Cb |    Cr |  wAvg | Enc% | Dec% |
  +------------+-------+-------+-------+-------+------+------+
  | A2         | -0.17 |  0.48 |  0.48 | -0.12 |  100 |   99 |
  | A3         | -0.19 |  0.11 |  0.02 | -0.17 |  100 |   98 |
  | A4         | -0.06 |  1.38 |  1.81 |  0.07 |  101 |  100 |
  | A5         | -0.59 | -0.33 | -0.34 | -0.57 |  100 |   97 |
  | B1         | -0.10 |  0.88 |  0.67 | -0.03 |  101 |   99 |
  | B2         | -0.02 |  0.36 |  0.20 |  0.00 |  100 |   98 |
  | Avg w/o B2 | -0.18 |  0.55 |  0.54 | -0.12 |  100 |   99 |
  +------------+-------+-------+-------+-------+------+------+

STATS_CHANGED

[yeqing-wu]

Hi all, this fix has -0.03% RA gain and -0.05% LD gain with VCWG CTC 130 frames on top of @urvangjoshi's "Fix inter tx type cost table fill for small tx sets" (commit: 5315fe2)