Issue with encoding integers and floats with the decimal schema

[hariso]

Bug description

This issue has been found in the Postgres connector, but we may want to try fixing it here.

Sometimes, the Go type for a value is identified as an integer, e.g. int64. However, the generated schema can be a decimal schema. For example, in the Postgres connector, the NUMERIC(5) will be translated into this schema:

{
    "name": "column5",
    "type": "bytes",
    "logicalType": "decimal",
    "precision": 5
}

However, the value itself that we get is int64. Encoding the data with the given schema fails with:

err: could not marshal into avro: column5: avro: int64 is unsupported for Avro bytes

The encoding will work if the integer is converted to a byte array.

Steps to reproduce

const Column5PrimitiveType = `{
    "name": "conduit_testsource_read_690_payload",
    "type": "record",
    "fields":
    [
        {
            "name": "column5",
			"type": "bytes",
			"logicalType": "decimal",
			"precision": 5
        }
    ]
}`

func TestEncodeDecodeInt64Serde(t *testing.T) {
	is := is.New(t)

	serde, err := commons_avro.Parse([]byte(Column5PrimitiveType))
	is.NoErr(err)

	int64Val := int64(4)
	x := map[string]interface{}{
		"column5": int64Val,
	}

	_, err = serde.Marshal(x)
	is.NoErr(err)
}

Version

v0.4.0

[raulb]

@hariso will try to find more information about what happens in Postgres.

[hariso]

Here's how it looks like in the Pg connector: when we get a numeric, this code converts it into an int64 or float64. Then we get to the above issue.

I've also tried with a big.Rat, but that didn't work either. Providing a bytes array works. However, if a connector does that (i.e. converts a numeric value into a bytes array), that means that using that value in a non-SQL destination or a processor is not as straightforward.