Unique variants per VCF

[davmlaw]

Ideally we'd like to do this:

class CohortGenotype(models.Model):
    unique_together = ("collection", "variant")

But that caused a conflict after normalizing a varinat to dupe, as we no longer do rmdup in variant normalization as it makes the bcftools old variant record disappear (vcf_preprocess line ~116)

Perhaps we can remove duplicates ourselves as they are going to be right after each other (as file is sorted)

[davmlaw]

Had problem again, dupe variants were 2 apart in array, so started using a hash check instead

TODO: there is a very slight chance that dupes could be split across multiple processors (due is at end of 1 split file, start of another). Should raise an issue to fix that