Transformers should include concepts that fail to normalize

[korikuzma]

Currently, we only include concepts that succeed in VICC normalization. However, in the CDMs we also want to be able to include concepts that fail to normalize. For these cases, we'll simply add an extension ({"name": "vicc_normalizer_failure", "value": True}).

I'm not really sure how we want to handle this in /search... @mcannon068nw may have some guidance. We can create a separate issue for this. For now, we'll skip loading concepts in the DB that have this extension.

[korikuzma]

MOA does not have internal identifiers for therapy, genes, or diseases.

In cases where normalization succeeds, we use

f"moa.{norm_resp.{therapy|disease|gene}.id}"

(examples:
moa.normalize.therapy.rxcui:1727455, moa.normalize.disease.ncit:C2926, moa.normalize.gene.hgnc:427) for the MappableConcept.id.

I'm not sure what we want the id to be for concepts that fail normalization. Currently, I'm just doing:

def _sanitize_name(name: str) -> str:
    """Trim leading and trailing whitespace and replace whitespace characters with
    underscores

    :param name: Name to sanitize
    :return: Sanitized string with whitespace characters replaced by underscores
    """
    return re.sub(r"\s+", "_", name.strip())

id_ = f"moa.{therapy|disease|gene}:{_sanitize_name(name)}"

@ahwagner is this okay? Would you like something different? We require an id for these concepts in the database.

[ahwagner]

Seems like a reasonable approach to me. 👍

[github-actions]

Closed by #427.