Replace assert-based validation with explicit errors in modular_division

Replace assert-based validation with explicit errors in modular_division #17283