2 tests fails on elderly python 3.6 (ubuntu 18.04)

[yarikoptic]

Describe the bug

I was trying to build backports of debian package (only 4.3.3 ATM in Debian) for NeuroDebian and whenever it built/tested on newer systems just fine, I ran into two tests fails on ubuntu 18.04

======================================================================
ERROR: test_unicode_contents (pyfakefs.tests.fake_open_test.FakeFileOpenTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-pyfakefs-4.3.3/pyfakefs/tests/fake_open_test.py", line 97, in test_unicode_contents
    f.write(text_fractions)
  File "/build/python-pyfakefs-4.3.3/pyfakefs/fake_filesystem.py", line 4907, in write_wrapper
    ret_value = io_attr(*args, **kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character '\xdc' in position 0: ordinal not in range(128)

======================================================================
ERROR: test_write_str_read_bytes (pyfakefs.tests.fake_open_test.FakeFileOpenTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-pyfakefs-4.3.3/pyfakefs/tests/fake_open_test.py", line 117, in test_write_str_read_bytes
    f.write(str_contents)
  File "/build/python-pyfakefs-4.3.3/pyfakefs/fake_filesystem.py", line 4907, in write_wrapper
    ret_value = io_attr(*args, **kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character '\xc4' in position 0: ordinal not in range(128)

Just didn't want to stay silent ...

How To Reproduce

can try to provide a reproducer if needed. most likely in a docker container, .e.g docker run -it --rm neurodebian:nd18.04

Your environment
NeuroDebian base build images, this one for ubuntu 18.04 with python 3.6 (3.6.5-3; update to 3.6.9 from bionic-updates gave no remedy)

Please run the following and paste the output.

root@smaug:/tmp/buildd/python-pyfakefs-4.3.3# alias python=python3
root@smaug:/tmp/buildd/python-pyfakefs-4.3.3# python -c "import platform; print(platform.platform())"
Linux-5.10.0-0.bpo.3-amd64-x86_64-with-Ubuntu-18.04-bionic
root@smaug:/tmp/buildd/python-pyfakefs-4.3.3# python -c "import sys; print('Python', sys.version)"
Python 3.6.5 (default, Apr  1 2018, 05:46:30)
[GCC 7.3.0]
root@smaug:/tmp/buildd/python-pyfakefs-4.3.3# python -c "from pyfakefs.fake_filesystem import __version__; print('pyfakefs', __version__)"
pyfakefs 4.3.3

testing with current master

root@smaug:/tmp/buildd/pyfakefs# git describe
fatal: No annotated tags can describe '4ff65237e50661a1cd62e77ab529dfe0460c4f99'.
However, there were unannotated tags: try --tags.

(that is why I don't like github releases mechanism since they use non-annotated tags)

root@smaug:/tmp/buildd/pyfakefs# git describe --tags
v3.4.3-281-g4ff6523

(that is why I dislike off branch releases , well -- they could still be merged back in to at least get a sane describe)

I am also getting those two fails

======================================================================
ERROR: test_unicode_contents (pyfakefs.tests.fake_open_test.FakeFileOpenTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/pyfakefs/pyfakefs/tests/fake_open_test.py", line 97, in test_unicode_contents
    f.write(text_fractions)
  File "/build/pyfakefs/pyfakefs/fake_filesystem.py", line 5177, in write_wrapper
    ret_value = io_attr(*args, **kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character '\xdc' in position 0: ordinal not in range(128)

======================================================================
ERROR: test_write_str_read_bytes (pyfakefs.tests.fake_open_test.FakeFileOpenTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/pyfakefs/pyfakefs/tests/fake_open_test.py", line 117, in test_write_str_read_bytes
    f.write(str_contents)
  File "/build/pyfakefs/pyfakefs/fake_filesystem.py", line 5177, in write_wrapper
    ret_value = io_attr(*args, **kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character '\xc4' in position 0: ordinal not in range(128)

----------------------------------------------------------------------
Ran 2038 tests in 2.279s

FAILED (errors=2, skipped=848, expected failures=2)
Test failed: <unittest.runner.TextTestResult run=2038 errors=2 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=2038 errors=2 failures=0>

/tmp/buildd/pyfakefs# python -c "from pyfakefs.fake_filesystem import __version__; print('pyfakefs', __version__)"
pyfakefs 4.6.dev0

[mrbean-bremen]

Thanks for the report and PR! I dimly remember this kind of errors - I think they had to do with the encoding set in the Docker container. I will try to find the related issue before merging the PR that skips the tests.

[mrbean-bremen]

Found it. From a private conversation with the maintainer:

The first 2 (errors) use the default local encoding, which is expected (in the tests) at least to be able to encode Latin1. For some reason, this seems to be ASCII on that machine

This has been fixed by adding an encoding to the docker container in PR #456.

This is probably not an option in your case, though I'll wait for your opinion on that. Note that this does not depend on the Python version, just on the installed locale, so skipping it for a specific Python version is probably not the corrrect way to handle this. I guess we could skip if we could check the encoding (not sure about this - is there even a Python encoding for pure ASCII?), or we could catch the exception in the test and skip the test if it occurs.

[yarikoptic]

well -- for myself I "addressed" the issue by skipping just to get that package built.
Setting locales might indeed be trickier or not sufficient unless I also install locales etc. IMHO since 3.6 is soon (end of this year IIRC) to be gone anyways, I would not sweat too much about it (e.g. condition in addition on the locales etc) and just skip it one way (as in the #624) or with some extra care (locales).
on another thought: may be (since the idea is to parallel real file system) you could just sense first whether you can perform a similar write to a regular file using python itself... if it doesn't blow there with 3.6 but blows with pyfakefs -- there might be some magic missing in pyfakefs making it behave differently from regular python calls.

[mrbean-bremen]

may be (since the idea is to parallel real file system) you could just sense first whether you can perform a similar write to a regular file using python itself...

Thanks for the suggestion. Actually, we run most of the tests both in pyfakefs and in the real fs (at least in the CI tests), so I know from that previous occurrence that these tests also fail in the real fs (I omitted that part of the conversation). I will probably go with just ignoring the exception in the test if it occurs (skip the test in this case).

[mrbean-bremen]

@yarikoptic - I added a PR that does just that (skip the test if the exception occurs), you may have a look to check if this would be ok.