What is stdout default encoding on Windows ?

[ikus060]

Hello,

I would need some help to understand a problem I'm facing with stdout encoding as the observed behavior doesn't match my expectation.

I have a console application bundle with pyinstaller v5.13.2. I can't use v6 for compatibility reason. That application print text to the console using sys.stdout.write("some unicode string"). This is working well in my staging environment with Windows 10 english version. With additional debugging, I can confirm the stdout encoding is set to utf-8 and errors is set to surrogateescape.

But, when deploying the same binary to different Windows environment. The encoding of stdout is not set to utf-8. Instead it it set to cp1252. It might be related to the fact it's a french installation ?

Expected behavior:

• According to some bug report and documentation. I assume UTF-8 to be the default encoding for stdout on Windows OS.

Current behavior:

• It seams the encoding of stdout might be updated according to locale.

Could someone clarify so I make proper change to the application in order to get it fixed ?

Thanks

[rokm]

Are you perhaps redirecting stdout? As per python docs on sys.stderr, "On Windows, UTF-8 is used for the console device. Non-character devices such as disk files and pipes use the system locale encoding (i.e. the ANSI codepage)."

This seems consistent between unfrozen python and a frozen application (made with v5.13.2), at least on my test system; i.e., sys.stdout.encoding is utf8 without redirect, and cp1252 if I redirect stdout to a file (and this raises UnicodeEncodeError: 'charmap' codec can't encode characters in position ... : character maps to <undefined> error when trying to print e.g. a Kanji text).

[rokm]

Reading more about sys.stdout, I read it possible to override the encoding of sys.stdout using PYTHONUTF8=1. Is it the recommanded approach for pyinstaller application ?

PYTHONUTF8=1 should work with v5.13.2 where we explicitly read and parsed the environment variable in the bootloader
https://github.com/pyinstaller/pyinstaller/blob/v5.13.2/bootloader/src/pyi_pythonlib.c#L262-L275
and then use the value to set the global flag.

In contemporary v6 versions, though, the frozen application ignores PYTHON* environment variables. (A subset of) interpreter options can be set (permanently) at build time via --python-option flags: https://pyinstaller.org/en/stable/spec-files.html#specifying-python-interpreter-options

[ikus060]

@rokm With pyinstaller v6. Is it possible to enforce UTF8 mode with options = [('X utf8', None, 'OPTION)], # Force UTF8 ?

[rokm]

Yes. EDIT: unless you have evidence to the contrary?

[ikus060]

I think the syntax if different with pyinstaller v5. I tried options=['-X', 'utf8'], and it doesn't seams to have the right effect.

[rokm]

I never said that it would work with v5, though...