Unicode filenames breaks MM

[schnuti]

Copy (FTP upload) a file with e.g. the name Übergrößen länge.jpg to the image folder.

Expected result

The file should appear in MM with "funny" characters as in J3.x
alt. skip files with unicoded names.

Actual result

No files or folders at all are displayed after a short "progress".

System information (as much as possible)

XAMPP local test, Win10

[schnuti]

This is because json_encode($array) throws an error when strings in the array is non UTF-8 as with Windows.
Solution 1: json_encode($array, JSON_PARTIAL_OUTPUT_ON_ERROR) skips the non UTF-8 strings.
Solution 2: require PHP 7.1 for Joomla. That version handles the Windows filesystem.

[laoneo]

#463 should fix this issue.

[laoneo]

Issue can be closed as the pr got merged. Thanks for reporting.

[schnuti]

@laoneo
Did you really test with a Windows system and PHP version < 7.1 ? With a PHP version > 7.0 or Linux the issue is solved. I still havn't seen anything about that Joomla 4 will raise the requirement to PHP > version 7.0.
As I store utf8 filenames in the DB of my local system, I require PHP > 7.0. i.e. I'm not able to make a quick test. Better for Joomla is to raise the requirement.

[schnuti]

Well, I remembered, I still have an older XAMPP instance with PHP 7.0.4. After downloading the code from here I made a quick test. As I presumed - the problem remains. You have my two alternative solutions above. Image 1: fresh installation, Image 2: Übergrößen-länge.jpg added. = broken view. You can reopen the issue.

[laoneo]

It works on Linux, hope somebody else can help here out.

[kasvith]

I am using XAMMP with PHP 7, Windows 10 x64

I copied a file with unicode characters to the images directory and here is the result

In NMM

But after i tried to rename it to Übergrößen länge nmm the result is follows

[laoneo]

That's a correct behavior as it strips out illegal characters when editing in MM. @schnuti do you use the latest code from the dev branch?

[kasvith]

If we want to keep the file name with UTF
Its needed to have this extension installed in PHP according to this article

http://blog.garr.co.uk/php/2015/09/22/php-windows-and-utf-8-filenames.html

[laoneo]

The UTF issue is handled in core joomla/joomla-cms#16878. Here we need to care only about when somebody uploaded a file with UTF characters trough FTP.

[schnuti]

Again -> from my post above

This is because json_encode($array) throws an error when strings in the array is non UTF-8 as with Windows.
Solution 1: json_encode($array, JSON_PARTIAL_OUTPUT_ON_ERROR) skips the non UTF-8 strings.
Solution 2: require PHP 7.1 for Joomla. That version handles the Windows filesystem.

To test you need a Windows system with PHP 7.0x i.e. < 7.1

If the PHP version requirement for Joomla 4 is raised to >= 7.1 this issue can be closed!
If it remains at PHP version 7.0 the json_encode statement has to be edited to include the above constant. In that case you will not see the Übergrößen-länge.jpg on systems with PHP 7.0 but will appear on systems with PHP 7.1
I do hope that we do not need Solution 1! If you want to implement it anyway (it makes no harm) it's better you write the PR so I can be a tester on my Windows system.

Clear enough?

@laoneo My new test yesterday (Jan 11) was made with a download the same day.
I have to include the new Joomla instance with the IDE, if you want me to search for the line to edit. I knew the line as I started this issue.

[laoneo]

@schnuti I will discuss that issue in the PLT about a possible version increase to 7.1 for J4 as 7.0 will be EOL by December 2018 anyway.

[schnuti]

@laoneo
My view is clear. The "vendor" versions required should be the latest "stable" versions at the time Joomla 4 goes live. I find Joomla 3 feels old because of the hard bindings to old versions of Php, JQuery, Bootstrap ... and the development cycles are getting even shorter.

For PHP

Each release branch of PHP is fully supported for two years from its initial stable release.

[schnuti]

@laoneo
To make sure that what I said is still true, I installed the same J40 MM code on a system with PHP 7.1.10.
As you can see the unicoded filename appears as expected. I didn't though find the line/lines to modify for PHP 7.0 after a quick search. I guess the code has bin modyfied a lot since last time I made a search.

[laoneo]

At the moment we stick with PHP 7.0 as min requirement. We can have a look on the issue again at a later point when we are in beta or rc phase when the PHP 7 version is fixed.

[schnuti]

OK! We just keep this issue open then.