If cleanup.php doesn't fully wipe the DB, it can cause failures.

[Ipstenu]

There's really no other place to post these errors. I get them running the normal tests. Nothing on my end changed that I know of. I've tried upgrading all the things.

1. Tests_User_Capabilities::test_all_caps_of_users_are_being_tested
   User with administrator role has capabilities that aren't being tested
   Failed asserting that two arrays are equal.

--- Expected
+++ Actual
@@ @@
 Array (
+    59 => 'export_others_personal_data'
+    60 => 'erase_others_personal_data'
+    61 => 'manage_privacy_options'
 )

/phpunit-test-runner/wp-test-runner/tests/phpunit/tests/user/capabilities.php:381

2. Tests_User_Capabilities::testPrimitiveCapsTestsAreCorrect
   These primitive capabilities are not tested

Failed asserting that Array &0 (
    59 => 'export_others_personal_data'
    60 => 'erase_others_personal_data'
    61 => 'manage_privacy_options'
) is identical to Array &0 ().

/phpunit-test-runner/wp-test-runner/tests/phpunit/tests/user/capabilities.php:423

I'm not sure how/where this broke (I was out for a while)

[Ipstenu]

Okay it LOOKS like cleanup.php isn't nuking the DB tables, which I thought it was supposed to?

[Ipstenu]

Confirmed. Once I wiped out the database and let it rebuild clean, it was fine. So yeah, that;'s a thing,

[getsource]

Thanks @Ipstenu ! In core, manual cleanup isn't necessary (as far as I'm aware), so I'm wondering what might be different in this particular case.

[Ipstenu]

The best guess I have is that somehow something was corrupted, and since the cleanup doesn't wipe the DB, there it stayed. Usually the DB changes are additive to WP, but it might be smart to tweak that in advance of DB changes down the road.

[timbutler]

I can confirm this was the error with the Conetix tests failing as well. Manually deleting the database tables and re-running corrected the issue.

[mrxkon]

I've been using the code in #138 on my setups since i also wanted a clean start always. Most likely it's fine as it is to be merged & used in general (no issues on my end at least 😁 ) but feel free to point out any extra ideas.

I didn't want to go via a $wpdb route as including wp-load or anything like that would most likely start to return warnings for headers etc and wanted an as clean as possible output for logging so I decided to just go for a pretty straightforward mysqli way.

Note I've been using this on "local" setups, not sure if the remote setup would need something different (and I have no way to test it atm).

[ekamran]

I was able to reproduce this on current trunk, so this is still valid.

The stale data survives because WordPress boots before the test tables are dropped in tests/phpunit/includes/install.php. Data such as user roles can already be loaded into memory, and then gets written back into the fresh tables.

For the repro, I added one extra capability to the administrator role directly in the database and ran the suite normally. It produced the same failure from the original report, and dropping the test tables before the run fixed it.

#138 goes in the right direction, but two gaps showed up while testing: with WPT_SSH_CONNECT, cleanup needs to run where the tests ran, and a database cleanup failure should not stop file cleanup. I am opening a PR that handles those cases, drops all base tables matching the configured test prefix, includes multisite sub-site tables like wptests_2_posts, leaves unrelated tables untouched, and keeps the database password off the command line.